ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks

Speakers: Saurabh Jha, Research Staff Member, IBM; Rohan Arora, PhD Student, University of Illinois at Urbana-Champaign; Yuji Watanabe, Takumi Yanagawa, Yinfang Chen, Jackson Clark, Bhavya Bhavya, Mudit Verma, Harshit Kumar, Hirokuni Kitahara, Noah Zheutlin, Saki Takano, Divya Pathak, Felix George, Xinbo Wu, Bekir Turkkan, Gerard Vanloo, Michael Nidd, Ting Dai, Oishik Chatterjee, Pranjal Gupta, Suranjana Samanta, Pooja Aggarwal, Rong Lee, Jae-wook Ahn, Debanjana Kar, Amit Paradkar, Yu Deng, Pratibha Moogi, Prateeti Mohapatra, Naoki Abe, Chandrasekhar Narayanaswami, Tianyin Xu, Lav Varshney, Ruchi Mahindru, Anca Sailer, Laura Shwartz, Daby Sow, Nicholas Fuller, Ruchir Puri

Conference: ICML 2025

YouTube: https://slideslive.com/39044115

Overview

In the rapidly evolving landscape of artificial intelligence, the promise of AI agents to automate complex, real-world tasks has garnered immense attention. However, the true capabilities of these agents, particularly in high-stakes operational environments, remain largely underexplored and inadequately benchmarked. This talk introduces ITBench, a novel and comprehensive benchmark designed to rigorously evaluate AI agents on diverse, real-world IT automation tasks. Developed through a collaborative effort between IBM and the University of Illinois at Urbana-Champaign, ITBench aims to bridge the gap between theoretical agentic capabilities and practical deployment in critical IT infrastructure.

Saurabh Jha, the lead presenter, underscored the critical need for such a benchmark by illustrating the immense scale and complexity of modern IT management. From managing individual laptops to orchestrating thousands of interconnected machines, IT systems are prone to failures, security breaches, and cost inefficiencies that demand rapid, precise intervention. The talk highlights the significant economic and operational impact of IT outages, exemplified by the CrowdStrike incident, which caused widespread disruption and billions in economic losses. ITBench steps in as a vital tool to assess whether current AI agents can truly assist or even autonomously manage these intricate, high-consequence scenarios, providing a grounded, realistic, and rigorous evaluation framework for the community.

The core motivation behind ITBench is to accelerate the development of robust and reliable AI agents for IT operations. By exposing the current limitations of state-of-the-art large language models (LLMs) and agentic frameworks in this domain, the benchmark provides clear directions for future research. It emphasizes challenges unique to IT, such as multimodal data interpretation, long-horizon planning, context discovery in vast information spaces, and the critical need for safe state-changing actions. The findings from initial evaluations on ITBench reveal a substantial performance gap, indicating that despite significant advancements, current AI agents are far from ready for autonomous deployment in enterprise IT environments, thereby setting a crucial agenda for the AI and ML research communities.

Background

▶ Watch: Introduction to IT automation tasks and daily challenges (0:00)

Modern IT infrastructure is characterized by an unprecedented scale, diversity, and complexity. What begins as managing a single laptop—restarting systems, installing antivirus, or monitoring API costs—quickly escalates to orchestrating thousands or tens of thousands of interconnected machines in an enterprise setting. This vast environment is managed by specialized human roles: Site Reliability Engineers (SREs) for failure management, CISO (Chief Information Security Officer) teams for compliance and security, and FinOps teams for cost optimization. These are just a few examples within a broader ecosystem of IT management personas and domains.

The challenges faced by human IT professionals are multifaceted. They must synthesize information from truly multimodal data sources, including time series data, dashboards, code, and system events, often under immense time pressure. The sheer scale and diversity of software and hardware, coupled with intricate interdependencies across networks, make diagnosis and remediation profoundly difficult. Crucially, IT operations involve state-changing actions, meaning interventions can have real-world, often irreversible, consequences. Unlike writing code in an isolated environment, deploying an action in an IT system carries inherent risks, demanding careful risk assessment to avoid exacerbating problems or introducing new vulnerabilities.

The economic and operational impact of IT failures underscores the urgency of this problem. The talk cited the 2024 CrowdStrike outage, which paralyzed critical services globally—airlines, hospitals, banks, and emergency services—affecting an estimated 85 million devices and incurring an economic impact of approximately $54 billion. Even for CrowdStrike's own experts, recognizing and fixing the problem (a code update issue) took 80 minutes, with full recovery spanning multiple days. This incident vividly illustrates the limitations of human intervention in complex, rapidly evolving IT crises and highlights the potential for AI agents to accelerate problem resolution. While LLM agents offer a compelling vision for automating and assisting humans in these tasks, their application in such high-stakes, real-world environments necessitates a robust and domain-specific evaluation framework that accounts for the unique challenges of IT. This need for a specialized benchmark, moving beyond general-purpose agent evaluations, is precisely what ITBench addresses.

Key Findings

▶ Watch: Multimodal IT problem-solving and LLM agent potential (4:00)

The primary and most striking finding from the initial evaluation of ITBench is the significant performance gap exhibited by even the most advanced, state-of-the-art LLM agents. Despite the hype and rapid progress in large language models, their current capabilities fall far short of what is required for effective and safe IT automation.

Specifically, the study revealed:

• Low Pass Rates: When evaluated on SRE (Site Reliability Engineering) scenarios, GPT-4o, considered one of the best available models at the time of the study, achieved a pass-at-one rate of only about 11-14%. The performance was even lower for compliance and security scenarios, as well as FinOps tasks, where pass rates hovered around 2-5%.
• Consistent Underperformance Across Models: Subsequent evaluations with other leading models, including Gemini 1.5 Pro, Claude, and Kimi K2, showed very similar performance, typically ranging from 10% to 12% pass rates. This consistency across different advanced models suggests a fundamental limitation not just of a single model, but of the current generation of LLMs and the underlying agentic frameworks when applied to the IT domain.
• Lack of Domain Knowledge: The low success rates are attributed to a significant lack of domain-specific knowledge within these LLMs, making them ill-equipped to understand the nuances, terminology, and operational logic inherent in IT systems.
• Deficiencies in Agentic Frameworks: Beyond mere knowledge, the results also highlight limitations in the theoretical underpinnings and practical implementations of current agentic frameworks. These frameworks struggle with the complexity, long-horizon planning, and critical decision-making required for IT automation.
• Surfacing Core Challenges: ITBench effectively surfaces several critical challenges that current agents struggle with:
• Context Discovery: Agents fail to intelligently identify and utilize relevant information from vast, token-rich IT environments.
• Conflicting Evidence: They struggle to perform causal analysis and correctly interpret situations where multiple, potentially conflicting pieces of evidence are present.
• Non-LLM Friendly Language: Technical jargon, logs, and error messages often present language patterns that LLMs do not robustly understand.
• Long-Horizon Planning: Many IT tasks require extended sequences of actions, and current agents exhibit poor performance in multi-turn reasoning.
• Catastrophic Actions: A significant concern is the agents' propensity to take actions that can worsen the system state, demonstrating a lack of robust safety and risk assessment mechanisms.

These findings collectively underscore that while AI agents hold immense promise for IT automation, current capabilities are nascent, necessitating substantial advancements in LLM domain adaptation, reasoning, planning, and safety mechanisms before they can be reliably deployed in real-world IT environments.

Technical Deep Dive

▶ Watch: Introducing ITBench: a realistic IT automation benchmark (6:00)

ITBench is meticulously designed to create a realistic and challenging evaluation environment for AI agents in IT automation. Its technical foundation is built upon mimicking real-world conditions and providing a robust framework for assessing agent performance.

Design Principles of ITBench

1. Realistic Environment Grounding: ITBench scenarios are not synthetic; they are grounded in real-world IT incidents derived from internal IBM systems. This ensures that the problems presented to agents reflect actual operational challenges, including the nuances of data, system states, and potential failures. The benchmark incorporates three initial, high-impact domains: Site Reliability Engineering (SRE), Compliance and Security (CISO), and Financial Operations (FinOps).
2. Domain Complexity Abstraction: A key design goal was to make the highly complex IT environment accessible to AI scientists. ITBench hides the underlying domain complexity, providing a standardized interface and toolset that allows researchers to focus on agent development rather than grappling with intricate IT system configurations. This abstraction layer is crucial for fostering broad community participation.
3. Rigorous Evaluation Framework: The benchmark employs a comprehensive evaluation system. Beyond a simple pass/fail metric, it provides partial scoring based on the number of steps taken by the agent. Crucially, it offers real reasoning feedback, explaining why an agent failed or succeeded, which is invaluable for debugging and improving agent design. This granular feedback helps researchers understand the specific points of failure in an agent's reasoning process.

Agent Interaction Model

The baseline agent provided with ITBench, and the expected interaction model for agents, is based on the popular ReAct framework. This framework operates on a continuous loop of:

• Thought: The agent reasons about the current situation and the next step.
• Action: Based on its thought, the agent selects and executes an action using available tools.
• Observation: The agent receives feedback from the environment as a result of its action.

A critical distinction in ITBench is that the agent's "tools" are not merely for observation; they enable real-time state-changing actions. This means agents can actively modify the environment, such as deploying software, changing configurations, or deleting resources. This capability introduces significant risk and complexity, as incorrect actions can lead to catastrophic system failures, making careful risk assessment and robust error recovery mechanisms paramount.

Core Technical Challenges Surfaced by ITBench

ITBench is specifically engineered to highlight fundamental limitations in current AI agents:

1. Context Discovery and Engineering: Real IT environments generate an immense volume of data—logs, metrics, configurations, code. Feeding all this data (potentially billions of tokens) directly into an LLM's context window is infeasible. Agents must exhibit smart context discovery, intelligently identifying and prioritizing relevant information. The benchmark challenges agents to navigate this information overload effectively.
2. Handling Conflicting Evidence and Non-LLM Friendly Language: IT systems often present ambiguous or contradictory information. Agents must perform a form of causal analysis to discern the true root cause amidst conflicting alerts or events. Furthermore, much of IT data (e.g., error messages, log formats, specialized jargon) is not "LLM-friendly," leading to misinterpretations. The talk provided an example where modern LLMs confidently chose the wrong causal event when presented with two alerts and two potential causes. This highlights a lack of uncertainty awareness and robust reasoning in complex, ambiguous scenarios.
3. Long-Horizon Planning: IT tasks are rarely single-step operations. Successful resolution often requires a sequence of many interdependent actions. The evaluation showed that even for successful tasks, GPT-4o averaged 93 turns. This extended planning horizon poses two major problems:

• Compounding Errors: If the per-turn accuracy is less than perfect (e.g., 0.9), even a small error rate compounds significantly over 93 turns, leading to a high probability of overall task failure. Agents need to develop robust strategies for recovery and self-correction.
• Cost and Latency: Many turns mean more API calls, increased latency, and higher operational costs, especially with large models.

1. Catastrophic Actions and Safety: The ability to perform state-changing actions introduces the risk of catastrophic failures. An agent might, for instance, attempt to "fix" a problem by deleting an entire system or critical software components, akin to deleting Microsoft Office to solve a minor bug. This underscores the critical need for agents to incorporate sophisticated safety protocols, risk assessment, and an understanding of the potential impact of their actions before execution, preventing the system from entering a worse state than it started.
2. Multi-faceted Challenges: ITBench also implicitly and explicitly surfaces other complex challenges, including data heterogeneity (different formats, sources), data complexity (relationships, structure), robustness problems (handling unexpected inputs), long context reasoning (maintaining coherence over extended interactions), agent-human interaction (for collaborative problem-solving), and multi-agent coordination (for distributed IT operations).

By integrating these technical complexities, ITBench provides a rigorous environment for advancing the capabilities of AI agents, pushing them beyond simple question-answering or code generation towards truly autonomous and safe operation in critical domains.

Experimental Setup & Results

▶ Watch: ReAct baseline agent and critical state-changing actions (8:00)

The experimental setup for evaluating AI agents on ITBench focused on assessing the current capabilities of leading large language models in performing real-world IT automation tasks.

Benchmark and Domains

The core of the experimental setup is the ITBench benchmark itself. It encompasses scenarios across three critical IT domains:

1. Site Reliability Engineering (SRE): Tasks related to detecting, diagnosing, and resolving system failures and ensuring service uptime.
2. Compliance and Security (CISO): Scenarios involving identifying security vulnerabilities, ensuring adherence to regulatory policies, and responding to incidents.
3. Financial Operations (FinOps): Tasks focused on optimizing cloud spending, managing resource allocation, and identifying cost inefficiencies.

All scenarios within ITBench are derived from real IT incidents, ensuring their practical relevance and complexity. The benchmark is designed such that, for the most part, these scenarios are solvable by human experts, although some intentionally unsolvable variations are included to test agent robustness.

Baselines and Models Tested

The study established a baseline using a ReAct-based agentic framework. This framework, which cycles through "Thought," "Action," and "Observation," is a well-known paradigm for building agents that interact with environments. The tools available to the agent within ITBench allow for both observation and, critically, state-changing actions.

The following cutting-edge large language models were used to power the ReAct agent:

• GPT-4o
• Gemini 1.5 Pro
• Claude
• Kimi K2

These models represent some of the most advanced general-purpose LLMs available at the time of the study, making them strong candidates for demonstrating the state-of-the-art in agentic capabilities.

Metrics and Evaluation

The primary evaluation metric was the pass-at-one rate, indicating the percentage of scenarios an agent successfully resolved on its first attempt. Beyond this binary metric, ITBench provides:

• Partial scoring: Agents receive partial credit for successfully completing intermediate steps, offering a more nuanced view of performance than a simple pass/fail.
• Reasoning feedback: The framework offers insights into why an agent succeeded or failed, crucial for debugging and improving agent logic.
• Number of turns: The average number of actions an agent took to complete a task was also tracked, providing insight into the planning horizon and efficiency.

Headline Results

The experimental results revealed a stark reality regarding the current capabilities of AI agents in IT automation:

• Overall Low Performance: Even the best model, GPT-4o, achieved only an approximate 11-14% pass rate for SRE scenarios.
• Domain-Specific Weaknesses: Performance dropped significantly for other critical domains, with compliance, security, and FinOps scenarios seeing pass rates of roughly 2-5%.
• Consistent Across Models: Gemini 1.5 Pro, Claude, and Kimi K2 showed very similar performance, generally in the 10-12% range across all domains. This indicates a systemic limitation across current LLM architectures and agentic approaches rather than an isolated issue with a single model.
• Long Horizon Planning: For the limited number of tasks where GPT-4o did succeed, it required an average of 93 turns to reach a solution, highlighting the extensive planning and interaction required for these complex tasks.

These headline numbers unequivocally demonstrate that current AI agents are not yet equipped to autonomously handle real-world IT automation tasks. The significant gap between current performance and the requirements for reliable operation underscores the need for substantial research and development in areas such as domain-specific knowledge integration, robust reasoning under uncertainty, and advanced planning and safety mechanisms within agentic frameworks.

Practical Implications

▶ Watch: Current state-of-the-art AI agent performance on ITBench (9:00)

The findings from ITBench carry profound practical implications for various stakeholders involved in the development, deployment, and management of IT systems. The current limitations of AI agents in this high-stakes domain necessitate a cautious and strategic approach.

For Practitioners and Infrastructure Teams

For IT practitioners, SREs, CISO teams, and FinOps teams, the immediate implication is clear: current AI agents are not ready for autonomous deployment in critical IT environments. The low success rates and the risk of catastrophic actions mean that relying on these agents for unassisted problem resolution is highly dangerous and could lead to significant operational disruptions, security breaches, or financial losses. Instead, AI agents might currently serve best as assistants or co-pilots, offering recommendations or automating low-risk, well-defined sub-tasks under human supervision. The focus should remain on human-in-the-loop systems where AI augments human capabilities rather than replaces them. Infra teams must also recognize that integrating such agents requires robust observability and rollback mechanisms to mitigate the risks associated with state-changing actions.

For Model Builders and AI Researchers

For model builders and AI researchers, ITBench provides a clear roadmap for future development. The benchmark highlights several critical areas requiring significant innovation:

• Domain Knowledge Integration: LLMs need better mechanisms to assimilate and reason over vast, technical, and often jargon-filled IT domain knowledge. This could involve specialized pre-training, fine-tuning on IT-specific datasets, or novel knowledge retrieval and grounding techniques.
• Robust Reasoning and Causal Analysis: Agents must improve their ability to handle conflicting information, perform accurate causal analysis, and understand the intricate dependencies within complex systems. This includes developing better uncertainty quantification and reasoning under ambiguity.
• Long-Horizon Planning and Error Recovery: The need for 93 turns on average for successful tasks points to a lack of efficient planning and recovery capabilities. Research into more sophisticated planning algorithms, hierarchical task decomposition, and self-correction mechanisms is crucial.
• Safety and Risk Assessment: The potential for catastrophic actions necessitates the development of robust safety protocols. Agents must be able to assess the risk of an action before execution, predict its potential impact, and adhere to predefined safety constraints. This could involve formal verification techniques, reinforcement learning with safety constraints, or human preference learning.
• Context Engineering: Techniques for intelligently discovering and summarizing relevant context from massive data streams are paramount to avoid overwhelming LLM context windows and improve efficiency.

Tradeoffs and Limitations

Deploying AI agents in IT automation involves significant tradeoffs:

• Risk vs. Automation: The potential benefits of automation (speed, efficiency) are currently heavily outweighed by the high risk of errors and catastrophic failures.
• Cost of Development vs. Reliability: Building agents that are sufficiently reliable for IT operations requires substantial investment in research, testing, and safety validation.
• Interpretability vs. Autonomy: For high-stakes actions, interpretability of an agent's reasoning is critical for human oversight, which can sometimes conflict with end-to-end autonomous approaches.

A key limitation is the absence of a direct human baseline within the ITBench evaluation, though the scenarios are derived from real human-solvable incidents. While the benchmark makes domain complexity accessible, the actual deployment of agents still requires addressing challenges like integration with legacy systems, enterprise-specific policies, and human-agent trust.

Ultimately, ITBench serves as a vital catalyst for the community, providing a standardized, challenging environment to accelerate the development of AI agents that are truly capable, safe, and reliable enough to revolutionize IT operations. The call to action is for continued collaboration, contributing both agents and new scenarios to expand the benchmark's scope and drive progress.

Key Takeaways

• IT Automation is Critical and Complex: Modern IT infrastructure presents high-stakes, multimodal, and long-horizon challenges, with significant economic impact from failures (e.g., the $54B CrowdStrike outage).
• Current AI Agents Fall Short: State-of-the-art LLM agents (GPT-4o, Gemini 1.5 Pro, Claude, Kimi K2) achieve alarmingly low pass rates (e.g., 11-14% for SRE, 2-5% for security/FinOps) on real-world IT automation tasks in ITBench.
• ITBench Uncovers Core Challenges: The benchmark highlights fundamental limitations in LLMs, including poor context discovery, struggles with conflicting evidence and non-LLM friendly technical language, difficulties in long-horizon planning (averaging 93 turns for success), and the risk of catastrophic state-changing actions.
• Rigorous and Realistic Evaluation: ITBench provides a grounded, real-world benchmark derived from actual IT incidents, offering not just pass/fail but also partial scoring and reasoning feedback to guide agent development.
• Not Ready for Autonomous Deployment: Current AI agents are not yet suitable for autonomous, high-stakes IT operations and should primarily function as human assistants or co-pilots in supervised settings.
• Call for Community Collaboration: ITBench is an open platform inviting AI scientists to develop agents and domain experts to contribute new scenarios, fostering collaborative advancement in robust AI for IT.

About the Speaker(s)

The talk was presented by Saurabh Jha, a Research Staff Member at IBM, and a collaborator with the University of Illinois at Urbana-Champaign. The extensive list of co-authors indicates a significant collaborative effort involving researchers from both IBM and the University of Illinois at Urbana-Champaign, reflecting a deep interdisciplinary engagement between industry and academia to tackle the complex challenges of AI in IT automation.