AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds

AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds

Speakers: Yinfang Chen, Manish Shetty, Gagan Somashekar, Chetan Bansal, Saravan Rajmohan

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

In an era increasingly reliant on complex cloud infrastructure, the stability and performance of services are paramount. Yet, production incidents remain an inevitable and costly reality, leading to significant revenue loss, user dissatisfaction, and decreased productivity. A recent study highlighted that a staggering 60% of these incidents are directly related to cloud operations, stemming from issues in underlying infrastructure, deployment processes, or inter-service dependencies. Manually detecting, localizing, and mitigating these sophisticated cloud incidents is not only tedious but also unsustainable, underscoring an urgent need for AI-driven agents to automate incident management. This talk introduces AIOps Lab, a groundbreaking, open-source framework designed to address this critical need by providing a comprehensive platform for the design, development, and rigorous evaluation of AI agents specifically tailored for cloud operations tasks.

The presentation, co-led by Gagan Somashekar and Yinfang Chen, details the motivation behind AIOps Lab, its architectural components, and its unique approach to simulating realistic cloud incident scenarios. Developed through a collaboration involving UIUC, UC Berkeley, and IISC, AIOps Lab stands out by bridging a significant gap in the current AI agent evaluation landscape. While the field has seen a proliferation of benchmarks for software development tasks, robust and standardized frameworks for cloud operations (AIOps) have been conspicuously absent. AIOps Lab aims to fill this void, enabling researchers and practitioners to build, test, and compare AI agents that can truly contribute to achieving autonomous, resilient cloud environments.

Background

▶ Watch: Introduction and the high cost of cloud incidents (0:00)

The evolution of AI agents, particularly following the advent of powerful models like GPT-4 in early 2023, has opened new avenues for automation beyond simple code generation. Early evaluation frameworks like Intercode and Agent Bench demonstrated the potential of these models to perform autonomous tasks by interacting with environments through OS commands or database queries. This trajectory accelerated with the release of Sweep Bench in late 2023, a coding benchmark that challenged models to fix GitHub issues within real repositories, leading to the emergence of advanced coding agents such as Cognition AI's Devin and Microsoft's Masai. These developments clearly showcased the growing capability of AI to tackle complex software engineering challenges.

However, as highlighted by the speakers, the focus of this agent-centric innovation has been disproportionately skewed towards software development tasks, primarily code generation and bug fixing. A critical oversight has been the neglect of equally demanding and financially impactful operations tasks, particularly incident management in cloud environments. This disparity is stark: while numerous agents and frameworks exist for development, offerings for cloud operations are sparse and suffer from significant limitations. Existing AIOps solutions typically lack standardization in metrics, task definitions, and interfaces, making comparative evaluation difficult. They are often siloed, addressing only a single operational task rather than the full incident lifecycle. Furthermore, many are static, relying on Q&A datasets rather than real execution environments, and a substantial number remain proprietary, hindering broader research and community collaboration. This fragmented and immature landscape for AIOps agent evaluation forms the foundational problem that AIOps Lab seeks to resolve, aiming to standardize and democratize the development of AI agents capable of truly managing autonomous clouds.

Key Findings

▶ Watch: Critical gap: ignoring cloud operations tasks for agents (4:00)

The central finding and contribution of this work is the identification of a critical void in the evaluation of AI agents for cloud operations, and the subsequent introduction of AIOps Lab as a holistic framework to address it. The speakers meticulously articulated the disparity between the mature state of agent evaluation for software development and the nascent, unstandardized landscape for AIOps. AIOps Lab is presented as the solution to this problem, offering a comprehensive, open-source platform specifically designed to develop, evaluate, and benchmark AI agents for incident management in cloud environments.

The framework is built upon four core goals:

1. Standardization: Establishing a common set of tasks, metrics, and baselines for AIOps agents, enabling fair comparison and progress tracking.
2. Realistic Environment: Utilizing real services, workloads, and agent actions within a live execution environment, moving beyond static datasets.
3. Scalability: Designing a framework extensible enough to incorporate new applications, tasks, faults, and integrate with any type of AI agent.
4. Reliability: Providing robust observability across the entire operations lifecycle, ensuring agents can accurately perceive and interact with the system state.

AIOps Lab’s key contributions include not just a live open-source benchmark with over 100 pre-defined problems, but also a framework that empowers users to build their own custom scenarios. It provides flexible interfaces for developers, ensuring it is not limited to specific architectures (e.g., microservices) or cloud setups. Crucially, it defines agent-cloud interfaces that allow AI agents to interact with the cloud environment, collect telemetry, and submit solutions. The framework supports the full spectrum of incident management tasks, including detection, localization, root cause analysis, and mitigation, with a particular emphasis on the challenging mitigation aspect. By leveraging fault injection techniques, AIOps Lab creates authentic error states, providing a realistic testbed for AI agents to diagnose and resolve complex cloud incidents.

Technical Deep Dive

▶ Watch: Introducing AIOps Lab: a holistic framework (7:00)

AIOps Lab is architected as a sophisticated ecosystem designed to simulate, orchestrate, and evaluate AI agent interactions within a dynamic cloud environment. At its core, the architecture comprises several interconnected components: Agents, the Service and Task environment (representing the cloud), Workloads, an Orchestrator, a Telemetry Collector, and an Automatic Evaluator. The Orchestrator acts as the central hub, stitching these components together to manage the lifecycle of an incident scenario from injection to resolution.

The framework defines specific goals for AI agents within this environment, aligning with the stages of incident management:

• Detection: Agents must identify anomalous behaviors or deviations from normal system operation.
• Localization: Pinpointing the exact component, service, or layer where a problem or error resides.
• Root Cause Analysis (RCA): Determining the underlying cause of an incident, going beyond mere symptoms.
• Mitigation: The most challenging task, requiring agents to actively fix the problem within the running system, potentially involving configuration changes, service restarts, or code patches.

To create realistic incident scenarios, AIOps Lab employs fault injection, a technique that deliberately introduces errors into the system. This approach is superior to simply replaying past incidents because it allows for controlled, reproducible, and customizable problem states without requiring access to proprietary historical data or code. The framework categorizes faults into two main types:

• Symptomatic Faults: These inject observable symptoms into the system, such as network loss, power failure (simulated), CPU overhead, or memory overhead. These mimic the immediate manifestations of a problem without necessarily being the root cause themselves.
• Functional Faults: These represent the true underlying problems or root causes. Examples include misconfigurations within an application or service, or code bugs leading to issues like a broken image in a container. AIOps Lab provides a comprehensive fault library to facilitate the instantiation of these diverse problem types.

For agents to effectively interact with and understand the state of the cloud environment, observability is crucial. AIOps Lab integrates a Telemetry Collector that gathers various forms of telemetry data—logs, tracings, and metrics—from different layers and components of the simulated cloud. This collected data is then streamed back to the AI agents, allowing them to perceive the system's current state, identify anomalies, and diagnose problems.

The agent's workflow within AIOps Lab follows a structured sequence:

1. Agent Registration: AI agents are registered with the AIOps Lab framework.
2. Problem Deployment: The framework deploys a specific cloud context and problem scenario, potentially involving a microservice application (e.g., a social network application).
3. Fault Injection: A designated fault (symptomatic or functional) is injected into the deployed environment.
4. Workload Trigger: A synthetic or real workload is triggered to manifest the error, making the incident observable (e.g., generating HTTP 400 or 500 error codes).
5. Agent Interaction: The agent uses provided interfaces (the agent-cloud interfaces) to query the Telemetry Collector, gather data, and potentially execute actions within the cloud.
6. Solution Submission: Once the agent believes it has completed its assigned task (e.g., detected, localized, or mitigated the fault), it calls a submission API to indicate completion.
7. Automatic Evaluation: The Automatic Evaluator then assesses the agent's performance against the predefined ground truth solution for the problem.

A problem in AIOps Lab is constructed using three primary components:

• Task: A Python class defining the specific objective, such as LocalizationTask for identifying the problematic component.
• Context: Specifies the application under test (e.g., a social network application), the workload to apply, and the specific faults to inject.
• Exact Solution: The ground truth answer required for successful completion of the task (e.g., the name of the problematic microservice, such as "user service"). If the agent's submitted answer matches this, it's considered a success.

AIOps Lab offers flexible deployment options to cater to various research and development needs. Users can choose a local deployment setup, running the entire cloud environment and framework on a single machine, which is particularly useful for resource-constrained researchers. Alternatively, for more realistic and scalable evaluations, the framework supports deployment on real cloud clusters. This dual approach ensures accessibility while also providing the capability for high-fidelity testing.

Experimental Setup & Results

▶ Watch: Challenges building a realistic cloud incident benchmark (8:00)

The primary focus of the AIOps Lab presentation is the introduction and detailed architectural design of the framework itself, rather than presenting exhaustive experimental results from evaluating specific AI agents against its benchmark. The talk emphasizes the capability of the framework and the existence of its benchmark.

The speakers state that the AIOps Lab benchmark currently supports over 100 problems. These problems are constructed using the three-component structure (Task, Context, Exact Solution) and cover the four agent goals: detection, localization, root cause analysis, and mitigation. The framework's ability to inject both symptomatic faults (e.g., CPU/memory overhead, network loss) and functional faults (e.g., misconfigurations, broken images) ensures a diverse set of real-world incident scenarios.

While specific performance metrics of AI agents are not detailed in the presentation, the speakers do highlight the ease of agent registration into the AIOps Lab environment and the framework's capability to facilitate trajectory analysis of agents. This suggests that the platform is designed to provide insights into how agents interact with the environment and arrive at their conclusions, which is crucial for debugging and improving agent performance.

Regarding the scalability of the benchmark itself, a question from the audience regarding the addition of new problems acknowledged that while 100+ problems is a good start, it is a current limitation compared to benchmarks like Sweep Bench, which boasts over 2,000 problems. However, the framework addresses this by providing flexible interfaces that empower developers to deploy their own applications and define custom scenarios within the AIOps Lab environment, thereby allowing the community to extend the benchmark's problem set. The dual deployment options (local and real cloud clusters) further enhance the experimental flexibility for users, catering to different resource availabilities and fidelity requirements.

Practical Implications

▶ Watch: Agent goals: detection, localization, and mitigation tasks (11:00)

AIOps Lab holds significant practical implications for a diverse range of stakeholders in the cloud and AI/ML ecosystem, addressing long-standing challenges in operationalizing AI for autonomous cloud management.

For practitioners and infrastructure teams, AIOps Lab offers a standardized, realistic, and open-source platform to rigorously test and validate AI agents before deploying them into production environments. This is crucial for building confidence in AI-driven automation, reducing the risk of introducing new vulnerabilities or regressions, and ultimately enhancing system reliability. The framework's ability to simulate diverse and complex incident scenarios through fault injection means that agents can be trained and evaluated against real-world challenges, moving beyond theoretical models or static datasets. This can lead to more robust and effective AI agents that can proactively detect, diagnose, and resolve issues, reducing human toil and minimizing the impact of incidents.

For model builders and AI researchers, AIOps Lab provides a much-needed benchmark and development environment focused specifically on cloud operations. This allows researchers to shift their focus from predominantly development-centric agent tasks to the critical domain of incident management, fostering innovation in areas like anomaly detection, intelligent localization, automated root cause analysis, and autonomous mitigation. The standardized metrics and task definitions facilitate fair comparisons between different AI agent architectures and algorithms, accelerating progress in the field. Furthermore, the open-source nature of the framework encourages community collaboration, enabling shared development of problems, applications, and agent solutions.

The framework directly addresses several limitations identified in existing AIOps solutions:

• Lack of Standardization: AIOps Lab provides a common ground for defining tasks, metrics, and interfaces, making it easier to compare and integrate different tools and agents.
• Siloed Solutions: By supporting the full incident management lifecycle (detection to mitigation), it encourages the development of holistic agents rather than single-task solutions.
• Static Evaluation: The use of real execution environments with live services and workloads ensures that agents are evaluated under realistic, dynamic conditions.
• Proprietary Nature: Its open-source availability democratizes access to advanced AIOps research and development, fostering a more collaborative ecosystem.

While the scalability of adding new problems is acknowledged as an area for continued development, the flexible interfaces provided by AIOps Lab empower users to extend the benchmark with their own applications and scenarios. This ensures that the framework can evolve with the dynamic nature of cloud technologies and operational challenges. Ultimately, AIOps Lab is a foundational step towards realizing truly autonomous clouds, enabling organizations to build more resilient, efficient, and self-managing infrastructure.

Key Takeaways

• AIOps Lab is an open-source framework designed to address the critical gap in evaluating AI agents for cloud operations, shifting focus from software development to incident management.
• It provides standardization for tasks, metrics, and baselines, operating in realistic environments with real services and workloads.
• The framework utilizes fault injection (symptomatic and functional) to create diverse and authentic cloud incident scenarios.
• AIOps Lab supports the full incident management lifecycle, including detection, localization, root cause analysis, and mitigation by AI agents.
• It offers flexible agent-cloud interfaces for developers to build custom scenarios, integrate various agents, and collect comprehensive telemetry (logs, traces, metrics).
• The framework supports both local deployment for accessibility and real cloud cluster deployment for high-fidelity, scalable evaluations.

About the Speaker(s)

The AIOps Lab project is a collaborative effort, with the presentation co-led by Gagan Somashekar and Yinfang Chen. Yinfang Chen played a pivotal role, leading the core development of AIOps Lab during an internship. The work benefited from the support of Manish Shetty, and involved significant contributions from Chetan Bansal and Saravan Rajmohan. The research was conducted in collaboration with leading academic institutions, including the University of Illinois Urbana-Champaign (UIUC), UC Berkeley, and the Indian Institute of Science (IISC), highlighting a strong academic-industrial partnership in addressing complex cloud operational challenges.