Lessons From Building a Cloud Attack Simulation Program

Lessons From Building a Cloud Attack Simulation Program

Speakers: Pavel Lineitsev, Detection and Response Team, Conland

Conference: fwd:cloudsec

YouTube: https://www.youtube.com/watch?v=F6DXOYLREGo

Overview

In the dynamic landscape of cloud security, ensuring the efficacy of detection and response capabilities across heterogeneous cloud environments is a monumental challenge. Pavel Lineitsev, from Conland's Detection and Response Team, presented an insightful talk at fwd:cloudsec, detailing his team's journey and lessons learned from building an internal cloud attack simulation program. The presentation offered a unique defender's perspective, moving beyond theoretical discussions to practical implementation.

The core motivation behind Conland's initiative stemmed from the complexities of operating across all three major cloud service providers—AWS, GCP, and Azure. This multi-cloud setup inherently leads to hundreds of distinct detection rules, each requiring meticulous maintenance, tuning, and continuous validation. A critical pain point was the silent failure of logging and data pipelines, where detection rules could cease functioning without immediate notice, only to be discovered during an active incident. This talk outlines how Conland addressed these challenges by developing a robust attack simulation platform, Simun, to gain better confidence in their detections and improve overall security posture.

The article delves into the technical specifics of building such a program, highlighting the limitations of existing open-source tools and the subsequent development of a custom solution. It explores the architectural decisions, key features like customizable attack techniques, log injection, and comprehensive metrics, and discusses the profound defensive implications for organizations grappling with similar multi-cloud security dilemmas. Lineitsev's experience serves as a valuable blueprint for security teams aiming to proactively validate their defenses against evolving cloud threats.

Background

▶ Watch: Why build a cloud attack simulation program? (0:40)

The genesis of Conland's cloud attack simulation program was rooted in a critical operational need: to ensure the continuous functionality and efficacy of their security detections across a sprawling multi-cloud infrastructure encompassing AWS, GCP, and Azure. Managing hundreds of unique detection rules and their corresponding data and logging pipelines across these diverse environments proved to be a constant uphill battle. The inherent differences in logging mechanisms, event formats, and API structures across CSPs meant that pipelines were prone to silent failures—SQS queues overfilling, event hubs throttling, or log sync filters unexpectedly breaking. Such failures could render entire detection rule sets ineffective, often going unnoticed until a real incident occurred, leading to significant visibility gaps and increased risk.

Before embarking on complex attack simulations, Lineitsev emphasized the importance of foundational work: ensuring robust data ingestion. This involves validating that all necessary logs are consistently flowing to the security lake or platform, and that underlying infrastructure components like queues and event hubs are operating within expected parameters. Only after establishing this baseline could the team confidently move towards end-to-end detection testing, conceptualized in three stages: detonate, ingest, and detect. "Detonate" involves simulating an attack activity, "ingest" refers to the (largely out-of-control) logging pipeline, and "detect" encompasses the security rules and enrichment processes.

In their search for existing solutions, Conland evaluated tools from DataDog, specifically Threatest and Stratus Red Team. Stratus Red Team is a well-known open-source tool that facilitates the execution of predefined attack techniques. It operates by deploying ephemeral infrastructure via Terraform, performing specific malicious actions, and then cleaning up the environment, providing a clean slate for validation. Threatest, while less widely known, implements the full end-to-end detection testing flow. It leverages Stratus to execute attacks and then interacts with the security information and event management (SIEM) system to verify if expected alerts were generated. A key mechanism for correlation in Threatest is the injection of unique strings into user agents during simulated attacks, allowing it to link observed SIEM events back to specific test activities. An example scenario involves referencing a specific Stratus technique and an expected alert, with a configurable timeout to ensure timely detection (e.g., within 10 minutes, not 10 hours).

Despite their utility, Threatest and Stratus Red Team presented several bottlenecks for Conland's specific requirements. Threatest, in particular, suffered from a lack of recent maintenance, with inconsistent development activity. Crucially, it was designed primarily for the DataDog Security Platform, requiring significant custom development to integrate with other SIEMs like Splunk, Panther, or Elastic, which Conland utilized. Furthermore, adding custom simulations relevant to an organization's unique threat landscape or specific red team findings was not straightforward; it typically necessitated forking Stratus or contributing upstream, which could be cumbersome for internal-only techniques. Finally, Threatest's command-line interface (CLI) deployment model was unsuitable for scheduled, consistent, and easily troubleshootable execution of hundreds of tests, lacking robust logging and operational features required for an enterprise-grade program. These limitations propelled Conland to develop their own "spiritual fork" of Threatest, named Simun.

Key Findings

▶ Watch: Detonate, Ingest, Detect: Core testing stages (2:47)

Conland's journey into cloud attack simulation, culminating in the development of Simun, yielded several critical findings that underscore the necessity and benefits of a dedicated, custom-built program.

Firstly, the talk highlighted the inherent fragility of multi-cloud detection pipelines. The speaker explicitly stated that "detection rules you have building are no longer working and you don't even know, and you will find out during an incident." This problem, experienced firsthand by Conland, became the primary driver for building Simun. The continuous, automated testing provided by Simun acts as an early warning system against such silent failures, validating not just the detection logic but also the underlying logging and data ingestion infrastructure.

A significant finding was the inadequacy of existing open-source tools like DataDog's Threatest and Stratus Red Team for enterprise-scale, multi-cloud environments with diverse SIEM integrations. While these tools offer a valuable starting point, their limitations in terms of maintenance, specific SIEM platform lock-in, complexity of adding custom attack techniques, and CLI-centric deployment model necessitated a more robust and flexible solution. Simun was born out of this need, demonstrating that for organizations with complex requirements, a tailored approach is often essential.

The program's most impactful outcome, beyond theoretical coverage metrics, was its ability to catch critical operational failures. Lineitsev recounted an incident where "our SIEM rules have been corrupted some of them due to an upgrade between major versions... lots of detections have stopped working." Simun's continuous testing process promptly alerted the team to this issue within an hour, allowing them to rectify it before it led to widespread visibility loss during a potential incident. This real-world validation underscores the tangible value of a continuous attack simulation program in maintaining operational resilience.

Finally, the talk emphasized the importance of driving the attack simulation roadmap with real-world threat intelligence and organizational needs. Instead of creating simulations for "interesting" but irrelevant scenarios, Conland's approach was heavily influenced by the findings of internal red team exercises. As Lineitsev explained, "we kind of trying to use what we really have at our table to drive this forward... real driven by business needs mostly." This ensures that the simulations directly address known gaps and vulnerabilities specific to the organization's environment, leading to a more effective and targeted improvement of defensive capabilities. The metrics tracked by Simun, such as MITRE ATT&CK coverage and SIEM rule coverage, further reinforce this data-driven approach, providing clear insights into detection efficacy and areas needing attention.

Technical Deep Dive

▶ Watch: Overview of Stratus Red Team and Threatest (3:28)

Conland's Simun platform emerged as a "spiritual fork" of Threatest, designed to overcome its limitations and provide a more comprehensive, customizable, and enterprise-grade solution for cloud attack simulation. The development roadmap for Simun evolved from essential "must-haves" to desirable "nice-to-haves," shaping its feature set.

A foundational "must-have" for Simun was the ability to integrate with any SIEM platform. Unlike Threatest's DataDog-specific integration, Simun was engineered to wire into various SIEMs, including Splunk, Panther, and Elastic (which Conland uses). This required building flexible connectors and ensuring that logging, troubleshooting, and result interpretation were consistent and reliable, making it a viable day-to-day operational tool. Lineitsev noted that achieving this baseline integration, along with a robust deployment model, could be a reasonable stopping point for organizations not requiring extensive custom attack techniques.

However, for organizations like Conland, which required deep customization, Simun introduced a proprietary pack subsystem. This subsystem was developed as an alternative to directly forking Stratus Red Team, which is an actively developed project and maintaining a fork would incur significant overhead. The pack subsystem allows for the definition of attack techniques that are completely customizable and highly parameterizable. This is crucial for environments where security controls, network configurations, or resource tagging requirements (e.g., specific tags on all cloud resources) differ across regions, accounts, or organizational units (OUs). Simulations can be tuned with various knobs to deploy and test specific scenarios under precise conditions. The detonate function within these packs adheres to a stricter contract for inputs and outputs compared to Stratus, which helps control situations where Terraform configurations might be mismanaged, enhancing reliability for an enterprise-grade tool.

A significant innovation within Simun is its log injection mechanism. This feature addresses the challenge of simulating activities that are either impossible or impractical to perform directly in a testing environment, such as logging in as a root user, creating API tokens in a SaaS application like Octa, or altering identity provider (IdP) settings. For these scenarios, Simun allows users to define templates for log sources (e.g., simple JSON structures) and then dynamically inject logs into the SIEM. These injected logs can be customized on the fly with different actors, regions, IP addresses, event names, and event types. While this approach sacrifices the end-to-end testing of the logging pipeline itself, it remains highly effective for validating the detection logic (the "detect" stage) for critical, hard-to-simulate events.

Simun also extends its capabilities with customizable matching and analysis features. An explore mode enables security teams to run new attack techniques and observe what alerts are triggered in their environment, facilitating discovery and validation of new detections. A collect mode is designed for troubleshooting; if detections are not firing as expected, this mode helps gather raw logs from the security platform to diagnose issues such as over-filtering or misconfigurations. Furthermore, Simun supports custom indicator matching beyond the default user-agent correlation. For certain data sources or attack types, user agents may not be available or unique enough for correlation. In such cases, Simun allows the use of other unique indicators, like specific IP addresses, to link simulated activity to observed logs. An illustrative example provided was the simulation of credential theft from AWS, followed by an attempt to use those credentials from a random region in Asia or GCP, or using EKS tokens from a Kubernetes pod in an external context, where IP addresses become the primary correlation mechanism.

Recognizing the limitations of a CLI tool for complex operations and user collaboration, Conland integrated a sophisticated web UI into Simun. This UI significantly improves usability, facilitates contributions from various team members, and centralizes management of the program. To further enhance scalability and manageability, Simun introduced the concept of connectors. These connectors allow the platform to manage multiple sandbox accounts, which might reside in different OUs or organizations, enabling seamless switching between scenarios without the need to run multiple instances of the simulation tool.

Crucially, Simun incorporates robust metrics to track the program's effectiveness and guide its evolution. Two key metrics are emphasized:

1. Diversity of attack simulations: This metric helps visualize if simulations are overly focused on specific attack types (e.g., exfiltration or scanning) while neglecting others, prompting teams to broaden their coverage.
2. Mapping between detection rules and simulations: This provides a clear understanding of how much of the actual detection rule base is covered by existing simulations, helping answer critical questions like "how many critical detections do we cover?" and identifying gaps.

The overall flow of Simun mirrors Threatest but with significant enhancements. Users set up connectors instead of local credentials, install custom or Stratus-adapted packs, run simulations, and then retrieve detailed results, often visualized through the web UI. The talk included screenshots demonstrating the scenario management, a scenario builder that pulls attack techniques and rules from installed packs and the SIEM, a completed assessment for an Azure disk export technique showing match times and logs, views of Stratus and custom packs, log injection examples (simple JSON for API token creation), and customization options for tags and regions. Visualizations of MITRE ATT&CK coverage and SIEM coverage were also shown, directly linking detected techniques to relevant scenarios and highlighting uncovered detections.

Finally, Conland's commitment to community learning is reflected in their decision to open source Simun. While not mandating its use, they offer it as a valuable example of how organizations can build upon tools like Threatest, leveraging modern AI capabilities to accelerate development. The open-source bundle includes a Stratus adapter to convert Stratus techniques into Simun simulations, allowing immediate benefit from existing attack libraries, and a sample Simun pack for reference.

Demo / Proof of Concept

▶ Watch: Limitations and challenges of existing tools (6:00)

During the presentation, Pavel Lineitsev showcased several aspects of Simun's functionality through screenshots of its web UI and attempted a live demonstration. While the live demo encountered technical difficulties and did not run as intended, the UI screenshots provided a clear visual understanding of the platform's capabilities and user experience.

The demonstration started with a view of a scenario, which is the core organizational unit in Simun. The UI showed details such as schedules for automated runs, target accounts for simulations, and a history of past runs. It also displayed the underlying YAML configuration, illustrating how scenarios are defined and managed.

A key feature highlighted was the scenario builder, a graphical interface designed to simplify the creation of new simulations. Instead of manual YAML editing, this builder allows users to select attack techniques from installed packs, configure custom outputs, and link to detection rules pulled directly from the SIEM. This centralized UI streamlines the process of defining and customizing simulations, making it more accessible for detection engineers.

Lineitsev then presented a screenshot of a completed assessment for an Azure technique involving disk export. This view detailed the simulation ID, the time taken for execution and detection matching, and relevant logs, providing clear evidence of the simulation's outcome and the detection's performance.

The packs section of the UI was also shown, illustrating how Simun incorporates both the Stratus Red Team techniques (via a custom adapter) and Conland's own custom-built attack simulations. This demonstrated the flexibility of the platform to leverage existing community resources while also enabling internal, organization-specific threat modeling. An example of a log injection was displayed, showing a simple JSON structure used to simulate API token creation, emphasizing the customizability of injected log events.

Further customization capabilities were shown, allowing users to inject custom tags and specify different regions for their simulations. This is particularly useful for organizations with strict resource tagging policies or region-specific security controls, enabling highly granular testing.

Finally, the UI showcased two critical metrics visualizations: a MITRE ATT&CK coverage matrix and a SIEM coverage dashboard. The MITRE matrix dynamically pulls data from all installed packs to visualize the organization's coverage against the MITRE ATT&CK framework. The SIEM coverage dashboard, drawing data directly from the SIEM, provided a clear overview of which detection rules were covered by existing simulations, which were not, and even linked back to the relevant scenarios, providing actionable insights for improving detection efficacy.

Although the live demo portion did not execute as planned, the comprehensive UI screenshots effectively conveyed the power, flexibility, and user-centric design of Simun, demonstrating how Conland operationalizes its cloud attack simulation program.

Defensive Implications

▶ Watch: Introducing Simun: A custom attack simulation tool (8:00)

Pavel Lineitsev's talk on building a cloud attack simulation program carries profound implications for defensive security strategies, especially for organizations operating in complex multi-cloud environments. The insights gleaned from Conland's journey offer actionable recommendations for enhancing detection and response capabilities.

Firstly, the most significant implication is the absolute necessity of proactive and continuous validation of security controls. The core problem Simun addresses—silent failures in logging pipelines and detection rules—is a universal challenge. Defenders can no longer assume their detections are working simply because they were configured correctly once. Implementing an automated, scheduled attack simulation program is crucial to catch these "silent breaks" before they are exploited during an actual incident. This shifts security from a reactive posture to a proactive one, ensuring that the critical "detect" stage of the kill chain is always operational.

Secondly, the program emphasizes the importance of comprehensive coverage across diverse cloud environments. For organizations leveraging AWS, GCP, and Azure, the inherent differences in their ecosystems mean that a detection strategy for one CSP may not translate effectively to another. An attack simulation program, particularly one with connectors like Simun, enables defenders to systematically test their controls across all cloud providers, accounts, and regions, identifying and remediating blind spots that might otherwise go unnoticed.

Thirdly, the focus on custom threat modeling is a vital defensive strategy. Instead of relying solely on generic threat intelligence, organizations should integrate insights from internal red team exercises, vulnerability assessments, and real-world incidents into their simulation program. The pack subsystem in Simun, allowing for highly customizable and parameterizable attack techniques, empowers defenders to create simulations that precisely mirror the tactics, techniques, and procedures (TTPs) most relevant to their specific threat landscape, ensuring that their defenses are tailored to their highest-priority risks.

Furthermore, the implementation of log injection mechanisms has significant implications for testing hard-to-simulate scenarios. For sensitive actions (e.g., root user logins, critical SaaS platform API calls), direct simulation might be disruptive or forbidden. Log injection provides a safe and effective way to validate the detection logic for these high-impact events, ensuring that even if the logging pipeline itself isn't fully tested end-to-end, the SIEM rules designed to catch them are functional. This enables defenders to gain confidence in detections for critical, high-privilege activities without risking production environments.

The emphasis on custom indicator matching (beyond user agents) highlights the need for flexible correlation strategies. Defenders should be prepared to use various unique identifiers—such as IP addresses, unique resource names, or specific metadata—to correlate simulated attack activity with observed logs. This is particularly relevant for complex attack chains involving credential theft and lateral movement across different cloud services or even external environments, where traditional user-agent correlation may be insufficient.

Finally, the focus on metric-driven security improvements offers a clear path for defensive teams to demonstrate value and prioritize efforts. By tracking attack diversity and SIEM rule coverage, defenders can identify areas of over-focus or neglect in their simulation efforts. This data-driven approach allows security leaders to make informed decisions about resource allocation, prioritize the development of new detection rules, and justify investments in security tooling and personnel, ultimately leading to a more mature and resilient security posture. The rapid detection of a corrupted SIEM rule set, as recounted by Lineitsev, serves as a powerful testament to the program's ability to prevent significant visibility loss and maintain operational security.

Key Takeaways

• Continuous validation is non-negotiable for multi-cloud security: Operating across AWS, GCP, and Azure introduces significant complexity, making logging and detection pipelines prone to silent failures. An automated, end-to-end attack simulation program is crucial for continuously validating that security controls are functional and effective, preventing critical visibility gaps during incidents.
• Existing tools provide a starting point but often fall short for enterprise needs: While DataDog's Threatest and Stratus Red Team offer valuable foundations, their limitations—such as platform lock-in (DataDog SIEM), difficulty in adding custom attack techniques, and CLI-centric deployment—necessitate custom solutions for organizations with complex, multi-cloud, and diverse SIEM environments.
• Customization and flexibility are paramount for tailored threat modeling: Building an internal solution like Simun allows for highly customizable and parameterizable attack techniques (via the pack subsystem), enabling organizations to create simulations that precisely reflect their unique threat landscape, specific red team findings, and varying security controls across different accounts, regions, or OUs.
• Beyond direct simulation: log injection for hard-to-test scenarios: For actions that are impossible or impractical to perform directly (e.g., root logins, sensitive SaaS API calls), a log injection mechanism provides a powerful alternative. While sacrificing end-to-end pipeline testing, it effectively validates the detection logic for critical, high-impact events.
• Robust operational features and metrics drive program success: An effective attack simulation program requires more than just attack execution. Features like a web UI, connectors for multi-account management, and comprehensive metrics (e.g., MITRE ATT&CK coverage, SIEM rule coverage, attack diversity) are essential for usability, scalability, demonstrating value, and guiding continuous improvement.
• Real-world threat intelligence should drive simulation development: The most impactful simulation programs are those guided by actual business needs, red team exercises, and observed threats, rather than arbitrary "interesting" scenarios. This ensures that defensive efforts are focused on the most relevant and critical risks.

About the Speaker(s)

Pavel Lineitsev is a member of the Detection and Response Team at Conland. His work focuses on building and implementing internal security programs, specifically in the realm of cloud attack simulation. With a background rooted in defensive security, Pavel brings a practitioner's perspective to the challenges of maintaining robust detection capabilities across complex, multi-cloud environments. His insights are particularly valuable for security teams grappling with the practicalities of validating and improving their security posture against evolving cloud threats.