Breach & Attack Simulation - Continuous Security Validation (incl. live demo)

Raphael Ruf (Cyber Security Consultant · Swiss Post Cyber Security)

Swiss Cyber Storm 2024 · Day 1 · Panorama

Overview

In an era of increasingly sophisticated cyber threats and rapidly evolving attack surfaces, organizations invest significant resources into security controls. Yet, a fundamental question often remains unanswered: are these expensive tools truly effective, correctly configured, and providing the intended protection? Raphael Ruf, a Cyber Security Consultant at Swiss Post Cyber Security, addressed this critical challenge in his talk at Swiss Cyber Storm, introducing Breach and Attack Simulation (BAS) as a solution for continuous security validation. His presentation, complete with a live demonstration, illuminated how BAS platforms enable organizations to proactively test their defenses against real-world attack scenarios, ensuring their security posture remains robust against the most current threats.

Watch on YouTube · Slides

Visual summary for Breach & Attack Simulation - Continuous Security Validation (incl. live demo) by Raphael Ruf
Visual summary for Breach & Attack Simulation - Continuous Security Validation (incl. live demo) by Raphael Ruf

Key moments

  1. 0:54 What is Breach and Attack Simulation?
  2. 6:00 Deeper insights by integrating with security controls
  3. 10:40 Simulator deployment and advanced malware actions
  4. 16:00 Live demo: SafeBreach dashboard and scenarios
  5. 20:00 Live demo: Analyzing results and remediation
  6. 25:30 BAS: A perfect addition to manual pen tests
  7. 28:00 Q&A: Mobile device attack simulation coverage
  8. 29:00 Q&A: SaaS and cloud security validation

Breach & Attack Simulation - Continuous Security Validation (incl. live demo)

Speakers: Raphael Ruf, Cyber Security Consultant, Swiss Post Cyber Security

Conference: Swiss Cyber Storm

YouTube: https://www.youtube.com/watch?v=gghXitMW3gQ

Overview

In an era of increasingly sophisticated cyber threats and rapidly evolving attack surfaces, organizations invest significant resources into security controls. Yet, a fundamental question often remains unanswered: are these expensive tools truly effective, correctly configured, and providing the intended protection? Raphael Ruf, a Cyber Security Consultant at Swiss Post Cyber Security, addressed this critical challenge in his talk at Swiss Cyber Storm, introducing Breach and Attack Simulation (BAS) as a solution for continuous security validation. His presentation, complete with a live demonstration, illuminated how BAS platforms enable organizations to proactively test their defenses against real-world attack scenarios, ensuring their security posture remains robust against the most current threats.

Ruf's talk underscored the shift from reactive security measures to a proactive, continuous validation model. By deploying specialized simulators within an environment and continuously launching real attacks against them, organizations can gain objective, data-driven insights into the efficacy of their EDRs, network controls, SIEM integrations, and hardening policies. This approach helps identify critical gaps and misconfigurations before they can be exploited by malicious actors, thereby transforming the traditional "snapshot" view of security assessments into an ongoing, dynamic process of validation and improvement.

The importance of this methodology cannot be overstated. In a landscape where new vulnerabilities and attack techniques emerge daily, relying solely on periodic penetration tests or compliance audits leaves significant windows of exposure. BAS, as presented by Ruf, offers a vital layer of assurance, allowing security teams to continuously fine-tune their defenses, optimize their security investments, and significantly reduce overall organizational risk through automated, evidence-based validation.

Background

▶ Watch: What is Breach and Attack Simulation? (0:54)

Modern enterprises deploy a complex array of security tools designed to protect their assets from a multitude of threats. This typically includes Endpoint Detection and Response (EDR) solutions for endpoint protection, various network security controls (firewalls, intrusion detection/prevention systems), and a Security Information and Event Management (SIEM) system (such as Splunk or Microsoft Sentinel) to aggregate logs, perform analysis, and facilitate incident response. While these tools represent substantial investments, organizations frequently grapple with fundamental questions regarding their actual effectiveness.

Raphael Ruf highlighted common concerns that drive the need for continuous validation: "Do I have the right solutions, or are they configured the right way? Am I protected against the most current threats? Is the SIEM integration of all those tools actually working? Do the use cases I have cover the attacks that might happen? If something happens, how is the reaction of the SOC? Do I have any gaps or what are the risks?" These inquiries stem from the inherent complexity of integrating diverse security technologies and the dynamic nature of the threat landscape. Traditional methods, such as manual penetration tests or vulnerability assessments, offer only a snapshot in time. A penetration test report, for instance, is often outdated the moment it's delivered due to continuous changes in the environment or new threats emerging.

Ruf, drawing from his background as a mechanic in Operational Technology (OT) and his current role in vulnerability management and OT penetration testing at Swiss Post Cyber Security, emphasized the need for an automated mechanism to continuously validate these controls. This necessity led to the adoption of Breach and Attack Simulation (BAS) platforms, which provide an automated, proactive, and continuous method to ensure that security investments are indeed delivering the expected level of protection against evolving cyber threats.

Key Findings

▶ Watch: Simulator deployment and advanced malware actions (10:40)

The core finding presented by Raphael Ruf is that Breach and Attack Simulation (BAS) is synonymous with continuous automated security validation. Its primary objective is to automatically and continuously verify that all deployed security controls are correctly configured and operating as intended. This moves beyond a static "snapshot" assessment, providing ongoing assurance that defenses remain effective even as configurations change or new threats emerge.

A critical aspect of BAS, as implemented with the SafeBreach platform, is its use of real attacks executed by simulators deployed within the target environment. These simulators mimic both external attackers (from the cloud) and internal lateral movement, running actual attack techniques. Crucially, the platform integrates deeply with existing security controls, such as EDRs and SIEM systems. This integration allows for a nuanced assessment: not merely a binary "attack successful" or "attack failed," but also whether the attack was detected and logged by the relevant security tools (e.g., "attack successful but detected"). This full-circle validation provides invaluable insights into the detection capabilities of specific tools and the overall coverage of the security stack.

The typical BAS process involves several key steps:

  1. Continuous Simulation: Running a variety of attack scenarios, or "playbooks," which are regularly updated (typically within 24 hours of new threat emergence) to reflect the latest attack techniques.
  2. Results Analysis and Prioritization: Reviewing the outcomes of these simulations to identify successful attacks or detection gaps.
  3. Remediation: Acting on the findings to improve detection rules, refine use cases in the SIEM, enhance hardening policies, or even initiate projects for new security solutions (e.g., Data Loss Prevention).
  4. Risk Reduction and Repetition: Continuously iterating through this cycle to drive down risk and maintain a strong security posture.

This systematic approach allows organizations to precisely understand which security control covers which part of an attack chain and how effectively, moving beyond assumptions to data-driven insights.

Technical Deep Dive

▶ Watch: Live demo: Analyzing results and remediation (20:00)

The architecture of a typical Breach and Attack Simulation deployment, as exemplified by SafeBreach, consists of three main components: simulators, the Management Console, and attack playbooks.

  1. Simulators: These are lightweight software agents deployed on representative systems within the customer's environment. Crucially, these are not production systems but rather dedicated machines (clients and servers) that mirror the production environment in terms of operating system, image, hardening policies, and patching cycles. Their purpose is to act as both targets and launchpads for simulated attacks. One simulator is typically deployed in the cloud, acting as an external attacker or a target for data exfiltration. These agents are responsible for executing the actual attack actions.
  1. Management Console: This is the central brain of the BAS system. It can be deployed in the cloud by SafeBreach, on-premise within the customer's network, or even in an air-gapped configuration for highly sensitive environments. From this console, security teams manage all deployed simulators, configure and schedule attack scenarios, and analyze the results. It serves as the primary interface for interacting with the BAS platform.
  1. Attack Playbooks: These are extensive libraries of pre-defined attack scenarios and actions, meticulously curated and regularly updated by SafeBreach. These playbooks reflect the latest threat intelligence, including emerging attack groups and techniques. This ensures that the simulations remain relevant and challenging, constantly testing defenses against current attack methodologies, often with updates available within 24 hours of a new threat's identification.

A typical on-premise deployment involves a dedicated manager appliance within the customer's network. This manager communicates with a cloud-based SafeBreach simulator and multiple customer-provided simulators (e.g., a Windows client, a Linux server) installed with the agent software. The manager also integrates directly with the customer's existing security controls, such as on-premise SIEMs like Splunk or cloud-based SIEMs like Microsoft Sentinel, to collect detection logs and events.

With this basic setup, the platform can simulate a wide array of attack vectors:

  • Infiltration: Attacks originating from the cloud simulator targeting internal client or server simulators.
  • EDR Validation: Testing the endpoint detection and response capabilities of the EDR on individual simulators.
  • Lateral Movement: Simulating an attacker moving between different internal simulators (e.g., from a compromised client to a server).
  • Exfiltration: Testing the ability to extract data from internal simulators to the cloud simulator.

This allows for the simulation of a complete attack chain through the enterprise's security stack.

For optimal coverage, a minimum of five simulators is recommended: one in the cloud (external attacker), one client, two servers (e.g., Windows and Linux), and a specialized Advanced Actions Simulator. This fifth simulator is an isolated system, completely disconnected from the customer's production network. Its purpose is to safely execute real malware in a pre-execution state (loading into memory) to test EDR capabilities without the risk of actual ransomware deployment or system compromise. This design mitigates the slim but non-zero risk associated with running certain advanced attack techniques. Organizations can expand their simulator deployment to cover various operating system versions (e.g., Server 2016, 2022, 2012), different hardening policies, or specific client types (e.g., administrator workstations).

To guide customers, SafeBreach offers predefined scenario baselines. The "Initial Baseline" covers fundamental attacks like infiltration, host-level attacks, lateral movement, exfiltration, email-based attacks, endpoint and network specific tests, and ransomware simulations. As customers mature, they can progress to a "Full Baseline" or add custom scenarios, for instance, to simulate attacks from a specific threat actor group that has targeted a competitor. This structured approach not only helps customers prioritize improvements but also allows for benchmarking against peers.

Beyond the technical deployment, a continuous service model is essential. Swiss Post Cyber Security provides quarterly status reports and enhancement sessions, helping customers prioritize findings, identify low-hanging fruits, and develop mitigation strategies. Furthermore, integration with the SIEM allows for immediate notification if security scores drop due to configuration changes or new vulnerabilities, ensuring continuous monitoring and rapid response.

Demo / Proof of Concept

▶ Watch: BAS: A perfect addition to manual pen tests (25:30)

Raphael Ruf provided a live demonstration of the SafeBreach platform, showcasing its intuitive dashboard and powerful capabilities. Upon logging in, users are presented with a concise overview, including an arbitrary "security score" (e.g., 30 out of 100 in the demo environment, intentionally vulnerable for demonstration purposes). This score is often benchmarked against other SafeBreach customers, providing context. The dashboard visually segments the security posture across different domains like network, endpoint, and email, offering an immediate glance at overall security health.

The core of the platform lies in its extensive scenario library. Ruf navigated through a vast collection of pre-prepared attack scenarios, categorized for ease of use. These include "Getting Started" scenarios covering infiltration, endpoint, lateral movement, and outbound attacks, alongside specialized categories for endpoint security, network security, email security, data security, cloud security, and even scenarios grouped by specific attack groups (e.g., APTs). This library allows security teams to select and configure simulations tailored to their environment and specific concerns.

Ruf illustrated a ransomware execution scenario, detailing its multi-stage attack chain: email attack, malware transfer, malware drop, pre-execution phase, ransomware execution, and finally, the encryption phase. A key design principle highlighted was the atomic nature of these tests. Even if the initial email security layer successfully blocked all 622 simulated malicious emails, the platform would still proceed to test the subsequent stages, such as malware transfer and drop, independently. This ensures comprehensive validation across the entire attack chain, preventing a single strong defense from masking weaknesses further down the line.

Configuring a scenario involves selecting the appropriate simulators to act as attackers and victims. For an email attack, an external cloud simulator might target an internal client simulator. For lateral movement, two internal simulators would be chosen. Endpoint-specific tests run directly on the individual endpoint simulators.

Upon completion of a scenario, the platform presents detailed results, including a specific score for that scenario. With SIEM integrations, the view branches out to show precisely what each security control (e.g., Microsoft Defender) detected. The "Insights" section is particularly valuable, offering SafeBreach's recommendations for remediation. These insights are grouped by topic, such as "prevent user with insufficient permission from performing 37 system information collection attacks." Ruf emphasized that these remediations are rarely simple patch installations. Instead, they typically involve adjusting hardening policies, refining EDR rules, or even identifying larger architectural gaps, such as the absence of a Data Loss Prevention (DLP) system, which might necessitate a new project to evaluate and implement a solution. The BAS tool can then be used to test different DLP solutions during a Proof of Concept (POC) phase.

For security analysts, the platform provides deep-dive capabilities. For any simulated action, an analyst can view its description, recommended remediation, and crucially, the simulation logs detailing exactly what was executed on the simulator. Furthermore, with SIEM integration, all relevant security events collected by controls like Defender (e.g., detection of Mimikatz) are directly linked, allowing analysts to quickly debug why an attack might not have been detected or why logs were missing. Finally, the demo concluded by showing the centralized management of all simulator agents from the platform, eliminating the need for direct access to individual systems for updates or status checks.

Defensive Implications

▶ Watch: Q&A: SaaS and cloud security validation (29:00)

The insights gained from Breach and Attack Simulation have profound implications for defensive security strategies. Firstly, BAS provides objective, continuous validation of security controls, allowing organizations to quantify the effectiveness of their significant investments in EDR, SIEM, network security, and other tools. This moves beyond assumptions and delivers concrete data on what is working and what is not.

Secondly, by running simulations continuously, organizations can fix issues before they become problems. If a configuration change inadvertently degrades a security control's performance, the BAS platform will detect it quickly, providing immediate notification through SIEM integration. This proactive stance significantly reduces the window of vulnerability that attackers might otherwise exploit.

Thirdly, Raphael Ruf stressed that BAS is a perfect addition, not a replacement, for manual penetration tests. While pen testers might uncover forgotten vulnerable systems or outdated software that BAS, operating on designated simulators, would miss, a pen test report is inherently a snapshot and quickly becomes outdated. BAS, on the other hand, offers continuous, real-time validation of the effectiveness of the deployed security stack against current threats. It complements the deep, human-driven insights of a pen test with automated, persistent verification.

A crucial distinction highlighted is that BAS targets production security controls (e.g., the EDR, SIEM, network firewalls protecting the simulators), not production systems themselves. The simulators are representative, isolated systems. However, defenders must be aware that simulations can trigger legitimate security responses. For instance, if an organization has automatic host isolation policies, a simulator might indeed get isolated during an attack simulation, demonstrating the real-world impact of the defensive controls.

Finally, the findings from BAS often point to deeper defensive improvements than simple patching. Remediations frequently involve refining hardening policies, optimizing EDR rules, adjusting SIEM use cases, or even identifying the need for entirely new security solutions like a Data Loss Prevention (DLP) system. This drives a continuous process of security posture improvement, ensuring that defenses are not only in place but are also optimally configured and responsive to the evolving threat landscape.

Key Takeaways

  • Continuous Validation is Essential: Breach and Attack Simulation (BAS) provides automated, continuous validation of security controls, moving beyond static assessments to ensure ongoing effectiveness against evolving threats.
  • Real Attacks, Real Insights: BAS utilizes real attack techniques executed on representative systems, offering objective data on the true efficacy of EDRs, network controls, and SIEM detections.
  • Deep Integration for Full Visibility: By integrating with existing security controls, BAS provides nuanced results, differentiating between successful attacks, detected attacks, and blocked attacks, showing precisely which control caught what.
  • Atomic Testing Ensures Comprehensive Coverage: Attack chains are broken down into atomic tests, ensuring that even if an early stage of an attack is blocked, subsequent stages are still tested to uncover deeper vulnerabilities.
  • Focus on Policy and Architecture: Remediation efforts often involve refining hardening policies, optimizing EDR configurations, enhancing SIEM use cases, or addressing architectural gaps (e.g., lack of DLP), rather than just applying patches.
  • Complement to Penetration Testing: BAS serves as a powerful complement to manual penetration tests, offering continuous, automated verification of security posture while pen tests provide human-driven, in-depth vulnerability discovery at a specific point in time.

About the Speaker(s)

Raphael Ruf is a Cyber Security Consultant at Swiss Post Cyber Security, an entity formerly known as Tractive before its acquisition by Swiss Post. With a unique background as a mechanic, he possesses an Operational Technology (OT) foundation, which he leverages in his work, including conducting penetration tests in OT environments. At Swiss Post Cyber Security, Raphael is primarily responsible for vulnerability management and the implementation and management of breach and attack simulation initiatives.

Reviews

Dr. Zero (Offensive Security Researcher) — WEAK

This was a SafeBreach product demo delivered as a methodology talk. Automated unit testing for your EDR is a standard operational practice, but watching a dashboard walkthrough doesn't belong on a conference stage.

Heather Calloway (CISO) — WEAK

This is effectively a SafeBreach product demonstration wrapped in a methodology presentation. While continuous security validation is a mandatory capability for justifying security spend, your leadership team doesn't need to watch a consultant navigate a vendor dashboard.

→ Top-rated talks at Swiss Cyber Storm 2024

All talks from Swiss Cyber Storm 2024