Chaos Stack: Designing Layered Tabletop Exercises for Complex Crisis Simulation

Chaos Stack: Designing Layered Tabletop Exercises for Complex Crisis Simulation

Speakers: John Hollenberger (Fortinet)

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=LuLwHg8Fi4M

Overview

In the realm of cybersecurity, traditional tabletop exercises often fall short in preparing organizations for the chaotic reality of a real-world incident. John Hollenberger's talk, "Chaos Stack: Designing Layered Tabletop Exercises for Complex Crisis Simulation," introduces a novel approach designed to inject the necessary complexity, pressure, and multi-faceted challenges that characterize actual security breaches and operational disruptions. This methodology, dubbed Chaos Stack, aims to transcend the linear, single-threaded scenarios typically encountered in conventional exercises, forcing participants to make critical decisions under duress and with incomplete information.

Hollenberger, representing Fortinet, articulates a compelling case for moving beyond predictable simulations to embrace a more dynamic and realistic training environment. The core premise is that real incidents are inherently messy, fast-paced, and multi-threaded, demanding adaptive thinking, rapid prioritization, and robust cross-functional communication. By intentionally layering crises and introducing new information every 5 to 10 minutes, Chaos Stack replicates the overwhelming nature of a genuine incident, preparing teams not just for technical challenges but also for the psychological and organizational pressures that accompany them.

This presentation is crucial for any organization seeking to enhance its incident response capabilities, validate its playbooks, and identify critical gaps in its security posture. By simulating the "fog of war" that defines a major cyber crisis, Chaos Stack empowers teams to build resilience, refine decision-making processes, and foster seamless collaboration across diverse departments, from IT and security to legal, public relations, and executive leadership. The insights offered provide a practical framework for designing exercises that truly challenge and prepare an organization for the inevitable complexities of a security event.

Background

▶ Watch: Introduction and Traditional Tabletop Shortcomings (0:00)

Traditional tabletop exercises, while foundational, frequently fail to accurately model the inherent chaos and multifaceted nature of real-world security incidents. Hollenberger highlights several critical shortcomings: they are often linear and single-threaded, presenting one problem at a time with ample time for discussion before progressing. This results in clean timelines, where participants focus on a specific aspect for 15-20 minutes, leading to minimal cross-functional pressure. Exercises might isolate technical issues from business or executive concerns, preventing a holistic understanding of impact. Furthermore, they often provide full information and exhibit little consequence chaining, meaning decisions rarely pivot the scenario or lead to escalating, unforeseen problems. The scenario is typically fixed, offering no dynamic response to participant actions.

The fundamental reason these traditional approaches fall short is that real incidents are anything but clean or predictable. They are messy, fast, multi-threaded, and multifaceted, with problems emerging from every angle simultaneously. Information is invariably incomplete, time pressure is intense, and roles and ownership often become unclear as events unfold. Consequences escalate rapidly, even while decisions are still being formulated. Hollenberger vividly illustrates this with an audio recording of a chaotic Los Angeles fire incident, where firefighters faced an explosion, multiple injuries, and a complete lack of clear information amidst overwhelming noise and pressure. This analogy underscores the need for incident response training to mirror such high-stakes, confusing environments.

To further contextualize this from a cyber perspective, Hollenberger references a real-world ransomware incident handled by his team in the educational sector. This incident, spanning just 24 hours, demonstrated the rapid escalation and concurrency of attacks that traditional exercises often miss. Within a single day, the organization experienced:

• Encryption of Nanx logical drives in the early hours.
• Large-scale encryption impacting both Windows and Linux systems.
• The creation of 5,000 rogue user accounts.
• Attempts at EDR bypass.

All these events unfolded in less than 24 hours, demanding simultaneous attention and decision-making across multiple vectors. This example powerfully articulates why organizations need to prepare for a "storm" of problems rather than a linear sequence, necessitating a methodology like Chaos Stack to build adaptive thinking, prioritization skills, and the ability to make critical decisions under extreme pressure.

Key Findings

▶ Watch: Real-World Multi-Threaded Ransomware Incident Example (8:00)

The central discovery and contribution of John Hollenberger's talk is the Chaos Stack methodology itself, a layered approach to tabletop exercises designed to simulate the authentic complexity and pressure of real-world cyber incidents. The core idea behind Chaos Stack is to create a layered crisis where new information is constantly introduced, often before earlier issues are fully resolved. Unlike traditional exercises, injects (new pieces of information or developments) are delivered every 5 to 10 minutes, maintaining a continuous state of flux and urgency. This frequent injection of information is crucial for replicating the chaotic, non-linear nature of actual incidents, thereby building stress and pressure that are absent in more controlled simulations.

Key findings and design pillars of Chaos Stack include:

• Multiple Domains Impacted: Exercises are designed to involve and affect various organizational domains simultaneously, including cyber, physical security, operations, executive leadership, technical teams, and legal. This necessitates broad cross-functional participation and highlights interdependencies.
• Decisions Impact Outcomes: A critical element is the dynamic nature of the exercise. The facilitator must be prepared to pivot the scenario based on the decisions made (or not made) by the participants. This means pre-planned outcomes or alternative injects must be ready to reflect the consequences of participant choices, making the exercise highly interactive and realistic.
• Replication of Real-World Complexity and Pressure: The overarching goal is to generate a feeling of anxiety and pressure, mirroring the "fog of war" experienced during a live incident. This environment is intentionally created to foster adaptive thinking, improve prioritization skills, and enhance communication under stress, ultimately preparing teams for actual incidents.
• Building Adaptive Thinking and Prioritization: By presenting multiple, concurrent problems, Chaos Stack forces participants to determine what takes precedence when resources are strained and information is incomplete. This directly combats the tendency to address issues sequentially, as often happens in traditional exercises.
• Forcing Decision-Making Under Pressure: Participants cannot simply "talk through" a problem until an easy answer emerges. New injects frequently arrive as decisions are being formulated, demanding quick assessments of whether to continue with a current task or pivot to a higher-priority issue.
• Overlapping Crises and Competing Priorities: The methodology mandates the inclusion of multiple events at once—for example, a ransomware outbreak quickly followed by a media leak, a physical disruption (e.g., door systems, cameras, manufacturing operations going down), and a regulatory inquiry. This creates a scenario where various stakeholders in the room will naturally have competing interests and priorities, leading to productive chaos.
• Facilitator's Role as "Bad Guy": The facilitator is responsible for strictly adhering to the rapid inject timeline, ensuring the pressure never lets up. They must also be ready to introduce additional injects or modify the scenario based on participant actions, requiring flexibility and quick thinking.
• Common Domains of Impact: While customizable, typical domains involved include cyber security, legal/regulatory, operations, public relations, leadership, and governance, ensuring a truly cross-functional audience and challenging every aspect of an organization's response capability.

These findings collectively underscore that effective incident preparedness goes beyond technical competence; it demands robust organizational resilience, agile decision-making, and seamless communication across all levels of an enterprise, all of which Chaos Stack is designed to cultivate.

Technical Deep Dive

▶ Watch: Step-by-Step Layered Scenario Example Walkthrough (13:00)

The Chaos Stack methodology is structured around a four-layer framework designed to progressively escalate and complicate a crisis, mimicking the rapid deterioration of a real incident. Each layer introduces new challenges, ensuring participants are continuously confronted with evolving circumstances and competing priorities.

The general structure of a Chaos Stack exercise unfolds as follows:

• Layer 1: Starts the Crisis: This initial inject kicks off the scenario, often presenting what appears to be a localized or non-cyber issue.
• Layer 2: Complicates the Crisis: The second layer adds pressure and stress, introducing new information that makes the initial problem more complex or widespread.
• Layer 3: Disrupts the Response: This layer typically introduces a critical piece of information that forces a re-evaluation of the response strategy, potentially revealing the true nature of the incident (e.g., confirming a cyber intrusion).
• Layer 4: Forces Executive Decisions: The final layer introduces high-level business, legal, or reputational pressures, requiring strategic decisions from leadership.

Hollenberger provides an example timeline illustrating this rapid layering:

• 10 minutes in: Layer 1 introduces a system outage.
• Soon after: Layer 2 adds a safety impact.
• Minutes later: Layer 3 confirms a cyber intrusion.
• Immediately following: Layer 4 brings in media and legal pressure.

All these critical developments are designed to hit participants within a 10 to 20-minute window, creating intense pressure and forcing rapid, multi-threaded problem-solving.

When designing injects, Hollenberger recommends a granular approach. If an inject has multiple bullet points, instead of presenting them all at once, reveal them every few minutes. For instance, if an inject is scheduled for every 10 minutes, each bullet point could be revealed every 2-3 minutes, maintaining a constant flow of new information and discussion points.

A detailed example scenario demonstrates the practical application of Chaos Stack:

Overarching Scenario: Corporate headquarters experiences an unexpected outage; building management systems (BMS) are failing, affecting multiple departments.

1. Inject 1 (Layer 1 - The Initial Outage):

• Initial Reveal: "The badge system is failing." (Focus: Facilities, Maintenance)
• 2-3 mins later: "The HVAC system resets." (Adds to facilities, potential operational impact)
• 2-3 mins later: "The phone system goes down." (Introduces IT/VoIP, broader communication issues)
• Discussion Focus: Facilities, maintenance, and IT discuss immediate response. At this stage, it's unclear if it's a cyber incident.

1. Inject 2 (Layer 2 - Complications and Escalation):

• "The HVAC system is online and operating, but parts of the building don't have climate control." (Ongoing operational issue, suggests deeper problem)
• "Conference areas are inaccessible." (Business impact)
• "Employees are reporting building alarms and system alerts on endpoints and servers." (Starts hinting at IT/security issues, but still ambiguous)
• Discussion Focus: Participants must triage multiple, seemingly disparate problems. Who owns what? Is there a common cause?

1. Inject 3 (Layer 3 - Cyber Intrusion Confirmed & Red Herring):

• "IT starts detecting unusual traffic from the building control servers." (Strong indicator of cyber activity)
• "There are unexpected logins." (Further cyber confirmation)
• "Similar incidents reported at other companies." (Suggests a coordinated attack or widespread vulnerability)
• "The VoIP vendor reached out and said, 'Sorry, we put a patch out there and it might have shut your system down.'" (A red herring! This intentionally misdirects the team. Early on, the phone system issue might have consumed significant discussion, only for it to be a benign, unrelated problem. This tests the team's ability to pivot and not get bogged down by false leads.)
• Discussion Focus: The team must quickly integrate new cyber intelligence, discard the red herring, and shift focus to the true threat. This is where the CISO and security teams become central.

1. Inject 4 (Layer 4 - Executive and External Pressure):

• "The operations director is pressuring for the systems to be back online, but the CISO wants the incident to be contained." (Classic organizational conflict: business continuity vs. containment.)
• "The news is involved, asking about safety and data exposure." (Public relations and reputational risk)
• "Legal is asking about reporting obligations." (Regulatory and compliance pressure)
• "Executives are asking for an immediate status update in 20 minutes." (Intense time pressure on leadership)
• Discussion Focus: High-level strategic decisions, stakeholder management, crisis communication. This layer forces trade-offs and requires clear leadership.

This scenario, unfolding within roughly 20 minutes, creates overlapping crises, competing priorities, resource strain, and escalating consequences. It forces participants to grapple with decision interdependency, where early delays or missteps lead to bigger problems (e.g., missed legal input causing greater exposure, poor communication leading to reputational damage).

The design framework for Chaos Stack involves:

1. Objective Setting: Clearly defining the goals (e.g., testing communication, validating a playbook).
2. Domain Definition: Identifying all relevant cross-functional teams.
3. Scenario Choice: Selecting a realistic core incident relevant to the organization.
4. Layer Addition: Building out the injects according to the 4-layer structure and rapid timing.

A crucial technical aspect is the concept of consequence mapping or "if-then" planning. The facilitator must anticipate participant actions and prepare alternative injects. For example:

• "If within the first five minutes customer service or IT does not escalate to cyber, we're going to present Layer 2 (sooner)."
• "If the web and database server are not isolated within 5 minutes, then we're going to introduce Layer 2B, which indicates domain admin credentials are compromised."

This dynamic pivoting based on participant decisions is what elevates Chaos Stack beyond a mere walkthrough, making it a true simulation of adaptive incident response. The facilitator effectively "plays a game" with the participants, responding to their choices in real-time.

Demo / Proof of Concept

▶ Watch: Ultimate Goals: Simulate Fatigue, Uncertainty, Escalation (25:50)

While "Chaos Stack: Designing Layered Tabletop Exercises for Complex Crisis Simulation" does not feature a live software demonstration or a tangible proof-of-concept tool, the entire presentation serves as a highly detailed and effective walkthrough of the methodology in action. John Hollenberger's comprehensive explanation of the layered inject structure, coupled with a specific, multi-stage scenario, functions as the core demonstration of how the Chaos Stack approach is implemented and experienced.

The example scenario provided in the talk acts as the primary proof of concept. It illustrates, step-by-step, how a seemingly simple corporate headquarters outage can rapidly escalate into a complex, multi-domain cyber crisis. By breaking down the scenario into four distinct layers and detailing the content of each inject, Hollenberger effectively "runs" a mini-tabletop exercise for the audience.

This demonstration highlights:

• Progressive Escalation: How a crisis begins as a facilities/IT problem (badge system, HVAC, phone system failures) and quickly evolves into a confirmed cyber intrusion (unusual traffic from building control servers, unexpected logins, similar incidents elsewhere).
• Introduction of Red Herrings: The inclusion of the VoIP vendor's patch explanation showcases how unrelated issues can distract teams, testing their ability to prioritize and identify true threats.
• Competing Internal Pressures: The conflict between the operations director (demanding system restoration) and the CISO (prioritizing containment) clearly demonstrates the internal organizational friction that real incidents create.
• External Stakeholder Engagement: The involvement of the news and legal counsel illustrates the broader reputational and regulatory implications that extend beyond technical response.
• Time and Executive Pressure: The demand for an immediate executive status update within 20 minutes emphasizes the intense time constraints and high-stakes communication required.

Through this carefully constructed narrative, Hollenberger demonstrates the core principles of Chaos Stack: the rapid succession of injects (every 5-10 minutes), the involvement of multiple functional teams, the dynamic shift in priorities, and the escalating consequences based on unfolding information. The "if-then" logic for facilitator pivots, where participant decisions (e.g., failing to escalate to cyber, not isolating servers) trigger specific, more severe follow-up injects (e.g., earlier Layer 2 presentation, domain admin compromise), further reinforces the interactive and realistic nature of the methodology.

Ultimately, the detailed scenario walkthrough serves as a compelling proof that Chaos Stack is not merely a theoretical concept but a practical, actionable framework for designing tabletop exercises that genuinely challenge and prepare organizations for the multifaceted chaos of a real cyber crisis.

Defensive Implications

▶ Watch: Q&A: Evaluating Exercise Results (Focus on Gaps) (31:00)

The Chaos Stack methodology offers profound implications for defenders, providing a robust framework to significantly enhance organizational resilience and incident response capabilities. Organizations should proactively integrate these principles into their security training and preparedness programs.

1. Embrace Realistic Chaos: Defenders must move beyond linear, predictable exercises. Adopt Chaos Stack to simulate the messy, fast-paced, multi-threaded nature of real incidents. This means designing scenarios where information is incomplete, multiple problems emerge simultaneously, and time pressure is constant.
2. Foster Adaptive Thinking and Prioritization: Implement exercises that force teams to prioritize under pressure. When multiple crises (e.g., ransomware, physical disruption, media leak) hit concurrently, who does what? This helps identify gaps in roles, responsibilities, and escalation procedures.
3. Build Cross-Functional Collaboration: Ensure exercises involve a diverse group of stakeholders: cyber security, IT, legal, public relations, operations, executive leadership, and even physical security. Chaos is amplified when multiple teams with different objectives must coordinate. This helps in validating communication channels and decision-making hierarchies across the organization.
4. Practice Decision-Making Under Pressure: Design scenarios with "if-then" logic where participant decisions directly influence the unfolding narrative. This forces difficult trade-offs, such as:

• Containment vs. Business Continuity: When should systems be taken offline for containment, even if it impacts productivity and revenue?
• Public Communication vs. Legal Caution: What information can be released, and when, balancing transparency with legal risk?
• Forensics Preservation vs. Operational Restoration: How long should compromised systems be preserved for forensic analysis versus bringing operations back online?

1. Identify Gaps in People, Processes, and Technology: Use Chaos Stack exercises not as a scoring mechanism, but as a diagnostic tool. Look for:

• Information Gaps: Where are teams lacking critical data?
• Communication Breakdowns: Are escalations happening effectively? Are key stakeholders informed?
• Process Deficiencies: Are incident response plans and playbooks robust and up-to-date? Are they being followed?
• Decision Fatigue: Does the exercise reveal a tendency for teams to become overwhelmed, and how can this be mitigated?

1. Develop Dynamic Facilitation Skills: Facilitators must be prepared to pivot the scenario in real-time, introducing new injects or modifying existing ones based on participant actions. This requires pre-planning potential outcomes and having additional content ready.
2. Start with Realistic Core Incidents: Tailor scenarios to what realistically could happen to your organization. If you're a cloud-native company, focus on cloud application outages or executive account compromises, not necessarily building system failures. The initial inject should ideally appear as a non-security IT or operations problem, evolving into a security incident.
3. Utilize Provided Templates: Leverage the timeline template, scenario builder worksheet, and facilitator guide provided by Hollenberger to structure the exercise design process. These tools help map out injects, define objectives and domains, and plan for dynamic pivots.
4. Strategic Information Control: When preparing, gather necessary information from teams through brief, targeted meetings (e.g., 30-60 minutes), but keep the overall scenario details confidential from participants to maintain realism and prevent pre-gaming.
5. Optimal Participant and Duration Planning: Aim for a participant count of less than 20 to ensure active engagement, ideally with a primary and secondary representative from each critical team. Limit exercise duration to two hours, with a maximum of three, to prevent overwhelming participants while allowing sufficient time for complex discussions.

By systematically applying the Chaos Stack methodology, defenders can transform their tabletop exercises from static reviews into dynamic, high-fidelity simulations that build true organizational resilience against the inevitable chaos of a real cyber crisis.

Key Takeaways

• Realism is Paramount: Traditional, linear tabletop exercises fail to prepare organizations for the chaotic, multi-threaded nature of real-world cyber incidents. Chaos Stack aims to replicate this authentic pressure and complexity.
• Layered Crisis Simulation: The methodology involves rapidly layering new information and challenges every 5-10 minutes, forcing simultaneous decision-making and prioritization across multiple domains.
• Cross-Functional Engagement is Critical: Effective Chaos Stack exercises require active participation from diverse teams—cyber, IT, operations, legal, PR, and executive leadership—to expose interdependencies and communication gaps.
• Facilitator Agility is Key: Exercise facilitators must be prepared to dynamically pivot the scenario, introducing new injects or consequences based on participant decisions, making the exercise highly interactive and outcome-driven.
• Focus on Gaps, Not Scores: The primary goal of Chaos Stack is to identify weaknesses in an organization's people, processes, and technology, as well as to combat decision fatigue and uncertainty, rather than simply grading performance.
• Structured Design for Chaos: Despite its chaotic nature, Chaos Stack benefits from a structured design framework, including objective setting, domain definition, scenario selection, and consequence mapping ("if-then" logic) to ensure effective learning outcomes.

About the Speaker(s)

John Hollenberger is a professional from Fortinet, specializing in cybersecurity and incident response preparedness. His expertise lies in designing and facilitating advanced tabletop exercises that challenge organizations to improve their adaptive thinking, prioritization, and communication skills under the intense pressure of simulated crises. Through his work, Hollenberger advocates for moving beyond traditional, linear training methods to embrace more realistic and complex simulations, ensuring teams are truly ready for the multifaceted nature of real-world security incidents.