All hands on deck needed for Everything Everywhere All At Once

Andrew Dettmer (Black & Veatch), Katrina Rosseini (CR-ISAC)

Critical Effect DC 2025 · Day 1 · Track II: Strategic Effect

Overview

In an era where critical infrastructure faces escalating and increasingly sophisticated cyber threats, the conventional wisdom of incident response planning is no longer sufficient. This pivotal talk, "All hands on deck needed for Everything Everywhere All At Once," delivered by Andrew Dettmer of Black & Veatch and Katrina Rosseini of CR-ISAC, challenges deeply ingrained assumptions about resource availability and response capabilities during a widespread cyber attack. The speakers introduced the concept of "Everything Everywhere All At Once" attacks, envisioning scenarios where entire swaths of industry are targeted simultaneously, rather than isolated facilities or fleets.

Watch on YouTube

Visual summary for All hands on deck needed for Everything Everywhere All At Once by Andrew Dettmer, Katrina Rosseini
Visual summary for All hands on deck needed for Everything Everywhere All At Once by Andrew Dettmer, Katrina Rosseini

Key moments

  1. 0:30 Introducing 'Everything Everywhere All At Once' attack
  2. 3:00 Bricking controllers: a new, insidious attack vector
  3. 11:00 Addressing critical infrastructure cybersecurity workforce shortage
  4. 14:00 Ukraine's lesson: civilians must be prepared beforehand
  5. 15:30 CR-ISAC's progress and call for strategic partners
  6. 17:00 Developing a civilian defense mindset across communities
  7. 20:00 Cyber insurance's role in driving cyber resilience

All hands on deck needed for Everything Everywhere All At Once

Speakers: Andrew Dettmer (Black & Veatch); Katrina Rosseini (CR-ISAC)

Conference: Critical Effect DC 2025

YouTube: https://www.youtube.com/watch?v=S1bAQsph75o

Overview

In an era where critical infrastructure faces escalating and increasingly sophisticated cyber threats, the conventional wisdom of incident response planning is no longer sufficient. This pivotal talk, "All hands on deck needed for Everything Everywhere All At Once," delivered by Andrew Dettmer of Black & Veatch and Katrina Rosseini of CR-ISAC, challenges deeply ingrained assumptions about resource availability and response capabilities during a widespread cyber attack. The speakers introduced the concept of "Everything Everywhere All At Once" attacks, envisioning scenarios where entire swaths of industry are targeted simultaneously, rather than isolated facilities or fleets.

The core message of the presentation is a stark warning: current incident response paradigms, often focused on singular events or localized disruptions, are ill-equipped to handle the systemic impact of coordinated, multi-sector cyber campaigns. Dettmer meticulously dissects the vulnerabilities inherent in critical infrastructure's reliance on fragile supply chains and inadequate cyber-resilient architectures, particularly within Industrial Control Systems (ICS) and Operational Technology (OT). Rosseini then expands on the critical shortage of skilled cybersecurity professionals and introduces the Civilian Reserve ISAC (CR-ISAC) as a proactive, community-driven solution to build a national cyber reserve workforce. This talk is a clarion call for a fundamental shift in mindset, urging both public and private sectors to collaborate on a collective, community-wide defense strategy.

Background

▶ Watch: Introducing 'Everything Everywhere All At Once' attack (0:30)

Andrew Dettmer initiated the discussion by drawing on his decade of experience at a Generation & Transmission (G&T) cooperative in the Midwest, where he oversaw cyber controls for a nine-plant fleet. His insights were sharpened by events like the Volt Typhoon announcement, which highlighted the adversary's preference for living off the land techniques on network devices. While traditional security measures focus on detecting such intrusions, Dettmer began to question the broader strategic implications for incident response planning. He observed that most tabletop exercises and IR plans center on scenarios where a single plant or, at best, an entire fleet is attacked. This narrow focus, he argued, fails to account for a more insidious threat: the "Everything Everywhere All At Once" attack, which targets not just an individual entity but multiple neighboring utilities or peers running similar Distributed Control Systems (DCSs).

This new threat model challenges two critical assumptions underpinning current incident response strategies: the availability of hardware and the availability of resources (personnel, specialized services). Dettmer recalled the COVID-19 backlogs as a recent, tangible example of how quickly supply chains can be disrupted, rendering the assumption of readily available hardware dangerously naive. Traditionally, the "worst possible day" at a power plant revolved around physical destruction—such as a hydrogen-cooled generator exploding or a turbine over-spinning at 3600 RPM, causing catastrophic structural damage and potentially dozens of deaths. However, Dettmer posits that a sophisticated adversary might opt for a less conspicuous, yet equally devastating, approach: the bricking of controllers. Such an attack, while not immediately causing visible physical destruction, could cripple operations for years by creating an insurmountable backlog for replacement hardware, akin to the three-year lead times for new generators or step-up transformers. Current ICS/OT systems, while often featuring physical redundancy, are typically designed to handle physical or electrical events, not cyber attacks. The fatal flaw lies in their architecture: primary and backup controllers, operator consoles, servers, and network hardware often reside on the same flat network, making them vulnerable to a single, widespread cyber event. This fundamental architectural weakness, coupled with the prevalence of end-of-support devices in production environments, significantly elevates the risk profile for critical infrastructure.

Key Findings

▶ Watch: Addressing critical infrastructure cybersecurity workforce shortage (11:00)

The talk unveiled several critical findings that demand urgent attention from critical infrastructure operators and policymakers:

  1. Shifting Risk Calculus for Cyber Resilience: The cost-benefit analysis for re-engineering ICS/OT networks to achieve true cyber resilience has fundamentally changed. While expensive, the potential for prolonged operational outages due to "bricked" controllers and supply chain disruptions now justifies significant investment. The speakers emphasized that this proactive re-engineering is far more advantageous when undertaken on an organization's terms, rather than reactively during an attack.
  2. The Death Knell for End-of-Support Devices: The continued operation of end-of-support (EOS) devices in production environments is an unacceptable risk. In a scenario where replacements are scarce, these systems become critical points of failure, exacerbating recovery efforts. Their immediate replacement or upgrade is imperative.
  3. Strategic Spares as a Necessity: While costly, maintaining full sets of spares for critical control system components is transitioning from a luxury to a necessity. This investment not only mitigates supply chain risks during an incident but also supports production lines, potentially easing future supply chain pressures.
  4. Inadequate Incident Response Planning: Current incident response retainers and agreements with OEMs and integrators are largely insufficient for "Everything Everywhere All At Once" scenarios. Organizations must explicitly question their IR providers and vendors about their plans for widespread, multi-entity attacks, verifying their capacity and readiness.
  5. Vulnerable Communication Networks: Reliance on corporate communication networks or even personal cell phones for incident communication is a significant risk. These channels are also susceptible to compromise or disruption, necessitating the development of resilient, out-of-band communication strategies.
  6. Limitations of Mutual Aid Models: While mutual aid agreements (like those for power distribution linemen post-tornado) are effective for physical events, a direct analogue for cyber incident response across critical infrastructure is not readily available or scalable. This highlights a gap in collective cyber defense capabilities.
  7. Critical Cybersecurity Workforce Shortage: A severe and growing shortage of skilled cybersecurity professionals, particularly those with Operational Technology (OT) experience, poses an existential threat to critical infrastructure resilience. NIST estimates a national shortfall of 700,000 professionals, and CESAI reports that 70% of critical infrastructure entities struggle to find the necessary skill sets.
  8. The Imperative for a Civilian Defense Mindset: Drawing lessons from Ukraine's experience, the talk underscored the need for a civilian defense mindset, where citizens and communities are prepared and organized before an attack. This extends beyond the traditional security community to encompass broader community resilience.
  9. CR-ISAC as a Workforce Solution: The Civilian Reserve ISAC (CR-ISAC), founded by Major General Vic Mas, was presented as a vital initiative addressing the workforce shortage by building a trained, vetted cyber reserve. The CR-ISAC aims to mobilize skilled individuals, including veterans, those transitioning careers, and even high school students, to serve as a deployable force during cyber disasters.
  10. Cyber Insurance as a Driver for Resilience: Similar to how insurance drove safety culture in the late 1960s and early 1970s, cyber insurance is increasingly positioned to push organizations towards adopting more robust cyber resilience measures, including investment in cyber-resilient redundancy.

Technical Deep Dive

▶ Watch: Ukraine's lesson: civilians must be prepared beforehand (14:00)

The technical heart of this talk revolves around the concept of bricking controllers and the inherent vulnerabilities within existing ICS/OT architectures. Dettmer highlighted that an adversary targeting critical infrastructure might strategically choose not to cause immediate physical destruction. While a hydrogen generator explosion or an over-spinning turbine would generate dramatic headlines, the long-term impact of cyber-induced operational paralysis could be far more devastating and difficult to recover from.

The core technical vulnerability lies in the prevalent flat network architecture of many industrial control systems. Even when redundancy is built into these systems—with primary and backup controllers, operator consoles, servers, and network hardware—they are often deployed on the same logical or physical network segment. This design choice, while simplifying management and reducing costs, means that a successful cyber intrusion into one component can easily propagate to its redundant counterpart. An adversary capable of compromising the network could then execute a coordinated attack to corrupt or render inoperable both the primary and backup controllers. The term "bricking" implies a state where the device's firmware or configuration is so thoroughly corrupted that it cannot be easily restored or rebooted, effectively turning it into a useless "brick."

The implications of such an attack are severe. Unlike a physical component failure, which might be addressed by switching to a pre-configured spare, bricked controllers would require entirely new hardware. This brings the discussion back to supply chain fragility. The lead times for highly specialized industrial controllers and their associated components can be exceptionally long, potentially mirroring the multi-year waits for major physical assets like new generators or step-up transformers. An adversary aiming for prolonged operational disruption would leverage this weakness.

To counter this, Dettmer suggests a fundamental re-engineering of ICS/OT networks to achieve cyber-resilient redundancy. This would involve architectural changes that logically and physically separate primary and backup systems, potentially isolating them into different security zones, using different network paths, or even employing diverse vendor technologies where feasible. The goal is to prevent a single cyber event from simultaneously incapacitating both the primary and redundant components. This is acknowledged as a "tall task" with significant costs, but the changing threat landscape makes the investment increasingly justifiable.

Furthermore, the prevalence of end-of-support (EOS) devices exacerbates this problem. These systems lack security patches, making them easier targets for adversaries to exploit and brick. Their continued presence in production environments creates a significant attack surface that can be leveraged for widespread disruption. Proactive replacement and modernization are not merely best practices but critical survival strategies. The talk also implicitly touches on the supply chain security aspect, suggesting that even if spares exist, their integrity could be compromised if they originate from untrusted sources or have been tampered with. The discussion on communication networks, including corporate and personal cell phones, highlights another technical vulnerability: a large-scale attack could target these channels to prevent effective incident coordination, underscoring the need for out-of-band, resilient communication systems.

Demo / Proof of Concept

▶ Watch: Developing a civilian defense mindset across communities (17:00)

The talk did not feature a live technical demonstration or a specific proof of concept of an attack or defense mechanism. Instead, Andrew Dettmer and Katrina Rosseini relied on real-world examples, historical events, and theoretical scenarios to illustrate their points. Dettmer leveraged his past experiences with Volt Typhoon and the COVID-19 supply chain backlogs to contextualize the threats. He painted vivid pictures of hypothetical "worst-day" scenarios, contrasting traditional physical destruction events with the more insidious and prolonged disruption caused by cyber attacks designed to "brick" industrial controllers. Rosseini, in turn, referenced the experience of Ukrainian officials following cyber attacks on their infrastructure, highlighting the critical lesson of preparing civilians proactively. These narrative and illustrative approaches served as powerful conceptual demonstrations of the vulnerabilities and proposed solutions, grounding the discussion in practical implications rather than specific technical exploits.

Defensive Implications

▶ Watch: Cyber insurance's role in driving cyber resilience (20:00)

The insights shared in this talk necessitate a comprehensive overhaul of defensive strategies for critical infrastructure. Operators must move beyond traditional incident response to embrace a holistic, community-wide approach to cyber resilience.

  1. Re-architect for Cyber-Resilient Redundancy: Critical infrastructure operators must undertake a fundamental re-engineering of their ICS/OT networks. This involves designing redundancy that is resilient to cyber events, not just physical or electrical ones. Implement logical and physical segmentation between primary and backup controllers, operator consoles, and network hardware. Consider diverse vendor technologies or geographically dispersed redundancy to prevent a single point of failure from a widespread cyber attack.
  2. Eliminate End-of-Support Devices: Proactive identification and replacement or upgrade of all end-of-support (EOS) hardware and software in production environments is non-negotiable. These systems represent unpatched vulnerabilities that adversaries will exploit, leading to easier "bricking" and prolonged outages.
  3. Strategic Investment in Spares: Establish robust strategic sparing programs for all critical control system components. This means acquiring and maintaining full sets of spares, factoring in potentially multi-year lead times for specialized equipment. This investment should be viewed as an insurance policy against supply chain disruptions during widespread attacks.
  4. Scrutinize Incident Response Retainers: Organizations must engage in detailed discussions with their incident response (IR) retainer firms, OEMs, and integrators. Explicitly inquire about their capacity and plans for responding to "Everything Everywhere All At Once" scenarios, not just isolated incidents. Understand their mutual aid agreements and how they scale resources in a regional or national crisis.
  5. Develop Resilient Communication Plans: Establish out-of-band, resilient communication protocols that do not rely solely on corporate networks or personal cell phones. This could include satellite phones, amateur radio networks, or other robust, secure channels to ensure command and control during widespread communication disruptions.
  6. Actively Support Workforce Development Initiatives: Address the critical cybersecurity skills gap by actively supporting and participating in initiatives like the Civilian Reserve ISAC (CR-ISAC). Encourage employees to volunteer, provide mentorship, and partner with CR-ISAC to help train and vet a national cyber reserve workforce, particularly those with OT experience.
  7. Foster Public-Private Partnerships: Strengthen collaboration with government agencies and industry peers through organizations like InfraGard (for FBI collaboration on insider threats and resources) and ICS for ICS (an ISA initiative for standardizing emergency response language with FEMA). These partnerships facilitate intelligence sharing, resource coordination, and collective planning.
  8. Cultivate a Civilian Defense Mindset: Adopt and promote a civilian defense mindset within organizations and communities. This involves educating and preparing employees and the broader community on their roles in cyber resilience, understanding that collective defense extends beyond the traditional security team.
  9. Leverage Cyber Insurance as a Catalyst: Recognize the growing influence of cyber insurance providers in driving improved cybersecurity posture. Engage with insurers to understand how policy requirements can incentivize investment in cyber-resilient architectures and practices, similar to how insurance drove safety culture in the past.

Key Takeaways

  • Widespread Attacks Demand New IR Paradigms: Traditional incident response plans, focused on isolated incidents, are inadequate for "Everything Everywhere All At Once" attacks that target entire sectors, necessitating a fundamental shift in planning.
  • Cyber-Resilient Redundancy is Critical: Existing ICS/OT redundancy, often on flat networks, is vulnerable to cyber events; re-engineering architectures to achieve true cyber resilience, separating primary and backup systems, is now essential.
  • Supply Chain and End-of-Support Risks are Paramount: Fragile supply chains and the pervasive use of end-of-support devices in critical infrastructure pose severe risks, making strategic spares and proactive system replacement non-negotiable.
  • Massive Workforce Shortage Requires Collective Action: The severe national shortage of skilled cybersecurity professionals, especially those with OT experience, demands national-level initiatives like the CR-ISAC to build a trained and deployable cyber reserve workforce.
  • Community Preparedness is Key: Lessons from Ukraine highlight the critical need for a "civilian defense mindset," where communities are organized and citizens are prepared before an attack, emphasizing public-private partnerships and collective resilience.
  • Cyber Insurance Drives Security Investment: Cyber insurance is emerging as a powerful force, akin to its role in driving safety culture, pushing critical infrastructure entities to invest in more robust cyber hygiene and resilience measures.

About the Speaker(s)

Andrew Dettmer is an Industrial Cybersecurity Consultant for Black & Veatch, a global engineering, procurement, consulting, and construction company. Prior to his role at Black & Veatch, Dettmer dedicated a decade to a Generation & Transmission (G&T) cooperative in the Midwest, where he was responsible for overseeing the cyber controls of a nine-plant fleet. His extensive experience in operational technology (OT) environments informs his deep understanding of critical infrastructure vulnerabilities and resilience strategies.

Katrina Rosseini serves as the Chair of the Executive Board of the Civilian Reserve ISAC (CR-ISAC), an organization dedicated to strengthening national resilience against cyber attacks by building a trained cyber reserve workforce. She was recently appointed to this position by Major General Vic Mas, the founder of CR-ISAC. Rosseini is also a doctoral candidate in engineering and cybersecurity at George Washington University. Her academic research has notably led to the development of Secure Grid, a context-aware behavioral adaptation methodology designed to effectively address vulnerabilities within critical infrastructure.

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Dettmer's half of the talk nails the grim reality of OT incident response: your backup PLC on the same flat VLAN will get bricked exactly three seconds after your primary. But the second half devolves into a naive pitch for a volunteer civilian cyber militia to fix a systemic engineering failure. Solid threat model, delusional remediation.

Heather Calloway (CISO) — STRONG ACCEPT

Dettmer and Rosseini outline the exact systemic risk scenario that should be driving critical infrastructure strategy: a coordinated ICS attack that exhausts national IR capacity and hardware supply chains simultaneously. Their warning that physical redundancy does not equal cyber redundancy is mandatory viewing for any board overseeing operational technology.

→ Top-rated talks at Critical Effect DC 2025

All talks from Critical Effect DC 2025