Automated wargaming of a Chemical Plant
Martyn (Doctoral Researcher and Security Analyst · ThreatSpike Labs)
Security BSides London 2023 · Day 1 · Track 3
Overview
In this compelling talk from BSides London, Martyn, a Doctoral Researcher at Brunel University and Security Analyst at ThreatSpike Labs, presented groundbreaking research on the automated wargaming of a chemical plant using reinforcement learning (RL). Titled "Automated wargaming of a Chemical Plant," the presentation delved into a sophisticated simulation environment where red and blue teams, represented by intelligent agents, vie for control over a model industrial control system (ICS). The core objective of this research is to move beyond traditional, static risk assessments and understand the dynamic, probabilistic nature of cyber-physical attacks in critical infrastructure.

Key moments
- 0:00 Speaker's background and talk overview
- 4:00 How chemical plants achieve high safety levels
- 7:00 Purdue model and layered safety defenses
- 10:00 Key historical ICS/OT cyber attacks
- 18:00 Reinforcement learning for automated wargaming setup
- 24:00 Discrete vs. continuous agent capabilities
- 28:00 Wargaming results: Red team quickly overwhelms blue
- 38:00 Future research directions and audience questions
Automated wargaming of a Chemical Plant
Speakers: Martyn, Doctoral Researcher and Security Analyst, ThreatSpike Labs
Conference: BSides London
YouTube: https://www.youtube.com/watch?v=gzGWsTq4fL0
Overview
In this compelling talk from BSides London, Martyn, a Doctoral Researcher at Brunel University and Security Analyst at ThreatSpike Labs, presented groundbreaking research on the automated wargaming of a chemical plant using reinforcement learning (RL). Titled "Automated wargaming of a Chemical Plant," the presentation delved into a sophisticated simulation environment where red and blue teams, represented by intelligent agents, vie for control over a model industrial control system (ICS). The core objective of this research is to move beyond traditional, static risk assessments and understand the dynamic, probabilistic nature of cyber-physical attacks in critical infrastructure.
Martyn, drawing on his unique background as a former metallurgist at a chemical plant, highlights the critical need for advanced defensive strategies in operational technology (OT) environments. His work demonstrates how multi-agent reinforcement learning can expose vulnerabilities, predict attacker behaviors, and evaluate the efficacy of defensive measures in real-time within complex industrial processes. This talk is particularly relevant for cybersecurity professionals, industrial engineers, and policymakers grappling with the escalating threat landscape in critical infrastructure sectors, offering a glimpse into future methodologies for securing these vital systems.
The research's significance lies in its ability to simulate attack and defense scenarios at speeds and scales impossible with human-led exercises, providing quantitative insights into the timeframes within which defenders must act. By exploring various threat actor capabilities and blue team responses, the wargaming model offers a practical framework for developing more resilient and proactive security postures against sophisticated cyber-physical threats, ultimately aiming to prevent catastrophic failures in real-world industrial settings.
Background
▶ Watch: Speaker's background and talk overview (0:00)
The journey into securing industrial control systems (ICS) is rooted in the very fabric of the Industrial Revolution, where manual labor was supplanted by automated processes driven by coal, steel, and steam. This evolution rapidly advanced with the advent of petrochemicals, combustion engines, and complex chemical processes, revolutionizing society but simultaneously introducing unprecedented levels of risk. The inherent danger of these systems, dealing with flammable liquids, exotic atmospheres, and phenomena like Boiling Liquid Expanding Vapor Explosions (BLEVEs), necessitated rigorous safety protocols. Paradoxically, despite these hazards, modern chemical plants are remarkably safe, largely due to centuries of accumulated knowledge in control theory and safety engineering.
Early control theory, exemplified by Maxwell's paper on governors, laid the groundwork for automatic negative feedback systems. The 20th century saw the integration of complex mathematics, like Laplace transforms and Nyquist stability criteria, with early computing devices, leading to the development of Programmable Logic Controllers (PLCs). These PLCs became the backbone of industrial automation, maintaining set points and executing critical operations. As industrial processes grew in complexity and interconnectedness, the need for higher-order coordination emerged, giving rise to the Purdue Enterprise Reference Architecture. This model stratifies industrial networks from physical devices (Level 0) through PLCs and Safety Instrumented Systems (SIS) (Level 1), Human-Machine Interfaces (HMIs) and Engineering Workstations (Level 2), Manufacturing Operations Systems (Level 3), up to enterprise IT systems and the cloud (Level 4/5), often separated by a DMZ. A common challenge in these environments is the prevalence of outdated operating systems, with many OT systems running software 15-20 years old, presenting significant security vulnerabilities.
Traditional safety defenses, such as the UK-specific HAZOP (Hazard and Operability Study) and Safety Cases, are methodical processes designed to enumerate risks and argue for safe operation. The widely adopted Swiss Cheese Model conceptualizes safety as layers of imperfect defenses, where an accident occurs only when the "holes" in these layers align. However, Martyn emphasizes a critical distinction: while effective against random failures, the Swiss Cheese Model falls short against a deliberate attacker who will actively seek out and exploit aligned weaknesses.
Historical incidents underscore this vulnerability. The Maroochy Shire sewage plant incident (1999) saw a disgruntled former contractor remotely accessing the system via radio telemetry, causing 250,000 gallons of sewage release. The infamous Stuxnet (2010), a sophisticated Advanced Persistent Threat (APT), exploited air-gapped networks using infected USBs to physically damage Iranian centrifuges by cycling them past their resonance frequency while simultaneously spoofing control room data. A key lesson from Stuxnet was the human element: operators noticed side-channel anomalies (unusual noise) despite false "all clear" signals from the PLCs. More recently, Triton malware (2019) targeted a Saudi Arabian refinery's Safety Instrumented Systems (SIS), though the payload was never deployed. The Colonial Pipeline ransomware attack (2021), though primarily affecting IT, led to an OT shutdown due to operator caution and alarm fatigue, highlighting the interconnectedness of IT and OT risks. Even water treatment plants have seen incidents, from the debated Oldsmar attack (2021), where sodium hydroxide levels were altered, to activists defacing PLCs in a Pennsylvania pumping station (2023), successfully mitigated by manual backups. These events demonstrate the tangible risks and the evolving nature of threats against critical infrastructure, necessitating a shift towards more dynamic and predictive security models, which Martyn's wargaming aims to address.
Key Findings
▶ Watch: Purdue model and layered safety defenses (7:00)
Martyn's automated wargaming of the chemical plant yielded several critical insights into the dynamics of cyber-physical attacks and defenses:
- Red Team Dominance and Rapid Shutdowns: Across most scenarios, the red team agents consistently managed to achieve their objectives, often leading to a plant shutdown. For continuous engagements (where agents could make fine-grained changes), the plant typically failed within approximately 165 seconds (3 minutes). Discrete engagements (where agents had a menu of drastic actions) could be even faster, with shutdowns occurring in as little as 20 seconds post-training, a stark reduction from 200 seconds pre-training. This timeframe is far too short for human incident response teams to react effectively.
- Blue Team Passivity: The blue team agents, despite being designed to defend, largely learned to perform "nothing" or, at best, initiate a safe but costly plant shutdown. They struggled to develop active, responsive defensive strategies that could counter the red team's aggressive actions. This suggests that current automated defense models may default to safety rather than proactive resilience.
- Specific Attack Vectors: The red team agents converged on several effective attack strategies:
- Draining the Reactor: A common failure mode in low-capability scenarios involved the red team manipulating set points to cause the control system to drain the reactor.
- Coolant Manipulation: Frequently, the red team would switch off the reactor coolant, leading to overheating and instability.
- Purge Gas Fraction: In continuous scenarios, even seemingly minor manipulations, such as altering the purge gas fraction, were sufficient to overwhelm the control system and cause plant failure.
- Fatigue Losses: While not explicitly demonstrated as the primary failure mode, the continuous agents were designed to explore subtle, sustained manipulations (e.g., continuously cycling a valve) that could induce fatigue losses in physical components, a vector that traditional range-based defenses might miss.
- Novel Plant Oscillations: A particularly interesting and previously unobserved phenomenon was the consistent appearance of a reactor pressure oscillation at around 165 seconds across multiple scenarios, often linked to the loss of coolant. This suggests the wargaming unearthed a previously unknown dynamic vulnerability in the test bed model.
- Process Model as Anomaly Detector: When the blue team was assisted by its own process model (digital twin), a spike in the inaccuracy of the twin's predictions often immediately preceded plant failure. This indicates that a diverging digital twin could serve as a valuable early warning indicator for real-world plant operators.
- Neural Network Behavior: The study observed that the reinforcement learning agents, particularly the red team, often "collapsed" on doing just one thing, even when capable of multiple simultaneous actions. This is a common characteristic of neural networks and suggests that while powerful, they can find optimal but narrow strategies.
- Visualization Challenges: Visualizing the behavior of systems with 51 parameters (inputs and outputs) proved to be a significant challenge, highlighting the complexity of understanding and managing highly automated industrial processes.
These findings collectively emphasize the urgent need for more sophisticated, dynamic, and automated defensive mechanisms in OT environments, moving beyond static safety assessments to actively counter determined and rapidly acting adversaries.
Technical Deep Dive
▶ Watch: Reinforcement learning for automated wargaming setup (18:00)
Martyn's research employs a multi-agent reinforcement learning (RL) framework to simulate the adversarial dynamics within a chemical plant. This approach provides a powerful computational method for exploring complex interactions where agents learn optimal strategies through trial and error within a simulated environment.
At its core, reinforcement learning involves an agent interacting with an environment. The environment sends an observation (state) and a reward (a scalar number, higher is better) to the agent. The agent, guided by a policy (π), translates this observation into an action, which is then sent back to the environment. The agent's goal is to maximize the long-term rewards, often incorporating a discount factor to prioritize immediate rewards. A noise term is also included to encourage exploration of different actions.
For this wargaming, Martyn utilized a two-agent system: a red team agent and a blue team agent. The environment sends separate observations and takes separate actions for each. The core test bed is a model chemical plant, a common example for control system studies, featuring three main vessels: a reactor, a separator, and a stripper. This plant has 12 manipulated variables (things that can be altered) and 41 measured variables (inputs and outputs). Crucially, the model already incorporates a traditional control system implemented with PLCs, allowing the agents to interact with and potentially subvert existing automation. The plant's objective is to produce "G" and "H" products, which are arbitrary organic chemicals. Martyn explicitly notes that the internal chemical details are less relevant than the control dynamics.
Scenarios are defined by the capabilities and intents of both the red and blue teams. The blue team's reward function consistently aims to keep the plant operational, running, and producing in-spec products. The red team's reward function typically opposes this, seeking to prevent production, cause out-of-spec products, shut down the plant, or, ideally, cause catastrophic damage like an over-pressurized reactor.
The agents operate within two distinct action spaces, representing different levels of sophistication:
- Discrete (Low Capability) Agents:
- Blue Team: Has a menu of strategic actions, such as resetting a PLC (providing an hour for remediation) or shutting down the entire plant (incurring high costs but allowing for complete infection removal). It can also choose to do nothing.
- Red Team: Acts with less subtlety, directly altering any measured variable, manipulated variable, or set point to extreme high or low values. This represents an attacker with full access but limited finesse. Programming these agents is simpler, as it involves evaluating discrete options.
- Continuous (High Capability) Agents:
- Blue Team: Possesses fine-grained control over the plant, able to make continuous, small changes to variables.
- Red Team: Cannot directly control the plant to the same extent as the blue team but can spoof signals and change set points continuously. This represents a more sophisticated attacker aiming for subtle, sustained disruption. These agents require more advanced RL techniques, specifically Deep Deterministic Policy Gradients (DDPG), a form of gradient descent for continuous action spaces.
An additional variant for the blue team introduced assistance from its own process model (digital twin). This twin attempts to predict the plant's state over the next 48 hours, allowing the blue team to react based on these predictions rather than solely on past experiences.
The reward functions for the blue team are designed to mimic real-world plant economics: earning "fake money" for producing in-spec products, paying for utilities, incurring environmental penalties for harmful purges (e.g., G chemical), and fixed CO2 costs. The red team's rewards are typically the inverse of these, or specific damage-oriented goals, such as reactor over-pressurization.
Training these neural networks involved a set of hyperparameters, which Martyn noted is still largely a process of trial and error. He observed that even relatively small networks could yield good results, as many parameters often converge to zero, meaning larger networks don't necessarily improve performance. While initially considering Modbus emulators, Martyn decided against them as they didn't significantly alter the conclusions—an attacker with arbitrary code execution can achieve their goals regardless of the underlying protocol.
Demo / Proof of Concept
▶ Watch: Discrete vs. continuous agent capabilities (24:00)
The "demo" of Martyn's research is the simulation itself, where the reinforcement learning agents interact within the model chemical plant environment. The results illustrate the stark differences in plant behavior under various red and blue team scenarios, both before and after the agents have undergone training.
Low Capability Scenario (Discrete Agents):
- Pre-training: At the outset, both red and blue agents act randomly. In this state, the red team typically causes the plant to fail within approximately 200 seconds (3 minutes). A common failure mode observed was production being out of specification due to the red team spoofing the control system's set point, causing it to drain the reactor. The coolant flow to the reactor (represented by a blue trace) also frequently dropped to zero. The blue team, acting randomly, couldn't prevent this.
- Post-training: After training, the red team's efficiency dramatically improved, able to take down the plant in as little as 20 seconds. The red team learned to consistently change the reactor level set point to drain the reactor and also disable the coolant. In contrast, the blue team agent in this scenario primarily learned to "do nothing" or take very safe actions that, while preventing further immediate damage, offered no effective response to the red team's rapid actions. Although the blue team managed to produce some product for about 20 seconds, the overall outcome was a decisive red team victory. Interestingly, some trained red team agents, when run without a blue team, sometimes missed their primary objective of taking down the plant, instead causing random chaos and incurring high carbon costs, indicating that the learning process isn't always perfectly aligned with the most direct path to damage.
Continuous Models Scenario (High Capability Agents):
This scenario aimed to explore more subtle, continuous attacks, including those that might induce fatigue losses in physical components.
- Pre-training: The plant broke in about 80 seconds due to the red team's mostly noisy, random actions. The agents struggled to learn the plant's dynamics, with the "loss" (inaccuracy of prediction) actually rising, indicating confusion.
- Post-training: Despite the ability to perform multiple actions simultaneously, the red team agents often converged on doing just one thing. In one notable instance, the red team successfully overwhelmed the control system by manipulating the purge gas fraction, a seemingly minor variable that a human operator might overlook. Again, this led to rapid plant failure. A recurring observation in these continuous scenarios was the emergence of a novel oscillation in reactor pressure at around 165 seconds, a phenomenon previously unrecorded for this specific test bed model, often coinciding with the loss of coolant. This suggests the wargaming can uncover unexpected dynamic behaviors.
Twin-Assisted Blue Team:
- Pre-training: Similar to other pre-training runs, the plant exhibited the 165-second oscillation, even with the twin-assisted blue team, indicating its linkage to basic plant dynamics like coolant loss.
- Post-training: While the blue team's overall prediction inaccuracy was slightly lower than the fully continuous agent, it remained high. Crucially, the process model's inaccuracy would often spike immediately before plant failure, acting as a potential early warning indicator. If the digital twin can no longer accurately predict plant behavior, it signals that things are going "badly."
Overall, the simulations consistently demonstrated that a determined red team, even with limited sophistication, could quickly cause plant failure, often within minutes. The blue team's current automated responses were largely insufficient, highlighting a significant gap in defensive capabilities against dynamic cyber-physical threats.
Defensive Implications
▶ Watch: Future research directions and audience questions (38:00)
Martyn's automated wargaming research provides critical insights for defenders in the ICS/OT space, challenging traditional security paradigms and pointing towards necessary evolutions in strategy:
- Rethink Traditional Safety Models: The research starkly demonstrates that the Swiss Cheese Model, while effective against accidental failures, is insufficient against a deliberate and intelligent attacker. Defenders must recognize that adversaries actively seek to align vulnerabilities, rendering passive, layered defenses largely ineffective when compromised. This necessitates a shift from purely preventative and reactive measures to proactive, adversarial-aware defense.
- Accelerate Incident Response Times: The simulated attack speeds, with plant shutdowns occurring in as little as 20 seconds for discrete attacks and around 165 seconds (3 minutes) for continuous ones, are far too rapid for human-centric incident response. This highlights the urgent need for automated and semi-automated defensive actions that can detect anomalies and initiate countermeasures within seconds, not minutes or hours. While a 10-minute window was achieved in some rare blue team wins, this is still a tight margin for a full IT response.
- Embrace Digital Twins for Anomaly Detection: The observation that a spike in the inaccuracy of a digital twin's predictions often precedes plant failure is a powerful defensive implication. Implementing digital twins or process models that continuously compare predicted plant states with real-time measurements could provide a highly effective, early warning system for sophisticated attacks that aim to subtly manipulate processes or spoof sensor data. Divergence between the twin and reality would immediately flag a potential compromise or attack.
- Monitor Side Channels, Not Just Reported Status: The Stuxnet lesson, reinforced by the wargaming's discovery of novel oscillations, emphasizes the importance of monitoring side channels (e.g., unusual noise, vibration, or unexpected process dynamics) independently of what PLCs or HMIs are reporting. An attacker with arbitrary code execution can easily spoof reported data; therefore, physical or secondary observations become crucial indicators of compromise.
- Develop Active Blue Team Automation: The blue team agents' tendency to "do nothing" or default to costly shutdowns underscores a gap in current automated defense strategies. Future blue team automation needs to be designed not just for containment but for active, intelligent counter-manipulation or rapid, targeted remediation that buys time without immediately incurring massive operational costs. This could involve dynamically adjusting parameters to mitigate attacker impact or restoring specific sub-systems.
- Consider Fatigue-Based Attacks: The exploration of continuous, subtle manipulations to induce fatigue losses in physical components points to an often-overlooked attack vector. Defenders must consider that attacks might not always aim for immediate catastrophic failure but rather for long-term degradation of equipment, which could bypass traditional alarms set for out-of-range values.
- Address Partial Breaches: While the wargaming primarily focused on assumed full compromise, the speaker noted that future work could explore scenarios where the red team only controls a single PLC. Defenses must be robust enough to detect and respond to such partial breaches, preventing escalation to full plant compromise.
- Mitigate Alarm Fatigue: The Colonial Pipeline incident highlighted the real-world impact of alarm fatigue. As more layers of the Purdue model are integrated into wargaming, including alarms, understanding how to present critical alerts to human operators without overwhelming them will be crucial for effective human-machine collaboration in defense.
- Improve Visualization and Understanding: The challenge of visualizing 51-parameter control schemes emphasizes the need for better tools and dashboards that can distil complex OT data into actionable intelligence for human operators and security analysts, enabling quicker comprehension and decision-making during an incident.
In essence, the research calls for a paradigm shift from a reactive, compliance-driven security posture to a proactive, analytically driven defense that leverages advanced computational methods to anticipate and counter dynamic cyber-physical threats in critical infrastructure.
Key Takeaways
- Traditional safety models are insufficient against deliberate attackers: The Swiss Cheese Model, effective for random failures, fails when an intelligent adversary actively seeks to align vulnerabilities.
- Cyber-physical attacks can cause rapid plant shutdowns: Automated wargaming revealed that red teams could achieve plant failure in as little as 20 seconds for discrete attacks and approximately 3 minutes for continuous ones, far too fast for human response.
- Automated blue team defenses need to evolve: Current automated defenses often default to passive "do nothing" or costly full shutdowns; more active, intelligent, and rapid counter-manipulation strategies are essential.
- Digital twins offer a promising avenue for anomaly detection: Divergence between a digital twin's predictions and real-time plant data can serve as a critical early warning indicator of an ongoing attack or compromise.
- Monitoring side channels is crucial: Beyond PLC-reported status, observing physical anomalies like unusual noise, vibrations, or novel oscillations (as discovered in the simulation) can reveal attacks that spoof control system data.
- Subtle, continuous manipulations (fatigue losses) are potential attack vectors: Defenders must consider attacks that aim for long-term equipment degradation, not just immediate catastrophic failures, as these may bypass traditional range-based alarms.
About the Speaker(s)
Martyn is a Doctoral Researcher at Brunel University, where he is pursuing postgraduate studies, albeit part-time. In his full-time professional capacity, he serves as a Security Analyst at ThreatSpike Labs. His work at ThreatSpike Labs involves hands-on experience in various cybersecurity domains, including red teaming, Security Operations Center (SOC) analysis, and penetration testing.
Prior to his career in computer science and cybersecurity, Martyn had a distinct professional background as a metallurgist at a chemical plant. This unique prior experience in the industrial operational technology (OT) environment directly informed his current research, providing him with invaluable practical insights into the physical processes, control systems, and inherent risks associated with chemical plants. His dual expertise bridges the gap between deep industrial process knowledge and cutting-edge cybersecurity research, making his contributions to the field of ICS/OT security particularly relevant and impactful.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Finally, an AI talk that isn't marketing bullshit. Martyn leverages his past life as a chemical plant metallurgist to build a reinforcement learning simulation that proves a critical point: human incident response in OT is a fantasy. When your red agent drains a reactor in twenty seconds, you need automated defense, not a SOC analyst staring at an HMI.
Heather Calloway (CISO) — MUST SEE
This research uses reinforcement learning to wargame cyber-physical attacks on chemical plants, proving that execution-phase physical failures happen in under three minutes. It is a necessary reality check for manufacturing and critical infrastructure CISOs. The core takeaway is that human-in-the-loop incident response is mathematically too slow to prevent physical OT damage.