Securing America’s Water Systems: Engineering-Based Approaches to Cyber Resilience
Gus Serino (I&C Secure, Inc.)
Critical Effect DC 2025 · Day 1 · Track II: Strategic Effect
Overview
In this insightful talk from Critical Effect DC 2025, Gus Serino, a seasoned control systems engineer and founder of I&C Secure, Inc., illuminates the critical vulnerabilities within America’s water systems and proposes a refreshingly pragmatic approach to cyber resilience. Serino, drawing from his extensive experience automating and securing water and wastewater processes for a large utility in Massachusetts, argues that while cyber threats to critical infrastructure are escalating, many traditional IT-centric security solutions are ill-suited for the operational technology (OT) environments of water utilities, often leading to maintenance challenges or even hindering operations. The core of his presentation advocates for engineering-based mitigations—simple, physical protective systems designed to prevent the most catastrophic cyber-physical outcomes, rather than solely relying on complex cyber technology.

Key moments
- 0:00 Introduction: Engineer's journey into water cyber security
- 2:15 Worst-case scenarios: Chemical poisoning and hydraulic attacks
- 5:10 Water systems: Inherently resilient vs. low cyber protection
- 8:10 Engineering-based approaches: Cyber Physical Safety Systems
- 11:00 Example: Overpressurization protection using hardwired switch
- 15:00 Example: Chemical overdose prevention with pH relay
- 20:00 Q&A: Importance of training, documentation, and testing
- 22:00 Q&A: Balancing connectivity, modernization, and risk
Securing America’s Water Systems: Engineering-Based Approaches to Cyber Resilience
Speakers: Gus Serino (I&C Secure, Inc.)
Conference: Critical Effect DC 2025
YouTube: https://www.youtube.com/watch?v=Esr8ZcUfrgU
Overview
In this insightful talk from Critical Effect DC 2025, Gus Serino, a seasoned control systems engineer and founder of I&C Secure, Inc., illuminates the critical vulnerabilities within America’s water systems and proposes a refreshingly pragmatic approach to cyber resilience. Serino, drawing from his extensive experience automating and securing water and wastewater processes for a large utility in Massachusetts, argues that while cyber threats to critical infrastructure are escalating, many traditional IT-centric security solutions are ill-suited for the operational technology (OT) environments of water utilities, often leading to maintenance challenges or even hindering operations. The core of his presentation advocates for engineering-based mitigations—simple, physical protective systems designed to prevent the most catastrophic cyber-physical outcomes, rather than solely relying on complex cyber technology.
Serino's perspective is grounded in the fundamental mission of water utilities: to deliver safe, treated potable water with sufficient pressure and volume. He emphasizes that cyber security in this context is not an end in itself, but a means to protect this vital mission. The talk delves into worst-case scenarios, such as rendering water chemically unsafe or causing hydraulic failures that damage infrastructure, and then systematically presents practical, non-cyber engineering solutions. This approach is particularly relevant for the many water utilities that are non-technical, underfunded, and lack robust cyber protections, offering a path to enhanced resilience that is both effective and sustainable.
By focusing on physical safety systems that self-limit dangerous conditions, Serino provides a blueprint for how water systems can build resilience from the ground up. His "Cyber Physical Safety Systems" concept, inspired by the work of Idaho National Labs (INL), offers a tangible way to fortify critical infrastructure against sophisticated attacks without requiring deep cybersecurity expertise or significant technological investment. This talk is a crucial call to action for defenders, engineers, and policymakers to rethink how they approach security in water systems, prioritizing engineering fundamentals to safeguard public health and critical services.
Background
▶ Watch: Introduction: Engineer's journey into water cyber security (0:00)
The primary mission of any water utility is to maintain pressure with sufficient volume of treated potable water, ensuring its safety and reliability for consumption, sanitation, and other critical dependencies. A typical water system involves several stages: intake (drawing water from ground wells or surface sources), treatment (removing particulates, contaminants, disinfecting, pH adjustment, fluoridation), transmission (moving water over distances), and distribution (delivering water to consumers and businesses). Protecting this entire chain is paramount, and it’s against this backdrop that cyber threats pose a unique and severe risk.
Serino highlights two primary worst-case scenarios that cyberattacks could precipitate:
- Chemically unsafe water: Adversaries could manipulate automated treatment processes, altering chemical dosing (e.g., disinfectants, pH adjusters) to render the water unsafe to drink. Many treatment processes are fully automated, making them potential targets.
- Hydraulic damage: Malicious actors could manipulate hydraulic components like pumps and valves. Rapid changes in flow or pressure can cause water hammer, overpressurization (leading to pipe bursts), or vacuum conditions (causing pipes or vessels to crush). Serino notes the immense force involved, citing 8 pounds per gallon of water moving quickly. He mentions the real-world example of the city of Wichita experiencing a main break after a power loss caused all pumps to restart simultaneously.
While temporary disruptions or ransomware attacks are also concerns, Serino focuses on these physical, irreversible damages, which he argues are more challenging to recover from than data loss. He also acknowledges the significant, though less immediate, impact of eroding public confidence if water is perceived as unsafe or unreliable.
The current state of cybersecurity in water systems presents a mixed picture. On the positive side, these systems are generally designed for resilience, often featuring redundancies like multiple intake sources, redundant treatment paths, and interconnections to adjacent systems. Operators are well-trained to respond to physical failures and restore service, demonstrating an innate ability to "hamper along." Furthermore, many systems retain the feasibility of manual operation, though this varies significantly and can be complex.
However, the negatives often outweigh these inherent strengths. A significant issue is the lack of regulation for cybersecurity in water, leading to low or non-existent cyber protections across many utilities. Where protections exist, they are typically due to an internal "champion" advocating for funding and resources. There's a widespread lack of understanding of cyber risk among non-technical staff and even management, who may be unaware of incidents like the Oldsmar, Florida water treatment plant hack (Mulesoft) or other state-sponsored attacks. This combination of minimal protection, limited funding, and a knowledge gap makes water systems particularly vulnerable.
Recognizing these challenges, Serino's approach is not new but draws on established principles. He recalls discussions with the EPA and American Waterworks in the 2010s about engineering solutions to cyber risks, even when the exact nature of cyberattacks was less understood. This concept was formalized and popularized by Andy Bachmann of Idaho National Labs (INL) in 2018 with the Cyber-Informed Engineering (CIE) and Consequence-driven Cyber-informed Engineering (CCE) programs. Serino's "Cyber Physical Safety Systems" directly aligns with these principles, aiming to "engineer out the biggest risks" by applying simple protective systems to prevent physical damage.
Key Findings
▶ Watch: Water systems: Inherently resilient vs. low cyber protection (5:10)
Serino's talk underscores several critical findings regarding the cyber resilience of water systems, particularly emphasizing the unique challenges and effective solutions for this vital sector:
- Catastrophic Cyber-Physical Threats are Real and Distinct: Water systems face specific worst-case scenarios—chemical contamination rendering water unsafe and hydraulic manipulation causing severe infrastructure damage (e.g., pipe bursts, vessel crushing). These outcomes are often irreversible and have immediate public health and economic consequences, distinguishing them from typical IT cyber incidents like data breaches or ransomware.
- Traditional IT Security Often Falls Short in OT Environments: Applying complex IT cybersecurity technologies (like MFA, VPNs, or advanced SIEMs) directly to operational technology (OT) environments in water utilities can be problematic. Many utilities are non-technical, underfunded, and struggle with maintaining such systems, potentially leading to operational disruptions (e.g., "MFA is not working again") that can make matters worse. This highlights the need for solutions tailored to the unique constraints and operational priorities of OT.
- Inherent System Resilience is a Foundation, Not a Guarantee: Water systems are often designed with redundancies (multiple intakes, treatment paths, distribution loops) and operated by skilled staff adept at managing physical failures. This inherent resilience provides a baseline, but operators are generally not trained for cyberattacks, and system design alone isn't sufficient to prevent malicious, coordinated attacks exploiting automation.
- Engineering-Based Controls Offer Robust, Cost-Effective Mitigation: The central finding is that "Cyber Physical Safety Systems"—simple, physical protective mechanisms—can effectively mitigate the highest-consequence cyber-physical risks. These systems are:
- Non-cyber in nature: They use control system design and hardware, not software-based cyber technology, making them less susceptible to cyber manipulation once installed.
- Self-limiting: They automatically prevent dangerous conditions, even if automation is compromised.
- Retrofit and design-in capable: They can be integrated into existing infrastructure or designed into new systems.
- Cost-effective: Many solutions can be implemented for a relatively low cost (e.g., a pressure switch system for under $10,000, independent monitors for ~$2,500).
- Require less specialized cyber expertise: Control system engineers can implement them without needing deep cybersecurity knowledge.
- Careful Implementation and Ongoing Maintenance are Crucial: While simple, these engineering controls must be implemented meticulously to avoid inadvertently causing operational problems. Furthermore, they are not "set-and-forget" solutions; regular testing, calibration, and maintenance programs (e.g., work orders every six months) are essential to ensure they function correctly when needed, especially for systems that may rarely trip.
- The Need for a Holistic, Design-Centric Approach: Serino advocates for incorporating cyber risk into the initial engineering design and "value engineering" processes. This means resisting the temptation to remove manual controls or critical hardwired connections for cost savings, ensuring that resilience is built in from the outset rather than bolted on later.
Technical Deep Dive
▶ Watch: Example: Overpressurization protection using hardwired switch (11:00)
Gus Serino's talk champions the implementation of Cyber Physical Safety Systems (CPS2) as a pragmatic and effective strategy for securing America's water infrastructure against cyber-physical attacks. The core concept is to apply simple, often hardwired, protective systems that prevent unintentional or intentional physical damage, thereby self-limiting the worst-case scenarios without relying on complex, vulnerable cyber technology. This approach aligns closely with the principles of Consequence-driven Cyber-informed Engineering (CCE) from Idaho National Labs (INL), focusing on engineering out the highest-consequence risks.
Serino outlines several concrete examples of how these CPS2 can be implemented:
- Overpressurization Control for Pumping Systems:
- Problem: Maliciously or accidentally turning on multiple pumps simultaneously can lead to severe overpressurization, causing pipe breaks (water hammer). Serino cites an instance at MWRA where fear of breaking old pipes due to multiple pumps running led to this solution.
- Solution: Install a pressure switch (e.g., a "green box" in his diagram) on the downstream side of each critical pump. This switch is set to a pressure just above the normal operational maximum.
- Mechanism: The pressure switch is hardwired directly to the pump control mechanism, bypassing the main automation network (PLC/SCADA). If the pressure exceeds the setpoint, the switch trips, shutting off the pump. The pump then requires a local manual reset, and an alarm is sent to operators, indicating the safety system engagement.
- Cost & Simplicity: Serino estimates the cost for such a system to be around $1,000s to under $10,000, involving a pressure switch, relay panel, conduit, wire, and an electrician. Its simplicity makes it robust and resistant to cyber manipulation of the primary control system.
- Hydraulic Control for Valves:
- Problem: Valves, especially large ones, can cause water hammer if moved too quickly. While operators typically move them slowly, some actuators are geared to allow rapid movement, creating a vulnerability if manipulated.
- Solution 1: Actuator Gearing Adjustment: Identify valves whose actuators can move too fast and adjust their internal gearing. This physically limits the speed at which the valve can open or close, preventing rapid pressure changes, even if a malicious command tries to slam it shut. The adjustment ensures the valve only moves as fast as legitimately needed, even in emergencies.
- Solution 2: Rupture Discs: For contained areas where rapid pressure spikes are possible, install rupture discs. These are sacrificial elements designed to fail at a specific pressure, directing the resulting water release to a known, controllable location. This prevents uncontrolled pipe ruptures in unpredictable places, allowing for faster repair and containment.
- Chemical Overdose Prevention (e.g., Sodium Hydroxide for pH Adjustment):
- Problem: An adversary could manipulate automation to overdose treatment chemicals (e.g., sodium hydroxide for pH adjustment), making water unsafe.
- Solution: Utilize the built-in relay capabilities of pH transmitters (or other critical process parameter sensors). These transmitters measure the actual pH of the water.
- Mechanism: If the pH exceeds a safe upper limit (indicating an overdose), the pH transmitter's relay closes a contact. This hardwired signal goes to a relay panel, which then directly shuts down the chemical feed pump, bypassing the main control system. An alert is also sent to operational staff.
- Benefits: This system self-limits the amount of chemical that can be added, preventing severe over-dosing and maintaining water safety.
- Protection for Critical Skid Systems (e.g., Ozone Generators):
- Problem: Highly specialized, self-contained equipment (like ozone generators) can be extremely difficult and time-consuming to replace if destroyed. Connecting them fully to the process control network exposes them to potential cyber destruction.
- Solution: Serino advocates for a strategy fundamental to CCE: instead of full network integration, hardwire essential signals from these critical skid systems directly to the main PLC.
- Mechanism: This allows operators to receive enough data and maintain sufficient control over the equipment for normal operation, but it isolates the system from the broader network, significantly reducing its exposure to cyberattacks that could render it inoperable for months.
- Maintaining Local/Manual/Local Automatic Operation:
- Problem: Over-reliance on centralized SCADA systems can leave utilities vulnerable if the primary control system is compromised or fails. "Value engineering" often removes local control panels to save costs.
- Solution: Design or retain the ability for systems to operate autonomously in a local, manual, or local automatic mode.
- Mechanism: This means having physical switches, buttons, and local controllers that can run pumps, valves, and even some treatment processes independently of the main network. While not ideal (lacking full diagnostics), it allows the system to "limp along" and maintain the primary mission (pressure, volume, basic treatment) during recovery from a cyber incident. Serino strongly advises against removing these capabilities during modernization efforts.
- Independent Monitoring Solutions:
- Problem: A sophisticated adversary might manipulate control systems and simultaneously mask their actions, making everything appear normal on the SCADA screen while dangerous conditions develop.
- Solution: Implement independent monitoring solutions at key process parameters (e.g., flow, tank levels, pressure).
- Mechanism: These are separate, often low-cost (Serino mentions ~$2,500 plus ongoing cell connection costs), sensors that operate outside the main control network. They can have their own cell connection to send alerts directly to operators if a discrepancy is detected between the reported SCADA values and the actual physical conditions. This provides an additional layer of situational awareness and helps detect masked attacks.
Crucially, Serino emphasizes that while these systems are simple, their careful implementation and ongoing maintenance are non-negotiable. Systems must be tested periodically (e.g., every six months) through work orders to ensure they remain functional, calibrated, and that operators are trained on how to respond when they engage. This holistic approach ensures long-term effectiveness and trustworthiness of the safety systems.
Demo / Proof of Concept
▶ Watch: Example: Chemical overdose prevention with pH relay (15:00)
Gus Serino's talk did not include a live demonstration or a new proof-of-concept for a specific attack or defense. Instead, the presentation served as a conceptual and illustrative deep dive, drawing upon his extensive real-world experience as a control systems engineer and his work with water utilities. Serino presented concrete examples of engineering-based security solutions that he and others have either implemented or advocated for, effectively using these as "proofs of concept" within the context of past projects and industry best practices. The detailed descriptions and diagrams of protective systems for overpressurization, chemical overdose, and critical equipment isolation served to illustrate the practical application and efficacy of his proposed Cyber Physical Safety Systems.
Defensive Implications
▶ Watch: Q&A: Balancing connectivity, modernization, and risk (22:00)
Gus Serino's presentation provides a clear, actionable roadmap for water utilities and critical infrastructure defenders to enhance their cyber resilience, particularly against the most damaging cyber-physical attacks. The defensive implications extend beyond traditional IT security, advocating for a fundamental shift towards engineering-informed defense.
- Prioritize Consequence-Driven Risk Assessment: Defenders must move beyond generic vulnerability assessments to identify and prioritize the specific worst-case scenarios that could lead to physical damage or public health crises (e.g., chemical contamination, hydraulic failures). This requires a deep understanding of the operational processes and potential physical impacts of control system manipulation.
- Embrace Engineering-Based Controls as a Primary Defense: The core takeaway is to strategically implement Cyber Physical Safety Systems (CPS2). These non-cyber, hardwired solutions are often more robust and less susceptible to sophisticated cyberattacks than software-based security measures. Utilities should actively seek opportunities to retrofit existing infrastructure with these systems and integrate them into all new designs.
- Reduce Network Exposure for Critical Assets: For irreplaceable or long-lead-time critical equipment (like ozone generators), minimize direct exposure to the broader network. Hardwiring essential signals to the main PLC provides necessary control and data while significantly reducing the attack surface, aligning with CCE principles.
- Preserve and Enhance Local/Manual Operational Capabilities: Resist cost-cutting measures ("value engineering") that remove physical control panels, switches, and buttons. Maintaining the ability to operate systems locally and manually is a vital fallback mechanism during a cyber incident, ensuring the continuity of essential services even if SCADA is compromised.
- Implement Independent Monitoring Solutions: Deploy separate, out-of-band monitoring for critical process parameters (e.g., flow, tank levels, pH). These independent sensors, potentially with cellular connections, can provide crucial situational awareness and detect anomalous physical conditions that might be masked or manipulated on the primary control system, acting as an early warning for sophisticated attacks.
- Establish Robust Maintenance, Calibration, and Testing Regimens: CPS2 are not "set-and-forget" solutions. Utilities must implement rigorous preventative maintenance programs, including periodic testing (e.g., every six months) and calibration, to ensure these safety systems remain functional and reliable. As Serino pointed out, "If you don't do that, when you're relying on something 10 years, it never kicks in and then finally you need it and it does no longer works."
- Invest in Operator Training for Safety System Engagement: Operators must be thoroughly trained on the purpose, function, and appropriate response protocols for each safety system. This includes understanding alarms, local reset procedures, and how to operate systems manually when a safety control engages. This ensures that when a system trips (which may be rare), operators know exactly what to do.
- Influence Secure-by-Design in Procurement and Modernization: Water utilities, engineers, and policymakers should exert influence on equipment suppliers and system integrators to incorporate cyber-physical resilience into their designs. This means demanding systems that inherently include engineering controls and local operational capabilities, rather than having to bolt them on later. Serino implicitly suggests that this requires a shift in market influence towards demanding secure by design for OT equipment.
- Integrate Cyber Risk into Engineering Design and Value Engineering: Cyber risk must be an integral part of the initial design process for any new system or modernization project. This ensures that protective measures, such as hardwired safety interlocks and manual overrides, are considered from the outset, preventing costly or risky omissions during "value engineering" phases.
By adopting these defensive implications, water utilities can build a layered defense strategy that combines traditional cybersecurity with robust, engineering-based physical protections, creating a more resilient and secure operational environment.
Key Takeaways
- Prioritize Engineering-Based Cyber Resilience: Traditional IT-centric cybersecurity solutions are often ill-suited for the unique operational technology (OT) environments of water utilities. Instead, focus on "Cyber Physical Safety Systems" (CPS2) — simple, hardwired engineering controls that prevent catastrophic physical outcomes.
- Mitigate Worst-Case Scenarios with Physical Controls: The most critical threats to water systems involve rendering water chemically unsafe or causing hydraulic damage (e.g., pipe bursts). Implement engineering solutions like pressure switches, geared actuators, rupture discs, and pH-triggered pump shutdowns to self-limit these high-consequence events.
- Reduce Network Exposure for Critical Assets: Isolate and hardwire critical, hard-to-replace equipment (e.g., ozone generators) from the main process control network to minimize their attack surface while still allowing essential monitoring and control.
- Preserve and Enhance Manual/Local Control Capabilities: Do not sacrifice manual or local automatic operational capabilities during modernization. These physical overrides are crucial for maintaining operations during cyber incidents or SCADA failures, ensuring continuity of service.
- Implement Independent Monitoring for Situational Awareness: Deploy separate, out-of-band sensors for key process parameters to detect discrepancies between SCADA readings and actual physical conditions, providing an early warning system against sophisticated, masked cyberattacks.
- Emphasize Maintenance, Testing, and Training: Engineering safety systems require rigorous ongoing maintenance, periodic testing, and calibration. Crucially, operators must be thoroughly trained on how these systems work and what to do when they engage to ensure their long-term effectiveness.
About the Speaker(s)
Gus Serino is a highly experienced control systems engineer and the founder of I&C Secure, Inc. His career began with designing process and automation systems for an engineering company. He then spent 15 years at MWRA, a large water utility in Massachusetts, where he was instrumental in automating water and wastewater processes and leading the cybersecurity efforts for these critical systems. Serino later worked at Dragos, continuing his focus on industrial control systems security. Driven by a desire to provide hands-on support to water utilities that often struggle to implement security recommendations, he founded I&C Secure, Inc. Serino's unique background combines deep expertise in control systems engineering with practical experience in securing critical infrastructure, making him a leading voice in developing engineering-based approaches to cyber resilience in the water sector.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Serino gets it. While vendors are trying to sell neural-network anomaly detection to underfunded municipal water plants, this guy is showing you how to defeat nation-state OT attacks with a $10,000 hardwired pressure switch. This is how you actually defend critical infrastructure—in the physical layer, where the malware can't reach.
Heather Calloway (CISO) — MUST SEE
This is required reading for any board overseeing cyber-physical risk or operational technology. It dismantles the assumption that securing critical infrastructure requires complex IT software, proving instead that capital engineering and physical safety limits are cheaper, safer, and more resilient.