Building an ICS/OT Lab

Building an ICS/OT Lab

Speakers: Gavin Dilworth

Conference: CHCon

YouTube: https://www.youtube.com/watch?v=pMjB-BGZlrM

Overview

In this insightful CHCon talk, Gavin Dilworth presents a comprehensive guide and retrospective on the intricate process of building a physical Industrial Control System (ICS) / Operational Technology (OT) lab. The presentation delves into the motivations, design considerations, component selection, and myriad challenges encountered during the construction of his multi-version water tank and pumping system. Dilworth, with over two decades of experience in security, shares his journey from initial conceptualization to a complex, fully physical lab, highlighting the stark realities of cost, time, and technical hurdles involved in such an undertaking.

The core of Dilworth's discussion revolves around creating an environment that accurately simulates real-world industrial processes, such as those found in power plants or wastewater treatment facilities. This realism is crucial for training cybersecurity professionals, understanding industrial protocols, demonstrating network segmentation, and analyzing the impact of "process upsets" – disruptions that can have significant physical consequences. While acknowledging the allure and educational benefits of a physical lab, Dilworth also critically examines the viability of more accessible, software-based or hybrid alternatives, providing a balanced perspective for anyone looking to delve into ICS/OT security research or training.

Ultimately, this talk serves as both a detailed technical blueprint and a cautionary tale. It provides invaluable practical advice for those considering a similar venture, outlining specific components, their costs, and compatibility considerations. More broadly, it underscores the importance of hands-on experience in understanding the unique security challenges within OT environments, offering pathways for engagement regardless of budget or resource constraints, and emphasizing that deep learning often comes from the very process of building and troubleshooting.

Background

▶ Watch: Introduction to ICS/OT labs and talk overview (0:00)

The realm of Industrial Control Systems (ICS) and Operational Technology (OT) encompasses the hardware and software used to monitor and control physical processes, ranging from manufacturing assembly lines to critical national infrastructure like power grids and water treatment plants. Unlike traditional IT environments, OT systems prioritize safety, availability, and integrity, with cybersecurity often being a secondary consideration until recent years. This unique operational context, coupled with the legacy nature of many systems, creates distinct security challenges that demand specialized understanding.

Gavin Dilworth's motivation for building an ICS/OT lab stemmed from a desire to provide practical, hands-on training for engineers and cybersecurity professionals. His initial concept, conceived in 2015, aimed to demonstrate how to securely transfer process data through a firewall using industrial protocols like OPC UA, explain the role of certificates in securing these communications, and illustrate network segmentation principles. This vision evolved through several iterations, culminating in his Version 5 lab, a physical water tank and pumping system.

Before embarking on a physical build, Dilworth explored various existing and accessible lab options. For purely physical processes, examples include using Lego for power distribution networks, the Lickster (a low-cost manufacturing simulation), or professional setups like those by IC Village seen at conferences like Defcon and Black Hat, which often feature complex power grid and water flow demonstrations. For simulated or virtualized environments, popular choices include OpenPLC (available via Docker or VM for writing ladder logic), Factory IO (a 3D rendering platform for simulating factories with conveyor belts, robots, etc.), and Fuxer (a free Docker-based SCADA visualization tool). Hybrid options, such as connecting Factory IO to OpenPLC, are also common for combining visual simulation with PLC logic. More comprehensive, all-in-one virtualized solutions like Griffix v3 offer a complete cyber-physical simulation environment within a container, demonstrating that extensive physical hardware is not always a prerequisite for effective learning. These alternatives often serve as a more accessible entry point due to their lower hardware requirements and ease of deployment.

Key Findings

▶ Watch: Speaker's custom tank system lab (Version 4/5) (6:17)

Dilworth's journey to construct a physical ICS/OT lab yielded several critical findings, primarily highlighting the significant investment of time, money, and expertise required for such an endeavor, alongside the unexpected technical complexities.

Firstly, the most striking finding is the prohibitive cost of building a comprehensive physical lab. Dilworth's Version 5 system, while a relatively modest water tank setup, accumulated a total cost of approximately $11,000 NZD, with a realistic estimate pushing it to $13,000 NZD once contingencies for unforeseen tools and replacements were factored in. This figure starkly contrasts with the speaker's initial expectations, leading him to conclude that, in hindsight, he "would have bought the Lego" if he had known the true expense. This underscores that while physical realism is valuable, it comes at a premium that often outweighs the practical benefits for many educational or research purposes.

Secondly, component selection and compatibility emerged as a major hurdle. Dilworth meticulously detailed issues such as:

• Valve ratings: Solenoid valves must be specifically rated for water or oil, not just air, to prevent rust and failure.
• Sensor depth: Submersible water level sensors need to match the specific depth range of the tank (e.g., a 100cm rated sensor for a 30cm tank, providing 30% range, is far better than a 5m rated sensor offering only 8-12%).
• Voltage compatibility: Ensuring all components (pumps, valves, PLC I/O cards) operate on the standard 24V DC to avoid power mismatches. A critical example was the PWM board intended for variable pump speed, which required 5V DC power despite the system operating on 24V DC, necessitating a DC-DC converter and ultimately leading to system failure due to an unknown issue (implied burnout).
• PLC I/O cards: The chosen Programmable Logic Controller (PLC) must have compatible input/output cards (digital and analog) that match the instruments being connected. This often requires extensive pre-purchase planning and can significantly increase costs if the wrong PLC or cards are selected.

Thirdly, the cost of proprietary engineering software licenses is a significant barrier to entry for learning about ICS/OT. Traditional vendor software from companies like Rockwell Automation or Siemens can cost between $11,000 and $20,000 per year. Dilworth circumvented this by opting for a Phoenix Contact PLCnext starter kit, which includes a free engineering software IDE, making hands-on PLC programming much more accessible. Similarly, he utilized the Ignition server for SCADA visualization, which offers a fully featured two-hour trial period, resettable by restarting the application or the machine, providing ample time for training and experimentation without incurring licensing fees.

Finally, the fragility of instruments and the time commitment were unexpected challenges. Physical components can be sensitive to even minor knocks, leading to breakages and leaks. The sheer amount of time required for planning, purchasing, wiring, programming, and troubleshooting a physical system, especially alongside a full-time job and family life, proved substantial. These factors reinforce Dilworth's ultimate advice: a physical process is not always necessary, and robust learning can often be achieved through more accessible, software-driven approaches.

Technical Deep Dive

▶ Watch: Total cost of physical lab and budget warning (20:30)

Dilworth's Version 5 lab is a sophisticated, multi-component system designed to simulate a water mixing and pumping process, offering a realistic platform for ICS/OT cybersecurity training. The core architecture comprises four individual water tanks, each with its own pump and associated valves and sensors, all controlled by a PLC and visualized via a SCADA system.

At the heart of the physical process are the tanks, which are repurposed CPU water cooling tanks. These were chosen for their cost-effectiveness ($250 NZD for four), pre-threaded inserts for easy piping connections, and suitable outlet/inlet configurations. Each tank is equipped with a submersible water level transmitter (specifically, a 100cm depth-rated sensor for a 30cm tank, providing better resolution than deeper-rated alternatives). The flow is managed by solenoid valves (24V DC), crucial for automated control, supplemented by manual valves for emergency isolation – a critical safety feature in any physical process. Dilworth emphasizes that these solenoid valves must be rated for water/oil, not just air, to ensure longevity and prevent corrosion.

The movement of water is driven by peristaltic or stellatic pumps, selected for their low volume capacity (320 milliliters per minute) to prevent rapid overflows during training sessions. These pumps operate on 24V DC, a standard voltage in industrial environments. Initially, Dilworth intended to implement variable speed control for the pumps using a PWM board (Pulse Width Modulation), but encountered a significant compatibility issue: while the system powered by 24V DC, the PWM board itself required a 5V DC supply. Although a DC-DC converter was used to address this, the system ultimately failed, forcing a reversion to a simpler on/off pump control.

The brain of the operation is a Programmable Logic Controller (PLC). Dilworth specifically chose a Phoenix Contact PLCnext starter kit. This PLC is critical for executing the ladder logic (a common PLC programming language, where instructions like XIC – Examine If Closed – are used) that dictates the process flow. The PLC communicates with the physical instruments via input and output (I/O) cards. Digital I/O cards handle binary states (on/off for pumps and valves), while analog I/O cards manage variable values (e.g., 0-10V from the level transmitters). The selection of appropriate I/O cards and ensuring their compatibility with instrument voltages and signal types is a complex, pre-purchase planning phase. To demonstrate real-world industrial setups, Dilworth also incorporated an optional Remote Input Output (RIO) card, allowing instrumentation to wire into the RIO, which then communicates with the PLC via Ethernet.

For visualization and control, the lab utilizes a SCADA (Supervisory Control and Data Acquisition) system. Dilworth selected the Ignition server, a widely used platform in industrial settings, known for its robust features and a convenient two-hour free trial that resets upon restart. This allows for extensive training on HMI (Human-Machine Interface) operation without licensing costs. Other SCADA options mentioned include Fuxer (free, Docker-based), Kipware, and Metricon.

Network security and segmentation are integral to Dilworth's training objectives. He included a firewall in his setup to demonstrate how to securely pass industrial protocols, such as OPC UA, between different network segments. OPC UA is a modern, secure, and robust industrial communication protocol that utilizes certificates for authentication and encryption, making it a prime candidate for demonstrating secure data transfer. The entire system is managed and programmed from an engineering workstation, a powerful NUC running Windows 11, capable of hosting virtual machines and containers to support the various software components.

The complete system incorporates a range of miscellaneous components, including fuses, terminal boxes, crimping bars, jumper bars, and ferrules, which contribute significantly to the overall cost and complexity, often overlooked in initial planning.

Demo / Proof of Concept

▶ Watch: Navigating expensive and restrictive software licenses (22:00)

While the talk meticulously details the design, construction, and component selection for the physical ICS/OT lab, it did not feature a live, fully functional demonstration of the Version 5 system. Dilworth candidly acknowledged that the lab, though mostly assembled, was not yet fully wired or operational at the time of the presentation. He had intended to showcase a complete working system with a video, but encountered several challenges, including the fragility of instruments (e.g., a cracked valve connector causing leakage) and the incomplete integration of the PLC. Therefore, the "demo" aspect of the talk focused more on illustrating the individual components, their configurations, and the intended process flow, rather than a live operational display. This transparency served to further emphasize the inherent difficulties and time investment involved in bringing such a complex physical system to full functionality.

Defensive Implications

▶ Watch: Recommendations for building an affordable virtual lab (28:00)

Gavin Dilworth's efforts in building a physical ICS/OT lab, and his candid discussion of its challenges, carry significant implications for cyber defenders. The primary takeaway is the critical importance of hands-on experience in understanding the unique operational context and security posture of industrial environments.

Firstly, the lab provides a safe, contained environment to experiment with industrial protocols like OPC UA. Defenders can learn how these protocols transmit data, how they can be secured with certificates, and how they interact with network devices. This practical understanding is invaluable for configuring firewalls, implementing Intrusion Detection Systems (IDS), and analyzing network traffic in real-world OT deployments. The demonstration of passing OPC UA securely through a firewall directly addresses the need for network segmentation – a fundamental security control in OT to limit the blast radius of attacks.

Secondly, the ability to induce "process upsets" in a physical (or even simulated) lab allows defenders to grasp the tangible consequences of cyberattacks on physical systems. Understanding that a misconfigured valve or an unauthorized pump command can lead to physical damage, environmental contamination, or safety incidents provides a crucial perspective that abstract IT cybersecurity training often lacks. This helps defenders prioritize controls based on impact to safety and operations, not just data confidentiality.

Thirdly, Dilworth's detailed account of component selection, compatibility issues, and the fragility of industrial instruments highlights the need for specialized knowledge within OT security teams. Defenders must understand the physical layer, including voltage requirements, sensor limitations, and the specific ratings of hardware, as vulnerabilities can arise from physical misconfigurations or component failures. This reinforces the need for cross-functional teams with both IT and OT expertise.

Finally, the discussion of the cost-prohibitive nature of traditional ICS/OT engineering software and the successful use of free/trial alternatives (e.g., Phoenix Contact PLCnext IDE, Ignition server) offers a blueprint for more accessible security training. Defenders can leverage these tools to build virtual or hybrid labs, democratizing access to learning about PLCs, SCADA, and industrial programming without incurring exorbitant licensing fees. This is crucial for scaling up the workforce capable of securing increasingly interconnected OT environments.

In essence, Dilworth's talk equips defenders with a deeper appreciation for the intricacies of OT, empowering them to design more effective security architectures, respond to incidents with greater contextual awareness, and advocate for appropriate investments in securing critical infrastructure.

Key Takeaways

• Physical ICS/OT labs offer unparalleled realism but are exceptionally costly and time-consuming. Dilworth's Version 5 lab cost roughly $13,000 NZD and required significant time investment, making him question the ROI for non-VC-funded individuals.
• Accessible and effective alternatives exist for learning ICS/OT security. Free software like OpenPLC, Cotus (two-hour PLC systems), Ignition server (two-hour SCADA trial), Fuxer, and complete containerized solutions like Griffix v3 can provide robust learning environments on a standard laptop.
• Meticulous component selection and compatibility checks are critical. Issues like solenoid valve ratings (water vs. air), level sensor depth ranges, and voltage compatibility (e.g., 5V PWM board in a 24V system) can lead to significant delays, costs, and system failures.
• Proprietary engineering software licenses are a major barrier to entry. Traditional vendor PLC software can cost $11,000-$20,000 annually. Utilizing starter kits with free IDEs (like Phoenix Contact PLCnext) or trial SCADA software (like Ignition server) is essential for affordable learning.
• Network segmentation and protocol understanding are fundamental security practices. Building a lab allows hands-on experience with securing industrial protocols like OPC UA through firewalls, demonstrating critical defensive strategies.
• The process of building itself is a profound learning experience. Despite the challenges, the journey of designing, troubleshooting, and integrating various components provides invaluable insights into the complexities and interdependencies of ICS/OT systems.

About the Speaker(s)

Gavin Dilworth is an experienced security professional with over 22 years in the field. His extensive background drives his passion for educating others on the intricacies of industrial control systems and operational technology cybersecurity. Dilworth embarked on the ambitious project of building a physical ICS/OT lab to create practical training content, particularly focusing on how to securely transfer industrial protocols like OPC UA through a firewall and implement effective network segmentation. He openly acknowledges the significant assistance received from Marty, an IT professional who provided crucial advice and even pre-coded solutions, and Sephix, an electrician who helped with the complex wiring of the physical lab components. His talk reflects a deep commitment to sharing knowledge and demystifying the often-opaque world of OT security for a broader audience.