Don’t Forget the Little Guy: Vulnerability Management in Operational Technology

Don’t Forget the Little Guy: Vulnerability Management in Operational Technology

Speakers: Kyling Ranahan, CTO, Bazo; Alex Asante, Security Consultant, A Coming

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=PfMsj0NvS-A

Overview

This talk, "Don’t Forget the Little Guy: Vulnerability Management in Operational Technology," delivered by Kyling Ranahan (CTO of Bazo) and Alex Asante (Security Consultant at A Coming), offers a critical examination of the unique and often overlooked challenges inherent in managing vulnerabilities within Operational Technology (OT) environments. Unlike traditional Information Technology (IT) systems, OT systems are designed to control physical processes, meaning their compromise can lead to real-world consequences such as power outages, industrial accidents, or environmental damage. The speakers articulate why a direct translation of IT vulnerability management practices to OT is ineffective, highlighting the fundamental differences in priorities, asset lifecycles, and operational constraints.

The presentation serves as an essential guide for anyone involved in securing critical infrastructure, from IT security professionals transitioning to OT to seasoned OT operators. It underscores the necessity of a distinct approach to vulnerability management that acknowledges the primacy of safety, reliability, and performance over traditional IT security concerns like confidentiality. By delving into the regulatory landscape, the practical difficulties of patching, and the limitations of current vulnerability intelligence, Ranahan and Asante advocate for a proactive, context-aware strategy that leverages emerging standards like Common Security Advisory Format (CSAF) to bridge the gap between security needs and operational realities.

This talk is particularly significant because it addresses a growing concern in cybersecurity: the protection of the foundational systems that underpin modern society. As OT environments become increasingly connected and targeted by sophisticated threat actors, understanding their unique vulnerabilities and implementing tailored management strategies is no longer optional but imperative. The insights shared are crucial for fostering more resilient critical infrastructure and ensuring the continuous, safe operation of essential services.

Background

▶ Watch: Introduction: Vulnerability Management in Operational Technology (0:00)

Operational Technology (OT) encompasses systems that manage industrial operations and physical processes, contrasting sharply with Information Technology (IT) systems primarily focused on data. Key terminology includes Industrial Control Systems (ICS), often used interchangeably with OT, which specifically refer to systems controlling industrial processes. Many OT environments overlap with critical infrastructure, defined by CISA in the US across 16 sectors, including energy, water, chemical, and transportation.

A foundational difference between OT and IT security lies in their core priorities. While IT traditionally adheres to the CIA triad (Confidentiality, Integrity, Availability) with confidentiality often paramount, OT flips this paradigm. Speakers emphasize that for OT, the true priorities during system design and operation are Safety, Reliability, and Performance. Availability, while critical, is a manifestation of reliability and performance. For an electric utility, ensuring consistent power delivery (availability) and preventing catastrophic failures (safety) always take precedence over data confidentiality. This distinction is crucial for effective communication with OT operators, who may not have dedicated security teams and prioritize operational uptime above all else.

The challenges in OT vulnerability management are multifaceted:

• Non-homogeneous Environments: Unlike IT, where large numbers of assets might use a few operating systems and vendors, OT environments often feature a smaller number of assets (e.g., 250 assets in a generation facility) but with a significantly wider array of vendors (e.g., 30) and operating systems (e.g., 12). This makes standardized patch management exceptionally difficult.
• Asset Discovery and Management: Even with robust systems, accurately tracking all assets in diverse OT environments is challenging, leading to missed assets and gaps in vulnerability coverage.
• End-of-Life (EOL) and Legacy Systems: It is common to find critical OT devices running incredibly old, unsupported operating systems like Windows NT93, Windows 3.1, Windows XP, or Windows 7. Replacements are often unavailable, forcing reliance on gray markets or continued operation of insecure systems. These devices were designed for operation, not security, often lacking basic controls like strong password enforcement or certificates.
• Geographical Dispersion: Utilities, particularly electric distribution and transmission, can have thousands of assets (e.g., substations) spread across vast geographical areas, sometimes tens of thousands of square miles and even international borders. This necessitates manual patching, which is highly impractical at scale.
• Availability Constraints: OT systems, especially in critical infrastructure, cannot simply be taken offline for patching like IT systems. Outage windows are extremely limited, often restricted to "shoulder seasons" (e.g., spring/fall for power plants) or short maintenance windows, making patch application a complex logistical challenge involving vendor and technician availability. A generator facility, for instance, incurred millions in fines for being offline for just two hours.
• Regulatory Compliance vs. Security: While regulations like NERC CIP (North American Electric Reliability Corporation Critical Infrastructure Protection) aim to improve security, they don't always equate to true security. Regulations often lag cyber threats, and their open-ended nature can lead to minimum "check-the-box" compliance rather than robust security. Over-committing to security measures beyond the regulatory minimum can paradoxically lead to non-compliance if those ambitious goals aren't fully met, even if the system is more secure than the baseline. Past incidents, like the 2003 Northeast Blackout (leading to NERC CIP) and the Colonial Pipeline attack (leading to TSA Security Directives), demonstrate how incidents drive regulation, even for previously unregulated sectors.

Key Findings

▶ Watch: Understanding OT, ICS, and Critical Infrastructure relationships (3:00)

The talk highlights several critical findings that underscore the distinct nature of OT vulnerability management:

1. OT Priorities Dictate Security Approach: Unlike IT's confidentiality-first approach, OT environments prioritize Safety, Reliability, and Performance. Any vulnerability management strategy must align with these operational imperatives, meaning that security measures cannot disrupt critical functions or introduce unacceptable risks to physical processes.
2. NERC CIP-007 R2 is a Major Hurdle: The NERC CIP-007 Requirement 2 (R2), which mandates a 70-day window for discovery, assessment, and application/mitigation of security patches, is identified as the most common cause of non-compliance in the industry. This stringent timeline, coupled with the complexities of OT environments (legacy systems, limited outage windows, non-homogeneous assets), makes adherence extremely difficult in practice.
3. Traditional Vulnerability Metadata is Insufficient for OT: Standard vulnerability information sources like CVEs (Common Vulnerabilities and Exposures), CWEs (Common Weakness Enumerations), and CPEs (Common Platform Enumerations), while crucial for IT, often lack the practical detail required by OT practitioners. Details like "buffer overflow" don't inform remediation steps, and CPE data is frequently missing, inaccurate, or outdated due to acquisitions and product lifecycle complexities. This forces OT engineers, who may lack deep cybersecurity backgrounds, to manually decipher applicability.
4. Vendor Security Advisories are Paramount, Yet Often Inaccessible: For OT, vendor security advisories are the primary and most reliable source of information for determining vulnerability applicability, patch availability, and mitigation strategies. They often contain critical details not found elsewhere, such as specific affected products and versions, CVE-to-patch mappings, workaround information (essential when patching is impossible), and vital revision histories needed for compliance evidence.
5. CSAF is a Revolutionary Solution with Dismal Adoption: The Common Security Advisory Format (CSAF), a JSON schema-based standard championed by organizations like BSI (German Federal Cyber Security Agency), offers a machine-readable, structured language for vendor advisories. CSAF provides fields for all the crucial information currently found in human-readable advisories, enabling automation of vulnerability management. Despite its potential to transform OT security by allowing automated applicability checks without dangerous active scanning, adoption is "dismal," with only 18 out of 447 CNAs (CVE Numbering Authorities) providing CSAF advisories at the time of the talk.

Technical Deep Dive

▶ Watch: OT's priorities: Safety, Reliability, Performance vs. CIA (5:00)

The core of effective vulnerability management in OT environments, particularly in regulated sectors, revolves around understanding and navigating specific technical and operational constraints. The speakers delve into the intricacies of NERC CIP-007 R2 and the critical need for machine-readable vulnerability intelligence.

NERC CIP-007 R2: The Patch Management Mandate

For electric utilities in North America, NERC CIP-007 R2 dictates the process for managing security patches. It establishes a strict 70-calendar-day window for addressing identified vulnerabilities, broken down into two main phases:

1. Discovery and Notification (Day 0): The clock starts when a utility discovers or is notified of an available cyber security patch for its assets.
2. Applicability Assessment (First 35 days): Within 35 calendar days, the utility must review and evaluate the patch to determine if it applies to their specific assets and environment. This involves understanding the vulnerability's impact and relevance.
3. Patch Application or Mitigation (Next 35 days): Upon completion of the applicability assessment, the utility has another 35 calendar days to either apply the patch or implement an existing/new mitigation plan.

The speakers highlight why this strict timeline is frequently violated:

• Missed Assets: In non-homogeneous environments with diverse technologies, it's easy to miss an asset during inventory, leading to unpatched systems and non-compliance.
• Vendor Patch Cadence: Vendors releasing daily patches can overwhelm utilities, making it difficult to keep up with the 70-day window, especially when a significant portion of that time is allocated for assessment and implementation.
• Limited Outage Windows: As discussed, OT systems cannot be shut down arbitrarily. Patching often requires scheduled outage windows, which are scarce and must be coordinated with operational teams, vendors, and other maintenance activities. Reflashing firmware, common in OT, is not a "hotfix" and requires dedicated downtime.

Data Needs and the Case for CSAF

A significant technical challenge lies in the inadequacy of current vulnerability data for OT contexts:

• Limitations of CVEs and CPEs: While CVEs provide standardized identifiers for vulnerabilities, and CPEs attempt to standardize product names, their utility in OT is limited. A CVE description might identify a "buffer overflow" (a CWE), but this generic information doesn't tell an OT engineer how to fix it or if it even applies to their specific, often proprietary, industrial device. Furthermore, CPEs are frequently missing, incorrect, or fail to accurately reflect product lines, especially after acquisitions or for specialized OT hardware. This means the primary IT method of determining applicability (e.g., automated scanning for CPEs) is often unreliable or impossible in OT.
• Crucial Role of Vendor Security Advisories: In the absence of reliable automated data, OT practitioners heavily rely on vendor security advisories. These documents are invaluable because they typically provide:
• Complete Affected Products and Versions: Specific, accurate lists of products impacted.
• CVE-to-Patch Mapping: Clear guidance on which patches address which CVEs.
• Mitigation/Workaround Information: Crucially, advisories often detail steps to mitigate a vulnerability without applying a patch, which is vital when patching is impossible due to legacy systems, operational constraints, or lack of hotfixes.
• Revision History: Essential for NERC CIP compliance, as auditors require evidence that checks were performed on specific dates. Utilities often screenshot advisories to prove due diligence, especially after instances of vendors backdating advisories.

CSAF: The Path to Automation

The speakers strongly advocate for the Common Security Advisory Format (CSAF) as the technical solution to these data challenges. CSAF, developed by Oasis and championed by BSI, is a JSON schema that provides a structured, machine-readable language for creating and sharing vendor security advisories. Key technical benefits include:

• Structured Data: CSAF defines specific fields for all critical information currently found in human-readable advisories, such as affected products, versions, CVEs, patches, and mitigation steps.
• Machine Readability: Because it's a JSON schema, CSAF enables automated parsing and processing of advisory data. This is revolutionary for OT, where active network scanning for vulnerabilities is often prohibited or dangerous (e.g., a scan could cause a resource-constrained OT device to shut down).
• Automated Applicability and Remediation Mapping: CSAF allows for programmatic determination of which vulnerabilities affect which assets and provides direct mapping to specific patches or workarounds. This significantly reduces the manual effort and specialized cybersecurity knowledge currently required.

Despite its clear technical advantages and potential for automation, CSAF adoption remains critically low, highlighting a significant gap between the available solution and industry practice.

Demo / Proof of Concept

▶ Watch: Utility priorities: Operations, Safety, and balanced Security (6:30)

The talk did not include a live demonstration or proof of concept. The speakers focused on theoretical and practical challenges, regulatory frameworks, and the conceptual benefits of improved data formats like CSAF.

Defensive Implications

▶ Watch: What's at stake: Beyond compliance for OT vulnerability management (8:00)

The insights from this talk provide critical guidance for defenders operating in or transitioning to OT environments. The core message is that a fundamental shift in mindset and strategy is required.

1. Adopt a Proactive, Context-Aware Mindset: Defenders must move beyond a reactive "patch everything" approach common in IT. In OT, the focus must be on understanding the unique operational context, prioritizing safety and reliability, and developing proactive strategies to manage vulnerabilities within those constraints. Simply porting IT solutions will fail.
2. Prioritize Mitigation and Segmentation over Patching (When Necessary): Given the prevalence of legacy, EOL systems (like Windows NT93 or Windows XP) and the difficulties of patching, defenders must prioritize effective mitigation strategies. This includes robust network segmentation (not necessarily air gaps, but strict access controls and isolation), restricting access to critical assets, and implementing compensating controls. The goal is to limit risk as much as possible, accepting that some residual risk will remain, especially for systems without available patches. An example cited was finding a worm in an "air-gapped" Windows XP system that had persisted for 12 years, underscoring the need for layered defenses.
3. Advocate for CSAF Adoption from Vendors: One of the most impactful defensive actions is to demand machine-readable security advisories from OT vendors. By providing advisories in CSAF format, vendors can enable automated vulnerability management, reducing manual effort, improving accuracy, and allowing OT operators to quickly identify applicable patches or mitigations without risking system shutdowns from active scanning. This collective demand from customers can drive broader adoption.
4. Strengthen Asset Inventory and Management: Accurate and comprehensive asset management is foundational. In non-homogeneous OT environments, it's easy to miss devices, leading to critical vulnerabilities. Robust asset discovery and inventory systems are essential to know what needs protection and what vulnerability intelligence applies.
5. Balance Compliance and Security Pragmatically: While compliance with regulations like NERC CIP is mandatory and provides a baseline, defenders must understand that compliance does not automatically equate to robust security. When designing security programs, aim for both security and compliance, but be realistic about what can be achieved and documented. Over-promising on security measures that cannot be consistently met can lead to non-compliance, even if the system is more secure than the minimum standard.
6. Invest in Cross-Functional Training and Communication: Many OT operators are engineers first, not cybersecurity experts. Effective defense requires bridging this knowledge gap. Security teams need to understand OT operations, and OT personnel need foundational cybersecurity training, particularly regarding the practical implications of vulnerabilities and mitigations. Communicating security risks in terms of safety, reliability, and performance will resonate better with OT stakeholders.

Key Takeaways

• OT vulnerability management is fundamentally distinct from IT, driven by different priorities where safety, reliability, and performance supersede confidentiality.
• Legacy systems, non-homogeneous environments, and severe availability constraints make traditional IT-style patching and vulnerability scanning impractical or impossible in many OT settings.
• Regulatory compliance, particularly NERC CIP-007 R2, imposes strict patch management timelines (70 days) that are frequently violated due to operational realities and data challenges.
• Vendor security advisories are critical for OT practitioners, as standard CVE/CPE data often lacks the specificity needed for applicability and remediation, especially for specialized industrial hardware.
• The Common Security Advisory Format (CSAF) is a vital, revolutionary standard for machine-readable advisories that could enable automation and significantly improve OT vulnerability management, yet its adoption by vendors remains "dismal."
• Defenders in OT must adopt a proactive, context-aware approach, prioritizing mitigation (e.g., segmentation, access control) when patching is infeasible, fostering vendor collaboration, and advocating for CSAF.

About the Speaker(s)

Kyling Ranahan is the CTO of Bazo, a company specializing in vulnerability remediation and prioritization specifically for Operational Technology (OT) environments. His expertise lies in understanding the unique challenges of securing industrial systems and developing practical solutions tailored to their operational constraints.

Alex Asante is a Security Consultant at A Coming, where his day-to-day work focuses on the operational technology field. His professional experience includes vulnerability assessment, NERC (North American Electric Reliability Corporation) compliance science, and navigating regulatory standards for OT. He brings a deep understanding of the practical implementation and challenges of security within critical infrastructure.