Efficient Vulnerability Management in Hierarchical Supply Chains

Efficient Vulnerability Management in Hierarchical Supply Chains

Speakers: Toby, Researcher, Siemens Research; Tom KD, Vulnerability and Incident Handler, Siemens

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=wrSuIFL3j_c

Overview

In an increasingly interconnected world, where software supply chains grow in complexity and depth, managing vulnerabilities efficiently has become a critical challenge for organizations of all sizes. This talk, presented by Toby and Tom KD from Siemens, delves into the intricate problems faced by a global industrial giant in maintaining security across its vast and diverse product portfolio. Siemens, known for developing everything from embedded industrial controllers to SaaS solutions and operating critical infrastructure, grapples with an exponential increase in reported vulnerabilities and the complexities introduced by nested dependencies.

The core of the presentation highlights the "Bermuda Triangle of undiscovered vulnerabilities" – a historical struggle with manual processes that stifled efficient vulnerability disclosure and management. With the advent of modern standards like SBOMs, VEX, and CSAF, alongside Siemens' evolution into a CVE Numbering Authority (CNA), the industry has seen improvements, yet new challenges emerge. The speakers articulate a vision for a robust, transparent, and automated vulnerability management system centered around empowering product teams while ensuring consistent information flow across a multi-layered supply chain, from upstream component developers to downstream critical infrastructure operators.

This article will explore Siemens' journey from reactive, manual vulnerability handling to a proactive, integrated approach. It will detail their proposed architectural solutions, including a sophisticated product mapping database and a central vulnerability intelligence system, designed to tackle issues like varying CVSS scores across contexts, the need for re-rating, and the critical importance of standardized data exchange. The insights shared are particularly relevant for large enterprises navigating complex product ecosystems and seeking to build resilience against supply chain attacks and widespread vulnerabilities like Log4j.

Background

▶ Watch: Introduction to speakers and Siemens' broad scope (0:00)

Siemens operates across an exceptionally broad and diverse technological landscape, creating unique challenges for vulnerability management. Their product range spans numerous categories:

• Software and Embedded Devices: From industrial controllers (e.g., PLCs for factories) produced in the tens of thousands, to infrastructure for trains, including traffic lights.
• SaaS Products: Cloud-based offerings for various industries.
• Integrated Solutions: Building and operating customized infrastructure for customers, such as entire factories, buildings, or stadiums.
• Operating Infrastructure: Managing not only IT infrastructure but also critical industrial control systems (ICS).

This vast scope means Siemens deals with a highly distributed and non-homogeneous array of projects, each with its own development methodologies, programming languages, and operational contexts. Historically, Siemens, like many large organizations, faced significant hurdles in vulnerability disclosure. Before becoming a CVE Numbering Authority (CNA) in 2016, the company struggled with a "Bermuda Triangle of undiscovered or unpublished vulnerabilities." Manual processes for assigning CVEs led to a backlog and an inability to efficiently communicate security information to the broader ecosystem. The shift to CNA status and the adoption of automation tools, such as the CVE Record Submission and Upload Service, significantly improved their capacity to allocate and publish CVEs.

However, this increased efficiency also coincided with an exponential rise in the sheer volume of vulnerabilities. The emergence of concepts like Software Bill of Materials (SBOMs) and Vulnerability Exploitability eXchange (VEX), while promising, further amplified the data management challenge. Customers began to demand specific information about the impact of vulnerabilities within the precise components they were using, rather than just generic upstream advisories. This created an "exponential problem in multiple dimensions" centered on managing the complex issues inherent in a hierarchical supply chain.

The talk illustrates a typical linear supply chain: a vulnerability in an upstream component like zlib might be embedded in OpenSSL, which is then used by Apache, integrated into an internal web server, then a subproduct, and finally deployed in a customer's solution or machine (e.g., a factory). This chain involves entirely different project types, from pure software development to system integration and operation. A critical problem arises if vulnerability information "drops" at any point, leaving downstream users unaware. Furthermore, the CVSS (Common Vulnerability Scoring System) score for a vulnerability changes dramatically as it moves through the supply chain. An initial CVSS score for zlib might decrease when embedded but could significantly increase if the embedding context exposes new attack vectors. This necessitates constant re-evaluation and re-rating of vulnerabilities based on their specific context and impact on the mission-critical systems downstream.

Key Findings

▶ Watch: The 'Bermuda Triangle' of undiscovered vulnerabilities (3:00)

The Siemens team's extensive experience in vulnerability management within a complex industrial ecosystem led to several key findings regarding the challenges and necessary solutions for efficient vulnerability handling:

1. Exponential Problem in Multiple Dimensions: The volume of vulnerabilities, combined with the depth of software supply chains and the diverse needs of upstream suppliers and downstream customers, creates an exponential increase in the complexity of vulnerability management. This is exacerbated by the need to understand specific impacts for individual components within integrated systems, not just generic upstream advisories.
2. Context-Dependent CVSS and Re-rating Necessity: A vulnerability's CVSS score is not static; it changes significantly as the vulnerable component is embedded and integrated into different products and operational environments. The initial root cause description is insufficient for operators who need to understand the safety, mission, and operational impact. Therefore, continuous vulnerability re-rating and re-evaluation are crucial to provide relevant information to different stakeholders along the supply chain.
3. Information Flow as a Critical Bottleneck: The primary challenge in hierarchical supply chains is ensuring that vulnerability information is seamlessly forwarded from upstream component developers to downstream operators. Any break in this chain can lead to undiscovered vulnerabilities in critical systems. This necessitates robust mechanisms for cross-organizational data exchange.
4. "One Tool to Rule Them All" Fails: Attempting to force diverse project teams (e.g., zlib developers versus airport operators) to use a single, monolithic vulnerability management tool is impractical and ineffective. Different teams have different use cases, technical implementations, and tooling requirements. The focus must be on enabling different tools to interact and leverage common data effectively.
5. Product Teams are Central, but Need Support: Product teams possess the essential expertise to assess vulnerabilities within their specific context. Any effective vulnerability management strategy must empower and incentivize their participation. However, they require standardized processes and tools that simplify their input and integrate with broader intelligence.
6. Component Mapping is a Major Pain Point: Identifying and consistently naming components across different tools, ecosystems (e.g., OpenSSL in Fedora vs. Debian vs. RedHat), and packaging variations is extremely difficult. Current standards like PURL and CPE are often insufficient in practice, leading to manual, labor-intensive mapping efforts. This is particularly challenging for legacy "brownfield" projects lacking modern package managers.
7. Data Transparency and Centralization are Paramount: To achieve synergies (rate once, reuse many times), there is a strong need for centralized collection and transparent sharing of vulnerability intelligence and rated vulnerability data across the entire organization. This includes a central SBOM database and a Vulnerability Intelligence system.
8. Addressing Duplicates and Gaps from Scanners: Organizations often use multiple vulnerability scanners, which can lead to duplicate notifications or, conversely, gaps in coverage. An adapted SBOM tailored for vulnerability management can act as a "routing mechanism" to ensure the right people are asked about the right components, minimizing duplicates and ensuring comprehensive coverage.

Technical Deep Dive

▶ Watch: Exponential rise in vulnerabilities and SBOMs (4:20)

Siemens' approach to tackling the complex problem of vulnerability management in hierarchical supply chains revolves around a vision centered on product teams and robust data exchange mechanisms. Their architecture is designed to overcome the challenges of diverse tooling, context-specific risk, and the sheer volume of information.

At the heart of their vision is the Product Mapping Database. This internal system addresses the critical issue of inconsistent component identification across various tools and ecosystems. While standards like PURL (Package Uniform Resource Locator) and CPE (Common Platform Enumeration) exist, the speakers note that in practice, these are often insufficient for their complex environment. For instance, OpenSSL might be packaged differently in Fedora, Debian, or RedHat, each effectively becoming a distinct component that needs to be tracked. The Product Mapping Database aims to establish a "ground truth" for components, linking these disparate identifiers and providing a consistent view of what a component is, regardless of its origin or packaging. This database is a cornerstone for all subsequent vulnerability intelligence and tracking efforts.

The next crucial element is their internal Vulnerability Intelligence system. This system serves as a central repository for storing the impact of vulnerabilities according to specific entities within Siemens' vast supply chain graph. It is fed by a variety of sources:

• External Sources: This includes public CVE (Common Vulnerabilities and Exposures) databases, CSAF (Common Security Advisory Framework) advisories (which they strongly advocate for), GitHub Security Advisories, and OSV (Open Source Vulnerabilities).
• Internal Scanners: Siemens employs multiple vulnerability scanners to cover different project types, programming languages, and environments. These scanners continuously identify specific vulnerabilities, feeding their findings into the intelligence system.

When a vulnerability is discovered and correlated with a product in scope, the Vulnerability Intelligence system notifies a central "vulnerability tracker" (the specific name varies internally, but its function is consistent). This tracker then initiates a request for information from the relevant product team. The product team, being the experts for their specific product, assesses the vulnerability's impact, exploitability, and severity within their unique context. This assessment is critical because, as highlighted in the background, the CVSS score and impact description can vary drastically depending on how a component is embedded and used. For example, a vulnerability in zlib might have a high CVSS base score, but its actual impact on a specific Siemens industrial controller could be lower if certain vulnerable functions are not called, or higher if the controller's operational context introduces new exposure.

The information provided by the product teams is then fed back into the Vulnerability Intelligence system. This closed-loop feedback mechanism ensures that the intelligence system holds the most accurate, context-specific data. This information can then be propagated along the supply chain. If a subproduct uses a component, and that subproduct's team rates a vulnerability, that rating can inform the teams building solutions that incorporate the subproduct, and so on, all the way to the end customer operating a factory.

Siemens also emphasizes the importance of pushing this context-specific information back into external databases, where possible, and into their public security advisories. This contributes to the broader security ecosystem and enhances transparency. The overall workflow also integrates a central SBOM database, which initially served for license clearing but is now being adapted for vulnerability management. This adaptation involves creating "adapted SBOMs" that contain only the necessary components for vulnerability management, excluding those already covered by project-specific scanners, effectively acting as a routing mechanism for vulnerability notifications to avoid duplicates and ensure targeted communication.

For "brownfield" projects—legacy systems, some 30-35 years old—the technical challenges are particularly acute. Introducing modern package management or fully automated SBOM generation is often impossible. For these systems, the solution relies more on expert manual evaluation and careful prioritization, as opposed to the more automated approaches feasible for "greenfield" (new, modern DevOps) projects.

Demo / Proof of Concept

▶ Watch: Illustrating a typical linear supply chain example (5:55)

The talk did not include a live demonstration or a specific proof of concept of their systems in action. Instead, the speakers presented a high-level architectural overview and conceptual workflows of their internal systems, such as the Product Mapping Database and Vulnerability Intelligence, illustrating how they envision and implement efficient vulnerability management within Siemens.

Defensive Implications

▶ Watch: Critical need for vulnerability information forwarding (7:35)

The insights shared by Toby and Tom KD offer significant defensive implications for organizations grappling with complex software supply chains:

1. Embrace Contextual Risk Assessment: Defenders must move beyond generic CVSS scores and adopt methodologies for contextual risk assessment. This means re-evaluating vulnerabilities based on how components are actually used within specific products and operational environments. Tools and processes should be established to facilitate this re-rating, recognizing that impact on a mission-critical system can differ vastly from an upstream component's base score. This might involve adopting frameworks that allow for environmental scoring or tailored impact analysis.
2. Invest in Robust Component Mapping: The "major pain point" of inconsistent component identification highlights the need for dedicated efforts in component mapping. Organizations should develop internal databases or leverage existing tools to create a "ground truth" for their software components, linking different identifiers (e.g., PURL, CPE, internal IDs). This is foundational for accurate SBOM generation and vulnerability intelligence correlation. Where automated solutions fall short, prepare for manual curation, potentially augmented by AI.
3. Build a Centralized Vulnerability Intelligence Platform: Establish a central Vulnerability Intelligence system that aggregates data from diverse sources – public CVEs, CSAF advisories, OSV, GitHub, and internal scanners. This platform should be designed to store not just raw vulnerability data, but also the context-specific impact assessments provided by product teams. This centralized hub ensures a single source of truth and enables consistent information flow.
4. Prioritize Information Flow Across the Supply Chain: Implement clear processes and technical mechanisms to ensure vulnerability information is seamlessly forwarded along the entire supply chain, from upstream dependencies to downstream deployments. This prevents information "dropping" and ensures all relevant stakeholders are informed. This could involve automated ticketing, standardized data formats (like CSAF), and integrations between different tracking systems.
5. Empower and Integrate Product Teams: Recognize that product teams are the ultimate experts for their specific products. Design vulnerability management workflows that actively empower their participation by providing them with streamlined tools and processes to assess and report impacts. Avoid "one tool to rule them all" approaches; instead, focus on interoperability between tools that cater to different team needs.
6. Leverage Adapted SBOMs as a Routing Mechanism: Explore the concept of adapted SBOMs for vulnerability management. These tailored SBOMs can help filter out redundant information or components already covered by specific scanning tools, effectively acting as a "routing mechanism" to direct vulnerability notifications to the correct responsible parties. This reduces noise and improves the efficiency of incident response.
7. Address Legacy Systems Strategically: For "brownfield" or legacy projects where modern package management is not feasible, defenders must accept that manual, expert-driven vulnerability assessment will remain critical. Develop specific, tailored processes for these systems, focusing on rigorous risk evaluation to inform patching decisions, especially given the high cost of patching embedded or deeply integrated systems.
8. Advocate for Industry Standardization: Contribute to and advocate for better industry standards for component naming, vulnerability data exchange, and SBOM/VEX utilization. The speakers' emphasis on the need for society/economy to improve on "naming things" underscores a collective responsibility to reduce friction in the security ecosystem.

By implementing these defensive strategies, organizations can build a more resilient and efficient vulnerability management program that scales with the complexity of modern software supply chains.

Key Takeaways

• Contextual Vulnerability Re-rating is Essential: CVSS scores are not static; vulnerabilities must be re-evaluated and re-rated based on their specific context within different products and operational environments to provide meaningful risk assessments.
• Information Flow is Paramount in Supply Chains: Seamlessly forwarding vulnerability information from upstream components to downstream deployments and operators is critical to prevent information loss and ensure timely remediation.
• Centralized Vulnerability Intelligence is a Must: A robust, internal Vulnerability Intelligence system fed by diverse sources (CVE, CSAF, OSV, scanners) and enriched by product team assessments is necessary for a single source of truth.
• Product Teams are Key Stakeholders: Empowering product teams with appropriate tooling and processes to assess context-specific vulnerability impacts is crucial, as they possess the deep expertise required for accurate risk evaluation.
• Component Mapping is a Foundational Challenge: Consistently identifying and mapping software components across various tools and packaging ecosystems is a significant hurdle that requires dedicated solutions, such as a Product Mapping Database, to create a "ground truth."
• Adapted SBOMs Enhance Efficiency: Using tailored SBOMs as a routing mechanism can help manage duplicates from multiple scanners and ensure vulnerability notifications reach the correct responsible parties, improving the overall efficiency of vulnerability management.

About the Speaker(s)

Toby is a seasoned professional in vulnerability management currently working as a Researcher at Siemens Research. He played a pivotal role in the early days of Siemens' vulnerability handling efforts, being one of the initiating members of Siemens ProductCERT. In this foundational role, Toby was responsible for setting up the entire infrastructure, workflows, processes, and organizational structure for vulnerability monitoring and incident response. His current work continues to focus on advancing vulnerability management within the research arm of Siemens.

Tom KD serves as a Vulnerability and Incident Handler at Siemens Product. He took over the responsibilities at ProductCERT after Toby's transition, inheriting a system that initially relied on a manual, Microsoft Office-based approach. Tom's expertise lies in improving these processes and focusing on the development of APIs for Siemens products to enable further automation in vulnerability handling, a key theme of their joint presentation. His work is instrumental in evolving Siemens' vulnerability management capabilities to meet the growing demands of modern software supply chains.