Getting Down and Dirty with SBOMs

Getting Down and Dirty with SBOMs

Speakers: Colby Prior, Security Operations Engineer, Octopus Deploy

Conference: Kawaiicon

YouTube: https://www.youtube.com/watch?v=Qp4Ute61-M

Overview

In his Kawaiicon talk, "Getting Down and Dirty with SBOMs," Colby Prior, a Lead Security Engineer at Octopus Deploy, delves into the critical importance and evolving landscape of Software Bill of Materials (SBOMs). The central theme of his presentation is transparency, emphasizing that in a world increasingly reliant on complex software supply chains, hiding vulnerabilities behind a "ghost" of unknown dependencies is no longer viable. Prior shares his practical experience from Octopus Deploy, a software vendor providing CI/CD solutions, which was compelled by customer demand to generate SBOMs for its commercial off-the-shelf product.

The talk highlights that while SBOMs are not a new concept, recent regulatory shifts and growing industry awareness are making them an indispensable tool for every organization. Prior passionately advocates for both software vendors to produce SBOMs and for customers to demand them, framing this as the only effective way to drive adoption and enhance collective security. He dissects the technical intricacies, challenges, and solutions involved in generating, managing, and utilizing SBOMs, particularly focusing on the Vulnerability Exploitability eXchange (VEX) format as a crucial component for effective communication during security incidents.

Prior's presentation serves as a comprehensive guide for anyone looking to understand the current state and future trajectory of software supply chain security. It bridges the gap between theoretical concepts and real-world implementation, offering actionable advice for developers, security professionals, and even sales teams. By exploring the nuances across different programming languages and build environments, he underscores the complexities involved while providing clear pathways to achieving greater transparency and resilience in software ecosystems.

Background

▶ Watch: Introduction to SBOMs and transparency (0:00)

The concept of a Software Bill of Materials (SBOM) has been evolving for well over a decade, initially gaining traction primarily as a mechanism for managing software licensing. Early formats like SPDX (Software Package Data Exchange) emerged to help organizations track open-source licenses, preventing the accidental inclusion of incompatible copyleft licenses in proprietary software. Later, CycloneDX, developed by OWASP, arose with a more explicit focus on security, aiming to provide a comprehensive inventory of software components to identify and mitigate vulnerabilities. While these formats initially had distinct focuses and preferred data structures (e.g., CycloneDX's early reliance on XML, which Prior expresses a strong dislike for), they have largely converged, both now supporting JSON and being largely convertible, albeit with potential minor data loss.

The fundamental problem SBOMs address is the pervasive lack of visibility into the components that make up modern software. Applications are rarely monolithic; they depend on a vast web of third-party libraries, which in turn depend on other libraries—a chain of transitive dependencies that can extend several layers deep. This complexity makes it incredibly difficult for organizations to understand their true risk posture or respond effectively to widespread vulnerabilities. The infamous Log4j incident of late 2021 served as a stark, global wake-up call, demonstrating how a single vulnerability in a widely used, deeply nested library could expose countless applications across industries. Had comprehensive SBOMs been readily available, organizations could have programmatically identified their exposure much more rapidly, drastically reducing response times from weeks to hours.

To manage the deluge of information an SBOM provides, tools like OWASP Dependency-Track have become indispensable. This open-source project offers a visual, centralized platform for ingesting, analyzing, and managing SBOMs across an entire organization's software portfolio. It not only highlights known CVEs (Common Vulnerabilities and Exposures) but also allows for policy enforcement based on licensing, component versions, and even advanced metrics like EPSS (Exploit Prediction Scoring System), which uses machine learning to predict the likelihood of a CVE being exploited in the wild.

Beyond technical solutions, the regulatory landscape is rapidly shifting to mandate SBOM adoption. In 2021, the US Executive Order 14028 on Improving the Nation's Cybersecurity notably mentioned SBOMs, though Prior points out its use of the permissive word "should" diminished its immediate impact. However, subsequent developments like the 2023 "Secure by Design" document, a collaborative effort by multiple national cybersecurity agencies, and especially the EU Cyber Resilience Act (CRA), announced in 2024 and coming into effect in 2027, introduce the much stronger term "shall." The CRA marks a significant turning point, making SBOMs a legal requirement for manufacturers of products with digital elements, at least for top-level dependencies. This regulatory momentum is finally cementing SBOMs as a foundational element of secure software development and deployment.

Key Findings

▶ Watch: EU Cyber Resilience Act: SBOMs will be legally required (7:30)

Colby Prior's talk underscores several critical findings that are shaping the current and future landscape of software supply chain security:

1. Mandatory Adoption is Imminent: While SBOMs have been a "good idea" for years, the regulatory environment, particularly with the EU Cyber Resilience Act's "shall" clause taking effect in 2027, is shifting them from optional best practice to mandatory requirement. This legislative push will fundamentally change how software vendors operate and how customers procure software, making SBOMs a standard artifact.
2. Customer Demand is a Primary Driver: Prior's personal experience at Octopus Deploy demonstrates that direct customer advocacy is a powerful catalyst for SBOM generation. Customers demanding SBOMs as a prerequisite for purchasing or using a product can force vendors to prioritize their creation, even over internal backlogs. This highlights the crucial role users play in driving industry-wide adoption.
3. The Transitive Dependency Challenge Persists: Despite the push for SBOMs, the initial regulatory focus (e.g., CRA) may only mandate "top-level dependencies." Prior stresses that true transparency and security require tracking all transitive dependencies, which represent the vast majority of components and potential attack surface. This gap represents an ongoing challenge for comprehensive supply chain security.
4. VEX is Essential for Practical Vulnerability Management: The realization that "vulnerabilities are in everything" necessitates a sophisticated communication mechanism beyond just listing CVEs. The Vulnerability Exploitability eXchange (VEX) format is presented as a crucial innovation. VEX allows vendors to provide context—whether a specific vulnerability in a component is actually exploitable in their product, why it's not, or what remediation steps are needed. This prevents "alert fatigue" and enables programmatic, actionable communication, saving immense time during incident response.
5. Language-Specific Nuances are a Major Hurdle: The talk meticulously details how different programming languages and ecosystems (Node.js, Python, .NET) present unique challenges for accurate SBOM generation. Issues like dev dependencies, compiler optimizations, runtime environments, and self-contained packages mean that a "one-size-fits-all" approach to SBOM tooling is ineffective. Developers must understand these nuances to generate truly representative and useful SBOMs.
6. Transparency Benefits Everyone: While some vendors fear "advertising" their vulnerabilities, Prior argues that providing SBOMs, especially when coupled with VEX, builds trust and streamlines incident response. It replaces reactive, email-based communication with proactive, programmatic data exchange, ultimately benefiting both vendors (reduced support overhead) and customers (faster risk assessment).

Technical Deep Dive

▶ Watch: Customers must demand SBOMs from software vendors (9:00)

The technical core of Prior's talk revolves around the practicalities of generating, managing, and contextualizing SBOM data, emphasizing the tools and formats that make this possible.

SBOM Formats: SPDX and CycloneDX

The foundation of any SBOM strategy lies in the chosen format. Prior discusses SPDX and CycloneDX as the two predominant standards. While SPDX originated with a focus on licensing and CycloneDX on security, they have largely matured to support both aspects comprehensively. A key development is their evolution from idiosyncratic formats (e.g., CycloneDX's original XML) to JSON, which has vastly improved tool support and interoperability. Although subtle differences remain, and some information might be lost during conversion, the ability to convert between them means that the choice of format is less critical than the commitment to producing an SBOM.

Centralized Management with OWASP Dependency-Track

Managing a multitude of SBOMs across numerous projects demands a centralized solution. Prior highly recommends OWASP Dependency-Track, an open-source project that can be easily deployed (e.g., via Docker Compose). Dependency-Track provides a powerful graphical user interface (GUI) for visualizing and tracking component health. It flags out-of-date libraries (green for up-to-date, yellow for minor patches, red for major versions behind) and known vulnerabilities (CVEs).

Beyond basic vulnerability tracking, Dependency-Track enables:

• Organizational Overview: Querying across all projects to identify where a specific vulnerable library (like Log4j) is being used.
• Policy Enforcement: Configuring alerts based on licensing types (e.g., avoiding copyleft), specific CVE severities, or even EPSS (Exploit Prediction Scoring System) scores, which use machine learning to estimate the likelihood of a CVE being exploited.
• Custom Alerting: Setting up alerts for non-vulnerability-related issues, such as changes in package maintainer terms and conditions.

Vulnerability Exploitability eXchange (VEX)

The most significant technical innovation discussed is VEX (Vulnerability Exploitability eXchange). Acknowledging that every piece of software will have vulnerabilities, VEX provides a standardized, programmatic way to communicate the exploitability of a known CVE within a specific product. While Prior calls it "more like a vibe," he notes that specific standards like CycloneDX VEX exist.

A VEX file (typically JSON) links a CVE ID to a specific product identifier (Package URL - PURL) and provides a status regarding its impact on the product. Key statuses include:

• Affected: The product is vulnerable.
• Not Affected: The product is not vulnerable, even if the underlying component has a CVE. This is crucial for reducing false positives and "noise."
• Reasons for Not Affected: VEX allows for detailed explanations, such as "code not reachable," "vulnerable code not present," "control flow does not reach vulnerable code," or "vulnerable configuration not enabled." This directly addresses situations where a component might have a CVE, but the way it's used in the product renders it harmless.
• Fixed: The vulnerability has been patched in a specific version.
• Under Investigation: The vendor is still assessing the impact.

VEX also includes an optional vendor response section for details like "will not fix" (for unsupported products), recommended update versions, or even temporary workarounds and rollback instructions. Dependency-Track integrates VEX, allowing users to suppress irrelevant vulnerabilities based on vendor-provided VEX data, significantly cleaning up dashboards and focusing efforts on truly exploitable issues.

A practical challenge with VEX is its lifecycle management. When a new version of a product is released, the VEX status of a vulnerability might change (e.g., a previously "not affected" function might become reachable). Prior shares Octopus Deploy's approach: they do not copy VEX forward for minor or major semantic versions but do for patch versions, based on a risk assessment. Dependency-Track doesn't offer this out-of-the-box, requiring custom scripting.

Language-Specific Challenges

Prior dedicates a significant portion of his talk to the "spooky" and often painful realities of generating accurate SBOMs across different programming languages:

• Node.js: The complexity stems from multiple package managers (npm, yarn, PNPM) and the distinction between dev dependencies and runtime dependencies. While pnpm-lock.yaml provides hashes and versions, its inclusion in the SBOM for external customers is debated. Including dev dependencies can inflate SBOMs by orders of magnitude (e.g., is-string showing up as a dependency), but excluding them might hide vulnerabilities in the developer's build chain. The key is to generate SBOMs at build time, capturing the actual versions used.
• Python: Similar to Node.js, Python's package management (requirements.txt, pip) and runtime flexibility make SBOM generation tricky. A requirements.txt might specify requests>=2.27.0 but not the exact version, making the runtime environment critical. pip freeze can capture exact versions but must be run during the build process and is OS-dependent.
• .NET (Prior's "White Whale"): This ecosystem presents the most significant challenges.
• Source Code vs. Build Artifacts: Early SBOM tools often scanned source code, but the .NET compiler makes the final decision on dependency versions, which can differ from what's listed in source. Accurate SBOMs require scanning build artifacts like the deps.json file generated after publishing.
• Runtime Dependencies: A "spicy CVE" in Kestrel (part of ASP.NET Core) highlighted a major issue: many SBOM tools exclude runtime packages because .NET applications are typically portable and rely on the host's .NET runtime. This means a critical vulnerability in a core framework component might not appear in the application's SBOM.
• Self-Contained Packages: .NET allows publishing as a self-contained package, which bundles the .NET runtime with the application. This changes the patching strategy: patching the host's runtime does nothing; the package itself must be updated. This nuance creates significant confusion and complexity for defenders.
• Golang and Rust: In contrast, Prior praises modern languages like Golang and Rust for their superior dependency management and SBOM tooling support, noting they largely bypass these complexities.

Tooling and Automation

Prior advocates for Trivy, an open-source, Golang-based CLI tool by Aquasec. Trivy is highly versatile, supporting numerous languages and integrating seamlessly into CI/CD pipelines. He demonstrates its use with GitHub Actions:

• The trivy-action can be used to generate an SBOM by pointing it at the correct file during the build process.
• The upload-artifact action stores the generated SBOM.
• Prior has also developed a public upload-sbom-github-action to automate the uploading of SBOMs to Dependency-Track, streamlining the entire process.

He also touches on Operational Bill of Materials (OBOMs) for Docker containers, capturing OS packages and runtime dependencies, which is particularly useful for containerized applications where the runtime environment is more controlled than a bare metal deployment.

Demo / Proof of Concept

▶ Watch: The pain of generating .NET SBOMs accurately (22:00)

While Colby Prior's talk did not feature a live, interactive demonstration or proof of concept in the traditional sense, he effectively walked the audience through the practical implementation of SBOM generation and management. He presented screenshots and code snippets illustrating the user interface of OWASP Dependency-Track for visualizing vulnerabilities and managing VEX, as well as GitHub Actions workflows demonstrating how Trivy can be integrated into a CI/CD pipeline to automate SBOM creation and upload to Dependency-Track. This approach provided clear, actionable examples of how these tools and processes work in a real-world development environment, rather than a live, step-by-step demo.

Defensive Implications

▶ Watch: Recommended tool Trivy and CI/CD integration (26:00)

The insights shared in "Getting Down and Dirty with SBOMs" provide several critical defensive implications for organizations, urging a proactive and systematic approach to software supply chain security:

1. Demand SBOMs from Vendors: The most immediate and impactful action for any organization consuming third-party software is to ask for SBOMs. As Prior emphasizes, customer demand is the most potent driver for vendors to produce these artifacts. Without an SBOM, assessing the true risk of commercial software remains a guessing game.
2. Generate SBOMs for All Software: For internally developed applications, open-source projects, and any software your organization maintains, generating SBOMs should become a standard practice. This fosters internal transparency, aids in compliance, and significantly improves incident response capabilities. Tools like Trivy and integration with CI/CD pipelines (e.g., GitHub Actions) make this highly achievable.
3. Implement Centralized SBOM Management: Tools like OWASP Dependency-Track are essential for aggregating, analyzing, and monitoring SBOMs across an entire software portfolio. This allows security teams to gain an organizational-wide view of their dependencies, enforce policies (e.g., license compliance, acceptable vulnerability scores), and quickly identify exposure to new, critical CVEs (like Log4j).
4. Embrace VEX for Contextualized Vulnerability Communication: Simply having an SBOM that lists CVEs isn't enough; it will inevitably lead to alert fatigue. Defenders should understand and advocate for the use of VEX (Vulnerability Exploitability eXchange). VEX allows vendors (or internal teams) to communicate whether a reported vulnerability in a component is actually exploitable within their specific product, reducing noise and enabling focused remediation efforts.
5. Understand Language-Specific Nuances: Be acutely aware that SBOM generation is not a "one-size-fits-all" process. Different programming languages (.NET, Node.js, Python) have unique dependency resolution mechanisms, build processes, and runtime considerations that impact SBOM accuracy. Defenders need to understand these intricacies to ensure the SBOMs they consume or produce are truly representative of the deployed software.
6. Prepare for Regulatory Compliance: With the EU Cyber Resilience Act and other regulations on the horizon, organizations need to proactively integrate SBOM generation and management into their development and procurement processes. This includes understanding the requirements (e.g., top-level vs. transitive dependencies) and preparing for the legal mandates coming into effect by 2027.
7. Shift to Proactive Incident Response: SBOMs, especially when combined with VEX, transform incident response from a reactive, email-driven chaos into a programmatic, data-driven process. Instead of emailing 50 vendors during a crisis, organizations can leverage SBOMs to quickly identify affected systems and VEX data to understand the actual impact, allowing for more efficient and effective mitigation.
8. Consider Operational Bill of Materials (OBOMs): For containerized environments, generating OBOMs that detail the operating system packages and runtime dependencies within Docker containers provides an additional layer of transparency, enhancing the security posture of the deployment environment itself.

Key Takeaways

• SBOMs are Shifting from "Should" to "Shall": Upcoming regulations like the EU Cyber Resilience Act (effective 2027) will mandate SBOMs, making them a legal requirement for software vendors and driving widespread adoption.
• Customer Demand is Crucial: Actively asking software vendors for SBOMs is the most effective way to compel their production and foster greater supply chain transparency across the industry.
• VEX is Essential for Practical Vulnerability Management: The Vulnerability Exploitability eXchange (VEX) format provides critical context to SBOMs, indicating whether a reported vulnerability is actually exploitable in a specific product, thereby reducing alert fatigue and streamlining incident response.
• Centralized Management is Key: Tools like OWASP Dependency-Track are vital for ingesting, analyzing, and managing SBOMs across an organization's entire software portfolio, enabling programmatic risk assessment and policy enforcement.
• Language-Specific Challenges Require Nuance: Generating accurate and comprehensive SBOMs requires understanding the unique dependency resolution, build processes, and runtime behaviors of different programming languages (e.g., .NET's compiler decisions, Node.js's dev dependencies).
• Automate SBOM Generation: Integrating tools like Trivy into CI/CD pipelines (e.g., via GitHub Actions) automates SBOM creation and upload, making the process efficient and scalable.

About the Speaker(s)

Colby Prior is a Lead Security Engineer at Octopus Deploy, a software vendor specializing in CI/CD solutions. His role involves navigating the complexities of software security within a commercial product environment, including the practical implementation of software supply chain security measures like SBOM generation. Prior brings a hands-on, pragmatic perspective to the challenges of securing modern software, drawing from his experience in a company that produces a self-hostable product, requiring deep insights into its constituent components. His work at Octopus Deploy has provided him with direct experience in addressing customer demands for transparency and preparing for evolving industry regulations.