Embracing the Era of Transparency: Automating VEX Application for Scalable, Context-Aware Security

Embracing the Era of Transparency: Automating VEX Application for Scalable, Context-Aware Security

Speakers: Jessica Butler (Software Engineer, NVIDIA), Kristina Joos (Product Manager, NVIDIA)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=_yTAUmVhN6o

Overview

In an era increasingly defined by software supply chain complexities, the challenge of effectively communicating vulnerability exploitability information to consumers remains a significant hurdle. This talk, delivered by Kristina Joos, a Product Manager for NVIDIA AI Enterprise, and Jessica Butler, a Software Engineer at NVIDIA, directly addresses this critical issue. Titled "Embracing the Era of Transparency: Automating VEX Application for Scalable, Context-Aware Security," the presentation chronicles NVIDIA's journey in adopting and evolving their Vulnerability Exploitability eXchange (VEX) practices. The core message revolves around NVIDIA's commitment to transparency, not just in publishing VEX records, but in making them genuinely consumable and actionable for their customers.

The speakers highlight a pervasive disconnect: while many organizations, including NVIDIA, are diligently generating machine-readable VEX statements for the CVEs found in their software, these efforts often fail to translate into tangible benefits for the end-user. Customers continue to face "red alerts" from their scanners, leading to manual investigations, delays, and a flood of support requests to vendors. NVIDIA's experience with its vast portfolio of AI software, which relies heavily on open-source components and generates hundreds of thousands of VEX documents annually, underscores the urgent need for systemic improvements in how VEX is published, discovered, and integrated into downstream security workflows.

This presentation is highly relevant for anyone grappling with software supply chain security, particularly those involved in producing or consuming complex software artifacts like container images. It provides a candid look at the practical challenges of VEX adoption beyond mere generation, proposing a forward-thinking solution leveraging OCI attestations and advocating for industry-wide collaboration to achieve truly scalable, context-aware vulnerability management. By sharing their story, NVIDIA offers valuable insights into the technical and process-related gaps that prevent VEX from fulfilling its promise of streamlined, proactive security communication.

Background

▶ Watch: NVIDIA's VEX journey and customer challenges (2:00)

NVIDIA, widely recognized for its hardware innovations, also develops and publishes a substantial volume of software, particularly within its NVIDIA AI Enterprise suite. This suite provides the foundational building blocks for production AI applications, encompassing everything from application development frameworks like NVIDIA NeMo and Omniverse, to models, efficient packaging (NIM), model serving, and GPU orchestration software, down to low-level operators like GPU Operator and Network Operator. Each of these components can comprise numerous artifacts, with many being containerized images. A single NVIDIA AI container image, for instance, can incorporate over 350 software dependencies and packages, a significant portion of which are open source.

This reliance on a vast and diverse software ecosystem, combined with the enterprise-grade stability requirements (avoiding API breaks), presents immense security challenges. The sheer scale—many products, many containers, and an ever-increasing number of CVEs—demands robust automation. NVIDIA recognized early on that merely patching vulnerabilities internally was insufficient; effectively communicating the exploitability status of these vulnerabilities to customers was paramount. This led them to embrace the "era of transparency" and specifically, Vulnerability Exploitability eXchange (VEX) records.

NVIDIA's VEX journey began over two years ago with its "production branch" for NVIDIA Enterprise. This branch offered stable container versions for nine months, with monthly CVE patches. Many of these patches were private backports, invisible to standard vulnerability scanners. Consequently, customers scanning these images would incorrectly flag patched CVEs as exploitable. VEX provided a crucial mechanism to communicate that a CVE, despite appearing vulnerable based on its version, was in fact resolved due to NVIDIA's internal patching. Initially, this process was manual and limited to a small set of containers and resolved CVEs, with VEX records made available for download on their container registry, NGC. However, as customer reports escalated and the volume of CVEs grew, it became clear that a manual approach was unsustainable. The need for a scalable, automated solution for VEX generation and distribution became an imperative.

Key Findings

▶ Watch: NVIDIA's sophisticated VEX automation engine (7:00)

NVIDIA's experience, despite significant investment in VEX generation, revealed a critical gap: the act of publishing VEX does not automatically equate to consumption and effective integration into customer security workflows. They found that even with machine-readable VEX documents available for all their images, customers continued to submit "scary and very long PDFs" of scanner reports, requesting manual review and clarification on CVEs. This led to a significant amount of manual triage for NVIDIA and deployment delays for customers.

The core problem stems from several interconnected issues:

1. Workflow Disconnect: The individual pulling the container image is often not the same person who later runs security scans or reviews the results. Images are frequently re-tagged, renamed, and distributed within internal registries, losing their original VEX attachments or metadata. When a scan occurs, typically right before deployment, the VEX information is often not readily available or accessible to the scanner or the security team.
2. Lack of Automated Consumption: NVIDIA's initial assumption that customers would inherently know how to find, interpret, and integrate VEX records into their internal processes proved incorrect. Many customers resort to manual methods, like converting JSON VEX files into spreadsheets for tracking, which entirely defeats the purpose of machine-readability and automation.
3. Scanner Incompatibility: A major barrier is the current state of vulnerability scanners. Most scanners do not natively discover, ingest, or filter results based on VEX documents attached to artifacts. This means even if VEX is present, scanners still report all detected CVEs, creating "red alerts" for issues that have already been assessed as non-exploitable or resolved by the vendor.
4. Ecosystem Immaturity: There is a lack of standardized, widely adopted mechanisms for VEX documents to travel with their associated software artifacts throughout the entire software supply chain—from vendor registry to customer deployment environment. This requires broader industry collaboration on discovery mechanisms and integration into common tooling.

In essence, NVIDIA discovered that while they had achieved scalable VEX generation, the ecosystem for scalable VEX consumption was still nascent, leading to persistent friction and manual overhead for both vendor and customer.

Technical Deep Dive

▶ Watch: The broken VEX consumption workflow problem (9:00)

NVIDIA's internal VEX generation process, internally dubbed Vex Stream, is a sophisticated, highly automated workflow designed to manage the immense scale of their software portfolio. When a container is scanned, the automation engine is triggered. This engine integrates several layers of logic and data sources:

1. Vendor VEX Integration: The system automatically pulls in VEX information from upstream vendors like Ubuntu and Red Hat, which informs the initial status of CVEs in their base images.
2. Parent Image Recognition: NVIDIA's internal builds are often based on specific internal parent images. The system recognizes these relationships and applies VEX statements applicable to the parent image to its child images, ensuring consistency and reducing redundant analysis.
3. Global VEX Rules: For certain CVEs, NVIDIA can define global VEX rules, applying a consistent status across a broad portfolio of products where the exploitability assessment is known to be the same.
4. Rules-Based Assessment Engine: This complex engine applies a series of internal rules to determine the VEX status for specific CVEs based on various factors, including vendor data, internal patches, and architectural context.
5. Human Review with Gentic Workflows: While heavily automated, some complex cases still require human intervention. Security analysts use Gentic workflows to review specific CVEs, assess exploitability (e.g., "is this package used?", "is the path exploitable?"), and then approve the proposed VEX statement.
6. Reporting Index and OpenVEX Generation: All generated VEX data is dumped into a reporting index, which allows for dashboard creation for leadership and serves as the source for creating OpenVEX documents. NVIDIA is currently using CycloneDX for VEX and is also exploring OpenVEX.

The output of this robust generation pipeline is a comprehensive set of machine-readable VEX documents. However, the challenge lies in making these documents seamlessly discoverable and consumable by customers. NVIDIA proposes leveraging Cosign attestations as a standardized mechanism for attaching VEX documents to OCI (Open Container Initiative) artifacts.

Cosign, a tool from the Sigstore project, enables signing and verification of software artifacts. Crucially, it also supports attestations, which are signed statements about an artifact. By using Cosign, NVIDIA can:

• Associate VEX with Artifacts: Attach VEX documents (e.g., in CycloneDX or OpenVEX format) directly to container images.
• Enable Signing: Cryptographically sign these attestations, providing integrity and authenticity guarantees for the VEX data.
• Facilitate Discovery: Cosign offers commands to discover and download attestations associated with an OCI artifact.

The proposed workflow involves:

1. Generating VEX documents from NVIDIA's internal Vex Stream and reporting index.
2. Using cosign attest to attach these VEX documents as attestations to the relevant container images in their NGC catalog.
3. On the customer side, when an image is pulled (e.g., via docker pull), the customer would then use cosign download attestation to retrieve the associated VEX documents.

NVIDIA acknowledges a current gap: this is still a two-step process (pull image, then download attestation). The ideal state would be for VEX attestations to automatically travel with the artifact during a docker pull or similar command. Furthermore, while the attestations ensure the VEX data is available, the ultimate goal is for customer-side scanners to automatically ingest and filter based on this VEX. They specifically mentioned testing with an "experimental option in Trivy," a popular open-source vulnerability scanner, which allows it to consume VEX automatically via OCI attestations. However, this functionality is experimental, not yet universally supported across all scanners, and has shown compatibility issues with newer OCI versions.

Despite these challenges, the use of Cosign attestations represents a significant step towards a more robust, standardized, and discoverable VEX distribution model, enabling the VEX data to persistently travel with the software artifact through different registries and environments.

Demo / Proof of Concept

▶ Watch: Demo: Scanner filtering with VEX via OCI attestations (19:00)

Jessica Butler walked through a conceptual demonstration of how the proposed tooling and workflow could operate today, highlighting both its capabilities and current limitations. The core of the demo illustrated the journey of a VEX document from its generation within NVIDIA's system to its consumption by a customer's scanner.

The demonstration began by showing a simplified script that would take NVIDIA's internal VEX data (derived from their reporting index, which consolidates vendor VEX, internal rules, and human assessments) and convert it into the OpenVEX format. This internal data includes classifications such as affected vulnerabilities with no fix available (e.g., 121 instances in the example), vulnerabilities under investigation, and those confirmed as "not affected" or "fixed" due to NVIDIA's patches or partner contributions.

Next, the demo showed how Cosign is used to attach these OpenVEX documents as attestations to a container image. The command cosign attest --type openvex --predicate openvex.json <image_reference> would be used to perform this attachment. A subsequent cosign verify-attestation command would then confirm that the VEX attestation has been successfully added and signed.

On the customer side, the process was outlined:

1. Image Pull: The customer pulls the container image from the NVIDIA NGC catalog or their internal registry.
2. Attestation Download: The customer would then execute cosign download attestation <image_reference> to retrieve the associated VEX documents. This step is currently manual and separate from the image pull, a noted area for future improvement.
3. Scanner Integration (Trivy): The demonstration then focused on how a scanner like Trivy could consume this VEX data. Without VEX, a standard Trivy scan would output a long list of vulnerabilities (e.g., 186 in the example), many of which NVIDIA has already assessed as non-exploitable or fixed. This leads to the "red alerts" that trigger customer inquiries.
4. VEX-Filtered Scan: Jessica then showed how Trivy's experimental option (--vex-oci-attestation or similar) allows it to automatically detect and consume VEX attestations attached via OCI. When this feature is enabled, Trivy would filter its results, suppressing vulnerabilities that are deemed "not affected" or "fixed" by the VEX document. The output would then show only the truly actionable vulnerabilities, such as those for which the vendor has confirmed no fix is available yet, or those still under investigation.

The demo effectively showcased the potential of this approach to dramatically reduce noise and provide context-aware scan results. However, it also underscored the critical limitations: the two-step download process for attestations, the experimental and non-standardized nature of scanner integration (Trivy's feature being experimental and facing OCI version compatibility issues), and the need for broader industry adoption to make this an end-to-end, seamless flow.

Defensive Implications

▶ Watch: VEX end-to-end flow requires ecosystem collaboration (20:00)

The insights shared by NVIDIA offer crucial defensive implications for organizations consuming and deploying software, particularly containerized applications. The primary takeaway is the urgent need to move beyond simply scanning for CVEs and integrate VEX (Vulnerability Exploitability eXchange) into the core of vulnerability management workflows.

1. Demand VEX from Suppliers: Defenders should actively request and prioritize software from vendors who provide VEX documents. This includes not just the presence of VEX, but its availability in machine-readable, standardized formats like CycloneDX or OpenVEX.
2. Advocate for Scanner Integration: The current state of scanner integration with VEX is a significant bottleneck. Defenders should pressure their security tool vendors (for SCA, DAST, SAST, etc.) to natively support VEX ingestion, discovery, and filtering via mechanisms like OCI attestations. The goal is for scanners to automatically suppress non-actionable alerts based on vendor-provided context, moving from "red alerts everywhere" to focused, actionable insights.
3. Implement VEX-Aware Workflows: Organizations should design their internal security pipelines to consume VEX. This might involve:

• Automated Attestation Retrieval: Integrating tools like Cosign into their CI/CD pipelines to automatically download VEX attestations alongside container images.
• Internal VEX Processing: Developing or adopting tools that can parse VEX documents and integrate their information into internal vulnerability tracking systems, ideally avoiding manual spreadsheet conversions.
• Policy Enforcement: Using VEX data to inform policy decisions, allowing deployments to proceed for images with known CVEs that have been declared "not affected" or "fixed" by the vendor's VEX.

1. Understand Exploitability Context: VEX empowers defenders with critical context beyond just a CVE ID and CVSS score. It clarifies whether a vulnerability is actually exploitable in the specific context of the software being used, if it has been internally patched by the vendor (even if the version number doesn't reflect it), or if it's irrelevant due to how the component is utilized. This context is invaluable for prioritizing remediation efforts and reducing false positives.
2. Contribute to Ecosystem Development: The challenges highlighted by NVIDIA underscore that a fully functional VEX ecosystem requires collective effort. Defenders should engage with industry initiatives, open-source projects (like Sigstore/Cosign, OpenVEX, Trivy), and standards bodies to push for better interoperability, standardized discovery mechanisms (e.g., automatic VEX movement with docker pull), and broader scanner support.
3. Internal VEX Generation: For organizations building their own software, NVIDIA's "Vex Stream" automation provides a blueprint. Consider implementing similar rules-based engines, leveraging vendor VEX, and establishing clear workflows (potentially with human review for complex cases) to generate accurate and timely VEX for their own products.

By embracing these defensive strategies, organizations can move from reactive, alert-fatigued vulnerability management to a proactive, context-aware security posture that leverages vendor transparency to its fullest potential.

Key Takeaways

• VEX Publishing != VEX Consumption: Simply generating and making VEX documents available is not enough; the industry needs better mechanisms for seamless discovery, ingestion, and integration into customer workflows and security tooling.
• Automation is Essential for Scale: Managing thousands of dependencies and hundreds of thousands of CVEs across a vast product portfolio necessitates highly automated VEX generation, as exemplified by NVIDIA's "Vex Stream" internal tool.
• OCI Attestations are a Promising Path: Using Cosign attestations to attach VEX documents (e.g., in CycloneDX or OpenVEX formats) to OCI artifacts offers a standardized way for VEX to travel with container images, ensuring authenticity and discoverability.
• Scanner Integration is the Next Frontier: The lack of native VEX consumption and filtering in most vulnerability scanners (e.g., Trivy's experimental feature) is a major impediment. Industry collaboration is crucial to enable scanners to automatically leverage VEX and reduce "red alerts."
• Contextual Analysis is Key: VEX provides critical context, clarifying whether a reported CVE is truly exploitable, has been patched by the vendor, or is irrelevant, allowing defenders to prioritize effectively and avoid unnecessary remediation efforts.
• Ecosystem Collaboration is Imperative: Achieving a truly transparent and efficient software supply chain security model requires collective effort from vendors, tool developers, and consumers to standardize VEX formats, distribution methods, and scanner integrations.

About the Speaker(s)

Kristina Joos is a Product Manager at NVIDIA, specifically working on NVIDIA AI Enterprise. Her role primarily involves collaborating with customers to ensure that NVIDIA's software meets their security and compliance expectations. She is a strong advocate for transparency in software security and has been instrumental in NVIDIA's adoption and evolution of VEX practices. This was her first time presenting at VulnCon.

Jessica Butler is a Software Engineer at NVIDIA. She works closely with Kristina Joos to develop and implement solutions for NVIDIA's VEX challenges. Jessica is the technical mind behind much of the automation discussed, including the internal "Vex Stream" tool. She was also scheduled to give a separate talk titled "Taming the Scanner Storm" at the same conference, which would delve deeper into the technical details of NVIDIA's VEX automation.