Taming the Scanner Storm: How VEX Brings Context to Vulnerability Data

Taming the Scanner Storm: How VEX Brings Context to Vulnerability Data

Speakers: Jessica Butler, NVIDIA

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=YKBIdPfQ2Zc

Overview

In the modern software supply chain, organizations face an overwhelming deluge of vulnerability data, often referred to as a "scanner storm." This talk, presented by Jessica Butler from NVIDIA, addresses the critical challenge of making sense of millions of vulnerability findings generated by daily container image scans. The core problem lies in the sheer volume and the lack of contextual information, forcing developers into inefficient, severity-only triage processes that often lead to chasing vulnerabilities with no actionable fixes or vendor support.

NVIDIA's innovative solution, dubbed the VEX overlay system or VEXstream, is a two-part pipeline designed to integrate vendor-provided Vulnerability Exploitability eXchange (VEX) data directly into vulnerability reports. By applying a sophisticated rules-based engine, this system automatically classifies and prioritizes findings, empowering developers to focus on genuinely actionable issues. This initiative, which began six months prior to the talk, has been successfully rolled out across all of NVIDIA's scanned container images, demonstrating significant improvements in vulnerability management efficiency and developer focus.

The talk highlights how this system transforms raw scanner output into intelligent, context-rich reports, moving beyond simple Common Vulnerabilities and Exposures (CVEs) severity ratings to incorporate vendor statements on exploitability, affectedness, and available patches. By providing clear guidance on which vulnerabilities require immediate attention versus those that are not affected or have no immediate fix, NVIDIA is taming its internal scanner storm and offering a blueprint for other organizations grappling with similar challenges in their software supply chain security.

Background

▶ Watch: Introduction and the "scanner storm" problem (0:00)

The genesis of NVIDIA's VEX overlay system stems from a pervasive problem in software development: the "scanner storm." Modern development practices, particularly those involving containerization, necessitate frequent and comprehensive security scanning. NVIDIA, for instance, scans approximately 34,000 container images daily, generating Software Bill of Materials (SBOMs) and updating vulnerability data. This extensive scanning, while crucial for security posture, yields an astonishing 26 million vulnerability findings across 24,000 distinct CVEs.

The traditional approach to managing this volume of data typically involves severity-based triage, where developers prioritize critical and high-severity findings. However, this method is fundamentally flawed due to a critical lack of context. Developers are often forced to manually research each vulnerability, delving into vendor advisories, National Vulnerability Database (NVD) entries, and upstream project information to determine if a fix exists, if the vulnerability is truly exploitable in their specific context, or if the vendor considers it a false positive or not affected. Jessica Butler revealed a stark statistic: 66% of these findings do not have vendor patches available, meaning that while an upstream fix might exist, the vendor has not yet incorporated it, or they deem its impact lower than the generic NVD severity. This disconnect leads to significant wasted effort, as developers chase issues they cannot fix, or that aren't relevant to their specific use case.

Furthermore, the lack of a standardized feedback loop from vendors exacerbates the problem. Different scanners may report the same vulnerability with varying statuses, or some might remove vulnerabilities that a vendor has declared "not affected," while others do not. This inconsistency leads to discrepancies in reports, causing confusion for internal teams and external customers who might use different scanning tools. NVIDIA experienced this directly, with customers questioning why certain vulnerabilities appeared in their scans but not in NVIDIA's reports, necessitating time-consuming investigations into false negatives or positives. The need for a centralized, normalized source of vendor-provided VEX data became apparent to address these challenges, enable consistent reporting, and provide actionable intelligence to developers.

Key Findings

▶ Watch: Two-stage pipeline architecture for VEX ingestion and overlay (6:20)

NVIDIA's implementation of the VEX overlay system has yielded significant, quantifiable improvements in their vulnerability management process, demonstrating its effectiveness in cutting through scanner noise. The primary achievement is the ability to apply vendor-informed decisions to a vast majority of vulnerability findings.

Key findings include:

• 94% VEX-Informed Decisions for No-Fix Findings: For vulnerabilities identified as having "no fix" available through traditional scanning methods, the VEX system successfully applied vendor data to 94% of findings for CVEs published in 2020 or later. This drastically reduces the manual research burden on developers.
• 82% of Vulnerabilities Not Actionable by Developers: A crucial insight is that 82% of the total vulnerability findings (across all severities) cannot be directly addressed by developers through simple actions like running updates or grabbing the latest package releases from distributions. This highlights the scale of the "noise" that VEX helps to filter.
• Significant Impact on Medium and Low Severities: The system particularly shines in managing medium and low-severity vulnerabilities. Out of 17 million such findings with no immediate fix, 84% were successfully informed by VEX data. This allows developers to avoid spending time on issues that vendors deem low impact or for which no patch is imminent.
• Prioritization of Critical and High Vulnerabilities: While the system excels at filtering lower-priority issues, it also ensures that critical and high-severity vulnerabilities receive appropriate attention. NVIDIA observed that vendors are more proactive in providing fixes or "not affected" statements for high-impact issues. For example, a relatively small number (around 2,200) of critical and high vulnerabilities were found to be "not affected" by vendor statements, compared to the millions of lower-severity findings.
• Auto-Triage for Recently Published CVEs: The system incorporates a mechanism to automatically classify recently published CVEs as "in triage." This provides a temporary VEX state, acknowledging that the vulnerability is new and under investigation, preventing immediate blocking of releases while vendor VEX data catches up.
• Improved Vendor Data Coverage by Distro: The effectiveness varies by distribution. Red Hat distributions showed very high VEX-informed ratings, while Ubuntu versions also had decent coverage. The system is still expanding its ingestion sources, with Alpine and Debian VEX data being future targets.
• Actionable Private VEX Data: The system effectively integrates NVIDIA's own private VEX data, such as internal patches or third-party vendor fixes, enabling "upgrade suggested" actions that scanners would otherwise miss. This accounted for a significant portion of actionable items in the final reports.

Overall, these findings illustrate how the VEX overlay system provides tangible benefits by transforming a chaotic "scanner storm" into a manageable and actionable set of vulnerability reports, allowing development teams to focus their efforts where they matter most.

Technical Deep Dive

▶ Watch: Rule engine logic: prioritizing VEX states and actions (10:00)

NVIDIA's VEX overlay system, or VEXstream, is architected as a robust, two-stage pipeline designed to ingest, process, and apply vendor-provided VEX data to internal vulnerability scan reports. This system is crucial for providing context and reducing the overwhelming volume of raw scanner output.

The first stage focuses on VEX Data Ingestion and Normalization. Recognizing that VEX data is often provided in disparate formats, NVIDIA built a system to daily pull VEX statements from various sources. Currently, this includes:

• Ubuntu VEX data from GitHub.
• Red Hat Common Errata Format (CEF) VEX data.

Future plans include incorporating Alpine and Debian VEX data.

Upon ingestion, these diverse VEX formats are normalized into a consistent internal representation. This involves mapping different vendor states (e.g., "triage," "under investigation") to a standardized set of internal classifications, which are then ultimately mapped to CycloneDX-compliant output statuses like not_affected, affected, fixed, or under_investigation. To optimize processing, the system uses a hash for each VEX document, ensuring that only changed documents are re-processed and added to a growing database. While this database is becoming "quite large," the focus remains on ensuring comprehensive coverage.

The second stage, the Processing and Rule-Based Engine, is triggered each time a container image scan is completed. An API call sends the scan report to the VEXstream system. The system then performs a lookup against its normalized VEX database, matching vulnerabilities by package, version, and vulnerability ID (CVE ID, GHSA ID, etc.). This lookup determines if relevant vendor VEX data exists for each finding.

Following the lookup, a sophisticated rule-based engine applies contextual logic to determine the final VEX action and status. This engine prioritizes actions based on NVIDIA's risk tolerance and operational needs:

1. Upgrade Suggested: If the VEX data (including NVIDIA's private VEX) indicates that a vulnerability is patched in a specific version, an "upgrade suggested" action is triggered. This is the highest priority, encouraging developers to apply known fixes. The system explicitly will not auto-VEX anything with an upgrade suggested, pushing for direct remediation.
2. Vendor VEX Not Affected: If no upgrade is suggested, but the vendor VEX explicitly states the component is not_affected or fixed (even if the scanner reports it), this status is applied. This prioritizes authoritative vendor statements over generic scanner findings, especially for private VEX data.
3. Scanner Fix Available: If neither of the above applies, but the scanner report itself indicates a fix is available, this status is used.
4. Vendor VEX Affected (with Rules): In certain cases, if the vendor states the component is affected but no fix is available, the system may auto-VEX it. This is typically applied for vulnerabilities where the vendor-stated impact is medium or lower, allowing developers to defer action or accept the risk without manual investigation. For critical and high-severity vulnerabilities, even if "vendor affected" with no fix, developers are still required to review and determine mitigations.
5. No VEX or Upgrade Found: If no VEX data is available, and no upgrade is found, the system still provides context. For recently published vulnerabilities (e.g., within the last 5 days), the system might automatically assign an "in triage" status, acknowledging that the issue is new and under review. For critical and high vulnerabilities, these are never auto-VEXed away; developers must investigate.

The rule engine is highly flexible, allowing rules to be added or modified at runtime, affecting future reports immediately. It can examine various fields from both the scanner report (package, version, vulnerability ID) and the VEX data. The output of this process enriches the original scan report with the determined VEX state, justification, and a link to the underlying vendor VEX data. This enriched data is then published, often to internal dashboards, providing developers with clear, prioritized actions and the rationale behind each VEX decision. This comprehensive technical approach transforms raw vulnerability data into actionable intelligence, significantly streamlining NVIDIA's security operations.

Demo / Proof of Concept

▶ Watch: Key takeaways: cutting scanner noise and maturing VEX ecosystem (22:00)

While the presentation did not feature a live, interactive demonstration of the VEX overlay system in action, the entire talk serves as a comprehensive proof of concept through its detailed architectural explanation and extensive quantitative results. Jessica Butler presented numerous data points and statistics directly derived from the operational system, showcasing its real-world impact across NVIDIA's entire fleet of container images.

The "Demo / Proof of Concept" is primarily conveyed through:

• System Architecture and Workflow: The detailed explanation of the two-stage pipeline, from daily VEX ingestion and normalization to the rule-based engine applying context to scan reports, illustrates the system's functional design.
• Quantitative Results: The core of the proof of concept lies in the impressive numbers presented: 94% VEX-informed decisions for "no fix" findings, 82% of vulnerabilities being non-actionable by simple updates, and 84% VEX-informed decisions for medium and low-severity vulnerabilities. These figures are direct measurements of the system's effectiveness.
• Example CVE Journey: A step-by-step walkthrough of how a newly discovered CVE is processed, from scanner detection to VEX lookup, rule engine application, and final report enrichment, serves as a conceptual demonstration of the system's flow and output.
• Output Reports and Dashboards: The speaker described how the system enriches existing reports with the VEX state, the rule that triggered it, and links to vendor data, which are then visualized in "simple dashboards." These outputs are the tangible results of the system's operation, guiding developers on what to prioritize.

In essence, the talk itself, with its deep dive into the system's mechanics and its statistical successes, acts as a powerful testament to the VEX overlay system's viability and significant benefits for large-scale vulnerability management.

Defensive Implications

▶ Watch: Q&A: pipeline clarification and future context/AI plans (23:30)

The NVIDIA VEX overlay system offers profound defensive implications for organizations struggling with the "scanner storm" and the inefficiencies of traditional vulnerability management. By providing critical context to raw scanner data, defenders can significantly enhance their security posture and operational efficiency.

Here are the key defensive implications:

1. Prioritized Remediation Efforts: The most immediate benefit is the ability to cut through noise and prioritize truly actionable vulnerabilities. By identifying the 82% of findings that are not directly fixable by developers through simple updates, or those where vendors have declared them "not affected" or "low impact," security teams can direct developer resources to the critical 18% (or less) that genuinely require attention. This reduces developer fatigue and ensures security efforts yield maximum impact.
2. Accurate Risk Assessment: VEX data provides a more accurate understanding of actual risk. A high-severity CVE from NVD might be less critical if the vendor has issued a VEX statement indicating it's not exploitable in their specific product or that its impact is lower in their context. This allows defenders to make informed decisions based on real-world applicability rather than generic severity scores.
3. Improved Communication and Trust: Consistent VEX-informed reports eliminate discrepancies between internal scans and customer-facing audits. When customers scan a product and find a CVE, NVIDIA can now provide a VEX-backed explanation (e.g., "vendor states not affected"), fostering trust and reducing time spent on resolving false positives or inconsistent findings.
4. Automated Decision Making: The rule-based engine allows for automated decisions on lower-severity vulnerabilities, freeing up security analysts and developers from manual triage of millions of findings. This ensures that security policies (e.g., "don't auto-VEX critical/highs") are consistently applied at scale.
5. Proactive Supply Chain Security: By ingesting VEX data from multiple vendors (Red Hat, Ubuntu), organizations gain deeper insights into the security posture of their upstream dependencies. This enables more informed decisions about which base images or packages to use, potentially shifting away from those with persistent, unfixable vulnerabilities.
6. Faster Release Cycles: Reducing the time developers spend chasing non-actionable vulnerabilities directly contributes to faster and more secure software release cycles. Teams can meet security gates with confidence, knowing that reported vulnerabilities have been properly contextualized and prioritized.
7. Adaptability to Evolving Threats: The flexible rule engine allows security teams to adapt their VEX application logic to new threats, organizational risk tolerance changes, or evolving vendor VEX practices without requiring extensive code changes to the core system. This means, for example, new rules can be quickly added to handle specific types of zero-day vulnerabilities or newly identified false positives.

In essence, the VEX overlay system transforms vulnerability management from a reactive, overwhelming task into a proactive, intelligent, and efficient process, enabling defenders to truly "tame the scanner storm."

Key Takeaways

• VEX Data is Essential for Context: Raw scanner data alone is insufficient. Ingesting and applying vendor-provided VEX (Vulnerability Exploitability eXchange) data is crucial to cut through the "scanner storm" noise and provide actionable context.
• Automate Exploitability Decisions: A two-stage pipeline with a rule-based engine can automate the classification and prioritization of vulnerabilities, especially for medium and low severities, based on vendor statements (e.g., not_affected, fixed, affected with low impact).
• Focus Developer Efforts: By filtering out non-actionable vulnerabilities (e.g., 82% in NVIDIA's case), development teams can concentrate their remediation efforts on the critical and high-impact issues that truly require a fix or mitigation.
• VEX Ecosystem is Maturing but Requires Effort: While more vendors are producing VEX data, the ecosystem is still evolving. Organizations need to invest in normalization, data ingestion, and custom rule engines to effectively utilize this data due to varying formats and cadences.
• Scanners Need to Catch Up: Current vulnerability scanners often rely on raw vulnerability feeds without overlaying public VEX data. Industry adoption of VEX integration into scanning tools would significantly reduce the need for organizations to build custom overlay systems.
• Trust and Review are Paramount: Before fully automating VEX decisions and publishing reports, it is crucial to thoroughly review and categorize initial data outputs to build confidence in the system's accuracy and ensure it aligns with organizational risk tolerance.

About the Speaker(s)

Jessica Butler is a security professional at NVIDIA, where she has been instrumental in developing and implementing the VEX overlay system (VEXstream). Her work focuses on tackling the significant challenge of managing vast amounts of vulnerability data, particularly within container images, to enable developers to prioritize and act on genuinely critical issues. Jessica's expertise lies in building systems that bring context to vulnerability findings, moving beyond traditional severity-only triage. She collaborates closely with colleagues, including Christina and Kajel, on these innovative projects at NVIDIA, demonstrating a commitment to improving software supply chain security practices.