Mind the Match: Why Vulnerability Matching Is Harder Than You Think

Mind the Match: Why Vulnerability Matching Is Harder Than You Think

Speakers: Alexandra Selldorff, Senior Staff Software Engineer, Manifest Cyber

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=_WYMpvXlmHI

Overview

In her VulnCon talk, "Mind the Match: Why Vulnerability Matching Is Harder Than You Think," Alexandra Selldorff of Manifest Cyber dissects the often-overlooked complexities and inherent challenges within the automated process of vulnerability matching. Selldorff argues that the prevalent assumption — that running a scanner yields a definitive list of CVEs to fix — is fundamentally flawed. Instead, she reveals a landscape rife with inconsistencies, false positives, and critical false negatives stemming from diverse scanner logic, identifier ambiguities, and fragmented vulnerability databases.

The presentation serves as a crucial wake-up call for security practitioners, urging them to move beyond blind trust in scanner outputs. Selldorff demonstrates, through compelling real-world experiments, how seemingly identical software components can produce vastly different vulnerability reports across various tools. Her insights highlight the significant operational burden and security risks introduced by these discrepancies, emphasizing the need for a deeper understanding of the underlying mechanisms and a more robust verification workflow to ensure accurate vulnerability management.

Background

▶ Watch: Can you trust vulnerability scanners? (Yes and No) (0:40)

The journey from a software component to an identified vulnerability is a multi-step process, each fraught with potential pitfalls. It typically begins with a Software Bill of Materials (S-BOM) generator or a direct scanner analyzing a repository or container. These tools parse manifest and lock files to identify dependencies, including transitive dependencies that might be several layers deep. The quality and depth of this initial package identification are critical, as missed or misinterpreted packages at this stage will inevitably lead to missed vulnerabilities downstream.

Once packages are identified, they must be resolved to standardized identifiers. The two primary identifiers discussed are Common Platform Enumeration (CPE) and Package URLs (PURLs). CPEs, designed for commercial software and widely used in the National Vulnerability Database (NVD), suffer from free-form vendor, product, and version fields, making them susceptible to issues with forks, distro repackaging, and backports. PURLs, on the other hand, are ecosystem-native, seamlessly integrating with package managers like NPM, PyPI, Maven, and Cargo, and are increasingly preferred in modern advisories like Open Source Vulnerability (OSV). However, PURLs lack a standardized qualifier spec, leading to inconsistent usage across different generators and databases.

These identifiers are then used to map against various vulnerability databases. Each database has its own strengths and weaknesses: NVD offers high-quality, manually reviewed data but often suffers from enrichment lag; cve.org can provide earlier hints via name/vendor; OSV and GitHub Security Advisories (GHSA) introduce alias divergence, where a single vulnerability might have multiple IDs; and distro-specific advisories (e.g., Red Hat) provide the most accurate version information for their respective ecosystems but are highly specific. Finally, matching logic within scanners determines how these identifiers are compared against database entries, ranging from strict exact matches to more forgiving fuzzy matching that normalizes for variations in capitalization or punctuation, each choice impacting the number and accuracy of reported vulnerabilities.

Key Findings

▶ Watch: Why the same image yields wildly different findings (6:00)

Selldorff's talk unveiled several critical findings that underscore the fragility of current vulnerability matching practices, primarily through empirical demonstrations of scanner divergence and identifier mismanagement.

A stark illustration of this divergence was her experiment with a Red Hat 8 image scanned by two popular tools, Grype and Trivy. The results showed an astonishing 80.5% divergence in the number of findings, with one reporting 852 CVEs and the other 3,719. This massive discrepancy was attributed to several factors:

1. Package-Level Differences and Suppression Logic:

• Kernel Headers: Grype automatically suppresses vulnerabilities related to kernel-headers packages, recognizing them as development packages that don't typically imply a runnable kernel, thus reducing noise. Trivy, however, surfaces these, leaving the decision to the user. This highlights a fundamental difference in scanner philosophy regarding what constitutes an "actionable" vulnerability.
• Setuptools: Trivy completely missed Python packages like setuptools, failing to identify them during the initial package discovery phase. This demonstrates a critical failure at the very beginning of the pipeline, leading to false negatives and potentially missing high-severity CVEs.
• Tomcat: For a Java JAR like Tomcat, Trivy correctly identified 16 CVEs, while Grype found zero. This was due to differing matching logic and identifier resolution. Trivy utilized its internal Java index, whereas Grype inferred the Maven Group ID from the directory path, constructing an incorrect PURL qualifier that prevented a match against vulnerability databases.

1. Additional Disagreements Beyond Package Identification:

• Fix Information: Scanners often disagree on whether a fix exists. For binutils CVEs, Grype reported no fix, while Trivy provided no information, leading to wasted team time chasing non-existent patches or indefinitely monitoring unfixable issues.
• Diverging Scores: Different CVE Numbering Authorities (CNAs) assign varying CVSS scores to the same vulnerability. Scanners may pull from different sources (NVD, Red Hat, etc.), leading to conflicting criticality assessments and misprioritization.
• Incorrect Fix Applicability: A scanner might report a fixed version, but that version might not be applicable or pullable into the specific image or environment, again leading to wasted effort.
• Diverging ID Space: A single vulnerability can be referenced by multiple IDs (CVE, GHSA, PySec, RHSA). Without proper alias resolution (like that provided by OSV), a single underlying vulnerability can appear as dozens of distinct findings, inflating reports and causing redundant work.

1. Impact of S-BOM Generators and Formats:

• Generator-Scanner Mismatch: An experiment using Sift and Trivy S-BOM generators with Grype, Trivy, and OSV scanners revealed that scanning a Sift-generated S-BOM with Grype resulted in a 66% loss of findings compared to scanning a Trivy-generated S-BOM with Grype. This was traced to the use of PURL qualifiers. Sift inserts an upstream qualifier (e.g., vim-minimal being part of vim) into the PURL, which Grype understands. Trivy, however, stores this information in a different S-BOM field, which Grype doesn't check, leading to missed vulnerabilities. This highlights that not all S-BOMs are the same even for the "same" components.
• S-BOM Format: While Trivy showed consistent results whether generating CycloneDX or SPDX S-BOMs, Grype demonstrated that using SPDX could lead to genuine findings being missed compared to CycloneDX. This challenges the assumption that S-BOM formats are interchangeable for downstream scanning.

These findings collectively demonstrate that the choice of scanner, S-BOM generator, and even S-BOM format, along with their internal logic and identifier handling, profoundly impacts the accuracy and completeness of vulnerability reports.

Technical Deep Dive

▶ Watch: Scanner disagreements: Missing packages and matching logic (7:00)

The technical underpinnings of vulnerability matching failures are complex, rooted in the design choices of various tools and standards.

At the core of the problem lies package identification. S-BOM generators and scanners parse manifest files (e.g., package.json, pom.xml) and lock files (e.g., package-lock.json, Gemfile.lock) to enumerate direct and transitive dependencies. However, their ability to correctly identify these packages can vary significantly. Some tools might only identify top-level dependencies, while others delve into transitive layers. Failures here, such as Trivy missing Python packages in Selldorff's experiment, mean that packages are simply not included in the S-BOM or internal package list, making subsequent vulnerability matching impossible.

Once packages are identified, they are resolved to identifiers to facilitate matching against vulnerability databases.

CPEs are structured as cpe:/<part>:<vendor>:<product>:<version>:<update>:<edition>:<language>. While widely adopted by NVD, their "free-form" nature for vendor, product, and version fields introduces ambiguity. Concepts like forks, distro repackaging, and backports are not well-modeled, leading to difficulties in accurately associating a CPE with a specific software variant or understanding complex version ranges.

PURLs (Package URLs) offer a more ecosystem-native approach, designed to identify packages across different language ecosystems (e.g., pkg:maven/org.apache.tomcat/[email protected]). PURLs are increasingly favored by modern advisories like OSV. However, their Achilles' heel is the lack of a standardized qualifier spec. Qualifiers are key-value pairs appended to a PURL that provide additional context (e.g., ?upstream=vim). Different tools interpret and utilize these qualifiers divergently. Sift, for instance, uses an upstream qualifier to link vim-minimal to vim, indicating a potential vulnerability inheritance. Trivy, however, might store this relationship in a different S-BOM field, causing Grype (which expects the upstream qualifier in the PURL) to miss the connection and thus the associated vulnerabilities. This non-standardized use of qualifiers creates a critical interoperability gap between S-BOM generators and scanners.

The process then moves to vulnerability databases, each with its own structure and data characteristics:

• NVD is the primary source for CPE-based vulnerabilities, providing comprehensive details and CVSS scores. Its strength lies in its manual review process, leading to higher data quality, but this often results in enrichment lag, where a CVE is announced but its full NVD entry (including CPEs and detailed descriptions) is delayed.
• cve.org can sometimes provide earlier hints about a CVE using just name and vendor, before a formal CPE is assigned in NVD.
• OSV and GHSA are crucial for open-source ecosystems, often using PURLs. OSV, in particular, excels at alias resolution, linking multiple IDs (CVE, GHSA, PySec, RHSA) to a single underlying vulnerability, which is vital for reducing noise.
• Distro advisories (e.g., Red Hat Security Advisories - RHSA) offer the most accurate and specific vulnerability information for particular operating system distributions and their package versions, but this specificity means they are not broadly applicable.

Finally, the matching logic within scanners determines how identified packages (with their CPEs/PURLs) are compared against vulnerability database entries. This logic can range from strict "exact matches only" to more "fuzzy matching" approaches that normalize for capitalization, hyphens, or spaces to increase the likelihood of a match. While fuzzy matching can capture more genuine vulnerabilities by accounting for data inconsistencies, it also increases the risk of false positives. The decision to suppress common false positives, like kernel-headers vulnerabilities for non-kernel installations, is another critical logic choice that differentiates scanners and impacts report fidelity.

The interaction of these technical elements – package identification, identifier resolution (especially PURL qualifiers), diverse vulnerability database characteristics, and varying matching logic – creates the complex and often contradictory landscape of vulnerability matching.

Demo / Proof of Concept

▶ Watch: Verification workflow for validating scanner results (14:40)

Alexandra Selldorff illustrated the core challenges of vulnerability matching through a series of practical experiments, serving as compelling proofs of concept for her arguments.

The foundational demonstration involved scanning a Red Hat 8 image using two popular open-source scanners, Grype and Trivy, straight "out of the box" without any custom configurations. This direct comparison yielded an 80.5% divergence in the number of reported vulnerabilities, with Grype identifying 852 CVEs and Trivy reporting 3,719. This stark difference immediately highlighted that scanner outputs cannot be blindly trusted. Selldorff then meticulously broke down this divergence by examining specific package examples:

• The kernel-headers package: Grype's default logic suppressed these, considering them development components unlikely to be exploitable, while Trivy surfaced them. This showcased differing approaches to noise reduction.
• The setuptools package: Trivy failed to identify this Python package entirely, leading to missed vulnerabilities. This exposed a fundamental weakness in Trivy's package identification for certain ecosystems.
• The Tomcat Java JAR: Grype failed to match any CVEs for Tomcat, while Trivy found 16. This was attributed to Grype's incorrect inference of the Maven Group ID from the directory path, leading to a flawed PURL qualifier, whereas Trivy correctly identified the JAR using its internal Java index.

A second set of experiments focused on the impact of S-BOM generation on scanner outputs. Selldorff generated S-BOMs from the same container using two different generators: Sift and Trivy. She then scanned these S-BOMs with Grype. The results showed that when Grype scanned a Sift-generated S-BOM, it lost 66% of findings compared to scanning a Trivy-generated S-BOM. This significant loss was attributed to how Sift and Trivy handle PURL qualifiers. Sift embeds an upstream qualifier (e.g., vim-minimal being linked to vim) directly into the PURL, which Grype is designed to interpret. Trivy, however, stores this crucial metadata in a different field within the S-BOM, which Grype does not consult, leading to missed vulnerability associations. This experiment concretely demonstrated that mixing and matching S-BOM generators and scanners without understanding their specific metadata handling can severely impact vulnerability coverage.

Finally, Selldorff also investigated the influence of S-BOM formats, comparing CycloneDX and SPDX outputs. While Trivy's scanner produced consistent results regardless of the S-BOM format, Grype showed that generating an S-BOM in SPDX instead of CycloneDX could lead to missed findings. This challenged the common assumption of format interoperability and highlighted that even the chosen S-BOM schema can introduce subtle but critical differences in vulnerability detection. These practical demonstrations effectively illustrated the complex interplay of package identification, identifier resolution, and scanner logic that makes vulnerability matching far more challenging than it appears.

Defensive Implications

▶ Watch: Scaling vulnerability matching: Best practices and strategies (17:00)

The insights from Selldorff's talk provide critical guidance for security defenders aiming to improve their vulnerability management practices. The overarching message is to move beyond passive acceptance of scanner outputs and adopt a more proactive, analytical approach.

Firstly, do not blindly trust scanner outputs. The demonstrated 80.5% divergence between Grype and Trivy on the same image is a stark reminder that every tool makes different logic choices. Defenders must understand why a scanner matched a particular vulnerability, delving into the package identification, identifier resolution (CPEs, PURLs, especially qualifiers), and specific matching logic employed by their chosen tools.

Secondly, map your S-BOM generator to your downstream scanner. The 66% loss of findings when Grype scanned a Sift S-BOM due to qualifier handling underscores the importance of this pairing. If a third party provides an S-BOM, inquire about the tool used to generate it and ideally scan it with the corresponding scanner. Furthermore, keep S-BOM generators and scanners synchronized in terms of versioning. Changes in metadata handling, particularly for PURL qualifiers, can occur between versions, leading to lost findings if tools are not upgraded in tandem.

Thirdly, establish a robust verification workflow. When a scanner flags a CVE:

1. Understand the matching logic: Why did the scanner connect this package to this vulnerability? (e.g., vim-minimal to vim via an upstream qualifier).
2. Validate against the original data source: Go directly to NVD, OSV, or distro advisories to confirm the vulnerability details.
3. Verify package installation and use: Ensure the package and its reported version are genuinely installed and actively used on the system, as S-BOM generators can make mistakes.
4. Cross-reference with multiple data sources: If NVD, OSV, and Red Hat disagree on applicability or score, analyze the discrepancies to make an informed risk decision.

Fourthly, document and share false positives and suppressions. Manual verification is time-consuming. If a category of findings (e.g., kernel-headers vulnerabilities for non-kernel installations) is consistently deemed a false positive, codify a suppression rule. Sharing these rules across repositories and teams prevents redundant manual work and allows security teams to focus on genuine threats. This also applies to complex versioning issues (e.g., non-SemVer) that require manual analysis; once understood, codify the logic.

Finally, defenders should be strategic about leveraging vulnerability databases and scanners:

• OSV is critical for alias resolution. Normalize diverse vulnerability IDs (CVE, GHSA, PySec) to a single CVE ID to reduce noise and avoid fixing the same vulnerability multiple times.
• Prioritize CVSS scores based on trusted sources. Decide which CNA (NVD, Red Hat, etc.) you trust most for scoring and integrate that into your prioritization framework.
• More scanners do not automatically mean better coverage. While useful for cross-validation, simply running all available scanners can introduce thousands of false positives, increasing workload without necessarily improving signal. Conversely, relying on just one scanner can lead to critical false negatives. Defenders should understand the strengths and weaknesses of each scanner for different package types and use them judiciously, potentially suppressing certain package types for specific scanners or combining outputs intelligently.

By adopting these defensive strategies, organizations can transform vulnerability matching from a source of frustration and risk into a more accurate, efficient, and reliable security control.

Key Takeaways

• Scanner outputs are not definitive: Tools like Grype and Trivy can have over 80% divergence on the same image due to differing package identification, matching logic, and data sources.
• Identifier nuances are critical: CPEs struggle with open-source complexities, while PURLs, despite being ecosystem-native, suffer from non-standardized qualifiers that lead to significant finding discrepancies between S-BOM generators and scanners.
• S-BOM generator/scanner pairing matters: Mixing S-BOM generators (e.g., Sift) with scanners (e.g., Grype) not designed to interpret their specific metadata (like PURL qualifiers) can lead to a substantial loss of identified vulnerabilities (e.g., 66%).
• Verification is essential: Implement a workflow to understand scanner matches, validate against original data sources, confirm package installation, and cross-reference information to minimize false positives and negatives.
• Optimize and automate: Utilize OSV for alias resolution to reduce redundant findings, prioritize CVSS scores from trusted sources, and codify manual verification work into suppression rules to scale efforts and reduce analyst fatigue.
• Strategic scanner use: More scanners don't always mean better security; understand each tool's strengths and weaknesses, and use them strategically rather than blindly combining them, to avoid overwhelming noise.

About the Speaker(s)

Alexandra Selldorff, a Senior Staff Software Engineer at Manifest Cyber, is a passionate advocate for improving vulnerability matching processes. Her expertise lies in dissecting complex security challenges within software supply chains, a topic she deeply enjoys discussing. Her work focuses on understanding the intricacies of vulnerability databases, package identifiers, and scanner logic to provide actionable insights for practitioners.