CVE Decaf: Brewing Better and More Actionable Data Quality

CVE Decaf: Brewing Better and More Actionable Data Quality

Speakers: Jerry Gamblin, Security Research Engineer, Cisco; Jay Jacobs, Chief Security Evangelist, Empirical Security

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=B-HwXHcGIaA

Overview

In "CVE Decaf: Brewing Better and More Actionable Data Quality," Jerry Gamblin from Cisco and Jay Jacobs from Empirical Security address a critical and growing challenge facing the cybersecurity community: the insufficient quality and usability of Common Vulnerabilities and Exposures (CVE) data. As the volume of newly disclosed vulnerabilities continues to surge, the ability to process, prioritize, and act upon this information effectively hinges on the underlying data's quality. This presentation introduces a foundational framework for evaluating and improving CVE data, shifting the focus from mere quantity to actionable insights.

The speakers argue that despite the CVE program's long history, its data often falls short of meeting the needs of modern security operations, particularly in automated environments. Many CVE records lack essential details, are inconsistently formatted, or fail to capture critical context required for tasks such as identifying vulnerable assets, characterizing threats, planning remediation, and prioritizing responses. This talk outlines a structured approach to defining data quality for CVEs, drawing parallels from historical quality management principles and applying them to the unique demands of vulnerability intelligence.

The significance of this work cannot be overstated. In an era where security teams are overwhelmed by alerts and manual processes, high-quality, machine-readable CVE data is paramount for enabling efficient vulnerability management, threat intelligence, and automated defense. By dissecting the types of data quality gaps—ranging from missing information in existing fields to the absence of fields for crucial data points—Gamblin and Jacobs provide a roadmap for the CVE program and its stakeholders to evolve, ensuring that CVEs remain a vital and practical resource for global cybersecurity.

Background

▶ Watch: Introduction: The challenge of CVE data quality (0:00)

The concept of data quality is not new, yet its rigorous application to vulnerability management, particularly within the CVE program, has often been a luxury rather than a standard. The speakers trace the origins of data quality principles back to the 1920s, drawing inspiration from pioneers in product quality like W. Edwards Deming, Joseph M. Juran, and Walter A. Shewhart. These early thinkers emphasized understanding the causes of defects and defining "fitness for use"—a product's ability to meet consumer needs. Deming's philosophy, notably, shifted blame from individuals to processes, a crucial perspective when examining data quality within a large, distributed system like the CVE program.

As data became increasingly central with the advent of the internet in the 1990s, data quality emerged as its own distinct field. A landmark contribution came from Wang and Strong, who conducted a seminal survey of data consumers, culminating in a four-category, 15-dimension framework for assessing data quality, again rooted in the "fit for use" principle. This evolution continued into the 2000s with more solidified frameworks and taxonomies, including comprehensive syntheses by Batini et al. and influential textbooks that further refined data quality assessment methodologies. By the 2010s, standardization efforts gained momentum, with frameworks like the ISO/IEC 8000 series and the International Monetary Fund (IMF)'s Data Quality Assessment Framework (DQAF) providing structured approaches, particularly for economic statistics. The IMF's framework was noted as one of the first large-scale, high-visibility implementations of a data quality standard.

Jay Jacobs highlights that within the security domain, the luxury of discussing data quality has historically been rare, often due to the necessity of simply accepting whatever data is available. However, the CVE program has reached a point where such discussions are not only possible but imperative. Emphasizing a perspective championed by Bob Lord, co-chair of the CVE Consumer Working Group, the speakers advocate for treating the CVE record as a product. This product management mindset is crucial for developing CVEs that are truly "fit for use" by their diverse consumer base. The challenge lies in defining a common persona for CVE consumers, moving beyond individual CNA-specific needs to identify the essential data points required by the broader community for effective vulnerability management. The speakers acknowledge that many end-practitioners don't directly consume CVE JSON but rather rely on tools and scanners, which further underscores the need for machine-readable, high-quality data at the source.

Key Findings

▶ Watch: Four essential consumer tasks for CVEs (10:05)

The core of the talk revolves around a structured approach to assessing CVE data quality, built upon two main pillars: identifying critical consumer tasks and categorizing data quality gaps. The speakers propose four fundamental consumer tasks that CVE data must support to be truly "fit for use":

1. Identification: Determining if a vulnerability is present in an environment.
2. Characterization: Understanding the nature of the vulnerability and its potential impact.
3. Remediation: Knowing how to fix the vulnerability and how urgently to act.
4. Prioritization: Assessing the overall risk and urgency of addressing the vulnerability.

To address shortcomings in supporting these tasks, the talk introduces four categories of data quality gaps:

1. Population Gap: The necessary data field exists in the CVE record schema, but CNAs are not consistently filling it in. This is often the "easiest" to fix through motivation, training, or tools like CNA Scorecard (cnacorecard.org).
2. Design Gap: There is no existing field in the CVE record schema to capture a crucial piece of information needed for a consumer task. This requires schema changes or new standards.
3. Ecosystem Constraint: The schema might support the data, but external factors like tooling, backend systems, or product limitations prevent accurate capture or consumption of that data.
4. Organizational Knowledge: The data is highly specific to a local environment or internal processes and is thus unsuitable for inclusion in a global record like a CVE.

Through their analysis, the speakers highlight significant deficiencies in current CVE records:

• Low Adoption of Product Identifiers: Only approximately 28% of CVEs published in the last six months included machine-usable product information like CPE (Common Platform Enumeration) or PURL (Package URL). PURL adoption is particularly dismal, with fewer than 200 CVEs containing PURL information.
• Inadequate Versioning: A staggering 58% of CPE strings in the NVD (National Vulnerability Database) lack specific version information, making automated identification of vulnerable software nearly impossible. This is a design gap where versions, designed for developers, are overloaded for software identification.
• Missing Public Disclosure Information: Only about 20% of CVEs include data about public disclosure, often from a single CNA, hindering threat intelligence efforts.
• Limited Exploitation Details: There's a significant design gap in capturing information suitable for building detection rules or signatures, such as specific exploitation conditions.
• Ineffective Security Consequence Reporting: While CVSS (Common Vulnerability Scoring System) is present, the speakers found it to be ineffective at differentiating security consequences in practical use, reducing complex information to roughly three bits. This points to a design gap in how security consequences are represented.
• Poor Patch Information: Despite patches often existing, only about 7% of CVEs report patch information, indicating a major population gap.
• Negligible Workaround Usage: The workaround field is used in less than 1% of CVEs published in the last six months, primarily by a single CNA.

Finally, a critical finding is the introduction of Bob Lord's Automation Test: "If a data record requires human intervention or custom logic to interpret, it fails the automation test." The speakers conclude that, based on this rigorous standard, none of the current CVE fields fully pass the automation test, underscoring the profound challenge in achieving truly scalable and automated vulnerability management with existing CVE data.

Technical Deep Dive

▶ Watch: Lord's Automation Test: Machine-readability of CVEs (23:40)

The technical deep dive into CVE data quality begins by dissecting the four consumer tasks and then mapping identified gaps to these tasks and the four gap categories. This systematic approach reveals the precise points of failure in the current CVE ecosystem.

Consumer Tasks and Associated Data Quality Gaps

1. Identification: Is this vulnerability present in my environment?

• Core Challenge: Asset-to-vulnerability matching in an automated way. This requires precise, machine-readable identifiers for vulnerable software.
• Presence in the Record (Population Gap): The CVE record schema provides multiple places for product information (e.g., CPE, PURL). However, only about 28% of CVEs in the last six months contained either CPE or PURL. PURL, crucial for identifying open-source components, is severely underutilized (less than 200 CVEs). This lack of adoption means vulnerability scanners and asset management systems struggle to automatically link CVEs to installed software.
• Technology Estate (Ecosystem Constraint): Even if CVEs had perfect product information, organizations often lack accurate, up-to-date inventories of their software assets, making identification difficult regardless of CVE data quality. This highlights that improving CVEs is only one part of a larger ecosystem challenge.
• Version Information (Design Gap): The use of version strings, primarily designed for developers, is "overloaded" for software identification. This is a significant design flaw, as 58% of CPE strings in NVD lack a specific version. Without precise version ranges or affected versions, it's nearly impossible to determine applicability programmatically. Apple's approach of simply stating "everything before X is vulnerable" is cited as an alternative, but it still represents a challenge for granular identification.

1. Characterization: What exactly is this vulnerability, and should I worry about it?

• Core Challenge: Providing sufficient detail to understand the vulnerability's nature and potential impact, enabling the creation of detection rules.
• Public Disclosure (Population Gap): Knowing when a vulnerability was publicly disclosed is critical for threat intelligence. Currently, only around 20% of CVEs contain this information, often provided by a single CNA. This makes it difficult to track exploit trends or assess the immediacy of a threat.
• CWE (Common Weakness Enumeration) (Population Gap): The CWE field describes the nature of the fault or weakness. While present, inconsistent population hinders comprehensive analysis, prioritization, and categorization efforts.
• Component Information (Population Gap): Identifying the specific vulnerable component within a larger software package is often missing or not machine-usable, further complicating characterization.
• Exploitation Conditions (Design Gap): A major limitation is the absence of fields to capture detailed exploitation conditions necessary for building detection signatures (e.g., Snort rules). Jerry Gamblin argues that a "good CVE record" should allow someone to write a detection rule. The current schema does not adequately support this.
• Security Consequence (Design Gap): While CVSS (Common Vulnerability Scoring System) provides some indication of impact (Confidentiality, Integrity, Availability - CIA), the speakers argue that its practical application often oversimplifies the true security consequence, reducing it to a mere "three bits of information." A richer, more granular way to express security consequences is needed.

1. Remediation: How urgently should I act on this?

• Core Challenge: Providing clear, actionable information about how to fix a vulnerability.
• Patch Information (Population Gap): A significant point of frustration for consumers is that the CVE program does not require a patch to be part of a vulnerability disclosure. While many vulnerabilities do have fixes, only about 7% of CVEs report patch information. This is a population gap, as CNAs often possess this data but don't consistently include it. Microsoft's recent efforts to populate this field are highlighted as a positive example.
• Workarounds (Population Gap): Workarounds offer temporary mitigation when a patch isn't available. However, the workaround field is used in less than 1% of CVEs published in the last six months, primarily by a single CNA, demonstrating a severe underutilization of an important defensive data point.

1. Prioritization: How urgently should I act on this?

• Core Challenge: Differentiating between global and local prioritization needs.
• Global vs. Local Prioritization: Global prioritization requires broad intelligence (attacker activity, malware trends, online discussions), which often changes daily (e.g., EPSS - Exploit Prediction Scoring System). Jay Jacobs argues that EPSS scores, due to their dynamic nature, should not be embedded directly in static CVE records but should be linked or derived externally.
• Organizational Knowledge (Organizational Knowledge Gap): Local prioritization depends heavily on an organization's specific context, asset criticality, and internal configurations (e.g., firewall rules). This type of information is inherently local and cannot, and should not, be part of a global CVE record. The CVE record's role in prioritization is to provide accurate identification and characterization data that feeds local and global prioritization efforts. The CISA KEV (Known Exploited Vulnerabilities) list is cited as an example of a valuable local prioritization list that is often misapplied as a global one.

Lord's Automation Test

A critical concept introduced is Bob Lord's Automation Test: "If a data requires human intervention, it doesn't pass the automation test." This test serves as a stark metric for the machine-readability and scalability of CVE data. The speakers conclude that, by this standard, none of the current CVE fields fully pass, highlighting the pervasive need for custom logic or human interpretation across the board. This profoundly impacts the ability of organizations to automate vulnerability management at scale, forcing manual analysis even for high-volume tasks.

The framework presented by Gamblin and Jacobs provides a robust methodology for identifying, categorizing, and ultimately addressing these multifaceted data quality challenges, paving the way for more actionable and automated vulnerability intelligence.

Demo / Proof of Concept

▶ Watch: Q&A: Conflicts in CVE data quality ownership (25:15)

The speakers explicitly stated that this particular talk focused on the theoretical framework and mental exercise of defining CVE data quality, rather than presenting a live demonstration or proof of concept of their findings. As Jay Jacobs noted, "we had a whole outline of all the data tests we could run, all the things we could look at and dig in, and and there's none of that in here. It's we're we're going to be mainly talking about this theory this mental exercise that we went through on this."

They confirmed that a detailed paper accompanying the talk, available on their GitHub organization (CVE DQAF), provides a more in-depth analysis and would contain the specific data tests and findings that were not demonstrated live. Therefore, this presentation served as an introduction to their conceptual framework and the critical issues it addresses, with the practical data validation being documented in their written work.

Defensive Implications

▶ Watch: Q&A: Evolving CVE design for modern needs (29:10)

The insights presented in "CVE Decaf" carry significant implications for various stakeholders in the cybersecurity ecosystem, offering concrete directions for improving defensive postures and vulnerability management strategies.

For CNAs (CVE Numbering Authorities)

• Prioritize Data Population: CNAs must make a concerted effort to fill out existing fields in CVE records. This includes critical machine-readable identifiers like CPE and PURL, public disclosure dates, CWE classifications, and, most importantly, patch information. Jerry Gamblin's CNA Scorecard (cnacorecard.org) is a valuable resource for identifying and addressing these population gaps. Microsoft's recent initiative to consistently include patch information is an excellent example for other CNAs to follow.
• Understand Consumer Needs: CNAs should engage with consumer working groups to understand the diverse needs of CVE data users beyond their immediate customers. The goal should be to contribute to a globally useful record, not just one tailored to specific vendor requirements.
• Embrace Automation-Friendly Data: When submitting data, CNAs should strive for clarity, consistency, and machine-readability to help CVEs pass Lord's Automation Test. This means avoiding ambiguity and providing structured information wherever possible.

For Security Defenders and CVE Consumers

• Advocate for Better Data: Consumers should actively participate in discussions (e.g., the CVE Consumer Working Group) to articulate their requirements for data quality. Their feedback is crucial for driving schema improvements and encouraging CNAs to provide more comprehensive data.
• Don't Rely Solely on CVSS: The talk highlighted CVSS's limitations in truly differentiating security consequences. Defenders should use CVSS as one input but complement it with other prioritization signals (like EPSS, CISA KEV list for local context, and internal asset criticality) and richer characterization data when available.
• Invest in Asset Inventory: The "ecosystem constraint" of poor asset inventory directly impacts identification. Defenders must prioritize robust asset management and software bill of materials (SBOM) initiatives to leverage improved CVE data effectively.
• Expect Tooling Evolution: As CVE data quality improves, expect and demand better tooling that can consume and process this richer, more structured information automatically, reducing the need for manual analysis.
• Be Aware of Data Overwrites: The discussion regarding CISA vs. NVD enriching/overwriting CVE data highlights a governance challenge. Defenders should be aware of potential inconsistencies when consuming CVEs from different sources and advocate for clear policies that respect the originator's data.

For the CVE Program and Standards Bodies

• Redefine CVE's Purpose: The CVE program, designed 27 years ago, needs to evolve for the modern era of automation and high-volume vulnerability disclosures. This includes revisiting its core purpose and adapting its schema to meet current consumer demands.
• Address Design Gaps: Prioritize the creation of new fields or structures to capture critical information currently missing, such as detailed exploitation conditions for detection rule development and more nuanced representations of security consequences.
• Improve Governance and Consistency: Establish clear guidelines for data enrichment and modification by downstream entities (like NVD) to prevent conflicts and maintain data integrity. The goal should be to encourage CNAs to provide complete data upfront, which then becomes authoritative.
• Promote PURL Adoption: Actively promote the adoption and consistent use of PURL for open-source software, which is a significant blind spot in current CVE data.
• Support Automation: Every future development within the CVE program should be evaluated against Lord's Automation Test to ensure that new fields and data structures genuinely facilitate automated processing and reduce human intervention.

By collectively addressing these implications, the cybersecurity community can transform CVEs from a mere list of vulnerabilities into a powerful, actionable intelligence feed capable of supporting advanced, automated defensive operations.

Key Takeaways

• CVE Data Quality is a Critical Bottleneck: The current state of CVE data quality significantly hinders automated vulnerability management, effective prioritization, and accurate threat intelligence, exacerbated by the ever-increasing volume of new vulnerabilities.
• A Structured Framework is Essential: A framework built on four consumer tasks (Identification, Characterization, Remediation, Prioritization) and four types of data quality gaps (Population, Design, Ecosystem, Organizational Knowledge) provides a robust method for evaluating and improving CVE data.
• Significant Deficiencies Persist: Key data points are consistently missing (Population Gaps) or lack dedicated fields (Design Gaps). Examples include low adoption of CPE/PURL (~28%), missing version data in NVD CPEs (58%), inadequate exploitation details, and poor patch information (only ~7% reported).
• Automation is Severely Limited: According to Bob Lord's Automation Test, most current CVE data requires human intervention or custom logic, preventing scalable, automated processing and increasing manual overhead for security teams.
• Collaborative Effort is Required: Improving CVE data quality demands action from all stakeholders: CNAs must fill existing fields, the CVE Program must address design gaps, and consumers must advocate for their needs and invest in their own asset inventory capabilities.
• The CVE Program Must Evolve: The 27-year-old design of the CVE record needs a fundamental re-evaluation and modernization to meet the demands of contemporary security operations and the imperative for machine-readable, actionable vulnerability intelligence.

About the Speaker(s)

Jerry Gamblin is a Security Research Engineer at Cisco. He is also known for running CNA Scorecard (cnacorecard.org), a valuable resource that evaluates and displays data quality metrics for CVE Numbering Authorities (CNAs), helping to identify areas for improvement in CVE record completeness.

Jay Jacobs is the Chief Security Evangelist at Empirical Security. He serves as the co-chair of the CVE Consumer Working Group, working alongside Bob Lord, to advocate for the needs of CVE data consumers and drive initiatives aimed at enhancing the usability and quality of CVE records.