Confusing Value with Enumeration: Studying the Use of CVEs in Academia

Confusing Value with Enumeration: Studying the Use of CVEs in Academia

Speakers: Moritz Schloegel

Conference: USENIX Security

YouTube: https://www.youtube.com/watch?v=orDOVQpSlrE

Overview

This article delves into a critical examination of how Common Vulnerabilities and Exposures (CVE) identifiers are perceived and utilized within the academic security research community. Presented by Moritz Schloegel at USENIX Security 2025, the talk, titled "Confusing Value with Enumeration: Studying the Use of CVEs in Academia," exposes a fundamental misunderstanding: that a CVE ID inherently signifies real-world impact, verification, or even the existence of a genuine security vulnerability. The research, a massive collaborative effort, stemmed from a growing concern among security researchers that the academic pursuit of CVEs might be inadvertently devaluing the very metric it seeks to leverage.

The core of the presentation highlights a concerning trend where academic papers frequently cite the acquisition of new CVEs as a primary indicator of their research's significance or a tool's effectiveness. Schloegel, acknowledging his own participation in this practice, argues that this approach misrepresents the true purpose of a CVE, which is merely a unique identifier for a publicly known cybersecurity vulnerability. The widespread misconception regarding the verification process behind CVE assignments, particularly those issued by "CNAs of last resort," creates a "false sense of credibility" that can lead to detrimental consequences for both the academic community and the open-source projects it aims to secure.

This work matters because it challenges the metrics by which security research is often judged, pushing for a more nuanced understanding of impact beyond mere enumeration. By dissecting the lifecycle of CVEs claimed by academic papers and surveying the community's perceptions, the research reveals systemic issues that strain relations with maintainers, burden overworked project teams with "bogus" reports, and ultimately undermine the credibility of academic security findings. It calls for a re-evaluation of how research contributions are measured and communicated, fostering a healthier ecosystem for vulnerability disclosure and collaborative security improvement.

Background

▶ Watch: Introduction: CVEs as a proxy for impact (0:00)

The concept of a Common Vulnerabilities and Exposures (CVE) ID was established to provide a standardized, unique identifier for publicly known cybersecurity vulnerabilities. This allows for unambiguous communication about specific security flaws across different organizations, tools, and databases. For instance, the infamous Heartbleed bug is officially known as CVE-2014-0160, and Meltdown is CVE-2017-5754. While these high-profile vulnerabilities often gain memorable names, the vast majority of security bugs benefit from a unique identifier to facilitate tracking, patching, and discussion among developers, clients, and users. Fundamentally, a CVE ID is an identifier, not an implicit validation or an indicator of severity or impact.

Despite this clear definition, a prevailing trend in academic security research has seen CVEs adopted as a proxy for demonstrating real-world impact. It's common to find statements in academic paper abstracts or introductions such as, "We identified 19 bugs and obtained 11 new CVEs," or "For 15 of the bugs, the Chrome team assigned a CVE acknowledging the impact of our results." This practice suggests that the acquisition of a CVE ID is considered a significant achievement, validating the attack technique, tool, or overall contribution presented in the paper. The speaker himself admitted to co-authoring papers that employed such language, highlighting the pervasive nature of this trend across the field.

The problem arises when the academic community begins to confuse the enumeration (the assignment of an ID) with genuine value or verified impact. If a CVE is merely an identifier, its existence alone does not confirm the severity of a bug, the quality of its discovery, or even the agreement of the affected maintainers that a vulnerability truly exists. This discrepancy forms the foundation of the research presented, aiming to quantitatively analyze the prevalence of CVE use in academic papers, qualitatively investigate the fate of the underlying bugs, and survey the security community's understanding and opinions regarding CVEs to chart a path forward.

Key Findings

▶ Watch: Quantitative analysis: Rising trend of CVE use (4:30)

The research embarked on a comprehensive analysis, starting with a quantitative study of CVE usage in academic security papers. The team examined papers from the top four security conferences – USENIX Security, IEEE S&P, ACM CCS, and NDSS – over an extended timeline. They found a consistent trend: between 15% and 20% of papers across these conferences utilize CVEs, a rate that has stabilized over the past decade. Interestingly, ACM CCS showed a slightly lower percentage, attributed to its broader scope which includes diverse fields like cryptography, where CVEs are less frequently relevant. The analysis distinguished between papers merely referencing CVEs for benchmarking (e.g., testing a tool against known vulnerabilities like Heartbleed) and those explicitly claiming CVEs for their novel contributions.

The more critical aspect of the study involved a qualitative analysis of CVEs that were specifically claimed by academic papers for their tools, techniques, or attacks. Focusing on papers published between 2020 and 2024, the researchers manually reviewed and identified 304 papers that claimed a total of 1,803 unique CVEs. For each of these CVEs, they meticulously analyzed the public record, including CVE descriptions and other publicly available resources, to determine the outcome of the underlying bug and the maintainers' response.

The findings painted a concerning picture:

• Acknowledged and Fixed/Confirmed: 1,189 CVEs (approximately 66%) were acknowledged by maintainers, with most receiving a fix and some merely confirmed. This represents cases where both the reporter and maintainer generally agreed on the vulnerability.
• Reserved State: 274 CVEs (approximately 15%) were in a "reserved" state, meaning a CVE Numbering Authority (CNA) had assigned an ID but no public disclosure or details were available. For all practical purposes, the status of these bugs—whether fixed or even real—remained unknown.
• No Response: 288 CVEs (approximately 16%) received no public response from maintainers. This category was further broken down:
• 101 cases involved inactive projects, where maintainers had abandoned the project, leaving no one to address the reported vulnerability.
• 161 cases suffered from insufficient public information, making it impossible to determine the bug's fate.
• Ignored by Maintainers: In 26 cases (around 1.4%), maintainers explicitly ignored the CVE report. They continued to push new code and fix other bugs, but specifically bypassed the reported vulnerability, raising questions about its legitimacy or priority.
• Maintainers Disputed: Most alarmingly, for 52 CVEs (nearly 3%), maintainers publicly stated that the report was "not a bug" or "not a security vulnerability." This directly contradicts the academic claim of having found a vulnerability worthy of a CVE.

Overall, the study found that in 66% of cases, there was agreement between the reporter and maintainer regarding the CVE assignment. However, in 21% of cases, agreement could not be judged due to lack of information, and in 13% of cases, there was clear disagreement or the CVE was ignored. The speaker explicitly noted that this analysis was "maintainer-centric," acknowledging that maintainers could potentially be wrong or influenced by PR concerns, but emphasized the difficulty of making independent judgments on all codebases.

Technical Deep Dive

▶ Watch: Qualitative analysis: Studying 1803 claimed CVEs (6:00)

The mechanism behind CVE assignment is crucial to understanding the discrepancies observed in academic claims. CVE Numbering Authorities (CNAs) are organizations authorized to assign CVE IDs. These can be large corporations like Apple and Google, or significant open-source projects such as Python or the Linux kernel, which operate their own CNAs. When a vulnerability is reported to one of these entities, their internal security teams typically review the report, confirm the vulnerability, and then assign a CVE ID if appropriate. This process generally involves a level of internal verification and agreement between the reporter and the affected party.

However, a significant portion of the software ecosystem, particularly smaller projects hosted on platforms like GitHub or npm, is not covered by specific, dedicated CNAs. For these cases, there exist CNAs of last resort. The most prominent and widely used CNA of last resort is MITRE, the organization that manages the overall CVE program. When a vulnerability is found in a project without its own CNA, maintainers can contact MITRE to request a CVE ID to inform their users.

The critical distinction, and the root of many problems identified by this research, lies in the operational model of CNAs of last resort. Unlike dedicated CNAs that have the expertise and direct access to the codebase to verify reports, CNAs of last resort like MITRE do not verify whether a reported issue is a genuine bug or a legitimate security vulnerability. As the speaker explained, it would be an "impossible task" for them to act as judges over all codebases not covered by specific CNAs. They primarily function as an ID issuing service.

This lack of verification creates a significant loophole: anyone can contact a CNA of last resort, including the vulnerability reporter, and obtain a CVE ID without ever contacting or obtaining agreement from the project maintainers. The research uncovered numerous instances where academic reporters directly approached a CNA of last resort, secured a CVE, and only then did maintainers learn about the CVE assigned against their code, often retrospectively and against their will.

The data unequivocally supports this technical breakdown. The study found that most of the "questionable" CVEs—those with insufficient information, those ignored by maintainers, or those outright disputed as "not a bug"—were, in fact, assigned by CNAs of last resort. Furthermore, the CNAs of last resort assigned twice as many CVE IDs as regular CNAs in the dataset of academic claims. This disproportionate assignment by unverified channels directly correlates with the observed issues of disagreement and bogus reports.

The impact of this technical flaw extends beyond academia. The speaker cited a direct quote from a project maintainer of a major GitHub project: "But they seem like a known bad actor. lots of bogus CVEs and no response after that anymore. This is the problem with the whole security circus." This sentiment underscores how the ease of obtaining unverified CVEs from CNAs of last resort creates an unnecessary burden on already overworked maintainers, fosters distrust, and ultimately harms the collaborative spirit essential for improving security. The process, intended to help, instead generates additional, often fruitless, work for maintainers who must then spend time investigating and disputing claims that may not even be valid.

Demo / Proof of Concept

▶ Watch: Concerning outcomes: CVEs ignored or rejected as not bugs (8:55)

This talk did not feature a traditional technical demonstration or proof of concept in the conventional sense. Instead, the speaker integrated interactive elements during the live presentation, such as asking the audience questions like "Do you know what a CVE is?" or "Have you used CVEs in a paper?" These interactive moments served to engage the audience and immediately highlight the community's existing knowledge and perceptions, which were later corroborated by the formal survey results presented in the paper. The "proof" of the paper's claims came from the extensive quantitative and qualitative data analysis, rather than a live exploit or tool demonstration.

Defensive Implications

▶ Watch: Overall agreement on CVE validity (66%) (10:00)

The findings of this research carry significant implications for various stakeholders within the security ecosystem:

For Academic Researchers and Authors:

• Rethink Metrics of Impact: The most crucial implication is to move away from using the sheer number of CVEs as a primary metric for research impact or the quality of a security tool. Researchers should focus on the genuine, verifiable impact of their findings, which includes successful remediation, adoption of defensive techniques, or significant changes in security practices, rather than just an identifier count.
• Prioritize Responsible Disclosure: Emphasize direct, transparent communication with maintainers and vendors before seeking CVE assignment, especially from CNAs of last resort. Collaboration and agreement with the affected party should be paramount.
• Understand CVE Assignment Nuances: Researchers must educate themselves on the differences between CNAs and CNAs of last resort, and the inherent lack of verification in the latter. This understanding should inform how they report vulnerabilities and frame their research claims.
• Focus on True Vulnerabilities: Invest more effort in ensuring that reported issues are indeed security vulnerabilities and not just bugs, misconfigurations, or intended behavior, to avoid burdening maintainers with "not a bug" reports.

For Project Maintainers and Vendors:

• Be Aware of Unilateral CVEs: Maintainers, especially of smaller projects, should be aware that CVEs can be assigned against their projects without their direct involvement or prior notification, particularly from CNAs of last resort.
• Establish Clear Disclosure Policies: Having a well-documented security vulnerability disclosure policy can guide researchers on how to report issues, potentially leading to more constructive engagements and pre-emptive agreement on CVE assignments.
• Allocate Resources for Triage: Given the potential for "bogus" or unverified CVE reports, maintainers may need to allocate resources for triaging these reports, even if they appear to originate from legitimate CVE IDs.

For Conference Program Committees and Reviewers:

• Skepticism Towards CVE Counts: Program committees and individual reviewers should adopt a more critical stance towards papers that heavily rely on CVE counts as proof of impact. They should look for evidence of vendor collaboration, confirmed fixes, and actual security improvements.
• Educate Reviewers: There's a clear need to educate reviewers about the CVE assignment process, particularly the limitations of CNAs of last resort, to counteract the "false sense of credibility" identified in the survey.
• Evaluate Impact Holistically: Reviewers should evaluate the real-world impact of research holistically, considering factors beyond CVEs, such as the severity of found issues, the novelty of the technique, the practicality of the defense, and the level of engagement with affected parties.

For the CVE Program Itself:

• Consider Verification Mechanisms: While challenging, the CVE program might explore mechanisms to introduce a baseline level of verification for CVEs assigned by CNAs of last resort, or at least clearer labeling for unverified claims.
• Clarify Purpose of CVEs: Reiterate and emphasize that CVEs are identifiers, not certifications of validity or impact, to counter widespread misconceptions.

Ultimately, the defensive implications revolve around fostering a more mature, collaborative, and evidence-based approach to security research and vulnerability disclosure, moving away from a metric-driven system that can be easily gamed or misunderstood.

Key Takeaways

• CVEs are Identifiers, Not Validation: A CVE ID is a unique identifier for a vulnerability, not an inherent measure of its impact, severity, or independent verification.
• Academic Misuse as Impact Proxy: Academic security papers frequently misuse CVE acquisition as a proxy for demonstrating real-world impact, often leading to a "false sense of credibility."
• CNAs of Last Resort Lack Verification: CVE Numbering Authorities (CNAs) of last resort, such as MITRE, do not verify the legitimacy or security implications of reported vulnerabilities before assigning IDs, creating a significant loophole.
• Bogus CVEs Burden Maintainers: This process enables reporters to obtain CVEs without maintainer contact or agreement, leading to a significant number of "bogus" or disputed CVEs that burden project maintainers and erode trust within the security community.
• Widespread Misconceptions: A substantial portion of academics and reviewers mistakenly believe that all CVEs undergo mandatory verification, contributing to the problem.
• Rethink Impact Measurement: The security research community needs to critically re-evaluate its metrics for measuring real-world impact, moving beyond simple CVE counts towards more verifiable, collaborative, and outcome-focused indicators.

About the Speaker(s)

Moritz Schloegel is a researcher who led this extensive, collaborative study on the use of CVEs in academia. He candidly acknowledged his own prior involvement in the practice of claiming CVEs in academic papers, which lends a significant degree of self-awareness and integrity to his critique of the community's current practices. The work itself was described as a "massive collaboration," born from discussions among individuals from diverse fields at USENIX Security 2023, reflecting a collective concern about the direction of vulnerability reporting in research.