A Paradigm Shift in Vulnerability Identity: Why Vulnerability Databases Struggle

A Paradigm Shift in Vulnerability Identity: Why Vulnerability Databases Struggle

Speakers: Art Manion (Tharros Labs); Jay Jacobs (Lead Data Scientist, Empirical Security)

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=3_s61rBvIVo

Overview

In this thought-provoking talk at VulnCon, Art Manion and Jay Jacobs present a radical re-evaluation of how the cybersecurity industry approaches vulnerability identification and management. They contend that the existing frameworks, including prominent ones like CVE and NVD, are not merely imperfect but are fundamentally broken due to underlying philosophical challenges rather than solvable engineering problems. Drawing on decades of experience in the vulnerability space, including significant contributions to CVE and EPSS, Manion and Jacobs argue that attempts to fix these systems with "one more field" or "one more schema change" are doomed to fail because they do not address the root issues of naming, identity, and the nature of knowledge itself.

The speakers' central thesis is that the problems plaguing vulnerability databases stem from a lack of philosophical grounding in their design. They assert that the conventional approach, which treats vulnerability identification as a purely technical exercise, overlooks deeper questions about what a vulnerability is, how it should be named, and how its characteristics evolve over time. This talk serves as a critical examination of the status quo, aiming not to criticize past efforts but to pave the way for more robust and sustainable vulnerability management practices by first understanding the philosophical landscape.

Ultimately, Manion and Jacobs propose a set of nine guiding principles for vulnerability identity, born from their philosophical inquiry. These principles are intended to serve as a bedrock for designing future systems, ensuring that any engineering solutions are built upon a sound conceptual foundation. The implications of their findings are profound, suggesting that a paradigm shift is necessary for the industry to move beyond its current "hero mode" of manual analysis and towards truly machine-usable and effective vulnerability intelligence.

Background

▶ Watch: The fundamental brokenness of vulnerability data (0:00)

The journey to this talk began with a seemingly straightforward engineering challenge. In 2024, at the Vulnerability Forecasting TC (now known as VulnOp ticon), Art Manion and Jay Jacobs, driven by their extensive backgrounds in CVE, vulnerability coordination, and data science (Jay Jacobs leads the EPSS SIG), sought to define a Minimal Viable Vulnerability Record (MVVE). Their goal was to identify the absolute minimum number of fields and data definitions required to uniquely identify and deduplicate vulnerability records across different databases. The vision was a world where a small, standardized set of fields would enable seamless interoperability and clarity, leading to "sunshine and roses" in vulnerability management.

After a year of dedicated weekly meetings, the duo presented their progress at VulnCon, confident they were close to a solution with perhaps five to seven core fields. However, feedback and further introspection led them to a critical realization: their engineering approach was fundamentally flawed. They discovered that they "cannot engineer our way out of a philosophical problem." The core issue was not a lack of technical sophistication but a deeper conceptual misunderstanding of vulnerability identity. There is, in fact, no universal, minimum number of fields that every vulnerability record can share. This insight marked a significant turning point, shifting their focus from engineering a solution to exploring the philosophical underpinnings of the problem. Their work then delved into the philosophy of language and identity, seeking to understand why existing systems like CVE, CPE, and CVSS struggle so profoundly with what appear to be basic challenges of naming and description.

Key Findings

▶ Watch: Mill: A name should be unmeaning, just a tag (5:40)

The central discovery presented by Manion and Jacobs is that the persistent struggles of vulnerability databases are not engineering failures but rather manifestations of unaddressed philosophical problems concerning identity, naming, and the nature of knowledge. Their year-long pursuit of a Minimal Viable Vulnerability Record (MVVE) ultimately revealed that a fixed, universal set of descriptive fields for vulnerabilities simply does not exist. This led them to a deeper inquiry, drawing on the works of six prominent philosophers to articulate the core issues:

1. John Stuart Mill (1843) on Unmeaning Names: A name should be a blank peg, a serial number without inherent description. Current systems often embed descriptions into names (e.g., CPE), which, according to Mill, inevitably "ages poorly."
2. Gottlob Frege (1892) on Sense and Reference: Distinction between the objective "reference" (the actual moon) and the "sense" or mode of presentation (viewing the moon through a telescope). CVSS metrics, they argue, often try to encode private mental images rather than objective, observable facts, making the instrument flawed.
3. Bertrand Russell (early 20th century) on Tags vs. Descriptions: A name is a tag; it cannot be true or false. Many "names" in vulnerability systems (like CPE) are descriptions in disguise, assuming a singular identifying name can be recovered from software, which is often not possible.
4. Ludwig Wittgenstein (1953) on Language Games: The meaning of words is defined by their use within specific "language games." Attempts to enforce shared glossary definitions for terms like "vulnerability" or "severity" consistently fail because meaning is fluid and context-dependent.
5. John Searle (1950s) on Cluster Descriptions: A name is a peg on which to hang a "cluster of descriptions." Vulnerability records are not static forms but dynamic collections of assertions that may vary in truthfulness but collectively convey meaning. This model acknowledges that different descriptions may be needed for different contexts (e.g., open-source library vs. IoT firmware).
6. Saul Kripke (1970s) on Naming and Necessity: Naming requires a namer, an act, and a thing named (a "baptism"). In the context of vulnerabilities, while CNAs act as namers and assignment is the act, the exact nature of the "thing" being named (is a vulnerability a naturally occurring phenomenon or a defined construct?) remains an open question, hindering true identity.

From these philosophical insights, Manion and Jacobs derive nine principles for vulnerability identity, which are not a framework or solution but rather foundational guidelines. They emphasize that current systems, despite the best intentions and efforts of many contributors (including themselves), are "fundamentally not working" because they have attempted to engineer solutions without first resolving these deeper conceptual challenges. The key finding is a call to action: to build sustainable vulnerability management, the industry must first establish a sound philosophical basis for what it is trying to identify, describe, and manage.

Technical Deep Dive

▶ Watch: Searle: Vulnerability names are pegs for descriptions (13:00)

The core of Manion and Jacobs' proposed paradigm shift lies in their nine principles for vulnerability identity. These principles, derived from their philosophical exploration, offer a blueprint for rethinking the architecture and design of vulnerability databases, moving beyond incremental fixes to address fundamental issues.

Principle 1: Terminology Disputes are Unresolvable through Definition Alone

Drawing on Wittgenstein's concept of "language games," the speakers argue that the meaning of terms like "vulnerability," "risk," or "severity" is often settled through common use within specific contexts, rather than by strict glossary definitions. Attempts to enforce universal definitions, such as those for CVSS or even the core concept of a vulnerability, frequently fail. A recent example is the Linux kernel community's rejection of CVSS scores on their records, highlighting a fundamental disagreement on terminology and applicability. The problem is that while definitions might be locally coherent, they "fail globally." The proposed path forward is not universal consensus but rather the development of working, governed, per-organization definitions that explicitly define the concept of a vulnerability within its operational context.

Principle 2: Global Uniqueness Requires a Central Authority (for Naming)

The talk highlights the pervasive problem of multiple identifiers for the same vulnerability across different databases (e.g., CVE, NVD, OSV, GitHub security advisories for Log4Shell). While each ID might be locally correct, the lack of a singular, globally unique identifier creates significant challenges for deduplication and interoperability. This principle underscores the need for an authoritative naming act, a "baptism" as Kripke would describe it. Critically, vulnerability identification is predicated on software identity. Without a globally unique way to name software, it is impossible to uniquely name a vulnerability affecting it. The speakers advocate for naming software at its point of creation, by the vendor or creator, rather than retrospectively at the point of vulnerability discovery.

Principle 3: Vulnerability Records are Temporal and Evolving

Current vulnerability databases often treat records as static, monolithic entities. However, the reality is that knowledge about a vulnerability evolves significantly over time. Post-publication, a "pool of people with insight" expands, leading to new assertions, analyses, proofs of concept, and even corrections. The current CVE system, for instance, struggles with this, with examples like a Microsoft vulnerability's CVSS score being revised five times over 13 months with no recorded reason. The architecture of vulnerability management systems must treat change and updates as first-class properties, actively welcoming new information and providing mechanisms to resolve conflicts or disputes.

Principle 4: Software Identity is a Prerequisite for Vulnerability Identity

This principle reiterates and expands on the dependency of vulnerability identity on accurate software identification. A CWE (Common Weakness Enumeration) floating in the wild is not yet a vulnerability; it must affect specific software. Existing efforts like CPE (Common Platform Enumeration) struggle significantly, with the NVD containing over 3 million CPE strings, 58% of which carry no version information, making precise identification impossible. The ideal solution involves the software creator registering and naming their products at the point of creation, rather than relying on downstream vulnerability databases to guess or infer identity.

Principle 5: Record Quality is an Architectural Problem, Not a Data Problem

The quality of data in vulnerability records is not merely a matter of data entry but a fundamental architectural challenge, akin to designing good survey questions. Poorly structured questions lead to inconsistent and unusable data. The speakers propose four sub-principles for designing high-quality record fields:

• Atomicity: Each field should ask one question and expect one answer. CVSS confidentiality, for example, often bundles multiple questions into one.
• Observability: Fields should capture objective, observable facts, not subjective interpretations or "private mental images" (referencing Frege).
• Convergence: Given the same information, two analysts should ideally arrive at the same outcome when populating a field.
• Unsupported: Explicitly acknowledge and handle cases where an answer does not exist or is not applicable (e.g., an "unsupported" state rather than a blank field).

Principle 6: Terminology is Settled Through Common Use (and Explication Helps)

Revisiting Wittgenstein, this principle emphasizes that despite official glossaries, terms like "vendor," "supplier," "maintainer," or "developer" are used differently across various contexts. Manion suggests an approach based on Rudolf Carnap's concept of explication: rather than trying to redefine a commonly used term like "vendor," introduce a new, similar but distinct term (e.g., "supplier") and provide a precise, governed definition for it. This creates a clear distinction and avoids the cognitive ease of defaulting to established, but imprecise, meanings.

Principle 7: Vulnerability Records are Assertions with Provenance

Instead of monolithic records, vulnerability information should be viewed as a collection of individual assertions. Each assertion—e.g., a CVSS score, an affected version, an exploit status—should have clear provenance: who made it, when, why, and based on what evidence. The current system's lack of provenance means that when a field changes (like the repeatedly revised CVSS score), there's no way to understand the reasoning or the evolution of knowledge. An analogy given is treating each part of a vulnerability record like a GitHub issue, with a clear trail of contributions, discussions, and evidence over time.

Principle 8: Records Must Be Machine Usable, Not Just Machine Readable

The sheer volume of new vulnerabilities (a CVE record published every 10 minutes over the last 12 months) makes manual analysis unsustainable, forcing analysts into "hero mode." Current records are often "machine readable" (a machine can store and display the text), but not "machine usable" (a machine cannot interpret and act on the information without significant NLP or LLM processing). Drawing on Claude Shannon's levels of communication, the goal should be to move beyond mere transmission of symbols to achieving semantic (correct interpretation) and effective (producing correct action) communication for machines. This means moving away from verbose, unstructured descriptions towards structured, atomized data.

Principle 9: Knowledge is Distributed and Temporal

This principle synthesizes several previous points, asserting that knowledge about vulnerabilities is not static or centralized but rather distributed among many parties and evolves over time. The architecture of vulnerability systems must inherently support this dynamic nature. The original CVE paper from 1999 by Dave Mann and Steve Christie acknowledged this by describing CVE as a "logical bridge" and a starting point, implying a need for future evolution. Modern systems must be designed to ingest and manage this distributed, temporal knowledge, requiring features like assertions with provenance and flexible update mechanisms.

These nine principles represent a significant shift from an engineering-first mindset to a philosophy-first approach, urging the industry to build a more robust foundation for vulnerability identity.

Demo / Proof of Concept

▶ Watch: Principle 7: Records are assertions, not monolithic forms (38:00)

The speakers explicitly state that their talk does not present a new framework, standard, or engineering solution. Instead, it focuses on identifying the philosophical problems and proposing guiding principles. Therefore, no demo or proof of concept was provided during this presentation. Their work is foundational, aimed at informing future development rather than demonstrating an immediate, deployable solution.

Defensive Implications

▶ Watch: Q&A: Evolving vs. rebuilding identity solutions (53:05)

The insights from Manion and Jacobs' talk have significant implications for cybersecurity defenders, urging a critical re-evaluation of current practices and expectations:

1. Adjust Expectations for Vulnerability Data: Defenders should recognize that vulnerability data, particularly from large, centralized databases like NVD/CVE, will inherently be imperfect, inconsistent, and often incomplete. Expecting a singular, perfectly defined, and static record for every vulnerability is unrealistic given the philosophical challenges of identity and naming.
2. Prioritize Internal, Governed Definitions: While global consensus on terminology remains elusive, organizations should establish and rigorously adhere to their own working, governed definitions for key terms like "vulnerability," "severity," and "risk." This internal clarity, even if not universally shared, is crucial for consistent internal processes and effective communication within the organization.
3. Embrace the Temporal Nature of Vulnerability Knowledge: Vulnerability information is not static. Defenders must design their processes and tools to continuously ingest and integrate new information, updates, and assertions about vulnerabilities. This means moving beyond a "set it and forget it" approach and actively seeking out supplemental intelligence beyond initial CVE publication.
4. Invest in Robust Software Identification (Asset Inventory): Since vulnerability identity is predicated on software identity, a precise and up-to-date Software Bill of Materials (SBOM) and comprehensive asset inventory are non-negotiable. Defenders should push vendors to provide accurate and consistent software naming and versioning at the point of creation, rather than relying on ambiguous post-discovery identification.
5. Demand Machine-Usable Data: Defenders should advocate for and prioritize vulnerability intelligence feeds that are not just machine-readable (text descriptions) but truly machine-usable (structured, atomic, objective data). This will enable automated analysis, reduce reliance on manual "hero mode" efforts, and improve the efficiency and accuracy of vulnerability management programs.
6. Seek Provenance for Vulnerability Assertions: When consuming vulnerability data, defenders should question the source and timing of assertions. Systems that can provide provenance—who made a specific claim (e.g., CVSS score), when, and based on what evidence—will allow for more informed risk assessments and better understanding of data reliability.
7. Complement Centralized Data with Distributed Intelligence: Recognize that no single database holds all knowledge. Defenders should integrate information from diverse sources, including vendor advisories, open-source project security advisories (OSV, GHSA), security research blogs, and internal analysis, to build a more complete picture of their risk landscape.
8. Understand the Limitations of Scoring Systems: Be aware of the philosophical challenges inherent in systems like CVSS, which can struggle with objectivity and convergence. While useful, these scores should be interpreted with context and ideally supplemented with internal risk assessments that consider the specific environment and impact.

By adopting these defensive implications, organizations can move towards a more resilient and analytically sound approach to managing vulnerabilities, better equipped to handle the inherent complexities and ambiguities of the threat landscape.

Key Takeaways

• Vulnerability identification is a philosophical problem, not just an engineering one. Attempts to solve it with more fields or schema changes will fail without addressing fundamental issues of naming, identity, and the nature of knowledge.
• There is no universal "minimal viable vulnerability record." Vulnerability descriptions are best understood as a dynamic "cluster of assertions" that evolve and vary by context, rather than a fixed set of fields.
• Current systems like CVE, CPE, and CVSS struggle with fundamental naming and identity issues. They often embed descriptions into identifiers or attempt to quantify subjective human interpretations, leading to inconsistency and unresolvable disputes.
• Vulnerability records must be designed as temporal, evolving assertions with clear provenance. Systems need to actively support changes, updates, and the tracking of who made what assertion, when, and why, rather than treating records as static, monolithic entries.
• The industry desperately needs machine-usable data, not just machine-readable. The volume of vulnerabilities necessitates structured, atomic, and objective data that enables automated analysis and action, moving beyond human "hero mode."
• Accurate software identity is a prerequisite for effective vulnerability identity. This requires a central authority for naming software, ideally at the point of creation by the vendor, rather than being a retrospective, error-prone exercise by vulnerability databases.

About the Speaker(s)

Art Manion is a distinguished figure in the vulnerability space, currently with Tharros Labs. He has a long and active history with CVE, serving as a board member and participating in numerous working groups. His expertise spans vulnerability coordination and disclosure, having previously and currently supported CISA in the US and other organizations. Manion's extensive experience "staring at every form, every field, every database" has given him a deep understanding of the systemic issues in vulnerability data.

Jay Jacobs is the Lead Data Scientist at Empirical Security. He is widely recognized for his leadership in the EPSS SIG (Exploit Prediction Scoring System Special Interest Group), which he founded and continues to lead. Jacobs also actively participates in multiple CVE working groups and recently initiated the CVE Consumer Working Group, focusing on the needs and challenges faced by those who consume CVE data. His background in data science brings a critical analytical lens to the philosophical problems of vulnerability identity.