CVSS v4.0 By The Numbers

CVSS v4.0 By The Numbers

Speakers: Nick Leali, Co-chair of the CVSS SIG, Incident Manager at Cisco PERT

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=d5CdaSVqmwI

Overview

In this insightful talk, Nick Leali, a co-chair of the CVSS Special Interest Group (SIG) and an Incident Manager at Cisco PERT, delves into the numerical and qualitative shifts introduced by CVSS v4.0 compared to its predecessor, CVSS v3.1. Leali's presentation, "CVSS v4.0 By The Numbers," provides a data-driven analysis of how the new scoring standard impacts vulnerability assessments, risk management decisions, and operational processes for both vulnerability producers (like product security teams) and consumers (like incident response and vulnerability management teams). The talk highlights that CVSS v4.0 is not merely an incremental update but a fundamentally different framework with altered mathematical assumptions that lead to significant changes in assigned scores and, consequently, in the qualitative categorization of vulnerabilities.

The core of Leali's research addresses a critical question: how will the adoption of CVSS v4.0 change an organization's approach to vulnerability handling, advisory writing, and compliance? By examining various public datasets, including Cisco's internal scores, CVE Program data, and GitHub advisories, Leali quantifies these changes, revealing substantial shifts in average scores, the most common score adjustments, and, most importantly, the qualitative boundary shifts (e.g., a vulnerability moving from "High" to "Medium" or "Low" to "Critical"). This analysis underscores the necessity for organizations to proactively understand and prepare for these changes rather than treating v4.0 as a direct, drop-in replacement for v3.1.

The talk serves as a crucial guide for organizations contemplating or undergoing the transition to CVSS v4.0. Leali emphasizes that the numerical scores, despite ongoing debates about their absolute importance, remain influential in regulatory compliance, service level agreements (SLAs), and internal decision-making processes. By providing a practical tool for data comparison and sharing his findings, Leali empowers security professionals to make informed decisions about adopting v4.0, moving beyond mere base scores to leverage the full context of Base, Threat, and Environmental metrics for more accurate risk assessments.

Background

▶ Watch: Introduction to CVSS v4.0 differences (0:00)

The Common Vulnerability Scoring System (CVSS) has long been the industry standard for communicating the characteristics and impacts of software vulnerabilities. Developed and maintained by FIRST (Forum of Incident Response and Security Teams), CVSS provides a standardized method for rating the severity of vulnerabilities, enabling organizations to prioritize their remediation efforts. Prior to v4.0, CVSS v3.1 was the most widely adopted version, building upon the foundation laid by CVSS v2.0. However, each iteration of CVSS has aimed to address limitations and incorporate lessons learned from real-world application.

The problem CVSS aims to solve is the consistent and objective assessment of vulnerability severity. Without a common language, different organizations might rate the same vulnerability with vastly different levels of importance, leading to inconsistent remediation priorities and communication challenges. CVSS achieves this by using a set of metrics to generate a numerical score, which is then mapped to qualitative ratings like Low, Medium, High, and Critical. These ratings, in turn, often drive operational decisions, dictate compliance requirements (e.g., PCI DSS often references CVSS scores), and influence contractual obligations.

Despite its widespread use, CVSS has faced criticism, particularly regarding the over-reliance on the CVSS Base Score in isolation, often divorcing the score from the specific context of an organization's environment or the actual threat landscape. This led to a common mantra within the CVSS SIG: "CVSS does not equal risk." The base score, while foundational, lacks crucial context about whether a vulnerability is actively being exploited (Threat metrics) or how critical the affected system is within a specific infrastructure, including existing mitigations (Environmental metrics). This gap in contextualization often resulted in a "scoring inflation" where base scores tended to be high, leaving little room for adjustments based on real-world factors. CVSS v4.0 was developed to address these and other shortcomings, aiming to provide a more nuanced and flexible framework that better supports contextualized risk-based decision-making.

Key Findings

▶ Watch: Speaker's motivation and research question (4:00)

Nick Leali's analysis revealed significant and often unpredictable shifts in vulnerability scores and qualitative ratings when comparing CVSS v3.1 to CVSS v4.0 across various datasets. These findings underscore the critical need for organizations to conduct their own assessments rather than assuming a direct correlation between versions.

1. Cisco Internal Data (308 records):

• Average Change: Scores increased by an average of 0.36.
• Mode Change: The most common change was zero (111 records), meaning no change, but a substantial number of scores increased.
• Range: Increases went up to 2.6, while one score decreased.
• Qualitative Boundary Shift: 15% of scores changed qualitative boundaries. Notably, 42 increased from Low to Medium, and 5 from Medium to High. Intriguingly, some scores decreased, including a few moving from Critical to Medium, highlighting that the assessment can be "radically different" even if the vulnerability's nature remains unchanged.

1. CVE Program Data (3516 records as of March 14th):

• Average Change: A smaller average increase of 0.19.
• Mode Change: The most common change was a decrease of 1.0, which Leali noted as an interesting anomaly warranting further investigation.
• Qualitative Boundary Shift: A significant one-third (33%) of CVEs, totaling 1,111 records, changed qualitative boundaries. This implies that if an organization's vulnerability management program makes decisions based on these boundaries (e.g., for PCI DSS compliance), they would face 491 new medium advisories to track, while 359 CVEs would shift from High to Medium, potentially reducing perceived urgency.

1. GitHub Advisories - Reviewed Data (2720 records):

• Average Change: A larger average increase of 0.34.
• Mode Change: A substantial jump of 1.2.
• Qualitative Boundary Shift: 23% of reviewed advisories saw qualitative boundary shifts. Compared to CVE data, GitHub showed more vulnerabilities increasing from Medium to High, suggesting a net increase in perceived "Highs" and "Criticals" in v4.0 compared to v3.1 for this dataset. Leali also noted a wide range, with some increases of 8.7 and decreases, attributing some outliers to potential mis-scores or zero v3.1 scores.

1. GitHub Advisories - Unreviewed Data (3400 records):

• Average Change: A smaller average change, but a high number of boundary shifts.
• Qualitative Boundary Shift: A significant 33% of unreviewed advisories changed boundaries. This dataset also showed numerous "big shifts," including some from Critical to Medium, and others from Medium to Critical or Low to High.

Overall Summarization:

Leali concluded that while the overall average score change across all datasets might not appear dramatic, individual shifts can be substantial and disruptive. The ratio of increased, decreased, or unchanged qualitative boundaries varies significantly based on the type of vendor, products, and services involved. For instance, a vendor like Cisco might see a small number of new high/critical advisories, while organizations tracking CVE program data might see a large influx of new medium advisories. GitHub's reviewed data, on the other hand, indicated more high/critical vulnerabilities in v4.0. These findings collectively highlight that CVSS v4.0 is fundamentally different, and its impacts are highly dependent on the specific types of vulnerabilities and the context of the organization.

Technical Deep Dive

▶ Watch: Data sources and comparison methodology (5:00)

Leali's technical approach centered on a custom Python tool designed to systematically compare CVSS v3.1 and CVSS v4.0 scores from various data sources. The tool, which utilizes the NumPy library for numerical operations, allows for statistical analysis and visualization of score changes. This bespoke script was developed to answer the initial question of how moving to v4.0 would impact Cisco's internal product security incident response process, but it evolved into a more general-purpose utility for any organization seeking to understand these shifts.

The core functionality of the tool involves parsing vulnerability data, which can be provided as local files or JSON structures within a directory. For each vulnerability record, the tool expects to find both a CVSS v3.1 score and a CVSS v4.0 score, enabling a direct, paired comparison. The statistical comparisons performed by the tool include:

1. Raw Count of Records: Simply counts the number of vulnerability entries (typically CVEs) that contain both v3.1 and v4.0 scores, serving as the basis for the analysis.
2. Average Change: Calculates the mean numerical difference between the v4.0 score and the v3.1 score across the entire dataset. A positive average indicates a general increase in scores, while a negative average suggests a decrease.
3. Mode Change: Identifies the most frequently occurring numerical change between the two versions. This metric helps to pinpoint common shifts that might not be apparent from the average alone, as an average can be skewed by outliers.
4. Qualitative Boundary Shift: This is arguably the most impactful metric. The tool assesses how a vulnerability's qualitative rating (Low, Medium, High, Critical) changes from v3.1 to v4.0. Leali explicitly mentioned Cisco's common boundary thresholds, which are standard in the industry, but noted that users could configure their own. The tool generates visualizations (histograms and graphs) to illustrate these shifts, showing how many vulnerabilities moved up (e.g., Low to Medium, Medium to High), down (e.g., Critical to Medium), or remained the same.

The underlying mathematical differences between CVSS v3.1 and v4.0 are central to these observed shifts, even if the talk doesn't delve into the specific equations. Leali notes that v4.0's math and "some of the assumptions" are different, leading to changes in how decisions are made based on the bins (qualitative categories). A key design goal of v4.0 was to shift base scores up on average, creating more room for Threat and Environmental metrics to lower the score when context dictates. This means that a vulnerability with a high base score in v4.0, representing a worst-case scenario, might be appropriately reduced by specific threat intelligence (e.g., no active exploitation) or environmental factors (e.g., strong mitigations, low-value asset). This design philosophy is intended to encourage the use of the full vector string (Base, Threat, Environmental) rather than solely relying on the Base Score, thereby providing a more accurate and risk-informed assessment.

Leali also highlighted the ongoing efforts by the CVSS SIG to clarify documentation, provide more examples, and encourage the use of the full vector string. This includes efforts to enable per-platform assessments within vulnerability databases, allowing a single CVE to have different CVSS scores based on the specific operating system or product it affects (e.g., an OpenSSL vulnerability scoring differently on Red Hat Enterprise Linux versus Cisco IOS XE). This granular approach aims to bring CVSS scores closer to the reality of diverse deployment environments.

Demo / Proof of Concept

▶ Watch: Cisco's internal CVSS v3 vs v4 data analysis (7:00)

Nick Leali's presentation included a practical demonstration and discussion of his custom Python tool, which serves as a proof of concept for analyzing CVSS v3.1 to v4.0 score transitions. While the talk did not feature a live coding session, Leali presented several graphs and histograms generated by the tool using various datasets.

The tool, which Leali openly admits was developed by him as an "amateur" developer, is available on GitHub (with a QR code provided in his presentation for easy access). Its primary purpose is to allow organizations to "start the conversation about how the scores that you're generating will change into your own environment."

Key aspects demonstrated or discussed regarding the tool:

• Input Flexibility: The tool can ingest data from various sources, including local files or JSON records within a directory, making it adaptable to different organizational data storage methods.
• Output Visualization: It generates histograms and graphs that visually represent the distribution of score changes and qualitative boundary shifts. For instance, the Cisco data histogram clearly showed a rightward shift, indicating more score increases, with a prominent bar for zero changes. The qualitative boundary shift graphs visually listed the number of vulnerabilities moving between categories (e.g., "42 increased low to medium").
• Statistical Reporting: Beyond visualizations, the tool provides the numerical summaries discussed in the "Key Findings" section, such as average change, mode change, and raw counts of boundary shifts.
• Customization: While Leali used common industry qualitative boundaries for his analysis, he noted that the tool could be configured to use different thresholds, allowing organizations to align the analysis with their specific internal policies.

Leali encouraged audience members to "grab it" and use it for their own environments, despite his self-deprecating remarks about his coding skills. The emphasis was not on the tool's perfection, but on its utility as a starting point for organizations to perform their own data-driven analysis of CVSS v4.0 adoption impacts. This practical, open-source approach directly supports the talk's objective of helping security professionals prepare for the transition to the new standard.

Defensive Implications

▶ Watch: Visualizing qualitative boundary shifts with graphs (9:00)

The shifts observed between CVSS v3.1 and v4.0 carry significant implications for security defenders, requiring a re-evaluation of current vulnerability management, incident response, and risk assessment strategies. The core message for defenders is clear: do not rely solely on the CVSS base score.

1. Re-evaluate Qualitative Thresholds and SLAs: With up to a third of vulnerabilities potentially shifting qualitative boundaries (e.g., from Medium to High, or High to Medium), organizations must review and potentially adjust their internal SLAs, regulatory compliance frameworks (like PCI DSS), and remediation policies that are often triggered by specific CVSS severity levels. A vulnerability previously deemed "Medium" might now be "High," demanding quicker action, or vice-versa, potentially freeing up resources. The talk highlighted a scenario where 491 new medium advisories could suddenly appear in an environment if relying on CVE program data, requiring defenders to track down vulnerabilities that previously might not have met their internal thresholds.
2. Embrace the Full CVSS Vector String: CVSS v4.0 is designed to provide more context through its Base, Threat, and Environmental metrics. Defenders should move beyond just the base score and actively incorporate threat intelligence (e.g., is the vulnerability being actively exploited? EPSS scores can help here) and environmental factors (e.g., system criticality, existing mitigations, network segmentation) into their risk calculations. This allows for a more accurate, risk-based prioritization rather than a generic severity rating. Leali strongly advocated for this, noting that relying solely on base scores "leaves a lot to be desired."
3. Prepare for Divergent Vendor Scores: As different vendors adopt CVSS v4.0 at varying rates and apply the scoring based on their specific products and environments, defenders should anticipate receiving advisories with different CVSS scores for the same underlying vulnerability. The call for per-platform assessments underscores this, meaning an OpenSSL vulnerability might have a different CVSS v4.0 score depending on whether it's running on a Red Hat server or a Cisco device. Defenders need to be equipped to reconcile these differences and apply their own contextual understanding.
4. Leverage Multiple Scoring Systems: The survey results presented by Leali indicated that most organizations are already using CVSS in conjunction with other scoring and evaluation systems like SSVC (Stakeholder-Specific Vulnerability Categorization), EPSS (Exploit Prediction Scoring System), and internal proprietary systems. Defenders should continue and expand this multi-faceted approach, recognizing that CVSS provides a foundational severity, but other systems add crucial layers of exploitability likelihood and business impact.
5. Proactive Data Analysis: Organizations should utilize tools like Leali's (or develop their own) to analyze their historical vulnerability data against CVSS v4.0. This allows them to understand the specific impacts of the new standard on their unique environment, identify classes of vulnerabilities where scores tend to increase or decrease, and proactively adjust their processes and resource allocation.
6. Document Deviations: When internal qualitative ratings or risk assessments deviate from the raw CVSS score (e.g., a 9.8 CVSS score is internally deemed "Medium" due to strong mitigations), it is crucial to document the rationale. This transparency is vital for auditing, compliance, and consistent decision-making within the security team.
7. Engage with Producers: Defenders should encourage their vendors and software producers to provide richer CVSS v4.0 vector strings and contextual information, helping them to make better-informed decisions. The more context provided by the source, the less guesswork required by the consumer.

In essence, CVSS v4.0 demands a more mature and holistic approach to vulnerability management. It's an opportunity for defenders to move beyond a simplistic reliance on a single number and build more robust, context-aware risk assessment programs.

Key Takeaways

• CVSS v4.0 is fundamentally different from v3.1: It's not a mere incremental update; its altered math and assumptions lead to significant shifts in vulnerability scores and qualitative ratings.
• Qualitative Boundary Shifts are Significant: Up to one-third of vulnerabilities can change qualitative categories (Low, Medium, High, Critical), which directly impacts remediation priorities, compliance obligations, and incident response decisions.
• Scores Don't Change Uniformly: The impact of v4.0 varies greatly depending on the dataset (e.g., Cisco, CVE Program, GitHub) and the types of vulnerabilities, requiring organizations to conduct their own analysis.
• Avoid Sole Reliance on Base Scores: Defenders should move beyond just the CVSS Base Score and leverage the full Base, Threat, and Environmental vector string, along with other risk assessment tools, to make context-driven decisions.
• Proactive Planning is Essential: Organizations must understand the specific impacts of v4.0 on their own environment using tools like the one demonstrated, and be prepared to adjust their vulnerability management processes, SLAs, and compliance frameworks.
• Adoption is Growing, and Perceived as Worthwhile: Despite challenges, a significant majority of organizations that have adopted CVSS v4.0 report that the transition was worth the effort, especially producers of vulnerability scores.

About the Speaker(s)

Nick Leali is a prominent figure in the vulnerability scoring ecosystem, currently serving as one of the co-chairs of the CVSS Special Interest Group (SIG). His deep involvement with CVSS spans many years, recalling his experience with CVSS v2.0. Beyond his leadership role in the SIG, Leali is an Incident Manager on the Cisco Product Security Incident Response Team (PERT), bringing a practical, real-world perspective to vulnerability assessment and management. He has been a co-chair of the CVSS SIG since November 2023, demonstrating his commitment to the standard's evolution and widespread adoption. Leali's work, including the development of tools to analyze CVSS data, highlights his dedication to helping the community understand and effectively implement security standards.