Vulnerability Haruspicy: Using Woo To Confirm Your Biases

Vulnerability Haruspicy: Using Woo To Confirm Your Biases

Speaker: Tod Beardsley — runZero

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=JWcrvUR0-So

Reading time: ~8 minutes

TL;DR

Tod Beardsley of runZero dissects the three dominant vulnerability scoring systems — CVSS, EPSS, and SSVC — with the same sceptical rigor he would apply to reading sheep livers. His central argument: each system produces a kind of structured confirmation bias that feels like science, practitioners misuse all three in ways their designers did not intend, and the most actionable information in CVSS is buried in the vectors rather than the headline score. A companion paper accompanies the talk.

Introduction

Haruspicy is the Etruscan and Assyrian practice of divining divine will from the entrails of sacrificed animals — primarily the liver. The haruspex examines the organ's surface, identifies patterns among thousands of documented omens, combines the positive and negative signals, and delivers a verdict. The question asked must be binary, and the answer comes with a confidence level somewhere between guesswork and pattern-matching on noise.

Tod Beardsley opens his NorthSec 2025 talk by spending genuine time on this history, and not as a throwaway metaphor. His point is precise: when practitioners look at a CVSS score of 9.8 and treat it as ground truth about the urgency of a vulnerability, they are doing something structurally similar to reading a liver — extracting a signal from a process designed to produce consistent outputs rather than accurate predictions of real-world risk.

Beardsley brings significant institutional context to this critique. He spent time as a US federal government employee applying SSVC in operational settings, and he now works at runZero, where asset and vulnerability visibility is a core product concern. The talk is grounded in a data analysis he performed for an accompanying paper, covering 76 days of published CVEs from November 5, 2024 to January 20, 2025 — a date range he notes is "a lot funnier if you're in America."

CVSS: The Illusions of Precision

CVSS — the Common Vulnerability Scoring System — dominates vulnerability management. Its version 3.1 remains the most widely used, despite version 4 having been available for well over a year (with under two percent of current CVEs carrying a V4 score from any source). Eight vectors describe different aspects of a vulnerability; combining them produces a score from 0.0 to 10.0 with one decimal point of fidelity, then mapped to four criticality bands that practitioners immediately collapse to two: low and high.

▶ Watch: CVSS score distribution and its fractal bell curve (8:00)

The distribution of CVSS scores across Beardsley's 76-day sample — and an independent April-to-April comparison — reveals a near-identical curve: almost nothing in the 0–1 range, a complete desert between 1 and 4, a climb beginning at 4, a peak around 7.5, and a sharp drop above 8. This curve is not a coincidence or an artifact of the specific time period. It reappears in every time-slice of CVSS history. CVSS math is, by design, optimised to produce 7.5s. The scoring rubric's internal structure makes it nearly impossible to land in the 2–3 range and very easy to cluster around the high end.

▶ Watch: What the CVSS curve tells us about software and reporting (10:00)

Two interpretations of this distribution emerge. First, low and medium severity vulnerabilities are systematically underreported — not because they do not exist, but because the effort of filing a CVE is not worth it for low-severity findings, and vendors have little incentive to report issues they do not intend to patch. This means environments almost certainly contain a population of low-severity vulnerabilities invisible to CVE-based tooling. Second, the score as a number is nearly useless for prioritisation because the distribution is so compressed at the high end. The practical range of meaningful CVSS scores is not 0–10; it is roughly 7–10, and even within that range, the difference between 7.5 and 9.8 is often less important than context the score does not capture.

The Vectors Are Where the Action Is

Beardsley's most immediately actionable CVSS claim: stop looking at the number. Look at the vectors.

▶ Watch: CVSS vectors as the useful signal (14:00)

A vulnerability with an attack vector of Physical and a privileges-required value of User Account is easy to deprioritise if a camera covers the server room. The analyst does not need to compute a risk score; the vector tells them what conditions must exist for exploitation. Conversely, a vulnerability with attack vector Network, complexity Low, privileges None, and user interaction None is worth attention regardless of the headline score because those conditions describe something exploitable from the internet by an automated scanner with no human involvement.

The specific example Beardsley walks through is a CVSS 9.3: all vectors maximally bad except scope unchanged (the exploit is contained to the vulnerable component) and availability none (the system stays running). He notes the definitional tension between Integrity: High and Availability: None — if an attacker has arbitrary write on a machine, turning it off is trivial, making the availability score feel like an artefact of how the scorer thought about the question rather than a reflection of actual risk.

EPSS: Prediction via Machine Learning

The Exploit Prediction Scoring System, operational since 2021, takes a fundamentally different approach. Where CVSS describes a vulnerability's intrinsic characteristics, EPSS predicts the probability that a given CVE will be exploited in the wild within the next 30 days.

▶ Watch: EPSS architecture — markers, signals, and the yin-yang model (14:00)

EPSS operates as a two-sided model. The right side ingests vulnerability markers: CVSS vectors, description keywords (buffer overflow, authentication bypass, remote code execution), vendor identifiers, and product classifications. The left side ingests exploitation signals: whether the vulnerability has a public proof of concept, whether it appears in known exploit kits, social media discussion volume, and other threat-intelligence feeds. A machine learning model combines these inputs and produces a probability score between 0 and 1.

Beardsley's assessment of EPSS is genuinely positive relative to CVSS — it attempts to answer a more useful question (will this be exploited?) rather than an intrinsic one (how bad is the bug?). Its limitations are that its 30-day prediction window does not map well to most patch cycles, and that its training data inherits the same underreporting bias as the CVSS corpus: if low-severity vulnerabilities rarely appear in exploit databases, EPSS will consistently assign them low exploitation probability, which may not reflect their actual use by patient, targeted threat actors.

SSVC: Structured Gut Feelings

The Stakeholder-Specific Vulnerability Categorisation framework, developed at Carnegie Mellon's CERT/CC and adopted by CISA for its own triage workflow, abandons numerical scores entirely. SSVC represents vulnerabilities as decision trees: a series of binary or multi-value questions whose answers produce an action recommendation.

▶ Watch: SSVC — decision trees and structured gut feeling (18:00)

Questions in an SSVC tree include: Is the vulnerability being actively exploited? Is exploitation automatable? What is the technical impact? Does the organisation have safety implications? Answers route through the tree to a terminal outcome: defer, scheduled, out-of-cycle, or immediate action. The framework makes no attempt to produce a number, which means it cannot be used for checkbox security (you cannot tell a regulator "our SSVC is 7.2") but it also avoids the false precision that plagues numerical systems.

Beardsley's critique of SSVC is gentler than his critique of CVSS — it is, at least, honest about its subjectivity. Its challenge is that it requires consistent, expert human judgment at each decision node. The same vulnerability assessed by two analysts who weigh "technical impact" differently will produce different recommendations. SSVC externalises the bias rather than hiding it inside a mathematical formula.

Putting the Systems Together

No single system adequately replaces the others. The practitioner's reality, Beardsley acknowledges, is that vulnerability management programs mix CVSS thresholds for compliance requirements, EPSS scores to prioritise remediation effort, KEV (Known Exploited Vulnerability) lists for immediate triage, and lived experience to fill the gaps.

▶ Watch: Practical synthesis — what defenders actually do (22:00)

His recommendation is not to abandon scoring systems but to use them honestly. CVSS is useful for checkbox security and for quickly discounting vulnerabilities with unfavourable attack conditions. EPSS is useful for identifying where active exploitation activity is clustering right now. SSVC is useful for forcing a team to articulate its actual priorities rather than inheriting them from a vendor-assigned number. The KEV list is the closest thing available to a ground-truth signal about what threat actors are using today. None of these is infallible; all of them are better than nothing; and the biggest mistake is confusing the output of any one of them with a rigorous risk assessment.

Notable Quotes

"Is that what we're doing in vulnerability scoring? Let's find out."

"CVSS is excellent for checkbox security — you can say, 'You must be this tall in order to qualify for my vulnerability management system,' and this tall is 7.5, for some reason."

"Nobody reports low things. Why? Because it's a little bit of a pain in the ass."

"We are writing high-severity software bugs at about the same rate we always have."

"Vectors is where all the action is at. I don't care about the number at all."

Key Takeaways

• CVSS scores cluster at 7.5 by mathematical design. The distribution is not informative about real-world risk distribution; it reflects how the scoring formula was constructed.
• Low and medium vulnerabilities are systematically invisible. The underreporting bias in CVE databases means defenders likely face more low-severity exposure than any scoring system reflects — and threat actors chain them.
• Read the vectors, not the number. Attack vector, complexity, and privileges required tell an analyst what conditions must exist for exploitation — which is far more actionable than a headline score.
• EPSS asks a better question than CVSS. Predicting exploitation probability is more useful for prioritisation than scoring intrinsic characteristics, though its training data inherits CVSS's underreporting biases.
• SSVC makes bias explicit rather than hiding it. Decision-tree frameworks require defenders to articulate their priorities, which is both their strength and their operational challenge.
• Combine sources; trust none absolutely. CVSS for compliance thresholds, EPSS for active exploitation signals, KEV for ground-truth urgency, and SSVC for structured decision-making each contribute something the others cannot.