Vulnerability Haruspicy: Picking Out Risk Signals from Scoring System Entrails

Vulnerability Haruspicy: Picking Out Risk Signals from Scoring System Entrails

Speaker: Todd Carroll, RunZero (formerly CISA)

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=CW0Awo7pN5M

Reading time: ~8 minutes

Type: Briefing

TL;DR

RunZero researcher Todd Carroll walks through CVSS, EPSS, and SSVC — the three dominant vulnerability scoring and prioritization systems — exposing what each actually measures, where each breaks down, and how security teams can use them together without mistaking statistical noise for signal. His central thesis: vulnerability scoring systems are sophisticated tools that, if misread as oracles, become modern haruspicy — reading meaning into patterns that may be artifacts of how data is collected and reported rather than reflections of actual risk.

Introduction

Haruspicy is the ancient Assyrian and Etruscan practice of reading omens from animal livers. A trained haruspex would examine a sacrificial liver, consult a reference catalog of thousands of omens, and produce a forecast — is tomorrow's military campaign auspicious? Will the harvest be good? Modern critics call it p-hacking: finding patterns in data that appear statistically significant but are artifacts of the method, not reflections of the underlying reality.

Todd Carroll, who has worked at CISA and now at RunZero, applies this analogy to vulnerability management. CVSS scores drive patching priorities, regulatory mandates, and headlines. EPSS scores promise to predict which vulnerabilities will be exploited in the next 30 days. SSVC frames vulnerability response as a structured decision tree. All three are legitimate, useful tools — but each has failure modes that practitioners frequently miss, and understanding those failure modes is the difference between risk reduction and checkbox security.

CVSS: The Score Is Not the Vector

CVSS (Common Vulnerability Scoring System) is the dominant standard, present on 70 to 80 percent of all CVEs published in 2025, with CVSS 3.1 still the most widely deployed version despite version 4 releasing approximately 18 months ago.

▶ Watch: CVSS Vector Breakdown — What the Number Actually Means (04:00)

Carroll's core instruction to practitioners: throw out the number and read the vector string. The CVSS vector encodes the actual security characteristics; the number aggregates them into a figure that loses the nuance.

Walking through the vector components:

• Attack Vector (AV): Network (remotely exploitable), Adjacent (ARP/DHCP-range, not internet-routable), Local (requires existing shell access), or Physical (requires touching the device). The difference between N and L is the difference between a remotely wormable vulnerability and one that requires an existing compromise.
• Attack Complexity (AC): Low or High. A "High" complexity 9.8 is fundamentally different from a "Low" complexity 9.8 in terms of realistic exploitability.
• Privileges Required (PR): None, Low, or High. This determines whether an unauthenticated attacker or a privileged insider is the threat model.
• User Interaction (UI): None or Required. The difference between a server-side vulnerability and a phishing email attachment.
• Scope (S): Unchanged or Changed. Scope Changed (the "C") is why a vulnerability can score 10.0 rather than 9.8 — it means owning the vulnerable component also gains impact on other components. This is one of the most frequently misunderstood CVSS vectors.
• CIA Triad (C/I/A): Confidentiality, Integrity, Availability impacts. Carroll notes that Availability is often miscounted — RCE that gives full shell access technically gives the ability to run shutdown, so Availability may legitimately be scored as impacted even though the primary impact is code execution.

The Fractal Distribution: Why CVE Scores Cluster in the Sevens

Carroll presents a striking empirical observation: CVSS score distributions are fractal. Whether you look at a 76-day window, a one-year window, or a five-year window of CVE disclosures, the distribution has the same characteristic shape — sparse at low scores, a deep trough in the middle, a pronounced spike around 7.2-7.8, and a sharp falloff toward criticals.

▶ Watch: CVSS Score Distribution Analysis — The Fractal Curve (08:01)

This is not what one would expect if CVEs represented a complete, unbiased sample of all software vulnerabilities. A neutral distribution should show many low-severity vulnerabilities and few high-severity ones — the classic "lots of lame, few bad" curve. Instead, low-to-medium scores are almost entirely absent. Carroll's explanation: no one bothers to write a CVE, calculate a CVSS score, and publish an advisory for a CVSS 2. The overhead is not worth it. The reporting infrastructure itself is biased toward vulnerabilities worth reporting about — which maps to the "vulnerable enough to care about but not necessarily the worst possible thing" range.

The implication is uncomfortable: CVE is not an accurate census of vulnerabilities in the world. It is a record of vulnerabilities that someone decided to invest resources in disclosing. And at any timescale, the security industry appears to be doing the same thing over and over: reporting a lot of 7.8s, some 9s and 10s, a few near-zeros, and nothing in between.

CVSS Variants: PVSS and AI VSS

CVSS's influence has spawned derivative systems. Carroll briefly covers two:

Pipeline Vulnerability Scoring System (PVSS) by François Pouliot — structurally identical to CVSS but with attack vectors adapted for software supply chain contexts, based on repository visibility and pipeline access rather than network accessibility.

AI VSS — OWASP-endorsed, intended to express vulnerabilities in AI/agentic systems using CVSS-like vector notation, with additional fields reflecting AI-specific threat models. Carroll is skeptical of the announcement framing but acknowledges the space needs something.

EPSS: Exploitation Probability vs. Exploitation Evidence

EPSS (Exploit Prediction Scoring System) has been running since 2021. For every published CVE, it produces a percentage representing the probability that the vulnerability will be exploited somewhere in the world within the next 30 days.

▶ Watch: How EPSS Works and What It Actually Predicts (14:02)

The model ingests CVE metadata and CVSS vectors on one side and honeypot/IDS/EDR telemetry from commercial feeds (GreyNoise, Censys, Shodan, Shadowserver) on the other. Keywords in vulnerability descriptions influence scores. KEV (CISA Known Exploited Vulnerabilities) list membership influences scores. The mechanism is machine-learning-based and not fully transparent.

Carroll runs the math: summing EPSS scores across the current CVE database yields approximately 10,000 CVEs that are likely to be exploited somewhere in the next month. By contrast, the CISA KEV has just under 1,400 actively exploited vulnerabilities in its entire multi-year history, and VulnCheck KEV sits around 3,500. The 10,000 figure sounds alarming — but Carroll points out that EPSS is measuring whether a vulnerability is exploited anywhere on the internet, including by script kiddies with nuclei templates hammering every IP on the internet. When a nuclei template is published for a CVE, the EPSS score can jump 30-40 percentage points overnight.

As a case study, Carroll examines CVE-2025-2010, a SQL injection vulnerability in a WordPress job board plugin. Its EPSS score was near zero, ticked up slightly, then jumped to 38 percent on July 13th. The cause: someone published a nuclei template the day before. The template's publication — whether or not anyone actively exploited the vulnerability — drove the score surge.

▶ Watch: EPSS Case Study — CVE-2025-2010 and Nuclei Templates (18:02)

To help practitioners track these dynamics, RunZero released EPSS Pulse at the time of the talk — a daily, ungated, login-free dashboard showing EPSS score movements with sparklines for every CVE. Carroll describes it as a starting point for detective work: watching scores rise and fall each day to identify what may be getting weaponized.

LEV and SSVC: Filling the Gaps

LEV (Likely Exploited Vulnerabilities), built on top of EPSS by researchers Jono and Peter, asks whether the KEV lists are systematically missing exploited vulnerabilities. The answer appears to be yes. The methodology is mathematically sound but faces an uncomfortable epistemic limit: the researchers cannot measure the entire internet at all times, so the margin of error is unknown.

SSVC (Stakeholder-Specific Vulnerability Categorization) is the framework CISA uses internally to determine what to do after a vulnerability is identified. Carroll used it during his time at CISA.

▶ Watch: SSVC — The Decision Tree CISA Uses Internally (22:03)

SSVC is a decision tree, not a scoring system. It asks a sequence of questions: Is there exploitation evidence? What is the technical impact (total control vs. partial)? Can an attacker automate exploitation? What is the exposure of the target system? How important is the target system? Would human life be at risk if the system fails? The output is not a score but a response category: Track (put it in a drawer), Monitor (put it in a drawer but check periodically), Attend (wake up your boss), or Act (wake up everyone).

SSVC requires asset inventory knowledge most organizations lack — you cannot answer "does this affect my environment" without knowing what is in your environment. But CISA's ADP (Authorized Data Publisher) program now provides SSVC-aligned exploitation evidence and technical impact metadata directly in CVE records, reducing the manual work for CVE-identified vulnerabilities.

Notable Quotes

"Anytime you're using CVSS, I strongly, strongly, strongly encourage you to actually pay attention to the vector. Kind of throw out the number, because the vector is where the action is." — Todd Carroll ▶ 04:00

"The real vulnerability scoring system is the friends we made along the way, and that is real. You need humans, and you need friends — people in your trust group who have a track record of saying, 'Mm, that one looks bad,' before the worldwide event happens." — Todd Carroll ▶ 28:03

"About 10,000 unique CVEs are likely to be exploited in the next month somewhere on Earth. To me, that seems crazy high. That is bananas high." — Todd Carroll ▶ 16:02

"No scoring system can predict human intent. We can't Tom Cruise our way into predicting future crime and arresting the bad guys before they do the thing." — Todd Carroll ▶ 28:03

Key Takeaways

• Read the CVSS vector string, not just the score. A 9.8 with Attack Vector: Network, No Privileges Required, No User Interaction is a fundamentally different threat from a 9.5 requiring Local access and High Privileges — the vectors tell the story the number obscures.
• CVSS score distributions are fractal and biased. The consistent spike at 7.2-7.8 reflects what practitioners choose to disclose, not the actual distribution of software vulnerabilities — CVE is not a complete census.
• EPSS measures exploitation probability anywhere on Earth, not at your environment. A nuclei template publication alone can drive a CVE's EPSS score up 30-40 points; the score reflects exploitability interest, not necessarily targeted threat.
• SSVC is a decision framework, not a score — it answers "what do I do?" rather than "how bad is this?" It requires asset inventory knowledge but is the closest existing tool to context-aware risk prioritization.
• Human trust networks remain irreplaceable. No scoring system captures attacker intent in advance. Experts who consistently flag the right vulnerabilities early are more predictive than any algorithm currently available.

No slides PDF was listed for this talk.