Quantifying Swiss Cheese, the Bayesian Way

Quantifying Swiss Cheese, the Bayesian Way

Speakers: Stephen Shaffer, Engineer, Moderna

Conference: VulnCon

YouTube: https://www.youtube.com/watch?v=Wo77sqI_Sqs

Overview

Stephen Shaffer, an engineer at Moderna and co-chair of the EPSS Special Interest Group, presented a compelling and ambitious vision for a quantified vulnerability risk model in his VulnCon talk, "Quantifying Swiss Cheese, the Bayesian Way." Shaffer tackles a fundamental, yet often unanswerable, question in cybersecurity: "Are our efforts actually making us safer?" He argues that while current vulnerability management (VM) practices – whether traditional or risk-based – involve significant effort, there's a critical gap in empirically validating their effectiveness in preventing breaches and losses.

This talk introduces a novel framework that bridges the gap between raw vulnerability data and measurable business risk. Shaffer’s model, which he terms the Exploit Vector Incident Loss (EVIL) model, aims to move beyond qualitative risk assessments by integrating various data points, including assets, vulnerabilities, exploitation likelihood (using EPSS), control effectiveness, incident rates, and ultimately, monetary business impact. By leveraging Bayesian statistics, the model seeks to quantify the inherent uncertainty in security controls and provide an economically defensible approach to prioritize risk reduction actions.

Background

▶ Watch: Introducing the 'Quantified Vulnerability Risk Model' (1:13)

The operational reality of cybersecurity is characterized by an ever-increasing volume of known vulnerabilities and an unknown quantity of zero-days. While initiatives like "secure by design" are lauded, the immediate challenge for organizations is managing the existing, vast attack surface. Traditional vulnerability management often focuses on counting fixes or applying prioritization mechanisms like CVSS or EPSS thresholds. However, Shaffer highlights a critical flaw: these methods don't confirm if the actions taken genuinely reduce risk or prevent material loss.

Shaffer's work draws inspiration from the FAIR (Factor Analysis of Information Risk) taxonomy, aiming to build a compatible model from the ground up, starting with vulnerability exploitation and ascending to material incident likelihood and risk exposure. He emphasizes the need to define the "local risk model" by incorporating assets, vulnerabilities, exploitation likelihood, control effectiveness, incident frequency, and monetary impact. A key hurdle in empirically measuring control effectiveness is the "missing not at random" problem in statistics, where defenders can count prevention events (successes) but lack visibility into how many attacks (failures) bypass controls without detection. This makes it impossible to determine a true denominator for effectiveness. He also critiques the FAIR-CAM (Control Analytics Module) for its reliance on empirically unsourced and decaying control effectiveness measures, prompting him to develop his own approach.

Key Findings

▶ Watch: Using SME surveys and attack simulation for control effectiveness (13:00)

Shaffer's research yields several significant findings and contributions:

• Quantified Control Effectiveness: The talk introduces a novel, data-driven methodology using Bayesian updating to empirically measure and continuously refine the effectiveness of security controls in preventing exploitation. This moves beyond subjective qualitative assessments.
• The EVIL Model: The development of the Exploit Vector Incident Loss (EVIL) model provides a comprehensive, end-to-end framework for quantifying an organization's material exploit risk exposure. It integrates asset-level vulnerabilities, control effectiveness, and business impact to answer the question: "Which actions would reduce risk the most?"
• Economic Prioritization: The model shifts the focus from purely technical prioritization (e.g., high CVSS or EPSS scores) to economically defensible actions. By simulating the removal of vulnerabilities or improvements in controls, it quantifies the monetary value of risk reduction for each action.
• Patch Cadence Dominance: A surprising, yet empirically consistent, finding from the model's simulations is that consistent, broad patch cadence across an entire fleet of assets often delivers a significantly higher aggregate risk reduction than targeted remediation of even high-EPSS, one-off vulnerabilities on a smaller number of hosts. This reinforces the importance of foundational security hygiene.
• Managing Uncertainty: Shaffer acknowledges that all models are "wrong" but emphasizes the utility of a model that is "less wrong" than current approaches. The Bayesian methodology explicitly quantifies uncertainty, providing a more transparent and defensible basis for decision-making.

Technical Deep Dive

▶ Watch: The 'EVIL' model: linking exploit risk to business loss (27:50)

Shaffer’s model begins by building on his previous work of grouping EPSS (Exploit Prediction Scoring System) scores at the asset layer. This involves calculating the probability that at least one vulnerability on a given asset will be exploited in the wild within the next 30 days. The core innovation lies in incorporating control effectiveness into this calculation using a Bayesian approach.

The quantification of control effectiveness starts with a beta distribution, which models the probability of success given a number of observed successes and failures. Initially, with no data, this distribution is a flat line, representing complete uncertainty. Shaffer proposes two primary methods for updating this distribution:

1. Subject Matter Expert (SME) Surveys: Experts are asked three calibrated questions to quantify their opinion on a control's effectiveness. The first question gauges their self-reported expertise (on a Likert scale of 1-5), which serves as a weight. The subsequent two questions ask for their P50 (median) and P90 (90th percentile best-case scenario) estimates for how many out of 10 observed vulnerabilities a control would prevent exploitation. These two numbers are then used to reverse-engineer an initial beta distribution for the control, representing the aggregated, weighted expert opinion.
2. Breach and Attack Simulation (BAS) / Red Teaming / Penetration Testing: To overcome the "missing not at random" problem (where failures are often unobserved), controlled experiments are crucial. By launching a known number of attacks against infrastructure and observing successes and failures through telemetry, defenders can gain empirical data. This data (successes and failures) is then used to perform Bayesian updating on the beta distribution. The formula for updating is straightforward: alpha_posterior = alpha_prior + successes and beta_posterior = beta_prior + failures. This iterative process continuously refines the understanding of a control's effectiveness.

From the updated beta distribution, a single point estimate for control failure rate is needed. Shaffer recommends using the P5 (5th percentile) value, which provides an "extremely conservative" yet "defensible" estimate of the control's failure rate.

With control effectiveness quantified, Shaffer then details the asset-level risk calculation. For each asset, he identifies individual software components and their associated vulnerabilities. To avoid over-indexing, he takes the maximum EPSS score for vulnerabilities within each component. Each component is then mapped to its corresponding exploit vector (e.g., SMB adjacent, IIS), and the specific controls applicable to that vector are identified. By applying the previously calculated control failure P5 rate for each relevant control, a residual exploitation likelihood is determined for each component on the asset. Finally, by taking the complement product of these residual likelihoods across all components, the model calculates the overall 30-day asset-level exploit likelihood, adjusted for control effectiveness.

This asset-level exploit likelihood feeds into the broader Exploit Vector Incident Loss (EVIL) model. The 30-day likelihood is converted into a hazard rate, which allows for scaling the risk across different time horizons (e.g., quarterly, annually). These individual asset hazard rates are then aggregated (summed) to derive an organizational hazard rate. This organizational rate is translated into threat and loss event frequency (terms from FAIR), ultimately predicting the likelihood of a material incident that could impact the business. The model can then integrate loss data (e.g., calibrated using Scientia Iris data or internal financial metrics) to quantify potential monetary impact. The final output is a loss exceedance curve, which visually represents the probability of exceeding various loss thresholds, allowing organizations to compare potential losses against their risk tolerance and insurance coverage.

Demo / Proof of Concept

▶ Watch: Simulating remediation actions to quantify risk reduction (35:00)

While the talk did not feature a live, interactive demonstration of the fully implemented EVIL model, Shaffer provided a conceptual "simulation of removal" to illustrate its practical application and value. He presented a hypothetical scenario with three potential actions to reduce risk:

1. Remediating 50 hosts with a vulnerability having an EPSS of 0.8.
2. Improving EDR control effectiveness (e.g., from 20% to 40% prevention rate, requiring re-configuration and re-testing).
3. Implementing a broad patch cadence across the entire fleet (thousands of assets).

By running these actions through his model, Shaffer quantified their respective economic impacts. The simulation suggested that remediating the 50 high-EPSS hosts would reduce risk exposure by approximately $19.6K. Improving EDR control effectiveness provided a similar, albeit slightly higher, reduction. However, the most significant impact by far came from consistent patch cadence across thousands of assets, which resulted in an estimated $196K reduction in risk exposure.

This simulated outcome dramatically highlights the model's ability to provide an economically defensible prioritization. It challenged the intuitive assumption that focusing solely on "hottest" vulnerabilities (high EPSS) would always yield the greatest return on investment, demonstrating that widespread, moderate-per-asset risk, when multiplied across a large environment, can dominate the aggregate hazard.

Defensive Implications

▶ Watch: All models are wrong, but some are useful (43:00)

Shaffer's model offers profound implications for cybersecurity defenders, shifting the paradigm from reactive vulnerability management to proactive, economically driven exploit risk management:

• Quantifiable Justification for Investment: Defenders can move beyond qualitative "high risk" labels to present concrete, data-backed assessments of risk exposure and the monetary value of proposed security initiatives. This facilitates better communication with business leaders and justifies resource allocation.
• Optimized Resource Allocation: The model enables organizations to prioritize security actions based on their actual impact on reducing material loss. Instead of chasing every high-CVSS or high-EPSS vulnerability, teams can focus on interventions that yield the greatest economic benefit, even if those are broad, foundational efforts like patch cadence.
• Empirical Control Validation: By integrating Bayesian updating with breach and attack simulation, organizations can continuously measure and improve the effectiveness of their security controls. This moves control assurance from assumption-based to evidence-based.
• Shift from Vulnerability Count to Exploit Risk: The model emphasizes reducing the likelihood and impact of exploitation, rather than merely reducing the count of vulnerabilities. This is crucial in an environment where the sheer volume of vulnerabilities may become unmanageable, especially with the potential impact of AI on vulnerability generation.
• Informed Risk Tolerance: The output, particularly the loss exceedance curve, allows organizations to set and understand their risk tolerance in monetary terms, and compare potential losses against insurance coverage. This provides a clear benchmark for deciding which risks to mitigate and which to accept.
• Strategic Decision-Making: The EVIL model provides a framework for evaluating the economic viability of different security strategies, such as investing in automation for patch management versus highly targeted remediation campaigns. It encourages a focus on attack surface management and patch cadence as core, high-impact interventions.

Key Takeaways

• Empirical Measurement is Crucial: Cybersecurity efforts must move beyond qualitative assessments to empirically measure whether actions genuinely reduce exploit risk and prevent business loss.
• Bayesian Approach for Uncertainty: Bayesian updating, particularly with beta distributions and SME input combined with attack simulation, provides a robust method to quantify and continuously refine the effectiveness of security controls, accounting for inherent uncertainty.
• The EVIL Model for End-to-End Risk: The Exploit Vector Incident Loss (EVIL) model offers a comprehensive framework to link vulnerabilities, control effectiveness, and asset value to material incident likelihood and economic loss exposure.
• Patch Cadence Dominates Aggregate Risk: Simulations demonstrate that consistent, broad patch cadence across an entire environment can yield significantly higher aggregate risk reduction compared to targeted remediation of individual high-EPSS vulnerabilities.
• Focus on Exploit Risk, Not Just Counts: Effective vulnerability management should prioritize reducing the likelihood and impact of successful exploitation, rather than simply reducing the number of vulnerabilities.
• Models are Useful, Even if Imperfect: While no model is perfectly "right," a well-constructed, data-informed model like EVIL can be immensely useful for making economically defensible security decisions and measurably improving an organization's security posture.

About the Speaker(s)

Stephen Shaffer is an engineer at Moderna, bringing a strong background in data science and evidence-based methodologies to solve complex security problems. He is also the co-chair of the EPSS (Exploit Prediction Scoring System) Special Interest Group (SIG), demonstrating his commitment to advancing quantifiable approaches in vulnerability management. Shaffer's work is driven by a fundamental question: how can organizations truly validate that their cybersecurity investments are making them safer and preventing actual losses? His engineering mindset, coupled with an analytical approach, focuses on building defensible systems and models to quantify risk in an environment characterized by pervasive vulnerabilities and a lack of economic incentives for inherently safe technology.