Cybersecurity is Ready for Local Models
Michael Roytman (Co-founder and CTO · Empirical Security)
Overview
In his CypherCon talk, Michael Roytman, Co-founder and CTO of Empirical Security, made a compelling case for the widespread adoption of local models in cybersecurity. Roytman argued that while global models like CVSS and EPSS provide a baseline for vulnerability assessment, they inherently lack the specificity required to effectively manage risk within unique enterprise environments. His vision centers on empowering security engineers to build and deploy bespoke models that leverage an organization's own data, leading to significantly more efficient and accurate vulnerability prioritization.

Key moments
- 3:50 The 15% average remediation problem
- 6:40 Vendor models fail without local context
- 11:00 Each enterprise needs its own unique risk model
- 21:00 CVSS is an inefficient risk prioritization tool
- 39:30 Local models dramatically increase remediation efficiency
- 44:00 LLMs are not for security operations forecasting
Cybersecurity is Ready for Local Models
Speakers: Michael Roytman, Co-founder and CTO, Empirical Security
Conference: CypherCon
YouTube: https://www.youtube.com/watch?v=X1oRvn8VSiE
Overview
In his CypherCon talk, Michael Roytman, Co-founder and CTO of Empirical Security, made a compelling case for the widespread adoption of local models in cybersecurity. Roytman argued that while global models like CVSS and EPSS provide a baseline for vulnerability assessment, they inherently lack the specificity required to effectively manage risk within unique enterprise environments. His vision centers on empowering security engineers to build and deploy bespoke models that leverage an organization's own data, leading to significantly more efficient and accurate vulnerability prioritization.
The core premise of Roytman's presentation is that the cybersecurity industry has reached a pivotal moment where advancements in machine learning infrastructure and data engineering technologies make it feasible for individual enterprises to move beyond generic risk assessments. These "local models" are not merely about running on-device, but rather about modeling the preventative side of cybersecurity—such as exposure management and vulnerability prioritization—with a deep understanding of an organization's specific assets, configurations, and historical threat telemetry. This approach promises to transform how security teams allocate their limited resources, drastically improving both the coverage of remediated risks and the efficiency of those remediation efforts.
Roytman substantiates his claims with extensive benchmarking data derived from 500 enterprises, accumulated during his tenure as Chief Data Scientist at Kenna Security. This data illustrates the inherent inefficiencies of current global prioritization methods and highlights the profound impact that tailored, local models can have. Furthermore, he directly challenges the prevailing hype around large language models (LLMs) for deterministic security tasks, asserting that classical machine learning techniques offer a superior, more cost-effective solution for precise forecasting within an enterprise's unique context.
Background
▶ Watch: The 15% average remediation problem (3:50)
The cybersecurity landscape is characterized by an overwhelming volume of potential exposures, far outstripping an organization's capacity to address them. Roytman presented sobering benchmarking data from 500 enterprises, revealing a consistent "law" in cybersecurity: organizations, regardless of size or the volume of vulnerabilities observed (from 100 to 10 million), remediate approximately 15% of their vulnerabilities on median per month. This scale-invariant capacity limitation underscores a critical problem: security teams are drowning in alerts, and their efforts are often inefficiently deployed.
Roytman categorizes security data into two fundamental types: potential problems and events. Potential problems encompass any exposure that could lead to an event, such as misconfigurations, weak passwords, application security flaws (SQL injection, buffer overflows), cloud misconfigurations, and CVE-based vulnerabilities. Crucially, these are not guaranteed to become events. Events, on the other hand, are confirmed instances of malicious activity or telemetry indicating an attack. The challenge arises because the quantitative methods traditionally used for vulnerabilities, particularly CVEs, rely on a manual, human-driven reporting process, and existing vendor-supplied models often fail to account for the unique context of an enterprise.
Historically, security vendors have struggled with limited data science tools, leading to the creation of generic "high, medium, low" or "red, yellow, green" criticality models. Practitioners are then left to manually adjust these assessments based on their specific environment—considering mitigating controls, system criticality, or external telemetry from firewalls. This manual correction process is inefficient and highlights the inadequacy of global, one-size-fits-all models.
Roytman, as a co-author of the Exploit Prediction Scoring System (EPSS), provided insight into the evolution of vulnerability models.
- CVSS (Common Vulnerability Scoring System), introduced around 2005, primarily measures the technical severity of a vulnerability based on 6-12 variables. While widely used, the CVSS version two documents explicitly state it should not be used as a risk model.
- By 2011, outside-in vendors like SecurityScorecard and BitSight began offering quantitative "credit scores" for companies. Concurrently, Kenna Security pioneered inside-out models for scoring risk based on vulnerabilities and assets, though these were also global in nature.
- EPSS, published in 2021 (with a Black Hat talk in 2019), represents a significant leap forward. It's a probability-based forecasting model for CVEs, published daily for free at first.org. EPSS uses an XGBoost model trained on donated data (e.g., from AlienVault, Shadowserver, Cisco) and observed exploitation events to predict the probability of a CVE being exploited on the internet within the next 30 days. Over 130 vendors currently integrate EPSS.
Despite its advancements, EPSS, being a global model, has inherent limitations. Its effectiveness is largely confined to CVEs because a given CVE (e.g., a SQL injection) behaves technically similarly across diverse environments. However, non-CVE exposures, such as application security flaws or cloud misconfigurations, are fundamentally different. An application at a healthcare company is not comparable to one at a hotel, and their threat vectors and attack surfaces are unique. Attempts to build global application security or cloud models consistently failed because "every enterprise is a special snowflake," demanding models that account for specific data sets, legacy systems, and unique configurations. This limitation is exacerbated by the exponential rise of non-CVE exposures; Roytman cited an example of a company with 2 million CVEs but 8 million non-CVE based exposures—for which no quantitative models exist. The median vulnerability takes about six months to close, creating long exposure windows, and this remediation speed varies wildly between organizations, further emphasizing the need for tailored prioritization.
Key Findings
▶ Watch: Each enterprise needs its own unique risk model (11:00)
Roytman's talk unveiled several critical findings that challenge conventional wisdom in vulnerability management and advocate for a paradigm shift towards customized, local models:
- Global Models are Inherently Inefficient for Enterprise Risk Reduction: While global models like CVSS can achieve high risk reduction, they do so at an unacceptable cost in efficiency. Remediation of all CVSS scores of 7 and above, for instance, would require fixing 63% of all CVEs in the National Vulnerability Database (NVD) to reduce 86.8% of known exploited vulnerabilities. However, this comes with a dismal 5% efficiency—meaning only 1 in 20 remediated vulnerabilities actually carries exploitation risk. EPSS significantly improves this, achieving approximately 80% coverage for the same amount of effort that yields only 25% coverage with CVSS. Yet, even EPSS, being a global model, cannot fully optimize for individual enterprise contexts.
- Prevalence Adjustment is Crucial for Real-World Effectiveness: The smooth performance curves often presented for global models are based on the flawed assumption of having "one of everything" in an environment (i.e., evaluating against the entire NVD). When models are evaluated against a real customer's environment and adjusted for the prevalence of vulnerabilities on assets (e.g., an Adobe Reader flaw on 100,000 endpoints versus a risky AIX vulnerability on a single server), their performance can drastically change. Global models often exhibit erratic efficiency drops when evaluated with prevalence data, indicating a mismatch with actual enterprise risk profiles.
- Local Models Dramatically Boost Efficiency and Coverage: By incorporating enterprise-specific data—including asset prevalence and historical SOC telemetry—into a classical machine learning model, Roytman demonstrated a profound improvement. For a real customer, a local model achieved 75% coverage with an astounding 99.3% efficiency. This means nearly every remediation effort directly addressed a high-risk vulnerability, a monumental leap from the 5% efficiency of CVSS. This performance even outstrips EPSS, highlighting the power of context-aware modeling.
- LLMs are Suboptimal for Deterministic Forecasting and Prioritization: Despite the hype, Roytman firmly stated that Large Language Models (LLMs) are ill-suited for the deterministic forecasting and prioritization tasks required in security operations. LLMs excel at sequence completion and general knowledge (like writing emails), but they lack the specific, tailored context and precision needed for vulnerability risk prediction. Running LLMs for these tasks is not only less effective than classical ML but also significantly more expensive, especially considering that current LLM costs are often subsidized.
- Enterprises Possess the Necessary Data and Tools: Roytman emphasized that security teams already collect the disparate data needed to build these local models: batch-processed scanner data ("what could happen") and real-time stream telemetry from SOC/SIEM systems ("what is happening"). Modern ML infrastructure (e.g., DataBricks, Snowflake, AWS S3, Kafka) has made the integration and processing of these data types feasible for security engineers, eliminating the heavy lift it once was. The challenge is primarily conceptual and organizational, not technological.
Technical Deep Dive
▶ Watch: CVSS is an inefficient risk prioritization tool (21:00)
The core technical contribution of Roytman's talk lies in advocating for and detailing the principles behind local models—models specifically designed to describe an enterprise's unique environment, rather than running on individual devices. This approach contrasts sharply with the limitations of global models like CVSS and EPSS, which, while useful as base rates, fail to account for specific organizational context.
Roytman posits that security data fundamentally comprises two types:
- Exposure Data ("What Could Happen"): This includes findings from vulnerability scanners (e.g., Tenable, Qualys, Rapid7), application security tools (SAST, DAST), and cloud security posture management (CSPM) platforms (e.g., Wiz). Historically, this data is collected in batches (monthly, weekly).
- Ground Truth Behavioral Telemetry ("What Is Happening"): This comes from SOC/SIEM systems (e.g., Splunk), IDS, endpoint protection software, and OSQuery. This data represents real-time streams of observed exploitation activity, malicious events, and system behavior.
The critical insight is the need to combine these two data streams within a unified data lake (which can be as simple as an S3 bucket). This integration allows for the correlation of potential problems with actual observed events within the enterprise, providing the necessary ground truth for training effective local models.
For the predictive forecasting and prioritization tasks, Roytman strongly recommends using classical machine learning models, specifically XGBoost. He highlights that these models are:
- Memory-efficient: Training can be done on a DataBricks cluster or even a 32GB laptop for smaller organizations.
- Cost-effective: Training costs can be as low as "$5 to train and like a thousands of a cent per inference per vulnerability."
- Purpose-built: Unlike LLMs, which are designed for sequence completion, XGBoost is optimized for deterministic forecasting based on structured data.
The model's inputs should include:
- Vulnerability and Asset Data: What vulnerabilities exist on which assets in the environment, sourced from scanners. This is the most basic, yet crucial, contextual data.
- Prevalence: How many instances of a particular vulnerability exist across the asset infrastructure. This is a critical factor for adjusting the model's output to reflect real-world impact.
- SOC Telemetry: Historical data on observed exploitation attempts or successful events related to specific vulnerabilities or configurations within the enterprise.
To measure the performance of these models, Roytman emphasizes two key metrics:
- Coverage: The percentage of actually exploited vulnerabilities that would have been remediated by the model's prioritization.
- Efficiency: The percentage of remediated vulnerabilities that were truly high-risk (i.e., had exploitation events associated with them). High efficiency means minimal wasted effort.
Roytman demonstrated the inefficiency of traditional methods: prioritizing all CVSS 7+ vulnerabilities yields 86.8% coverage but only 5% efficiency. In contrast, EPSS, a global ML model, significantly boosts efficiency (e.g., 80% coverage with 10% effort). However, the real breakthrough comes with prevalence-adjusted local models. When a model is trained using the specific prevalence of vulnerabilities across an enterprise's assets, the performance curves become drastically different and more accurate. For a real customer, a local XGBoost model, incorporating asset prevalence and SOC telemetry, achieved 75% coverage with an astounding 99.3% efficiency. This is a stark improvement over global models that produce "smooth lines" when evaluated against the entire NVD, which assumes "one of everything"—a scenario never found in a real enterprise.
A crucial aspect of building these local models is data subsetting. While an enterprise might generate 15 terabytes of daily Splunk telemetry, not all of it is needed for model training. SIEM vendors are already proficient at detecting malicious activity. By focusing on the 17,000 CVEs with observed exploitation activity (a manageable number), security engineers can subset their data to "what is actually risky," making the model training process feasible without requiring massive computational resources.
Roytman explicitly addressed the role of Large Language Models (LLMs). He argued that while LLMs might aid in tasks like summarizing GitHub exploit code to extract features for a deterministic model, they are fundamentally ill-suited for the core forecasting and prioritization of vulnerabilities. LLMs are sequence finishers, not reliable forecasters. They provide general knowledge but lack the precise, context-specific reasoning required for deterministic security decisions, leading to higher costs and lower accuracy compared to tailored classical ML models. The speaker mentioned a GitHub repository where Empirical Security benchmarked global and local models against LLMs for vulnerability triage, finding classical ML to be superior for this specific task.
Demo / Proof of Concept
▶ Watch: Local models dramatically increase remediation efficiency (39:30)
While Michael Roytman's talk did not feature a live, interactive demonstration, he presented compelling benchmarking data and results from a real customer scenario that served as a powerful proof of concept for the efficacy of local models. He walked the audience through a detailed comparison of CVSS, EPSS, and a local XGBoost machine learning model, illustrating their respective performance in terms of coverage and efficiency when applied to an actual enterprise environment.
The key visual evidence included charts demonstrating how a local model, trained on an enterprise's specific asset prevalence and SOC telemetry, could achieve 75% coverage with 99.3% efficiency in vulnerability remediation. This dramatic improvement over the 5% efficiency of CVSS was a central point of the presentation, underscoring the practical benefits of the approach. Roytman also mentioned a public GitHub repository where Empirical Security has published benchmarks comparing these global and local models against LLMs for vulnerability triage, further substantiating his claims with accessible data.
Defensive Implications
▶ Watch: LLMs are not for security operations forecasting (44:00)
The insights from Michael Roytman's talk carry significant defensive implications for security teams striving to manage risk effectively in an increasingly complex threat landscape:
- Prioritize Local Context Over Global Scores: Defenders must move beyond exclusive reliance on generic, global vulnerability scores like CVSS or vendor-provided high/medium/low classifications. These scores, while providing a baseline, do not adequately reflect the unique risk profile, asset prevalence, or mitigating controls within an individual enterprise. The focus should shift to building or utilizing models that incorporate specific local context.
- Integrate Exposure and Telemetry Data: A fundamental step is to break down the traditional silos between "preventative" exposure management data (from scanners, app security, CSPM) and "reactive" behavioral telemetry from SOC/SIEM systems. Integrating these two types of data into a unified data lake (e.g., S3 bucket) is crucial for training local models that understand both what could happen and what is happening in the environment.
- Empower Security Engineers to Build Custom Models: The talk highlights that the necessary ML infrastructure (e.g., DataBricks, Snowflake, Apache Kafka) and classical ML algorithms (like XGBoost) are now mature and accessible enough for security engineers to build these local models "at home." Organizations should invest in upskilling their security engineering teams to leverage these tools, rather than solely relying on external vendors or dedicated data science teams.
- Optimize for Efficiency and Coverage: Defenders should explicitly measure and optimize their vulnerability prioritization efforts for both high coverage (reducing actual risk) and high efficiency (minimizing wasted remediation efforts). The demonstrated leap from 5% CVSS efficiency to 99.3% with a local model translates directly into massive cost savings for IT and security teams, as fewer non-risky vulnerabilities are unnecessarily addressed.
- Proactive Prioritization with Data: Instead of reactively scanning and then trying to prioritize all findings, organizations can use local models to proactively identify the highest-risk vulnerabilities before extensive scanning. This allows for targeted validation using existing security tools like OSQuery, IDS, or endpoint protection software, focusing resources on confirming the presence of known critical risks.
- Re-evaluate LLM Use Cases: While LLMs have their place (e.g., summarizing code, workflow automation), defenders should be wary of deploying them for deterministic forecasting and prioritization in security operations. Classical machine learning models are shown to be more accurate, reliable, and cost-effective for these specific tasks. Organizations should use the right tool for the job to avoid unnecessary expense and potential inaccuracies.
- Tailor Remediation SLAs: By understanding their organization's actual vulnerability prevalence and historical remediation capacity, security teams can establish more realistic and effective Service Level Agreements (SLAs) for different types of vulnerabilities. This data-driven approach allows for better resource allocation and communication with IT and business stakeholders.
Key Takeaways
- Global vulnerability models like CVSS and EPSS are insufficient for precise risk management within unique enterprise environments due to their lack of specific context and inability to account for asset prevalence and non-CVE exposures.
- Security teams can and should build "local models" using classical machine learning (e.g., XGBoost) tailored to their specific environment, leveraging existing data from vulnerability scanners, asset inventories, and SOC/SIEM telemetry.
- Integrating "what could happen" (exposure data) with "what is happening" (real-time behavioral telemetry) is crucial for creating accurate and effective local risk prioritization models.
- Local models, especially when adjusted for vulnerability prevalence on specific assets, dramatically improve remediation efficiency (e.g., from 5% with CVSS to over 99% in real-world scenarios) while maintaining high risk coverage.
- Large Language Models (LLMs) are generally ill-suited for deterministic forecasting and vulnerability prioritization tasks in security operations, offering less precision and higher costs compared to purpose-built classical ML models.
- Modern ML infrastructure and data engineering tools make building these local models feasible and cost-effective for security engineers, enabling significant ROI through reduced remediation efforts and better resource allocation.
About the Speaker(s)
Michael Roytman is the Co-founder and CTO of Empirical Security, bringing extensive experience in applying quantitative methods to cybersecurity challenges. Prior to his current role, he served as Chief Data Scientist at Kenna Security and held positions at Cisco. Roytman is notably recognized as one of the primary authors of the Exploit Prediction Scoring System (EPSS), an widely adopted open-source project that provides probabilistic forecasts for CVE exploitation. With 15-20 years dedicated to building deterministic models in cybersecurity, he advocates for data-driven approaches that empower security teams to make more efficient and informed decisions.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Roytman—the guy who literally co-wrote EPSS—gets on stage to explain why global scoring models are fundamentally broken for enterprise triage. He cuts through the generative AI hype to show how classical ML and local SOC telemetry actually fix vulnerability prioritization. It's a masterclass in deterministic forecasting that every SecOps team currently drowning in scanner alerts needs to watch.
Heather Calloway (CISO) — MUST SEE
Michael Roytman proves mathematically what every security leader already suspects: patching based on CVSS scores is a massive waste of IT headcount. He lays out a compelling, data-backed case for using localized machine learning to prioritize vulnerabilities based on actual environmental telemetry rather than generic vendor scores. This is mandatory viewing for anyone managing a patching SLA or an enterprise vulnerability program.