How not to do ML: Showing the Negative Impact of Improper CVE Feature Selection in a Live Exploit Prediction Model

How Not to Do ML: The Hidden Pitfalls of CVE Feature Selection in Live Exploit Prediction

Speaker: François Labrèche (Sophos)

Conference: NorthSec 2025 — May 15–16, 2025, Marché Bonsecours, Montreal

Watch on YouTube: https://www.youtube.com/watch?v=JOzjeDVM2sQ

Reading time: ~7 minutes

TL;DR

A machine learning model that scores 93% accuracy and 83% recall on historical CVE data can drop to 2% recall the moment it goes live. François Labrèche of Sophos describes exactly how that happened to an in-production exploit prediction model — and identifies four distinct encoding errors that caused future information to leak into training data, inflating historical metrics while completely destroying live performance. The talk is a rigorous, self-critical post-mortem that every security ML practitioner should read before shipping a vulnerability prioritization model.

Introduction

The security industry produces over 40,000 new CVE entries per year, but only a small fraction of those vulnerabilities ever receive a working exploit, and only a subset of those are actually weaponized in attacks. If machine learning can reliably distinguish exploitable vulnerabilities from the broader noise, vulnerability management teams can focus remediation effort where it matters most.

François Labrèche has a PhD in applying machine learning to cybersecurity problems and leads the XDR Threat Prioritization team at Sophos. His talk does not begin with a success story. It begins with a model that looked excellent on paper and was essentially useless in production. Labrèche's stated method — "I make mistakes, and then I present them at conferences" — belies a systematically rigorous investigation of exactly what went wrong and why.

The core finding is that exploit prediction is a uniquely dangerous domain for standard ML evaluation methodology. Several encoding choices that are harmless or even beneficial in other supervised learning contexts become sources of data leakage when the model is applied to vulnerabilities published today. The historical cross-validation metrics look great; the live metrics collapse; and the root cause in each case is that the model learned something about the future that it will never be able to see when running in production.

▶ Watch: Context and problem framing (0:00)

The Model: Features, Labels, and Baseline Performance

The model architecture is a random forest classifier trained on CVE metadata drawn from publicly available sources. Feature groups include: CVE descriptions encoded as topic vectors using Latent Dirichlet Allocation (LDA), which clusters all known CVE descriptions into 30 thematic topics; structural metadata such as the number of vulnerable products and number of references; CVSS scores and subscores; CVE publication and modification dates; and online discussion volume aggregated from sources including dark web forums (via Flare), Reddit, Pastebin, and Telegram.

The binary label is simply whether one or more exploits exist for the vulnerability — drawn from Exploit-DB, Packetstorm, and GitHub proof-of-concept repositories. Class imbalance (most CVEs have no exploit) is addressed with balanced class weights and threshold moving.

Evaluated with standard tenfold cross-validation, the model achieves 93% accuracy and 83% recall. These are strong results by any conventional measure, and they would ordinarily justify a production deployment. The model was deployed. Its live recall was 2%.

At 2% recall, the model classifies virtually every new vulnerability as unexploitable regardless of its actual risk. Since most CVEs genuinely have no exploit, accuracy remains high — but the model has become a sophisticated mechanism for ignoring the vulnerabilities that actually matter.

▶ Watch: Model overview and the cross-validation vs. live performance gap (4:00)

Issue 1 and 2: Date Features and Temporal Data Leakage

The first encoding error is straightforward: including CVE publication and modification dates as model features. When trained on historical data, date features capture real trends — certain vulnerability types cluster in certain time periods, and the model learns these patterns. But when applied to a new CVE published today, the publication date is always a date the model has never seen. Rather than providing signal, the date feature introduces noise that actively degrades predictions for new entries.

The fix is simple: remove date features entirely. Doing so costs only a small amount of historical accuracy but triples live recall — from 2% to 6%.

The second issue is subtler and is an evaluation-methodology problem rather than a feature-encoding problem. Standard tenfold cross-validation randomly shuffles the dataset before splitting it into folds, meaning training folds routinely include data from dates later than the test fold. For most ML tasks, this is fine. For vulnerability exploit prediction, it is not: the model effectively learns from the future. An exploit published six months after a CVE will influence the training of a model fold whose test set includes that same CVE from before the exploit existed.

The fix is a time-series split: training data must always predate test data. Implementing this brings the historical evaluation metrics closer to the live performance, removing the false confidence that standard cross-validation was providing.

▶ Watch: Date features and cross-validation temporal leakage (12:00)

Issue 3 and 4: Online Discussion Features and LDA Label Contamination

Online discussion volume is an intuitively attractive feature: CVEs that are being heavily discussed on cybercrime forums or Reddit are more likely to be exploited. The problem is temporal. Discussion volume is cumulative — the feature value for a historical CVE reflects months or years of accumulated discourse. A new CVE has been public for days or weeks at most. The historical training data conditions the model to expect discussion volumes that new CVEs cannot have yet, causing systematic misprediction.

Removing online discussion features produces a substantial jump in live recall. There are more sophisticated approaches — encoding discussion volume as a time-windowed feature rather than a cumulative count, for instance — but Labrèche removes the features to isolate and quantify their impact cleanly.

The fourth issue concerns the LDA topic model used to encode CVE descriptions. LDA is trained on all CVE descriptions available at training time, including CVEs that postdate the training-set vulnerabilities. A topic model trained on future data shapes the feature representation of past data in ways that the live model cannot replicate, because at prediction time the topic model will not yet have seen the CVEs published in the future. This is a more subtle form of the same leakage: the feature extractor itself is contaminated by future information.

Labrèche presents a revised model incorporating all four fixes. The live recall improves significantly, though there is honest acknowledgment that the model is still not where it needs to be for operational deployment. The value of the talk lies not in a final solved problem but in the explicit documentation of each failure mode — a catalogue of encoding errors that practitioners can check against their own pipelines.

▶ Watch: Online discussion features and the LDA contamination issue (16:00)

Applying the Model Beyond CVE Exploitation

Labrèche closes by noting that the same supervised learning architecture, once corrected, can be applied to other threat prediction tasks: predicting which threats will be actively used in attacks, which vulnerabilities will appear in ransomware toolkits, or which indicators are worth hunting for in telemetry. The core methodology — encode threat metadata, train on historical ground truth, predict against new threats — generalizes broadly across the vulnerability intelligence domain.

The prerequisite is getting the feature encoding right. The four issues documented in this talk are not specific to CVE data. They are general failure modes of any time-series prediction problem applied to cybersecurity data: date features that mismatch between training and deployment, evaluation splits that allow future data to contaminate training, cumulative features that behave differently for recent versus historical items, and upstream encoders (topic models, embeddings) trained on data that postdates the training set.

▶ Watch: Revised model results and generalizing to other threats (22:00)

Notable Quotes

"A model that is built using incorrect features can have a drastic impact on the true performance — true performance meaning when you actually want to use the model in a real-world scenario."

"We can see that the recall is down to two percent. What this means is the model is predicting every new vulnerability as not having an exploit. Essentially, this model is completely useless."

"Future information was bled in the training of the model through the feature encoding, essentially inflating its performance on historical data, making the model appear really good on paper and actually hurting it when you deploy it to new vulnerabilities."

"For the most part, I make mistakes, and then I present them at conferences."

Key Takeaways

• 93% accuracy can mean 2% recall in production. Strong cross-validation metrics do not guarantee live performance; the gap between them is where data leakage lives.
• Date features hurt more than they help for new CVEs. Every new vulnerability has a publication date the model has never seen. Remove date features from exploit prediction models.
• Use time-series splits, not random splits, for cybersecurity ML. Standard tenfold cross-validation allows future data into training folds, creating a systematic overestimation of live performance.
• Cumulative features mismatch between historical and live data. Online discussion volumes, social media mention counts, and similar cumulative signals will always look different for newly published CVEs than for historical ones.
• Upstream encoders must also be temporally constrained. A topic model or embedding trained on future data contaminates the feature space for historical examples in ways that live predictions cannot replicate.
• Self-critical post-mortems are as valuable as success stories. Labrèche's explicit documentation of each failure mode provides a practical checklist for practitioners building vulnerability prioritization systems.