Why Most ML Vulnerability Detection Fails (And What Actually Worked for Kernel Bugs)

Why Most ML Vulnerability Detection Fails (And What Actually Worked for Kernel Bugs)

Speaker: Jenny Guanyi Qu, AI Researcher, Pebblebed

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch: YouTube

Reading time: ~5 minutes

TL;DR

Most ML models applied to vulnerability detection fail because researchers start with complex architectures before establishing what simple baselines can already do. Jenny Qu, trained on math AI at Caltech and backed by Pebblebed Ventures, applied rigorous ML methodology to 125,000 labeled Linux kernel commits and discovered that context length, curriculum design, and hard negative selection matter more than model sophistication — and that the most surprising result came from a model that used just three numbers.

Introduction

The Linux kernel is one of the most important and least-well-monitored codebases in the world. Its 1,400 emails per day on the kernel mailing list drown volunteer maintainers. Patches get lost. There's no centralized tracking of fixes, and CI testing without coordination means more infrastructure doesn't reliably mean better coverage. And inside all of that noise, 13% of bugs hide for over five years — including one introduced in August 2006 that wasn't fixed until August 2025, a 19-year-old vulnerability sitting in production systems the entire time.

Jenny Guanyi Qu, a CTF player whose team placed third at DEF CON in 2025 and an AI researcher whose work is backed by Pebblebed Ventures, decided to bring ML to this problem. She came to it from an unusual angle: before pivoting to security, she was trying to build a math AGI at Caltech. The same techniques she was developing for mathematical reasoning turned out to be well-suited for characterizing what makes a code change dangerous.

"I realized that I can use the skills I had at Caltech for building math AGI for cybersecurity, which is a field I'm extremely passionate about," she told the [un]prompted audience.

▶ Watch: The Linux Kernel Problem (00:00)

The Dataset: 125,000 Labeled Vulnerabilities

The Linux kernel's "fixes" convention — where commits reference the commit they're fixing — creates a naturally labeled dataset. Qu identified 125,000 labeled vulnerability commits: each one tagged as either introducing a vulnerability or fixing one. This is a rare resource in security ML, where labeled data is often the binding constraint.

The bug distribution reveals the problem's difficulty. Race condition bugs hide the longest because they require specific conditions to trigger and are hard to reproduce consistently. Reference counting errors don't crash immediately — they leak memory slowly, sometimes going unnoticed for years. These long-lived bugs are also the highest-value targets for detection, since they represent the risks that human review has consistently failed to catch.

The top bug fixers in the kernel are also worth noting: Dan Carpenter, who has contributed over 2,000 bug fixes, also invented the "fixes" tag convention that made this dataset possible in the first place.

▶ Watch: Bug Lifetime Distribution (02:00)

The Most Important Lesson: Beat the Dumb Baselines First

Before training any neural model, Qu established three baselines:

• B1 — Three numbers only: lines added, lines removed, files changed
• B4 — Commit subject line only
• B6 — Commit subject plus the full diff

B1 — the three-number model — achieved an area under the ROC curve (AUC) of 0.779. That's a surprisingly strong result for a model that never looks at the actual code.

"I was like, this is not very relevant to the vulnerability," Qu admitted during Q&A. "So I was like, the model is not gonna do very well with only these three numbers. And what's surprising is the model is actually doing pretty well."

The reason is dataset distribution. Vulnerable commits are rare. A model that never learns anything meaningful can still achieve decent AUC by defaulting to "safe" predictions. The practical implication: if your ML model can't beat these trivial baselines by a meaningful margin, it isn't actually learning vulnerability patterns. It's a fancy tokenizer.

This sanity check is one of the most underappreciated steps in security ML research, and Qu's framing of it as a required precondition rather than an afterthought is methodologically important.

▶ Watch: Dumb Baselines and Why They Matter (04:00)

What Actually Improved Performance

After establishing baselines, three factors proved most important:

Context length unlocks code understanding. At 512 tokens, transformer models read commit messages but can't see much of the actual diff. At 8,000 tokens, they can see the full change and understand what it does. Qu found that increasing context window from 512 to 8,000 tokens produced "a much better result" — models at the smaller context window were essentially reading the conversation around the code, not the code itself.

Vulnerability patterns have a shelf life. Models trained on older data performed worse on newer vulnerabilities. The patterns of what makes a commit suspicious shift over time as kernel development practices change and new vulnerability classes emerge. This has direct operational implications: models need retraining on a cadence matched to how quickly the vulnerability landscape evolves.

Curriculum learning was backwards. Qu initially assumed that large, complex diffs with many changed files would be the hardest classification cases and designed training to start with simpler examples. The reality was the opposite: large diffs have more signal. Small diffs with minimal context changes are the genuinely hard cases — there's less information to work with, so the model has to infer more. Training with an abundance of easy positive examples (obviously safe commits) before introducing hard ones (subtly dangerous commits) proved important for establishing what "clearly safe" looks like before asking the model to distinguish subtle danger.

▶ Watch: Context Length and Training Insights (06:00)

Hard Negative Selection

One of the most practical findings concerned negative examples — commits that are safe but could look suspicious. Qu experimented with including a large number of hard negatives: safe commits that closely resemble vulnerable ones in superficial ways. The intuition was that this would sharpen the model's discrimination.

The result challenged her assumptions: the model also needed an abundance of "easy" negatives — obviously safe commits — to build a reliable sense of what normal looks like. Without that foundation, the model struggled to generalize from the hard cases. "The model needs to learn what is obviously safe before learning what is subtly dangerous, because most commits are obviously safe."

This has implications for anyone curating datasets for security ML: hard negative mining alone isn't sufficient. The training distribution needs to match the real-world distribution, which is overwhelmingly benign.

▶ Watch: Hard Negatives and Training Strategy (08:01)

The Gap Between Detection and Exploitation

Qu was candid about the limits of what her system can currently do. Detection is not exploitation. A model that flags a commit as suspicious provides a signal, but there's no automated path from "this looks dangerous" to "this is a confirmed, exploitable vulnerability with a working exploit chain."

"There's no way to verify. And even if there's crashes, crashes does not mean actual vulnerability. It does not mean you can actually exploit this bug. So there is a big gap between crashes to actual exploit chains."

The practical value proposition is prioritization: pointing human researchers — or in her example, Nicolas Carlini's team, which has hundreds of crash findings — toward the cases with the highest probability of being exploitable. Even if the model can't close the loop, sharpening the focus of human expertise is itself a meaningful contribution.

Her blog post on this research reached number one on Hacker News and is available at the Pebblebed website.

▶ Watch: Q&A on Limitations (12:01)

Notable Quotes

"If your model cannot beat these simple, dumb baselines, it's just a fancy tokenizer."

— 04:00

"Context length unlocks code understanding."

— 06:00

"The model needs to learn what is obviously safe before learning what is subtly dangerous, because most bugs are obviously safe."

— 08:01

Key Takeaways

• Establish dumb baselines before training complex models. A three-number model achieved 0.779 AUC on kernel vulnerability classification. Any ML approach that can't meaningfully exceed that isn't learning vulnerability semantics.
• Context length matters more than architecture. At 512 tokens, models read messages. At 8,000 tokens, they read code. The difference in classification quality is substantial.
• Vulnerability patterns have a shelf life. Security ML models trained on historical data degrade over time; retraining cadence should reflect how quickly the vulnerability landscape evolves.
• Curriculum learning should start easy, not hard. Small diffs are harder to classify than large ones. Start with clearly safe commits before introducing subtle danger.
• Detection is prioritization, not exploitation. The practical value is directing human expert attention to the most suspicious signals — not replacing the human judgment needed to confirm and exploit a vulnerability.

Slides Reference

No slides PDF was listed in the conference bundle for this session. The talk was a standalone presentation. Qu's full write-up is available as a blog post on the Pebblebed website; her Twitter handle is @GuanyiXu.