Source to Sink: How to Improve LLM First-Party Vuln Discovery

Source to Sink: How to Improve LLM First-Party Vulnerability Discovery

Speakers: Scott Behrens, Principal Security Engineer, Netflix; Justice Cassel, Application & GenAI Security, Netflix

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch: YouTube

Reading time: ~10 minutes

TL;DR

Netflix's security engineering team spent months iterating through architectures, benchmark failures, and demoralizing late nights to build an LLM-based vulnerability discovery system that actually works in production. Their conclusion: specialized agents beat generalist agents, individual vuln-class agents beat grouped ones, and rigorous evaluation infrastructure is the only thing that separates signal from noise as models keep changing.

Introduction

When large language models first became capable enough to hold a conversation about code, security researchers everywhere had the same thought: can we use these to find vulnerabilities? The answer turned out to be yes — but not in the way anyone initially imagined.

Scott Behrens and Justice Cassel from Netflix's security engineering team arrived at [un]prompted 2026 to share what eighteen months of iterative research actually looks like: the moments of genuine excitement, the baffling regressions, the architectural pivots, and the system they've landed on that's starting to produce consistent, meaningful results for first-party vulnerability discovery.

"We started out with simple prompting, thinking we've broken the barrier and everything's gonna be great," Cassel recalled. "And then we kind of realized that it's a little bit more difficult than that. Consistency was difficult, variance was huge, false positives kept coming up. Sometimes it would ignore entire vulnerability classes."

Their journey is one of the most honest accounts of what it actually takes to deploy LLM-based security tooling at scale — including the hard lessons that nobody publishes.

▶ Watch: The Initial Hope and Early Frustration (00:00)

The Architecture Evolution

The Netflix team's system didn't arrive fully formed. It evolved through at least four distinct architectural phases, each driven by real limitations they encountered:

Phase 1: Simple prompting — Direct questions to the model about code. Quick to implement, results were inconsistent and highly variable.

Phase 2: Agentic workflows — Introducing file tracking, state tracking, tracing, and tool integrations. Results improved significantly, and the team felt like they were onto something real.

Phase 3: IDE-based architecture — An internal stack change prompted a shift to working within the IDE they were using. Still producing good results, but not portable enough.

Phase 4: MCP with context injection — Moving to an IDE-agnostic approach through MCP enabled broader deployment but introduced new challenges around context management.

Current: Custom orchestration framework — An in-house system with a much richer feature set that the team is now moving toward productionization.

The iteration wasn't wasted. Each architectural change surfaced insights that shaped the current design. The MCP phase, for example, forced them to think carefully about what context actually needs to flow to each agent and what should be filtered out.

▶ Watch: Architecture Evolution (04:00)

Building Evaluations First

Before any of the architectural work could be validated, the Netflix team had to solve an evaluation problem. Without knowing what they were looking for, there was no way to benchmark whether changes were improvements or regressions. Prompt changes that made the model seem smarter on one test would break it on another. A model version update would completely change the performance profile.

Their evaluation approach required three things:

1. A labeled set of known vulnerabilities (true positives) plus known non-issues (dead code, commonly misinterpreted paths, false positive traps)
2. Coverage of multiple vulnerability classes — not just OWASP Top Ten
3. Automated metrics gathering so they could run tests continuously as the system evolved

The benchmark they built targets 41 true positives across the vulnerability classes they care about. That number became the yardstick against which every architectural and prompt change was measured.

"When models change, so do the best configurations for the results that we're looking for," Behrens explained. "That's why it's important for us to keep running them because every once in a while the model will just seem to append a couple more brain cells and you're like, 'I solved everything. We are the best.' And then the next five runs will look a little bit different."

They're releasing their evaluation repos with score sheets so other teams can run similar tests on their own tooling — addressing what they see as a gap in the current evaluation landscape, where benchmarks test base models rather than models-in-workflows.

▶ Watch: Evaluation Infrastructure (04:00)

Experiment 1: Specialized Agents Beat Generalists

The first major experiment tested a core intuition: would specialized agents outperform a single generalist security agent?

The "super agent" approach — a single agent with a massive monolithic rule file containing tons of security context — found approximately 20 of the 41 benchmark vulnerabilities. The specialized agent approach, where individual agents focus on specific vulnerability classes, found 35 out of 41.

The lesson wasn't just about recall. Behrens noted that specialized agents also improve the quality of findings: correct severity assignments make triage more efficient, worth the additional token cost.

The tradeoff is real — specialized agents cost more in tokens — but the miss rate from the generalist approach is too high to be acceptable for production security use cases.

▶ Watch: Specialized vs. Generalist Agents (06:00)

Experiment 2: Individual Vuln-Class Agents Beat Grouped Categories

The next hypothesis seemed reasonable: group related vulnerability types into category agents (injection vulnerabilities, network bugs, etc.) to save tokens while maintaining reasonable coverage. The team was wrong.

Grouping agents into categories instead of running them individually dropped coverage to 34 out of 41 vulnerabilities. Token costs were significantly lower, but the miss rate was unacceptably high. The team also observed what looked like an overfitting problem — the model seemed to latch onto category labels in ways that distorted which vulnerabilities it looked for.

"My general recommendation: don't use categories," Behrens said flatly.

The practical implication: for comprehensive vulnerability discovery, you need one agent per vulnerability class, running in parallel. The parallelism is structurally important — since individual vuln-class agents don't have dependencies on each other, they can run simultaneously, controlling the time cost even as the number of agents grows.

▶ Watch: Grouped vs. Individual Agents (08:00)

The Three-Stage Architecture: Enumerate, Discover, Trace

The current Netflix workflow is built around three sequential stages with heavy parallelism within each:

Stage 1 — File Enumeration: Programmatic filtering of the repository before any model is invoked. Tests, build artifacts, large images, example files, and other noise are excluded. Only files relevant to the scan proceed. This step is entirely deterministic — no model involved.

Stage 2 — Discovery and Priming (parallel): Three processes run simultaneously:

• Static analysis — Semgrep, CodeQL, AST, LSP, and regular expressions identify known patterns
• Architecture discovery — Determines what the codebase is doing (web framework, database usage, templating, etc.) so that only relevant agents are loaded later
• Data flow tracing — Source-to-sink tracing identifies where untrusted data enters the system and where it flows

"We're leveraging all the tools in the arcade — AST, LSP, regular expressions, Semgrep, CodeQL — all these things that are going to enable these models to have a better trace and understanding and comprehension of the actual code base," Cassel explained.

Stage 3 — Specialized Agent Scanning (parallel): Discovery gates determine which agents to run. If there's no database, the SQL injection agent doesn't load. If there's no templating, the SSTI agent doesn't run. HTML output sinks are not fed to the SQL injection agent. Only relevant files and sinks are routed to each agent. Agents run in parallel because they have no dependencies on each other.

After scanning, findings go through a deduplication and corroboration phase. If static analysis and the LLM independently flag the same issue, that corroboration is a strong signal of a true positive. A dedicated false positive agent then reviews the findings — in one demo case, it correctly identified that a parameter flagged as a SQL injection vector was actually validated against an allow list before reaching the query.

▶ Watch: The Full Architecture and Demo (14:00)

The Demoralizing Moments (and Why They Matter)

In a notably candid section, Behrens and Cassel shared the operational failures that shaped their approach:

• Spending half a day building a Claude experiment that turned out not to support custom tools — a complete throwaway
• Watching a 45-minute scan reach completion only to see "auto compact at 0%" and wondering whether the entire context had been compressed into noise
• Watching token costs spiral on a scan that was producing beautiful coverage, knowing that someone was going to get a pricing alert
• Claude Code spontaneously turning into "a JavaScript pile for whatever reason" and losing state
• The moment Cassel pinged Behrens in the morning: "Hey, did you see Opus 4.6 drop?" — triggering an evaluation scramble just before their talk

"I get this ping from Justice in the morning. He's like, 'Hey, did you see Opus 4.6 drop?' And it's like, dun dun dun," Behrens recounted.

The Opus 4.6 story has a nuanced punchline. On small codebases, the new model was "absolutely fantastic." On larger codebases — those over a certain token threshold — it was less clear-cut. The team's eighteen months of evaluation work gave them the infrastructure to characterize that difference quickly, rather than having to rebuild from scratch when every major model update arrives.

▶ Watch: Demoralizing Moments and Opus 4.6 (10:00)

Responsible Disclosure as Validation

While the system is still being productionized, the Netflix team has been using a version of it for responsible disclosure of vulnerabilities in open-source projects. Cassel mentioned that they've been submitting PRs to popular projects — and in at least one case, a maintainer they expected to ignore the reports patched all of them. "Shout out to you."

This real-world use is a form of validation that benchmark scores alone can't provide. Finding bugs that actual open-source maintainers acknowledge and fix is a meaningful signal that the system isn't just producing impressive-looking lab results.

▶ Watch: Real-World Validation (10:00)

Context Injection and Esoteric Vulnerability Classes

One of the more interesting findings from the Q&A: for OWASP Top Ten vulnerabilities with a vanilla prompt, most capable models will find them. The challenge is esoteric vulnerability classes — time-of-check time-of-use (TOCTOU), race conditions, subtle business logic flaws.

"Very rarely have we actually seen Opus find meaningful time-of-check time-of-use without having time-of-check time-of-use in its prompt," Behrens noted. "When you think about the hundreds and hundreds of vulnerability classes, if you're only looking for OWASP Top Ten with a vanilla model, you'll find it. But the minute you start stepping into more esoteric vulnerabilities, providing that context in a vanilla prompt or in a workflow results in significantly better coverage based on the way the models were trained."

This ties directly back to the specialized agent design: each agent carries explicit context about the vulnerability class it's looking for, rather than hoping the model will generalize from a broad security prompt.

The team also touched on cost realities during Q&A: comprehensive scanning can run $400-500 per repo per scan at the high end. Thoughtful context management — the enumeration, discovery, and tracing stages — produces roughly a 26% cost reduction on most runs, which at that price point is meaningful.

▶ Watch: Q&A on Context and Cost (22:01)

Notable Quotes

"Consistency was difficult, variance was huge, false positives kept coming up. Sometimes it would ignore entire vulnerability classes."

— Justice Cassel, 00:00

"You might change one line in the prompt, it's a genius now. You change it again, it's terrible. You add one tool, it works great. You remove it, it ignores entire vuln classes, and it's kind of frustrating."

— 04:00

"My general recommendation: don't use categories."

— Scott Behrens, 08:00

"We're evaluating base models, but that's just not how folks are running them in production. We're often running them in workflows, with custom harnesses."

— 20:00

Key Takeaways

• Specialized agents per vulnerability class outperform generalists significantly. The Netflix benchmark showed 35/41 true positives from specialized agents versus ~20 from a monolithic super agent — a gap that makes generalists inadequate for production security tooling.
• Don't group vulnerability classes. Category-based grouping dropped coverage to 34/41 and appeared to cause model overfitting on category labels.
• Enumeration and tracing before the model saves ~26% on cost while improving quality. Deterministic filters and static analysis tools do the work that doesn't require a model.
• Evaluation infrastructure is non-negotiable. Every model update, every prompt change, every architectural tweak changes the performance profile. Without automated benchmarks against known vulnerabilities, teams are flying blind.
• Esoteric vuln classes require explicit context. OWASP Top Ten vulnerabilities surface from general prompts; TOCTOU, race conditions, and business logic bugs require specific context injected into each specialized agent.

Slides Reference

Slides available: 2026-04-04-D2-S2-11-17-Source-to-Sink-How-to-Improve-LLM-Firs.pdf

Note: The slides PDF for this session contains content from a different [un]prompted talk ("Operation Pale Fire" — a red team engagement by Block's offensive security team targeting the Goose AI agent). The slides cover indirect prompt injection via Google Calendar using invisible Unicode characters, the goose:// recipe URL handler as an attack vector, and associated mitigations including stripping non-standard Unicode characters, displaying recipe content before execution, and ML-based prompt injection detection (Meta's Llama-Prompt-Guard-286M, Protect AI's deberta-prompt-injection-v2, and Project Bashcat). The article above is based on the YouTube video transcript from Behrens and Cassel's Source to Sink session.