Evaluating Threats & Automating Defense: How Google is Advancing Code Security

Evaluating Threats & Automating Defense: How Google Is Advancing Code Security

Speakers: Heather Adkins, VP of Security Engineering, Google; Four Flynn, VP Security and Privacy, Google DeepMind

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=B_7RpP90rUk

Reading time: ~9 minutes

TL;DR

Google and Google DeepMind researchers presented two integrated AI projects — Big Sleep for autonomous vulnerability discovery and CodeMender for autonomous patch generation — with the explicit goal of eliminating every software vulnerability on Earth. Both systems achieve zero false positives through multi-stage verification pipelines, and together represent a blueprint for moving from "finding bugs" to "fixing software at the speed of AI."

Introduction

The National Vulnerability Database had a 30,000-entry backlog of unanalyzed vulnerabilities as of the morning of the conference. Between 2024 and 2025, logged vulnerabilities receiving a CVE grew by 35 percent — and that figure undercounts the true rate because not every discovered bug gets assigned a CVE. This is the backdrop against which Heather Adkins and Four Flynn took the stage to describe what Google and Google DeepMind are building.

Adkins, who leads security engineering at Google, and Flynn, who leads security and privacy for DeepMind, presented not as vendors pitching products but as researchers making an audacious claim: it is technically feasible, within the next few years, to create and maintain code that is relatively bug-free. The work they outlined — two research projects called Big Sleep and CodeMender — represents their attempt to get there.

▶ Watch: Setting the Scene (00:00)

The Coming Vulnerability Flood

Adkins opened by describing a landscape that is already breaking. CVSS scoring, the standard system for ranking vulnerability severity, will likely need to be redesigned — not because the rubric is wrong, but because the volume of vulnerabilities AI can discover will make severity ranking meaningless as a triage mechanism. There will simply be too many to rank.

The near-term trajectory she described: open-source frameworks for pen testing and finding vulnerabilities in web applications already exist. Approximately one billion dollars in venture capital has flowed into startups focused on vulnerability discovery, pen testing, red team simulation, and attack automation. "Between all the good guys and all the bad guys, I think it's fair to say we are at a point where we'll be able to find every vulnerability in every system using agentic frameworks. That is a completely different world."

The implication for defenders is stark: if attackers can find every vulnerability, the only path to safety is eliminating them faster than they can be exploited.

▶ Watch: The Vulnerability Flood and CVSS (02:00)

The mission statement Adkins articulated was blunt and deliberate: "We simply must eliminate every software vulnerability on Earth. That is the legacy that people in this room need to leave behind to the next generation of technologists." She added that as humanity moves beyond Earth — to space infrastructure, connected systems at civilizational scale — it is not acceptable to still be dealing with cross-site scripting bugs.

Big Sleep: Autonomous Vulnerability Discovery

Big Sleep is Google's agentic reasoning system for finding deep vulnerabilities — not the shallow bugs that a prompt to any modern LLM can surface, but the kind that a top elite vulnerability researcher would find after weeks of deep engagement with a codebase.

A 2017 RAND Corporation study established the benchmark Big Sleep is designed to beat: an expert vulnerability researcher takes approximately one month to find a deeply embedded bug, and roughly 22 days to successfully exploit it. Big Sleep is designed to do this in hours or minutes.

▶ Watch: Big Sleep Architecture (04:00)

The system was built by studying how elite researchers actually work. Three observations drove the architecture:

Deep codebase expertise. Expert researchers don't generalize across many codebases simultaneously — they go deep and niche in one or two areas. They understand the code architecture, study past vulnerabilities in that codebase, and perform variant analysis. Big Sleep's phase one replicates this: the system builds deep contextual understanding of a target codebase, including its bug history.

Agentic hypothesis-and-revision loop. Phase two is the reasoning loop. The system runs continuously, feeding back to itself. It builds hypotheses — based on past bugs, it might identify a region of code it suspects is vulnerable, then design a test for that hypothesis. To do this, it has access to a debugger, a code browser, and a Python interpreter it can use to write its own test scripts. "It is not just prompting itself — it is building code, building test cases, testing its own hypotheses, observing success and failure, and iterating." Adkins described the feel of watching this process as analogous to watching a top researcher standing in a corner with coffee, turning a problem over in their mind for hours.

Verification with zero false positives. Once the system produces a crash it believes may be a valid vulnerability, it enters a verification phase. It analyzes debugger feedback and actually builds an exploit — a proof-of-vulnerability. This step is the key to achieving zero false positives. Every bug that exits Big Sleep is not a candidate; it is a provably real vulnerability. A report is then written entirely by Gemini (not a human), including the proof-of-vulnerability code and a step-by-step explanation detailed enough for a non-expert developer to understand and act on.

As of the morning of the conference, all but five of the bugs Big Sleep had published to Google's issue tracker had been fixed.

▶ Watch: BigSleep Results and Hand-off to CodeMender (10:00)

CodeMender: Autonomous Patching and Code Hardening

Finding a vulnerability is half the job. The harder half — as any developer will attest — is writing a correct, maintainable fix. CodeMender, presented by Flynn, handles this second half.

Flynn framed the broader ambition: a complete end-to-end system where legacy code is discovered, patched, and hardened; and new greenfield code generated through "vibe coding" frameworks is secure by default. CodeMender is the patching and hardening component of that picture. (The greenfield security piece exists but wasn't covered in depth during the session due to time constraints.)

▶ Watch: CodeMender Introduction (10:00)

A good patch must satisfy three criteria, according to Flynn:

1. It must actually fix the security vulnerability.
2. It must not break the code's functionality.
3. As much as possible, it should honor the idioms of the original developer — so the patch is easy for them to understand and absorb.

CodeMender is built to meet all three. Given a verified vulnerability from Big Sleep, the LLM agent generates a set of patch candidates using the same tool environment — debugger access, hypothesis testing, code execution. But generating candidates is not enough.

The verification gauntlet. The "secret sauce" in CodeMender, as Flynn described it, is the multi-stage patch verification system:

• Fuzzing before and after: The code is fuzzed both before and after the patch to confirm that functionality has not been destroyed.
• Formal verification: The patched section is formally verified to prove functional equivalence with the original.
• Differential testing: The system fuzzes with the patch in place and injects malicious inputs to confirm the vulnerability is no longer present.
• LLM-as-judge: Finally, the LLM itself evaluates the patch against a carefully crafted prompt designed to assess accuracy.

If no candidates pass validation, the failures are fed back into the LLM's context window as learning signal, and the system generates a new round of candidates. Only patches that pass all verification stages are submitted to the open-source community.

▶ Watch: CodeMender Verification Pipeline (12:01)

The numbers as of the conference: 178 fixes autonomously generated through CodeMender and merged into open-source codebases. The team has worked through multiple libraries, with libwebp as one noted example, and has done significant work in Chrome — including automatically hardening pointer usage across those codebases through generated patches.

The Quality-First Trade-off

Flynn acknowledged a legitimate criticism of their approach: they are perhaps being too cautious, too committed to verification before releasing anything. This is a deliberate choice. "We believe developer trust in the system is what will drive adoption, and adoption is what will let us accelerate the community forward."

He also noted an unsolved adjacent problem: even if Big Sleep finds every vulnerability and CodeMender generates a correct patch, organizations struggle to apply patches in a timely manner. The world needs to be able to patch code at the speed of AI. "If you know how to solve that with AI, come find me."

▶ Watch: The Patch Adoption Problem (16:01)

Q&A Highlights

During the Q&A, an audience member asked how Big Sleep compares to OpenAI's recent work on automated vulnerability discovery. Neither Adkins nor Flynn had done a formal head-to-head comparison, though they noted the space is active. Another question probed whether the techniques apply to business logic vulnerabilities without a clear crash signal — both confirmed that the techniques generalize beyond memory safety, and that the team is actively building web vulnerability coverage. Importantly, Adkins noted that Big Sleep is finding things that OSS-Fuzz — Google's long-running fuzzing infrastructure — is missing, which confirms the system is operating at the right depth.

▶ Watch: Q&A Session (16:01)

Notable Quotes

"We will all look back on 2026 as the year everything pivoted — and we will also look back on it as a simpler time." — Heather Adkins (00:00)

"We simply must eliminate every software vulnerability on Earth. That is the legacy the people in this room need to leave behind to the next generation of technologists." — Heather Adkins (02:00)

"It's not enough to find vulnerabilities; we need to help developers patch code too. A 'vulnpocalypse' is coming, and that is a word you should take away from this conference." — Four Flynn (10:00)

"Big Sleep is finding things the fuzzers are missing. That tells us we're targeting at the right depth in the code." — Heather Adkins (18:01)

Key Takeaways

• The NVD has a 30,000-entry backlog of unanalyzed vulnerabilities, and logged CVEs grew 35% year-over-year from 2024 to 2025. The vulnerability discovery problem will require AI-scale solutions.
• Big Sleep recreates expert researcher behavior — not just prompting an LLM to find bugs, but building deep codebase knowledge, running hypothesis-and-revision loops with a debugger and Python interpreter, and achieving zero false positives through exploit-verified proofs.
• CodeMender uses a four-stage verification pipeline — fuzzing, formal verification, differential testing, and LLM-as-judge — to produce patches that are correct, non-breaking, and developer-idiomatic.
• 178 open-source fixes have been autonomously generated and merged through CodeMender so far, with libraries including libwebp and Chrome's pointer hardening as concrete examples.
• Developer trust is the adoption bottleneck. Google has deliberately prioritized high-verification outputs over speed, betting that trustworthiness will drive the adoption that accelerates the community.

Slides Reference

Slides from this session are available as 2026-04-04-D1-S1-9-22-Evaluating-Threats-Automating-Defense-H.pdf. Key topics in the slides include: the vulnerability growth data, the Big Sleep three-phase architecture (codebase understanding → agentic loop → verification), the CodeMender verification pipeline, and the complete end-to-end vision for legacy and greenfield code security.