From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs

Saurabh Kumar (Security Researcher · Microsoft Edge)

[un]prompted.au 2026 · Day 1 · Main Hall

Overview

In a compelling presentation at [un]prompted.au, Saurabh Kumar, a Security Researcher at Microsoft Edge, unveiled a novel approach to automated vulnerability research using Large Language Models (LLMs). His talk, "From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs," detailed a four-phase harness process designed to autonomously identify and filter memory corruption bugs in large codebases like WebKit. What began as a weekend project evolved into a sophisticated LLM-driven auditing system that successfully identified eight distinct vulnerabilities in WebKit, demonstrating the significant potential of AI in augmenting security research efforts.

Watch on YouTube

Visual summary for From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs by Saurabh Kumar
Visual summary for From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs by Saurabh Kumar

Key moments

  1. 0:00 Introduction: Automating bug finding with LLMs in WebKit
  2. 3:50 Introducing the four-phase LLM auditing harness
  3. 8:00 Adversarial 'Challenge' phase to filter false positives
  4. 9:40 Automated PoC generation and ASAN testing ('Prove' phase)
  5. 13:00 First bug example: WebRTC heap overflow to primitive
  6. 18:00 Summary: 8 bugs found, key success factors
  7. 20:00 Q&A: Handling long-horizon context pollution
  8. 22:30 Q&A: Strategy for managing false positives

From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs

Speakers: Saurabh Kumar, Security Researcher, Microsoft Edge

Conference: [un]prompted.au

YouTube: https://www.youtube.com/watch?v=kQZ3EbO4T7Q

Overview

In a compelling presentation at [un]prompted.au, Saurabh Kumar, a Security Researcher at Microsoft Edge, unveiled a novel approach to automated vulnerability research using Large Language Models (LLMs). His talk, "From Diff to Primitive: An LLM Auditor that Found 8 WebKit CVEs," detailed a four-phase harness process designed to autonomously identify and filter memory corruption bugs in large codebases like WebKit. What began as a weekend project evolved into a sophisticated LLM-driven auditing system that successfully identified eight distinct vulnerabilities in WebKit, demonstrating the significant potential of AI in augmenting security research efforts.

Kumar's primary motivation was to automate the discovery of security flaws with minimal human intervention, specifically targeting memory corruption issues due to their ease of automation and reproducibility with tools like ASAN. The methodology not only led to the discovery of multiple use-after-free bugs, including two race conditions in the GPU process, but also facilitated the development of two primitive exploits, showcasing the system's capability to move beyond mere crash detection to controlled exploitation. This talk provides a crucial insight into the practical application of LLMs in uncovering high-impact browser vulnerabilities and offers a glimpse into the future of security auditing.

The significance of this work lies in its ability to address the pervasive challenge of sifting through numerous false positives generated by traditional fuzzing or early LLM-based analysis. By implementing a multi-phase, adversarial auditing process, Kumar's system effectively filters out weak findings, delivering actionable bug reports that are both reachable and exploitable. This innovative approach underscores the evolving role of AI in streamlining and enhancing the efficiency of vulnerability research, making it a critical development for both offensive and defensive security practitioners.

Background

▶ Watch: Introduction: Automating bug finding with LLMs in WebKit (0:00)

The genesis of this research stemmed from a desire to leverage LLMs for automated bug hunting, with a specific focus on high-impact vulnerabilities in complex software. Saurabh Kumar chose WebKit as his primary target due to its intricate architecture and pervasive use, making it a high-value target for security research. His scope was deliberately narrow: to identify memory corruption issues—such as out-of-bounds access and lifetime bugs—which are particularly amenable to automated detection and reproduction using tools like ASAN (AddressSanitizer). Less critical issues, such as denial-of-service (DOS) bugs, asserts, or null dereferences, were consciously excluded to maintain focus on "good bugs" with higher security impact.

Before developing the LLM-driven harness, Kumar established a robust knowledge base for WebKit. This involved creating a high-level architectural threat model encompassing process boundaries, Inter-Process Communications (IPCs), and the interactions between web content, UI, and GPU processes. Additionally, he built a vector database (vector DB) by scraping historical WebKit bugs, particularly those related to the WebJS attack surface. This vector DB served as a crucial resource, allowing the LLM agent to perform semantic searches for invariants and learn from past vulnerabilities when encountering new code.

The core problem Kumar aimed to solve was the inefficiency inherent in running generic LLM agents on vast codebases. Such agents often generate an overwhelming number of reports, many of which claim valid bugs but are ultimately unreachable or blocked by runtime guards—leading to significant manual verification overhead. The overarching goal was to filter out weak findings and produce only actionable bugs that could be directly reported, thereby minimizing the manual effort required for validation. This objective led to the development of a structured, four-phase harness process designed to progressively refine and validate potential vulnerabilities.

Key Findings

▶ Watch: Adversarial 'Challenge' phase to filter false positives (8:00)

Saurabh Kumar's LLM auditor proved remarkably effective, leading to the discovery and reporting of eight distinct vulnerabilities in WebKit. These findings highlight the significant potential of LLMs in automated vulnerability research, particularly when guided by a structured, multi-phase approach.

Among the eight bugs, the system initially reported five use-after-free vulnerabilities without requiring any primitive exploitation work. Notably, two of these were race condition use-after-frees found in the GPU process, demonstrating the auditor's ability to identify complex concurrency issues beyond the typical web content process.

A key contribution of this research was the ability to transition from mere crash detection to the development of primitive exploits. Kumar successfully demonstrated two such primitives:

  1. A heap overflow in WebRTC, which granted register control and a write primitive, ultimately leading to a PAC (Pointer Authentication Code) fail.
  2. A type confusion bug in TransformStream, which also resulted in arbitrary read/write capabilities and a subsequent PAC fail. This bug could additionally be converted into a heap buffer overflow.

The success of the LLM auditor was attributed to several critical design choices:

  • Multi-model Approach: The harness strategically employed different LLM models for specific phases: GPT-5.4 for initial commit scanning, Opus 4.6 for auditing and proof-of-concept generation, and a combination of Opus 4.6 and GPT-5.5 for the adversarial debate phase. Kumar explicitly stated that no cyber-specialized models were used, underscoring the effectiveness of general-purpose LLMs when properly orchestrated.
  • Structured Harness: The four-phase Prepare, Find (Audit), Challenge (Debate), and Prove (PoC) process was instrumental in progressively filtering false positives and validating findings. The Challenge phase, in particular, leveraging a prosecutor-defender-judge model, significantly improved the quality of reported bugs.
  • Robust Knowledge Base: The creation of a detailed WebKit threat model, including process boundaries and execution flows, combined with a vector database of historical bugs, was crucial. This knowledge base allowed the LLMs to understand the target environment deeply and learn from past vulnerability patterns, providing valuable context when the models encountered new code or got "stuck."

In summary, the project successfully demonstrated that LLMs, when integrated into a carefully designed and automated auditing framework, can effectively discover and validate high-impact memory corruption vulnerabilities in complex software like WebKit, even progressing to the point of primitive exploitation.

Technical Deep Dive

▶ Watch: First bug example: WebRTC heap overflow to primitive (13:00)

The core of Saurabh Kumar's LLM auditor is a sophisticated four-phase harness process: Prepare, Find (Audit), Challenge (Debate), and Prove (PoC). Each phase is designed to perform specific tasks, progressively filtering out weak findings and refining the identified vulnerabilities. A key aspect of this design is the multi-model approach, where different LLMs are strategically employed based on the task's complexity and requirements. Kumar used GPT-5.4 for fast, deterministic scans in the prepare phase, Opus 4.6 for the more complex auditing and proof generation, and a combination of Opus 4.6 and GPT-5.5 for the adversarial "prosecution and judge" roles. He emphasized that no cyber-specialized models were used, relying instead on the capabilities of general-purpose LLMs.

Phase 1: Prepare

The Prepare phase initiates the audit process by identifying interesting areas within the WebKit codebase. It features a commit scanner tool that, when pointed to a specific component directory, analyzes WebKit's commit history. The scanner's objective is to find "interesting commits"—those that involve security fixes, expose new web APIs, or introduce complex changes. A smaller, faster model, GPT-5.4, is used here to rank these commits based on their potential security relevance.

Once interesting commits are identified, the system creates clusters of files associated with them. For example, if a commit exposes a new web API, the scanner will group all related files into a cluster. This clustering mechanism ensures that the subsequent audit phase focuses on relevant and potentially vulnerable code segments. Kumar provided an example of a commit from 2025 (likely a typo, perhaps 2015 or current year for a future fix) that added constructors for RTCEncodedAudioFrame and RTCEncodedVideoFrame. This commit involved 22 changed files and was ranked highly (e.g., 18.6 threshold, with the LLM giving a score of 30) due to its involvement in WebRTC and the direct exposure of a web API to JavaScript. The high ranking was also influenced by the vector DB, which indicated a history of bugs in WebRTC. This cluster was then passed to the next phase.

Phase 2: Find (Audit)

In the Find (or Audit) phase, the LLM harness receives the file clusters identified in the Prepare phase. This phase leverages Opus 4.6 and is equipped with specific "skills" tailored to focus on memory corruption issues, such as out-of-bounds (OOB) and lifetime issues. The prompt instructs the LLM to search for these types of bugs within the provided code clusters.

If Opus 4.6 identifies a potential bug, it generates a report. Crucially, a sub-agent is then launched to analyze the reported finding. This sub-agent's role is to trace the executable code flow from source to sink, determining if the bug is actually reachable from an attacker-controlled input. If the bug is deemed reachable, the sub-agent scores it and sends it to the next, more rigorous, Challenge phase. This step is vital for filtering out theoretical vulnerabilities that cannot be triggered in practice.

Phase 3: Challenge (Debate)

The Challenge phase is an innovative and critical component of the harness, designed to drastically reduce false positives. Instead of simply asking "is this a valid issue?", Kumar implemented an adversarial debate system with three distinct roles, using Opus 4.6 and GPT-5.5:

  • Prosecutor: This model, which initially found the bug, argues why the finding is valid and provides supporting evidence from the code.
  • Defender: This model actively attempts to disprove the bug's validity. It scrutinizes the code for any runtime checks, guards, or other mechanisms that might block the bug's exploitation. Kumar shared an example where, for a WebRTC bug, the defender identified a setter limit already set on the JavaScript side, preventing pre-allocation of packet buffers.
  • Judge: This model evaluates the reports from both the prosecutor and the defender, cross-referencing their claims with the actual code. It scores the arguments and, if the finding is determined to be invalid or unexploitable, it is dropped at this stage. In the WebRTC example, the prosecutor countered the defender by finding an alternative path through the encoding transform stream to bypass the setter limit, demonstrating the dynamic nature of this debate. Only bugs that survive this rigorous adversarial review proceed to the Prove phase.

Phase 4: Prove (PoC)

The final automated phase, Prove (or PoC), aims to generate a working proof-of-concept for the validated bugs. In this phase, an environment is set up with a WebKit mini browser ASAN build—a version of WebKit compiled with AddressSanitizer to detect memory errors.

The proof agent, powered by Opus 4.6, receives the artifacts of the validated bug. Its task is to write a simple WebJS PoC or a WebKit layout test that triggers the vulnerability. This PoC is then executed against the MiniBrowser in ASAN mode. The system allows for a maximum of 10 tries for PoC generation. If an attempt fails, the agent records the reason for failure and what was missing, then attempts to generate a new PoC. If any of the attempts successfully trigger a crash in the ASAN build, the bug is confirmed and sent to a dashboard, marking the end of the automated discovery process. This phase ensures that only truly reproducible and exploitable bugs are reported.

Demo / Proof of Concept

▶ Watch: Summary: 8 bugs found, key success factors (18:00)

While the initial bug discovery and PoC generation up to crash detection were largely automated, the process of developing primitive exploits required a human-in-the-loop approach. Saurabh Kumar explained that Apple, for instance, often requires primitive PoCs (e.g., register control, arbitrary read/write) for fast-tracking bug fixes and awarding bounties, rather than just a crash report.

To achieve this, Kumar developed a semi-automated process for primitive exploitation:

  • Goal-Oriented Skills: He created specific "skills" for the LLM, starting with a basic goal like achieving register control. If that was met, the next skill would aim for arbitrary read/write.
  • Tool Access: The LLM was given access to tools, including an LLDB script, which could launch and debug the MiniBrowser. Crucially, for primitive development, the MiniBrowser had to be run in non-ASAN mode, as ASAN's early crash detection could prevent the exploit chain from fully developing.
  • Iterative Analysis: After each crash, the system would generate an IPS file (Apple's crash report format). The LLM would then read and analyze this IPS file, extracting information such as fault addresses, register states, and PAC (Pointer Authentication Code) details. If the current attempt didn't achieve the set goal, the LLM would save the attempt's details and try an alternative path. Manual intervention was sometimes required to re-guide the agent if it diverged from the exploitation goal.

Kumar demonstrated two specific bugs that achieved primitive exploitation:

  1. WebRTC Heap Overflow (CSRC Metadata):
  • Vulnerability: Located in the WebContent process, this was a relatively simple heap overflow. The issue arose when JavaScript-controlled CSRC metadata values were processed. A vector was initially created to hold 'n' values. However, instead of filling these 'n' slots, the code then pushed back another 'n' entries, resulting in a total of 2n entries. On the libWebRTC side, this led to writing past a fixed-size buffer.
  • Root Cause: The critical flaw was the presence of a RTC_DCHECK (a debug-only check) that verified the size. This DCHECK was removed in release builds, leaving the vulnerability exposed.
  • Exploitation: By crafting a CSRC metadata of 512 length, Kumar was able to write past the packet buffer. This bug ultimately allowed for register control, a write primitive, and led to a PAC fail.
  1. TransformStream Type Confusion:
  • Vulnerability: This bug involved a type confusion in TransformStream. The native code expected three stream objects, but the size was only checked by an assert (another debug-only check). Furthermore, there was no downcast checking to verify the object's actual type during type casting.
  • Root Cause: The combination of an assert for size validation and the lack of type verification during downcasting created two distinct vulnerabilities within the same context.
  • Exploitation: Using a JavaScript iterator, Kumar was able to trigger a crash and control a write increment. This led to a virtual call through corrupted data and eventually a PAC fail. The bug could be converted into both a heap buffer overflow and a type confusion leading to arbitrary read/write.

Similarities in Exploitable Bugs: Kumar highlighted that both primitive bugs shared key characteristics:

  • Both relied on debug checks (asserts or DCHECKs) that were absent in release builds.
  • In the Debate phase, the LLM found alternative paths to reach the vulnerable code (e.g., encoding transform stream for WebRTC, JS iterator for TransformStream) when direct routes were blocked. These patterns were recorded and fed back into the harness's knowledge base to improve future bug detection.

Defensive Implications

▶ Watch: Q&A: Strategy for managing false positives (22:30)

The research presented by Saurabh Kumar offers several critical defensive implications for software developers, security teams, and browser vendors:

  1. Reliance on Debug-Only Checks is Dangerous: The discovery of two primitive bugs directly attributable to assert and DCHECK statements being stripped in release builds is a stark reminder. Developers must understand that debug-only checks offer no security guarantees in production environments. All security-critical size checks, type checks, and boundary validations must be implemented with robust, always-on mechanisms that persist in release builds.
  1. Comprehensive Type Checking is Essential: The TransformStream type confusion highlights the need for rigorous type checking, especially during downcasting operations in C++ code. Relying solely on assertions or assuming correct types can lead to critical vulnerabilities. Defensive programming should include explicit runtime type checks (e.g., dynamic_cast with null checks, or custom type ID checks) when dealing with polymorphic objects or untrusted input.
  1. LLM-Augmented Auditing is a Future Threat (and Defense): The success of this LLM auditor demonstrates that AI can significantly accelerate vulnerability discovery, moving from simple crashes to primitive exploits. Defenders must recognize that attackers will increasingly leverage similar LLM-driven tools. Consequently, adopting LLM-based security auditing internally can be a proactive defense, allowing organizations to find and fix bugs before attackers exploit them. The "debate" phase, in particular, offers a robust model for filtering out false positives, which is a common challenge in automated security tools.
  1. Focus on Reachability and Exploitability: The harness's emphasis on ensuring bug reachability and PoC generation (even for crashes) before reporting is a valuable lesson. Security teams should prioritize fixing issues that are demonstrably reachable and exploitable over theoretical or difficult-to-trigger flaws. This helps allocate limited remediation resources effectively.
  1. Robust Threat Modeling and Knowledge Bases: The foundational work of building a WebKit threat model and a vector database of historical bugs was crucial for the LLM's success. Defenders should invest in similar knowledge management for their own codebases. Understanding process boundaries, execution flows, and past vulnerability patterns can guide both manual and automated security assessments, making them more effective.
  1. Adversarial Thinking in Design: The "prosecutor-defender-judge" model is an excellent example of integrating adversarial thinking into the security testing process. Developers and security architects could apply similar principles during design reviews, actively challenging assumptions and seeking alternative attack paths, much like the LLM's defender-prosecutor interaction.

In essence, Kumar's work underscores that the era of AI-driven vulnerability research is here. Defenders must adapt by strengthening their coding practices, embracing AI as a defensive tool, and adopting more sophisticated, multi-layered validation processes to stay ahead of evolving threats.

Key Takeaways

  • LLMs are powerful vulnerability research tools: General-purpose LLMs, even without specialized cyber training, can effectively discover and validate complex memory corruption bugs in large codebases like WebKit.
  • Structured, multi-phase auditing is crucial: A four-phase harness (Prepare, Find, Challenge, Prove) significantly reduces false positives and ensures the generation of actionable, reachable bug reports.
  • The "Debate" phase is a game-changer for validation: An adversarial model (Prosecutor, Defender, Judge) dramatically improves the quality of LLM-identified vulnerabilities by rigorously challenging their validity and reachability.
  • Debug-only checks are security liabilities: Relying on assert or DCHECK for critical size or type validations in development means these protections are absent in release builds, creating exploitable vulnerabilities.
  • Primitive exploitation still benefits from human guidance: While LLMs can generate crash PoCs, transforming these into full register control or arbitrary read/write primitives currently requires iterative human-in-the-loop analysis and goal setting.
  • Knowledge base and threat modeling are foundational: A well-defined threat model, understanding of execution flows, and a vector database of historical bugs are essential for guiding LLMs and maximizing their effectiveness in vulnerability discovery.

About the Speaker(s)

Saurabh Kumar is a Security Researcher on the Microsoft Edge browser security team. His work primarily focuses on vulnerability research within browsers and web security. Kumar's presentation at [un]prompted.au showcased his practical, innovative approach to leveraging Large Language Models for automated bug finding, demonstrating a deep understanding of browser internals and exploitation techniques. His commitment to automating security research, as evidenced by this project, underscores his expertise in advancing the field of vulnerability discovery.

Reviews

Dr. Zero (Offensive Security Researcher) — MUST SEE

This is how you do AI-assisted vulnerability research without the marketing bullshit. Kumar didn't just throw Opus at WebKit and pray; he built a multi-agent adversarial harness that actually solved the LLM false-positive problem and spat out eight real CVEs. If you want to know what the next generation of automated VR looks like, watch this talk.

Heather Calloway (CISO) — MUST SEE

A Microsoft Edge researcher automated the discovery and primitive exploitation of 8 WebKit CVEs using general-purpose LLMs. It proves that the barrier to entry for finding browser zero-days is collapsing. You don't need to send your board the technical slides, but your risk committee needs to understand that AI is fundamentally altering the economics of offensive research.

→ Top-rated talks at [un]prompted.au 2026

All talks from [un]prompted.au 2026