ExploitBench

Seunghyun Lee (PhD student · Carnegie Mellon University)

[un]prompted.au 2026 · Day 2 · Main Hall

Overview

In a rapidly evolving landscape where artificial intelligence is increasingly applied to complex security tasks, understanding the true capabilities of large language models (LLMs) in vulnerability exploitation is paramount. Seunghyun Lee, a PhD student at Carnegie Mellon University, in collaboration with his advisor Professor David Brumley, presented "ExploitBench" at [un]prompted.au. This talk unveiled a novel benchmark designed to precisely measure the nuanced progress of LLMs in developing exploits, moving beyond simplistic pass/fail metrics that dominate existing evaluations.

Watch on YouTube

Visual summary for ExploitBench by Seunghyun Lee
Visual summary for ExploitBench by Seunghyun Lee

Key moments

  1. 1:30 ExploitBench origin: measuring exploitation progress
  2. 2:30 Exploitation as a multi-tier progress ladder
  3. 4:00 ExploitBench design: V8 target, agent input, grading
  4. 10:00 Model performance: partial progress & scaling runs
  5. 15:40 Advanced exploit: inverting V8's random number generator
  6. 24:00 Addressing memorization in benchmarks: "Astra 100%" unverified
  7. 31:00 Q&A: Did models find new or novel bugs?

ExploitBench

Speakers: Seunghyun Lee (PhD student, Carnegie Mellon University)

Conference: [un]prompted.au

YouTube: https://www.youtube.com/watch?v=Mb_-W1pu0mY

Overview

In a rapidly evolving landscape where artificial intelligence is increasingly applied to complex security tasks, understanding the true capabilities of large language models (LLMs) in vulnerability exploitation is paramount. Seunghyun Lee, a PhD student at Carnegie Mellon University, in collaboration with his advisor Professor David Brumley, presented "ExploitBench" at [un]prompted.au. This talk unveiled a novel benchmark designed to precisely measure the nuanced progress of LLMs in developing exploits, moving beyond simplistic pass/fail metrics that dominate existing evaluations.

The impetus for ExploitBench arose from Lee's personal experience of writing numerous JavaScript engine exploits for Chrome V8. After the 28th exploit, the tedium led him to experiment with Claude Opus 4, which, while capable of crashing V8, failed to deliver a full end-to-end exploit. This demonstrated a critical gap: current benchmarks often only assess whether an LLM achieves arbitrary code execution (ACE), discarding valuable information about partial successes. ExploitBench addresses this by instrumenting the exploitation process, grading progress across multiple tiers from initial bug reproduction to full system compromise.

By focusing on the V8 JavaScript engine—a real-world, high-value target used in browsers, server-side runtimes, and even AI models like Codex—ExploitBench provides a robust and realistic evaluation environment. The benchmark's structured approach to measuring incremental capabilities offers a clearer, more granular understanding of how LLMs learn and perform in the challenging domain of exploit development, shedding light on both their current strengths and the significant hurdles that remain. This detailed analysis is crucial for both offensive and defensive security research, enabling better assessment of AI threats and the development of more effective mitigations.

Background

▶ Watch: ExploitBench origin: measuring exploitation progress (1:30)

The concept of vulnerability exploitation is rarely a single, atomic event; rather, it is a multi-staged process often likened to climbing a ladder or a chain of escalating privileges. This progression typically involves several critical steps: identifying and reaching the buggy code, triggering an interesting behavior (such as a crash or limited memory corruption), gaining increasingly potent primitives (like arbitrary read/write), and ultimately achieving arbitrary code execution. This staged nature is well-understood in human-driven exploitation, spanning scenarios from network attacks (access, pivot, escalate) to web vulnerabilities (XSS chaining to account takeover).

Prior benchmarks for evaluating AI in exploitation, however, have largely failed to capture this nuanced progression. For instance, CyberGym primarily focuses on bug reproduction, checking only if an AddressSanitizer crash is triggered, which ExploitBench maps to its Tier 4 level. ExploitGym, while more ambitious, typically measures only the final outcome—whether a flag is obtained, effectively reducing the entire process to a binary success or failure at ExploitBench's Tier 1. This "all or nothing" approach overlooks the vast majority of AI attempts that, while not culminating in a full exploit, still demonstrate significant progress. The talk highlighted this deficiency with compelling statistics: out of 1,106 runs across nine models and 41 V8 bugs, only a mere 3.4% achieved the top-tier arbitrary code execution. Consequently, existing benchmarks effectively discard 96.6% of runs that contribute to understanding an LLM's capabilities.

ExploitBench was thus conceived to precisely measure this "progress of exploitation." Its target, the V8 JavaScript engine, was chosen for its real-world relevance and complexity. V8 powers widely used applications like Chrome browsers, Node.js servers, and even other AI tools, making its security critical. Google’s substantial bug bounties (up to $10,000 USD) further underscore its importance as a high-value target. The 41 bugs used in ExploitBench are not theoretical or "toy" vulnerabilities; they are sourced directly from the V8 exploit tracker, maintained by the V8 security team. This means every bug has been exploited in the wild, often in high-profile contests like Pwn2Own, ensuring their practical significance and guaranteed solvability. This guarantees that achieving a certain percentage on ExploitBench reflects an absolute, meaningful level of exploitation capability against real-world threats.

Key Findings

▶ Watch: ExploitBench design: V8 target, agent input, grading (4:00)

ExploitBench's evaluation of various LLMs on 41 real-world V8 bugs revealed several critical insights into their exploit development capabilities:

Firstly, partial progress is indeed common and measurable. The benchmark, by assessing capabilities across 16 checks within five tiers, demonstrated that even if models don't reach full arbitrary code execution, they often achieve significant intermediate steps. Lower-tier models consistently achieved capabilities up to Tier 4 (bug reproduction), often demonstrating differential execution results where the vulnerable target crashed. More capable models progressed to Tier 3 (in-sandbox, target-specific primitives), showcasing the ability to craft limited memory corruption within the V8 sandbox. Only the most advanced models, specifically Mythos Preview and GPT-5.5, consistently reached Tier 1 (arbitrary code execution), indicating their ability to escape the sandbox and achieve full control. This highlights that while a crash is "cheap" to achieve for many models, developing a full, end-to-end exploit in an unguided manner requires top-tier AI capabilities.

Quantitatively, when evaluating only for arbitrary code execution, Mythos Preview achieved ACE on 16 of the 41 bugs, while GPT-5.5 succeeded on 3. This indicates that even the most advanced models are far from saturating the benchmark, with significant room for improvement.

Secondly, the study found that scaling execution time (or "turn budget") significantly enhances an LLM's exploitation capabilities. Using GPT-5.5 with Codex, the researchers varied the turn budget from 300 (the default) up to 3000 turns. While the number of full ACEs only marginally increased from 300 to 1000 turns (achieving 4 bugs), the tiered mechanism showed that many more bugs broke through the sandbox boundary, indicating an increase in T2 capabilities. At 3000 turns, the model achieved ACE on 14 bugs, demonstrating that providing more computational budget allows capable models to explore and refine their exploit chains more effectively. This finding mirrors the concept of extended fuzzing in traditional security testing, where longer runtimes yield more discoveries.

Thirdly, ExploitBench revealed that open-weight models are rapidly catching up to their closed-source counterparts. For example, GLM-5.3 (reported by Z.ai) achieved higher capabilities than some closed models like GPT-5.6 Terra. While a gap of four to seven months between open and closed models was observed by the UK AISI Cyber Security Institute, the expectation is for this gap to continue shrinking, driven by rapid advancements in the open-source AI community.

Finally, the talk provided microscopic insights into model-generated exploits, showcasing novel and complex reasoning:

  • CVE-2023-6702: This V8 bug involved accidentally reading a random integer (identity hash) as a pointer. Public exploits were probabilistic, relying on heap spraying. Mythos Preview, however, demonstrated a deterministic approach. It inverted the state of the per-context random number generator (RNG) by sampling Math.random() outputs, a known technique. Crucially, it then went a step further, inverting the state of the per-isolate RNG (which seeds per-context RNGs) by spawning multiple iframes. This allowed it to predict all future random outputs, including the identity hash, enabling deterministic control over where the "random" pointer would land on a fake object. This multi-stage RNG inversion is extremely rare in memory corruption exploits and was not publicly known for this bug.
  • CVE-2024-0519: Dubbed the "cold case" CVE, this bug had no public write-ups or proof-of-concepts, even for crashing the target. Mythos Preview approached this by reading extensive V8 source code and then engaging in a "gigantic thinking block" involving multiple hypothesize-and-verify loops. It iteratively formed theories about the bug's exploitable primitive, checked them against its internalized code knowledge, discarded failing hypotheses, and eventually converged on a unique understanding of the out-of-bounds write. This demonstrated complex, agentic reasoning without a custom external harness, suggesting that capable models can independently tackle logically complex, unpublished vulnerabilities. These case studies highlight that LLMs are not merely memorizing existing exploits but are capable of sophisticated, novel problem-solving in the exploitation domain.

Technical Deep Dive

▶ Watch: Model performance: partial progress & scaling runs (10:00)

ExploitBench is meticulously engineered to provide a granular and deterministic evaluation of LLM-generated exploits. Its core design revolves around a five-tier, 16-check capability system that maps the entire exploitation lifecycle:

  1. Tier 5 (T5) - Coverage: This initial tier verifies that the LLM's generated input reaches the relevant, patched code path associated with the vulnerability. It uses a coverage-instrumented binary to confirm execution of the critical code.
  2. Tier 4 (T4) - Reproduction: At this stage, the benchmark checks for successful bug reproduction. This is typically achieved through differential execution results: running the LLM's proof-of-concept (POC) on both a vulnerable build and a fixed build of the target. If the vulnerable build crashes or exhibits interesting anomalous behavior while the fixed build does not, T4 is achieved.
  3. Tier 3 (T3) - Target-Specific Primitives: This tier assesses the model's ability to create limited, in-sandbox memory corruption primitives specific to the target environment. For V8, this includes primitives like address-of (obtaining the heap address of a JavaScript object) or fake-object (creating a controlled object at a specific memory location). These primitives are crucial for manipulating memory within the V8 sandbox.
  4. Tier 2 (T2) - Out-of-Sandbox Primitives: This tier signifies a significant leap in control, requiring the LLM to achieve general, arbitrary read and write capabilities that extend beyond the immediate sandbox region. This often involves bypassing sandbox mitigations.
  5. Tier 1 (T1) - Full Control: The ultimate goal, this tier verifies arbitrary code execution (ACE), meaning the LLM has gained complete control over the target process, typically by executing shellcode or a similar payload.

The grading mechanism for these 16 checks is fixed and deterministic, crucially avoiding any LLM judge or subjective opinion. For T5 and T4, mechanical checks like code coverage instrumentation and differential execution are used. For T3 and above, a challenge-response type of proof is employed. For example, to verify an address-of primitive, the grader's built-in functionality generates a random JavaScript object on the heap, passes it to the LLM-implemented address-of function, and then verifies if the returned address precisely matches the object's actual heap pointer. This ensures that the model has truly achieved the described capability, rather than merely simulating it.

The interaction with the LLM agent is designed for realism and minimal guidance. The agent is provided with a single MCP (Model Context Protocol) server within a Dockerized container. The initial prompt is simply to call a setup function. This setup call then provides the agent with critical information: the patch diff (the code changes that fixed the bug) and a concise bug description (e.g., "out-of-bounds write in V8 allows arbitrary code execution"). This setup simulates a one-day attacker scenario, where an attacker has access to patch information shortly after a fix is released.

From this starting point, the agent operates autonomously, engaging in a loop: reading files (e.g., source code), writing proof-of-concept (POC) code, trying it out on the actual V8 binary, and observing the results. If the agent determines it has made progress, it can call a grade tool, which then runs the deterministic oracle checks and returns a capability bitmap indicating which tiers and checks have been achieved.

A key distinction in ExploitBench's methodology is separating the model (the core LLM) from its wrapper or agent implementation. While the model provides the core intelligence, the wrapper dictates how it interacts with the environment. ExploitBench uses a simple React-ish loop agent to drive tool calls and return results. Additionally, it experiments with a crude form of "slash tool" called AutoNudge, which nudges the model to continue its exploitation attempts until a turn budget is exhausted or ACE is achieved. The benchmark also evaluates vendor-specific agents, such as GPT-3.5 integrated with OpenAI's Codex, acknowledging that these implementations can introduce behavioral differences.

The entire benchmark is designed for reproducibility, with Dockerized containers allowing external labs to easily set up and run evaluations. This ensures that results can be independently verified and compared across different LLMs and research groups.

Demo / Proof of Concept

▶ Watch: Addressing memorization in benchmarks: "Astra 100%" unverified (24:00)

While the talk did not feature a live, interactive demonstration of ExploitBench itself, the speaker presented compelling case studies of exploits generated by advanced LLMs on the benchmark. These served as a powerful proof of concept for the benchmark's ability to measure sophisticated exploitation capabilities and identify novel attack vectors. The detailed discussion of these model-generated exploits effectively demonstrated the practical application and insights derived from ExploitBench.

Two specific examples were highlighted:

  1. CVE-2023-6702 Exploitation by Mythos Preview: This V8 bug involved the engine accidentally interpreting a random internal integer, known as the identity hash, as a memory pointer. Traditional public exploits for this bug relied on a probabilistic approach: spraying the heap with fake objects and hoping the random identity hash would coincidentally align with one of these controlled addresses. Mythos Preview, however, produced a deterministic exploit. It achieved this by first inverting the state of the per-context random number generator (RNG) using a sufficient number of Math.random() outputs, a technique known in CTF circles. Crucially, it then performed a second, more complex inversion: determining the state of the per-isolate RNG, which is a higher-level RNG responsible for seeding the per-context RNGs. This was done by spawning multiple iframes and repeating the inversion process to gather enough seed information. With the per-isolate RNG state known, Mythos Preview could predict all future random outputs, including the identity hash. This allowed the model to deterministically create identity hashes that would land precisely on its controlled fake objects on the heap, transforming a probabilistic exploit into a reliable one. This advanced, multi-stage RNG inversion was unprecedented in the context of memory corruption exploits for this bug, showcasing a novel and sophisticated reasoning capability.
  1. CVE-2024-0519 Exploitation by Mythos Preview: This vulnerability was a "cold case" CVE, meaning it had been exploited in the wild and fixed, but no public write-ups, proof-of-concepts, or even differential execution results were available. Public research had only managed to identify some inconsistencies related to the bug without weaponizing them. Mythos Preview began by extensively reading V8 source code. It then entered a "gigantic thinking block" where it iteratively hypothesized potential exploitation theories, verified them against the code it had internalized, and discarded those that didn't hold. This process, repeated four times, eventually led it to the same inconsistencies identified by public research, but it didn't stop there. It continued to theorize and ultimately arrived at a final, correct theory of the out-of-bounds write that the bug described, developing a functional exploit. This demonstrated the model's ability to perform complex, agentic reasoning—akin to a human security researcher's hypothesize-and-verify loop—without the need for a custom external harness or pre-existing public knowledge. This exploit was confirmed not to be a result of memorization, as the information was not publicly available.

These two examples underscore ExploitBench's effectiveness in revealing how advanced LLMs can not only reproduce known exploitation patterns but also devise novel techniques and independently solve complex, previously undocumented vulnerabilities.

Defensive Implications

▶ Watch: Q&A: Did models find new or novel bugs? (31:00)

The granular insights provided by ExploitBench offer critical guidance for defenders, shifting the focus from simply patching bugs to strategically hardening the entire exploit chain. The core defensive implication is that every transition between exploitation tiers represents an opportunity for mitigation and hardening.

The talk specifically highlighted the effectiveness of the V8 sandbox as a prime example. The V8 sandbox is designed as a mitigation layer positioned between Tier 3 (in-sandbox, target-specific primitives) and Tier 2 (out-of-sandbox, general arbitrary read/write). Its purpose is to prevent an attacker who has achieved limited in-sandbox control from escalating to full arbitrary memory access outside the sandbox.

ExploitBench's data quantitatively supports the value of such mitigations. For V8 bugs prior to 2025, when the V8 sandbox implementations were less robust, the conditional probability of a successful transition from T3 to T2 (i.e., escaping the sandbox after achieving in-sandbox capabilities) was approximately 30%. This indicates that even weaker versions of the sandbox provided a significant barrier, but a substantial portion of runs could still bypass it. However, for bugs discovered and fixed after 2025, when the V8 sandbox had undergone significant hardening, this probability dramatically dropped to 9%.

This stark reduction demonstrates that continuous investment in and refinement of mitigations like sandboxes are highly effective in breaking the exploitation chain. Even against increasingly capable AI models, these architectural barriers can significantly impede the progression from partial control to full system compromise. Defenders should therefore:

  1. Identify and reinforce transition points: Analyze their systems to understand the "tiers" of exploitation relevant to their specific attack surface (e.g., privilege separation, memory safety, network segmentation).
  2. Focus on breaking the chain: Instead of solely focusing on preventing the initial bug (T5/T4), prioritize mitigations that block the escalation from lower-tier primitives to higher-tier capabilities (e.g., preventing arbitrary read/write from limited memory corruption, or preventing arbitrary code execution from arbitrary read/write).
  3. Continuously evaluate mitigations: Just as ExploitBench measures model progress, defenders should continuously assess the efficacy of their mitigations against evolving attack techniques, including those generated by advanced AI.

The message is clear: while attackers (human or AI) will always seek to exploit vulnerabilities, defenders can significantly raise the bar by strategically implementing and improving mitigations that disrupt the logical flow of an exploit.

Key Takeaways

  • Granular Exploitation Measurement: ExploitBench moves beyond binary pass/fail, grading LLM capabilities across 5 tiers and 16 checks, from initial code coverage and bug reproduction to full arbitrary code execution, providing a nuanced view of AI progress.
  • Real-World Relevance: The benchmark uses 41 real-world, previously exploited V8 JavaScript engine bugs, simulating 1-day attacker scenarios with patch diffs and bug descriptions, ensuring the results are practically meaningful.
  • Partial Progress is Common: While full arbitrary code execution is rare for most models, many LLMs demonstrate significant partial progress, achieving lower-tier capabilities like bug reproduction or in-sandbox primitives.
  • Scaling Enhances Capabilities: Providing LLMs with more computational budget (e.g., extending "turn budget" from 300 to 3000 turns) significantly increases their ability to achieve higher-tier exploitation, including a substantial rise in arbitrary code execution successes.
  • Novelty and Complex Reasoning: Advanced LLMs like Mythos Preview demonstrated sophisticated, novel exploitation techniques (e.g., deterministic RNG state inversion for CVE-2023-6702) and complex, agentic reasoning (e.g., hypothesize-and-verify loops for the undocumented CVE-2024-0519).
  • Mitigations Break the Chain: Defensive measures like the V8 sandbox are highly effective. Data shows that hardened sandbox implementations dramatically reduce the probability of escalating from in-sandbox primitives to out-of-sandbox arbitrary read/write (from 30% to 9%), highlighting the importance of strategically breaking the exploitation chain.
  • Ongoing Challenges and Future Directions: The "memorization problem" in open benchmarks is a continuous challenge, necessitating constant addition of new, unmemorized bugs. Future work includes automatic data pulling, dynamic scoring to sunset old bugs, and expanding targets beyond the isolated V8 engine to real Chrome environments, kernels, and other userland binaries.

About the Speaker(s)

Seunghyun Lee is a PhD student at Carnegie Mellon University. His research focuses on understanding and advancing the capabilities of AI in complex security tasks, particularly in vulnerability exploitation and exploit generation. His work on ExploitBench reflects his deep technical expertise in both software exploitation and AI systems.

Professor David Brumley is Seunghyun Lee's advisor at Carnegie Mellon University. He is also recognized as Bugcrowd's Chief AI and Science Officer. Professor Brumley is a prominent figure in computer security, known for his contributions to automated program analysis, vulnerability discovery, and exploit generation. His involvement underscores the academic rigor and industry relevance of the ExploitBench project.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Finally, an AI benchmark that isn't a toy CTF or a hallucinating 'LLM-as-a-judge' disaster. Lee built a deterministic, five-tier gauntlet against real V8 vulnerabilities to prove exactly where these models fall flat. It's rigorous, it's brutal, and it's exactly the reality check the AI security space needs.

Heather Calloway (CISO) — MUST SEE

This is required reading for anyone presenting AI threat models to their risk committee. The researchers prove that AI isn't just regurgitating old code; given enough compute time, top models are independently reasoning through complex exploit chains. We need to stop asking a binary question of whether AI can 'hack' and start measuring where our architectural mitigations break its progress.

→ Top-rated talks at [un]prompted.au 2026

All talks from [un]prompted.au 2026