Black-hat LLMs

Black-Hat LLMs: How AI Is Rewriting the Rules of Vulnerability Research

Speaker: Nicholas Carlini, Research Scientist, Anthropic

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=1sd26pWhfmg

Reading time: ~12 minutes

TL;DR

Nicholas Carlini, a research scientist at Anthropic and one of the most respected voices in AI security, delivered a talk that the conference organizer introduced as a "global-level emergency." Using minimal scaffolding — a single Claude command-line invocation — Carlini and colleagues have found the first critical vulnerability in the Ghost CMS (a 20-year-old SQL injection), multiple remotely exploitable heap buffer overflows in the Linux kernel, and hundreds of additional potential bugs he has not yet had time to validate. His message: LLMs have become the most significant development in security since the internet, and the curve is not bending yet.

Introduction

The introduction Gadi gave before Carlini took the stage was notably different in tone from the rest of the conference. He described Carlini as someone who "has not just been the best at this for the past year or two years, but for a long while, and consistently makes the industry better just by doing his research." He noted that Carlini had agreed to speak "last second" and had rushed in from handling "a truly global-level emergency or situation" to give the talk.

Carlini's own opening was characteristically understated. He works at Anthropic. He was going to talk about "some of the things I'm interested in in language model security." The gap between that framing and what followed was significant.

The core claim he returned to repeatedly: language models can autonomously, and without fancy scaffolding, find and exploit zero-day vulnerabilities in critical software. This was not true even three or four months ago. It is true now. And the rate of improvement shows no sign of slowing.

"Language models really feel to me to be something that's roughly on the order of importance [of the internet], which is not something that I believed, like, three or four years ago."

▶ Watch: Opening — LLMs as the Most Significant Security Development Since the Internet (00:00)

The Scaffold: Brutally Simple

One of the more striking aspects of Carlini's presentation was the simplicity of the tooling. His "scaffold" — the framework within which Claude finds and exploits vulnerabilities — is a single command:

That is it. No custom fuzzing harness. No elaborate multi-agent pipeline. One prompt. The model reads code, identifies vulnerabilities, and writes a report.

He immediately acknowledged two limitations of this approach. First, it is not parallelizable — running the same prompt multiple times on the same codebase tends to find the same bug repeatedly. Second, it is not thorough — the model reviews some code but not all of it. His solution for both problems is equally simple: add a hint parameter pointing to specific files. Run one instance focusing on foo.c, another on bar.c, and so on. You can parallelize across files and ensure systematic coverage. "I can just do this for like all the files."

The critical insight this simplicity embodies: the base capability of the model — what a malicious actor could do without spending months building a custom exploitation framework — is already extremely high. "If someone who's malicious wants to go cause some harm, and they don't have to spend six months designing some fancy fuzzing harness, and they can just go do bad things with it, this is quite scary."

▶ Watch: The Scaffold — One Command to Find Zero-Days (02:00)

Case Study 1: Ghost CMS — The First Critical Vulnerability in 20 Years

The first case study was Ghost, a content management system with approximately 50,000 GitHub stars. Ghost has been around for 20 years and had never had a critical security vulnerability in its history.

Using the simple scaffold, the model found a SQL injection vulnerability in Ghost's tag filter order function. The vulnerable code was concatenating user input directly into an SQL query — a classic and well-understood class of bug. "Everyone knows this is a problem. It's not really a surprise to anyone. And yet, they've been around for twenty years, and they're going to be around for another twenty."

The more impressive part was the exploitation. The vulnerability was only exploitable via blind SQL injection — meaning the attacker cannot see query output directly, only observe timing differences (how long the query takes) or whether the application crashes. Carlini was not sure if the vulnerability was actually exploitable or merely a low-severity theoretical issue. He asked the model to build an exploit.

The model produced a complete blind SQL injection exploit that exfiltrated data using timing-based inference. What could it retrieve? Everything. Usernames, email addresses, hashed bcrypt passwords — "literally everything you could want from unauthenticated accounts." The model built a working exploit for a subtle, non-trivial vulnerability class without requiring any security experience from the operator.

"I probably could have built this attack. But there's some amount of nuance you need to get this right. And I didn't need any security experience to have this happen. So these models are really quite good at implementing exploits now."

▶ Watch: Ghost CMS — First Critical Vulnerability in 20 Years (06:01)

Case Study 2: Heap Buffer Overflows in the Linux Kernel

The second case study was the Linux kernel — one of the most hardened, scrutinized pieces of software in existence. Carlini introduced it with appropriate gravity: "I have never found one of these in my life before. This is like, very, very hard to do."

He walked through one specific vulnerability in the NFSv4 daemon — the Network File System v4 kernel daemon. The attack involves two cooperating adversaries. Here is the sequence:

1. Client A establishes an NFS session with a server, opens a file, and takes a lock with a 1,024-byte owner field.
2. Client B establishes its own session, opens the same file, and requests the same lock range.
3. The server must deny Client B the lock (since Client A holds it). The denial response encodes the existing lock owner — Client A's 1,024-byte owner field — but writes it into a buffer that is only 112 bytes long.
4. This produces a heap buffer overflow in the kernel.

Carlini noted that this is not the kind of bug you find by fuzzing. "You would never find this by fuzzing." It requires understanding that there are two cooperating adversaries, that one of them has a long owner field, and that the interaction between their requests creates an overflow in the server's response path. The entire slide describing this vulnerability was, Carlini noted, "copy and pasted from the report that the language model wrote."

The bug predates Git. It has been in the Linux kernel since 2003 — older than some members of the audience. A language model found it in 2026.

▶ Watch: Linux Kernel Heap Buffer Overflow — Found by LLM (08:01)

The Exponential Curve — and Why You Should Not Assume It Bends Soon

Carlini spent significant time on the trajectory of improvement. He showed data on smart contract exploitation — a useful proxy because smart contracts have dollar values attached, allowing you to measure capability in terms of how much money a model can recover from vulnerable contracts. Recent models can identify and exploit smart contract vulnerabilities to recover several million dollars from real, deployed contracts. The Y-axis on that chart is log-scale, and the trend is exponential.

He compared older models: Sonnet 4.5 (released six months prior) and Opus 4.1 (less than a year old) "can't find these bugs almost ever." The new models, released over the preceding three to four months, can. The capability shift happened at the edge of detectability and then became real very quickly.

The historical analogy he offered was solar power deployment forecasts from the International Energy Agency. Year after year, the IEA would project modest linear growth for solar. Year after year, actual deployment grew by 30-40% more than projected. "For more than half of the years, their prediction for what would happen in 2040 happened the next year." The IEA kept making the same mistake: assuming current rates would continue linearly, when the actual trajectory was exponential.

Carlini's warning: "We should not be them. The exponential may bend in six months. It may bend in two years. But people have been predicting deep learning will hit a wall for ten years, and at least as of yet, it has not."

He made his own capabilities assessment explicit: "It's pretty clear to me that these current models are better vulnerability researchers than I am. I used to do this somewhat professionally. I have CVEs to my name. I do not have... well." He then described having hundreds of Linux kernel crash reports he has not yet had time to validate — bugs he cannot report because he has not confirmed they are real, that he is not going to dump on open source developers as "potential slop."

▶ Watch: The Exponential Trajectory and the Solar Power Analogy (12:01)

What To Do: Defenders Must Move First and Move Now

Carlini's conclusion was a direct call to action. The same capability that lets him find hundreds of Linux kernel bugs is available — in theory — to anyone malicious who wants to use it. The window in which defenders can act before attackers is narrow and closing.

Three concrete calls to action:

1. Help with defensive AI-powered bug finding. Anthropic has Claude Code Security doing this work. DeepMind and OpenAI (the Aardvark project) are also working on it. He urged: "Speaking not as an Anthropic employee — I don't really care where you help, just please help."

1. Act on order-of-months timescales. "Waiting a year is going to be too long." The rate of improvement means that the capability gap between where models are now and where they will be in a year is enormous.

1. Solve the validation bottleneck. Carlini acknowledged that he currently has several hundred crash reports that have not been validated. The process of finding bugs is now faster than the process of verifying them. Scaling validation — whether through AI assistance or tooling — is the next critical bottleneck.

In Q&A, Carlini addressed the dual-use concern directly. He wants to allow security researchers to use models to find and fix bugs. He does not want to allow malicious actors to exploit things. "For a very long time in security, the dual-use nature favored the defender. I think this has been true. It's unclear if this will always be true in the future, especially for language models."

He also addressed the long-term endgame for a question from Michael Siegel of MIT. In the long run, defenders probably win: "In the limit, I'll just rewrite all the software in Rust and get rid of memory corruption vulnerabilities. In the limit, I'll formally verify all my protocols." But the transition period matters enormously. "We sort of want to make things go well for the people who are living through the thing."

▶ Watch: The Call to Action — Defenders Must Act Now (20:01)

Notable Quotes

"Language models can autonomously, and without fancy scaffolding, find and exploit zero-day vulnerabilities in critical software. And they are getting good, scarily fast." — Slide TL;DR, 00:00

"This entire slide was copy and pasted from the report that the language model wrote." — On the Linux kernel NFSv4 vulnerability, 10:01

"This bug has been in the kernel since 2003. It is older than some of you in this room. A language model found it." — 12:01

"It's pretty clear to me that these current models are better vulnerability researchers than I am. I used to do this somewhat professionally. I have CVEs to my name." — 18:01

"Waiting a year is going to be too long. We are going to have a huge number of bugs. I have so many bugs in the Linux kernel that I can't report because I haven't validated them yet." — 20:01

Key Takeaways

• LLMs can autonomously find and exploit zero-day vulnerabilities with minimal scaffolding — a single command-line prompt suffices to find the first critical bug in a 20-year-old codebase.
• The Linux kernel NFSv4 heap buffer overflow — found by an LLM, dating to 2003, requiring two cooperating adversaries to trigger — represents the kind of non-trivial, multi-party interaction bug that fuzzing would never find.
• The capability curve is exponential and has not bent: models released just three to four months ago can find classes of bugs that models six months old "can't find almost ever."
• The validation bottleneck is the next critical problem: bug-finding is now faster than bug-verification, leaving a backlog of unvalidated crash reports that cannot be safely disclosed.
• The window for defenders to act is measured in months, not years — and the dual-use nature of these capabilities means the same power available to researchers is also available to anyone malicious.

Slides Reference

Slides are available as 2026-04-04-D1-S1-16-29-Black-hat-LLMs-pdf.pdf. The deck leads with a repeated TL;DR: "LLMs can autonomously, and without fancy scaffolding, find and exploit 0days in critical software." It includes the full one-line scaffold command, the Ghost CMS SQL injection payload, the detailed NFSv4 two-client attack sequence diagram, and a discussion of exponential improvement trajectory and smart contract exploitation benchmarks.