Claude: Climbing a CTF Scoreboard Near You

Claude: Climbing a CTF Scoreboard Near You

Speakers: Keane Lucas

Conference: DEF CON 33

YouTube: https://www.youtube.com/watch?v=sbkeEwhWIks

Slides: https://media.defcon.org/DEF%20CON%2033/DEF%20CON%2033%20presentations/Keane%20Lucas%20-%20Claude%20-%20Climbing%20a%20CTF%20Scoreboard%20Near%20You.pdf

Overview

Keane Lucas from Anthropic's Frontier Red Team presented a detailed experimental study of Claude's performance on Capture the Flag (CTF) competitions across a broad range of security categories. The results were surprising: Claude, without specialized tooling, fine-tuning, or custom models, performed competitively on beginner and intermediate CTF challenges and showed meaningful capability on some advanced challenges. The talk is not a celebration of AI hacking capability so much as it is an early warning system: the current models are just the beginning, and the security community needs to start seriously reckoning with what AI-assisted hacking will look like at full scale. Lucas shared specific prompting strategies, category-by-category performance data, and honest reflection on where models fail today — and why those failures are temporary.

Background

▶ Watch: Introduction: Using Claude AI for CTF competition (0:04)

CTF competitions are structured security challenges where participants solve problems across categories like binary exploitation, reverse engineering, web vulnerabilities, cryptography, forensics, and OSINT. They range from beginner-accessible challenges on platforms like PicoCTF to elite competitions like DEF CON CTF Finals and PlaidCTF. The flag — typically a string in a known format — is extracted by successfully exploiting the challenge.

CTFs are widely used for security education, recruitment, and skill benchmarking. They also provide a clean, objective benchmark for evaluating AI security capability: the flag is either found or not.

Previous research on LLMs and CTFs had explored whether frontier models could solve challenges at all. Lucas's work systemized this, running Claude against a large, categorized set of challenges with consistent methodology, recording detailed traces of the model's reasoning, and using those traces to understand both the successes and the failures.

The framing of the talk is deliberately calibrated: Lucas is on the Frontier Red Team at Anthropic, which means he is professionally engaged in understanding the risks that frontier AI systems pose. He is not presenting this research to show that AI is definitively dangerous right now, but to provide data that informs the community's threat modeling before the capability curve steepens further.

Key Findings

▶ Watch: Climbing the CTF scoreboard: results and performance metrics (0:15)

• Claude solved a substantial fraction of beginner and intermediate CTF challenges across multiple categories, including binary exploitation, web security, reverse engineering, and cryptography.
• Web challenges were a particular strength — the model understood common web vulnerability patterns (SQL injection, XSS, authentication bypass, SSRF, path traversal) and could often solve these challenges with minimal guidance.
• Binary exploitation was more variable. Claude could identify vulnerability classes (stack overflows, format string bugs) and reason about exploitation approach, but often failed at the final step of producing a working exploit due to precision requirements (correct offsets, correct gadget selection).
• Prompting strategy significantly impacts performance. The difference between naive prompting and optimized prompting (agentic loop, tool access, structured reasoning prompts) was dramatic on hard challenges.
• The model fails in predictable ways that are likely to be corrected by future capability improvements: precision in exploit construction, persistent state management across tool calls, and recovering from dead-end approaches.
• The results are an early warning, not a ceiling. Models have improved since these experiments were conducted, and the trajectory suggests significantly more capability in the near term.

Technical Deep Dive

▶ Watch: Prompt engineering techniques for security problem solving (6:14)

Experimental setup:

Lucas ran Claude against a curated set of CTF challenges sourced from multiple competitions and platforms, spanning six primary categories:

• Web exploitation
• Binary exploitation (pwn)
• Reverse engineering (reversing)
• Cryptography
• Forensics
• Miscellaneous

The model was given challenge descriptions, file attachments (binaries, source code, network captures), and access to a set of tools including a Python interpreter, a shell environment, and the ability to read and write files. The challenge was presented as a conversation, and the model was allowed to iterate — running commands, analyzing output, forming hypotheses, and revising its approach.

Category performance analysis:

Web challenges: Claude performed best here, solving a high proportion of beginner-to-intermediate web challenges. The model accurately identified SQL injection points, constructed working payloads, recognized authentication bypass opportunities (JWT manipulation, cookie forging), and exploited SSRF conditions. The model's training data covers web security extensively, and the challenge types map directly to well-documented vulnerability patterns.

Binary exploitation: Performance was more variable. Claude consistently identified the vulnerability type — buffer overflows, format string vulnerabilities, use-after-free — and could construct a conceptual exploitation approach. The failure point was usually in the precise mechanics: calculating the correct offset to the return address, identifying the right ROP gadgets with exact addresses, and generating a working exploit script that handles ASLR. The model would often produce an almost-correct exploit that failed at runtime, and sometimes struggled to diagnose why.

Reverse engineering: Claude showed strong ability to analyze disassembly and decompiled code (via tools like Ghidra/radare2 output fed as text) for common patterns: xor-based obfuscation, custom encryption, license check logic. It performed well on challenges that mapped to pattern recognition but struggled with highly novel obfuscation schemes.

Cryptography: Performance bifurcated sharply between classical/educational crypto (Caesar, Vigenere, RSA with small keys, simple substitution ciphers) where Claude excelled, and implementation-level attacks (padding oracle, Bleichenbacher, differential cryptanalysis) where the model could describe the attack but frequently failed to implement it correctly.

Prompting techniques that improved performance:

1. Agentic loop with tool use: Rather than asking the model to solve the challenge in a single response, using a structured loop where the model proposes an action, executes it via tool call, observes the result, and updates its approach dramatically improved performance on multi-step challenges.

1. Explicit uncertainty prompting: Asking the model to explicitly state its confidence and list alternative hypotheses when it reached a dead end helped prevent the model from tunneling on an incorrect approach.

1. Step-by-step decomposition: For binary exploitation, asking the model to first identify the vulnerability, then identify the memory layout, then calculate offsets, then construct the payload as separate explicit steps (rather than asking for a complete exploit) improved final output quality.

1. Error interpretation prompts: When an exploit attempt failed, feeding the error output back and asking the model to diagnose the failure (rather than retry blindly) accelerated convergence.

Where the model reliably fails today:

• Precise numerical reasoning: calculating exact addresses, offsets, and sizes without running the binary in a debugger
• Long-horizon planning: maintaining a consistent strategy across many tool calls when intermediate results are ambiguous
• Novel challenge types: challenges designed specifically to avoid known patterns, particularly for elite CTF competitions

Demo / Proof of Concept

▶ Watch: AI solving binary exploitation challenges (8:44)

The talk included:

• A live demonstration of Claude solving a web challenge (SQL injection-based flag extraction), showing the complete reasoning trace from challenge description to flag
• A demonstration of Claude successfully identifying a buffer overflow and partially constructing an exploit, then showing where the process broke down and how prompting changes could improve it
• Aggregate performance charts showing solve rates by category and difficulty level
• Comparison of naive prompting versus optimized agentic prompting on the same challenge set

The speaker emphasized that the aggregate solve rate was not the most important statistic — the key insight was which types of challenges the model found trivial versus hard, and why.

Defensive Implications

▶ Watch: Where Claude struggled: hard problems and limitations (16:14)

CTF performance predicts real-world capability in specific categories. Web challenges in CTFs closely mirror real web vulnerabilities. A model that can solve CTF web challenges reliably can find real SQL injection, XSS, and authentication bypass vulnerabilities. This has implications for AI-assisted security assessment services and for the economics of web application security testing.

The trajectory matters more than the current state. Current models solve a meaningful fraction of beginner-intermediate challenges. The history of AI capabilities suggests that this fraction will increase substantially. Security professionals and organizations should be developing their response to AI-assisted attacks now, not when the capability arrives at scale.

Detection of AI-assisted attacks is a research gap. The methodical, rapid-iteration style of AI-assisted hacking — especially for web vulnerabilities — may produce distinctive patterns in server logs (unusual request sequences, systematic parameter variation, specific timing patterns). Research into detection of AI-assisted web scanning and exploitation is warranted.

Defensive use of the same capability. The same model capability that can attack web applications can be deployed defensively: AI-assisted code review, automated vulnerability scanning, and AI-augmented penetration testing are all legitimate and valuable applications. Organizations should explore defensive AI adoption proactively.

Safety considerations for AI and offensive tools. Lucas, as a Frontier Red Team researcher at Anthropic, reflected on the dual-use nature of this research. Anthropic's policies on AI and cybersecurity offense are an active area of development, and the publication of this research is itself a contribution to the policy and capability discourse.

Key Takeaways

1. Claude solves a substantial fraction of beginner and intermediate CTF challenges across security categories without specialized fine-tuning or custom tools.
2. Web exploitation is the strongest category, with performance directly predictive of real-world web vulnerability discovery capability.
3. Binary exploitation capability exists but is limited by precision requirements — a limitation that is likely to decrease as models improve.
4. Prompting strategy — particularly agentic loops, explicit uncertainty handling, and step decomposition — significantly improves performance on hard challenges.
5. The current results represent an early warning, not a ceiling; models available today are more capable than those used in these experiments.
6. The security community needs to develop threat models, detection strategies, and policy frameworks for AI-assisted attacks now, before the capability curve steepens further.

About the Speaker(s)

▶ Watch: Implications for CTF competitions and AI in security (18:44)

Keane Lucas is a member of the Frontier Red Team at Anthropic. His role involves evaluating the capabilities and risks of frontier AI models, including offensive security capabilities. His research combines traditional security skills with AI capabilities evaluation, providing a unique vantage point for assessing how AI models perform on adversarial tasks. He presented this research with the explicit intent of informing the security community about capability trajectory, not to endorse or enable AI-assisted attacks.