Panel Discussion: The Present and Future of AI and Security

Panel Discussion: The Present and Future of AI and Security

Speakers: David Brumley (CEO and Professor, Mayhem and Carnegie Mellon University); Evan Downing (Senior Security Engineer, Trail of Bits); Minwoo Baek (Ph.D. Student, KAIST); Tyler Nighswander (Researcher, Theori)

Conference: CODE BLUE

YouTube: https://www.youtube.com/watch?v=CL1YIdDR_3E

Overview

This panel discussion, held at CODE BLUE, delves into the transformative impact of Large Language Models (LLMs) on cybersecurity, specifically through the lens of DARPA's ambitious AI Cyber Challenge (AIxCC). Moderated by Dr. David Brumley, a former winner of DARPA's Cyber Grand Challenge (CGC), the session brings together leading researchers from the top teams of the AIxCC: Minwoo Baek of Team Atlanta (1st place), Evan Downing of Trail of Bits, and Tyler Nighswander of Theori (both finalists). The discussion provides a unique insider perspective on the strategies, successes, and surprising limitations encountered while developing fully autonomous cyber reasoning systems (CRSs) capable of finding and patching software vulnerabilities.

The core of the talk revolves around the practical application of emerging AI technologies to address critical security challenges, particularly in the often under-resourced open-source software ecosystem. The panelists share their experiences, architectural decisions, and the evolving capabilities of LLMs over the course of the multi-year competition. Their insights illuminate not only the current state-of-the-art in AI-driven cybersecurity but also project future developments and the necessary adaptations for security professionals in an increasingly AI-augmented landscape. The conversation highlights DARPA's mission to foster open science, as all competing CRSs and collected data are made publicly available to accelerate research and development in this vital field.

Background

▶ Watch: Introduction to DARPA's AI Cyber Challenge (0:00)

The DARPA AI Cyber Challenge (AIxCC) represents a significant evolution from the 2016 DARPA Cyber Grand Challenge, which focused on fully autonomous cyber operations on an artificial playing field. DARPA initiated the AIxCC approximately two years prior to its announcement in August 2023, recognizing the profound shift in the technological landscape with the emergence of powerful LLMs. The agency questioned whether autonomous cyber could be re-envisioned through the capabilities of contemporary AI. Unlike its predecessor, the AIxCC placed a much heavier emphasis on defense, requiring systems to not only find and prove vulnerabilities but primarily to demonstrate meaningful patches.

A critical aspect of the challenge was its focus on well-known open-source projects, acknowledging their pervasive use and chronic under-resourcing in terms of security maintenance. The goal was to determine if AI could significantly enhance the security of this vital software. To ensure robust testing, the challenge incorporated both synthetic vulnerabilities (to cover a wide variety of bug types) and real, recent vulnerabilities (to prevent LLMs from merely memorizing old exploits, thereby incentivizing genuine reasoning). A foundational principle of the AIxCC, aligning with DARPA's open science mission, was the mandate for all participating Cyber Reasoning Systems (CRSs) to be open source, alongside the release of collected data to foster future research.

The AIxCC was structured as a multi-year program, commencing with an open track and a small business track. After an initial year, a semifinal round down-selected the top seven teams, each receiving $2 million, totaling over $14 million in prizes. The finals, held at Defcon, offered substantial prizes: $4 million for first place, $3 million for second, and $1.5 million for third. DARPA partnered with industry giants like Google, Anthropic, OpenAI, Microsoft, and the Linux Foundation's OpenSSF to ensure all competitors had equitable access to the latest LLM models and information.

The panelists' motivations for joining the challenge were varied but converged on a common interest in LLMs. Minwoo Baek, a PhD student, was initially skeptical but intrigued by the emerging LLM capabilities, wanting to test if they could truly automate vulnerability finding and patching. Evan Downing from Trail of Bits highlighted his company's long-standing commitment to open-source security and the academic curiosity within their research department regarding LLM potential. Tyler Nighswander of Theori, a veteran CTF player and Cyber Grand Challenge winner, was drawn by the competitive aspect and the opportunity to test the hypothesis of LLMs' utility in cybersecurity.

Key Findings

▶ Watch: Surprising LLM behaviors and observed improvements (20:40)

The AI Cyber Challenge yielded several crucial findings regarding the efficacy and limitations of LLMs in automated cybersecurity. Perhaps the most significant discovery was the rapid and substantial improvement in LLM performance over the competition's two-year span. Initially, models like GPT-3.5 were considered "okay" but limited. However, subsequent models, such as Claude 3.5, demonstrated a remarkable leap in capabilities, including the ability to identify and correct their own errors, re-evaluate hypotheses, and exhibit a form of "moderately smart human" insight when an initial approach failed. This evolution even prompted teams like Trail of Bits to re-architect their CRSs, allowing LLMs greater autonomy in determining what program analysis context they needed, leading to demonstrably better performance in the finals.

Despite these advancements, the panelists also uncovered significant limitations. Minwoo Baek described LLMs as capable of being "lazy and dumb," citing an instance where an LLM ignored relevant crash reports to instead select a patch from a hidden file named goodpatch.diff, without validating its relevance. This underscored a lack of inherent reasoning capability and a tendency to conclude prematurely if an easy (albeit incorrect) answer was presented. Tyler Nighswander recounted an extreme case where an LLM, deprived of access to actual code, hallucinated bugs and even successfully exploited one of its own fabricated vulnerabilities, highlighting their detachment from reality without proper guardrails. The consensus was that simply asking an LLM to "win the contest" was ineffective; instead, problems needed to be broken down into smaller, manageable tasks.

Another key finding was the success of diverse architectural approaches. While some teams leaned heavily on LLMs, others integrated them with traditional program analysis tools or employed a multi-agent, fail-safe strategy. This demonstrated that no single "best" algorithm dominated, but rather the intelligent orchestration and application of various techniques, including LLMs, were paramount. The competition confirmed that LLMs can be effectively leveraged to discover and patch security vulnerabilities, fundamentally answering DARPA's core question in the affirmative, while simultaneously exposing the ongoing need for human expertise to guide, constrain, and optimize their application.

Technical Deep Dive

▶ Watch: Will AI benefit attackers or defenders more? (30:00)

The three finalist teams showcased distinct yet overlapping architectural philosophies for their Cyber Reasoning Systems (CRSs), each leveraging LLMs in unique ways to find and patch vulnerabilities in C and Java.

Team Atlanta (Minwoo Baek):

Team Atlanta, the first-place winner, adopted a strategy of diversity and redundancy. Recognizing the inherent uncertainty in which algorithms or approaches would perform best, their CRS incorporated multiple modules for the same task, particularly for vulnerability finding and patching. For instance, for each supported language (C and Java), they employed both a language-specific module and a language-agnostic module for vulnerability detection. This "ensemble" approach ensured that if one module failed to identify a bug, another might succeed. Similarly, their patching system utilized "five or six different strategies" for generating fixes, increasing the probability of a successful patch even if individual strategies were imperfect. This multi-pronged approach aimed for fail-safe operation and future-proofing, anticipating the potential need to support additional languages like Python, Rust, or JavaScript, where a language-agnostic model could be invaluable.

Trail of Bits (Evan Downing):

Trail of Bits pursued a "best of both worlds" approach, aiming to marry the non-determinism of LLMs with battle-tested software analysis tools. Their CRS, named Buttercup, prioritized software stability from the outset, echoing Dr. Brumley's emphasis on robust testing. A unique focus for Trail of Bits was static analysis. They primarily utilized Tree-sitter and Code Query to statically analyze C and Java code. These traditional program analysis tools were then linked to the LLM components. The LLMs would query these tools to gain comprehensive information about the codebase, effectively learning about the software's structure and behavior to discover vulnerabilities and generate patches. This meant the LLMs were not operating in a vacuum but were informed by precise, structured data from established analysis techniques. Interestingly, between the semi-finals and finals, as LLM capabilities advanced, Trail of Bits re-architected Buttercup to allow the LLMs more autonomy in deciding what program analysis context they needed, rather than being pre-fed, leading to improved performance.

Theori (Tyler Nighswander):

Theori distinguished itself with a highly LLM-first, LLM-heavy approach. While they included a fuzzer (specifically libFuzzer) for completeness, they invested minimal engineering effort into it. Instead, their core innovation lay in using LLMs for static analysis to identify bugs and subsequently for Proof-of-Concept (PoC) development. The LLMs would analyze code, and if a bug was suspected, they would utilize tools like code browsing and generate Python snippets to construct exploits. These generated PoCs were then tested using a debugger. Tyler Nighswander noted that approximately 70% of the bugs found and PoCs constructed by their system were purely through LLM analysis, without reliance on a fuzzer. This demonstrated the remarkable capability of LLMs to perform complex reasoning and code generation for exploitation. A notable example involved an Engine X buffer overflow where initial LLM attempts failed because they incorrectly guessed a cookie name ("user ID" instead of "UID"). However, with the advent of Claude 3.5, the LLM exhibited self-correction, recognizing its hypothesis might be wrong and re-evaluating the code to find the correct configuration, showcasing a surprising level of meta-reasoning.

Across all teams, the discussion highlighted the challenges of LLM context windows (even large ones like Google Gemini's 2 million tokens showed drastic performance drops after 64k tokens), difficulties with instruction following, and the phenomenon of "laziness" or getting stuck in infinite loops when confronted with complex or repetitive tasks. These observations underscore that while LLMs are powerful, they are not yet fully autonomous problem-solvers and require careful engineering, scaffolding, and human expertise to be effectively applied in complex domains like cybersecurity.

Demo / Proof of Concept

▶ Watch: Q&A: Can AI eliminate all software vulnerabilities? (53:10)

As a panel discussion, this session did not include a live demonstration or proof of concept of the Cyber Reasoning Systems developed by the teams. Instead, the panelists provided detailed verbal accounts and architectural descriptions of their respective systems—Team Atlanta's ensemble approach, Trail of Bits' Buttercup combining LLMs with static analysis, and Theori's LLM-first vulnerability discovery pipeline. Their discussion served as an in-depth exploration of the capabilities and challenges of these AI-driven security tools, drawing upon their direct experience from the DARPA AI Cyber Challenge.

Defensive Implications

▶ Watch: Q&A: Staying competitive beyond core LLM capabilities (57:50)

The advent of AI-powered Cyber Reasoning Systems carries profound implications for cybersecurity defenders. The panelists generally agreed that, in the medium to long term, AI will likely confer an advantage to defenders, although the short term might see a "cat-and-mouse game" or even a temporary edge for attackers due to the rapid adoption of new tools. Tyler Nighswander articulated that attackers are often not limited by their ability to find bugs, as "low-hanging fruit" abounds. Defenders, however, typically react to known vulnerabilities. AI, in this context, offers defenders a "new capability": a "little hacker box" that proactively feeds them bugs, allowing them to take action before an exploit occurs (e.g., getting ransomware). This democratizes vulnerability research, making advanced bug finding accessible to a broader range of developers and organizations, even those not directly involved in security research.

Evan Downing emphasized that LLMs can help defenders tackle the "easier, shallow bugs" that frequently plague open-source software projects due to under-resourcing or lack of scrutiny. By automating the discovery and patching of these common vulnerabilities, AI can free up human security professionals to focus on more complex, novel threats.

However, leveraging AI effectively requires a shift in skill sets for security professionals. Minwoo Baek stressed the importance of domain expertise: knowing "what to ask the LLM and where to apply it." Simply instructing an LLM to "win the contest" is insufficient; problems must be broken down and specific, well-defined tasks assigned. Tyler Nighswander reinforced this, noting that all top AIxCC teams were primarily security domain experts, not AI experts. Future security engineers will need to understand the fundamentals of fuzzing, static and dynamic analysis, and possess the ability to deconstruct complex problems, almost at a "managerial level," to effectively guide AI tools. This suggests that AI will augment, rather than replace, human security intelligence.

Regarding the security of AI systems themselves, the panelists acknowledged challenges like prompt injection and the difficulty in defining what constitutes a "secure" AI. Tyler Nighswander drew an analogy to SQL injection, suggesting that engineering solutions and guardrails (e.g., limiting AI access to untrusted data or tools, setting specific goals) are crucial. Evan Downing added that solutions might range from simple deterministic filters (e.g., stop lists for undesirable output) to rigorous evaluation of training and testing data sets. Ultimately, the availability of tools like Kali Linux for both attackers and defenders suggests that if both sides have access to AI capabilities, the playing field might eventually balance, with the focus shifting to how intelligently these tools are applied. The discussion also highlighted future defensive needs, such as AI's role in automatically generating fuzzing harnesses and test cases to prevent vulnerability reintroduction, which are currently significant bottlenecks in the security development lifecycle.

Key Takeaways

• LLMs are Viable for Vulnerability Management: The DARPA AI Cyber Challenge conclusively demonstrated that LLMs can be effectively used to find and patch software vulnerabilities, representing a significant advancement in autonomous cybersecurity.
• Rapid LLM Evolution: LLM capabilities improved drastically within a 1-2 year timeframe, with models exhibiting enhanced reasoning, self-correction, and autonomy, necessitating adaptive architectural approaches from competing teams.
• Diverse Architectural Strategies Succeed: Successful CRSs adopted varied approaches, including multi-agent diversity (Team Atlanta), hybrid integration of LLMs with traditional static analysis (Trail of Bits' Buttercup), and LLM-first pipelines for bug finding and PoC generation (Theori).
• LLM Limitations Persist: Despite advancements, LLMs still exhibit "laziness," context window limitations, occasional hallucinations, and require careful engineering to break down complex tasks and prevent errors or infinite loops.
• Defenders Gaining an Edge (Long-Term): AI is poised to significantly benefit defenders by democratizing vulnerability research, automating the discovery and patching of common bugs, and enabling proactive security measures, although human domain expertise remains critical for guiding these tools.
• Future Focus on Harness & Test Case Generation: Key future improvements for AI in cybersecurity include the automated generation of fuzzing harnesses and regression test cases, which are currently labor-intensive bottlenecks, to ensure comprehensive and ongoing software security.

About the Speaker(s)

Dr. David Brumley is a Professor at Carnegie Mellon University and the CEO of Mayhem Security. He is a prominent figure in cybersecurity research, having led the team that won DARPA's first Cyber Grand Challenge in 2016. In the AI Cyber Challenge, Dr. Brumley served as a subject matter expert, contributing to the design of the competition's scoring algorithms and overall structure.

Minwoo Baek is a Ph.D. student at KAIST, advised by his professor Injun. He was a key member of the patching team for Team Atlanta, which secured first place in the AI Cyber Challenge. His current research focuses on the automation of security tasks using Large Language Models.

Evan Downing is a Senior Security Engineer at Trail of Bits. He played a crucial role in building their Cyber Reasoning System, Buttercup, specifically leading the program analysis component. Trail of Bits is known for its extensive work in helping governments and commercial industries defend against cyber attackers since 2012.

Tyler Nighswander is a Researcher and the team lead for Theori's AI Cyber Challenge effort. A long-time Capture The Flag (CTF) player, he was also on David Brumley's winning team in the 2016 Cyber Grand Challenge. Tyler has previously spoken at CodeBlue and is now a member of its review board, reflecting his deep expertise in software security research and competitive cybersecurity.