The Importance of Knitting in an AI World

The Importance of Knitting in an AI World

Speakers: Maddie Stone, Security Researcher

Conference: BruCON

YouTube: https://www.youtube.com/watch?v=15Emm5ABfls

Overview

In her compelling BruCON keynote, "The Importance of Knitting in an AI World," Maddie Stone, a distinguished security researcher formerly leading Google Project Zero's efforts on in-the-wild zero-day exploits, delivers a critical examination of the cybersecurity landscape amidst the rapid ascent of Artificial Intelligence. Drawing from her unique experience of returning to the industry after a multi-year sabbatical, Stone highlights the profound shift in vulnerability discovery, exploit development, and defensive strategies brought about by advanced AI capabilities. The talk challenges conventional thinking, arguing that the traditional skills and processes that have long defined cybersecurity are now insufficient to tackle the "bug apocalypse" unleashed by AI.

Stone's central thesis posits that as AI increasingly automates and even surpasses human capabilities in complex technical tasks like binary reverse engineering and vulnerability analysis, the unique strengths of human ingenuity become paramount. She argues that fostering creativity, adaptability, curiosity, and community – qualities often cultivated outside the direct scope of technical work, such as through hobbies like knitting – are not merely soft skills but essential competencies for navigating the evolving security paradigm. The presentation, delivered with Stone literally knitting on stage, masterfully intertwines personal narrative with profound technical and strategic insights, underscoring the urgent need for the industry to rethink its approach to security in an AI-dominated future.

This talk is particularly significant because it comes from a researcher with deep practical experience in zero-day exploitation, offering an informed perspective on AI's impact. It serves as a call to action for the cybersecurity community to look beyond purely technical solutions and embrace a more holistic approach that values human-centric attributes. By illustrating how AI is fundamentally altering the threat landscape and overwhelming existing defensive mechanisms, Stone provides a roadmap for resilience, emphasizing that the ability to learn, adapt, collaborate, and think creatively will be the ultimate differentiator in securing systems against an intelligent, autonomous adversary.

Background

▶ Watch: Introduction and knitting on stage (0:00)

Maddie Stone's keynote draws heavily on her experience re-entering the cybersecurity world after a significant break. Having left the industry in April 2022 (returning in August 2024), she observed a dramatic acceleration in AI's capabilities that few, including herself, had anticipated. Prior to her sabbatical, while machine learning (ML) had been a staple in security for years (e.g., malware detection), it was largely perceived as a supplementary tool, not a replacement for human experts. Stone recounts an incident from her time on the Android malware team where an automated ML system, after a human flagged a sophisticated malware variant (95% legitimate settings app, 5% malicious), subsequently flagged over 10,000 legitimate settings apps for human review. This demonstrated the limitations of ML at the time, leading to a general industry sentiment that AI wouldn't soon master complex tasks like reverse engineering, vulnerability analysis, or exploit development.

However, upon her return, Stone witnessed a starkly different reality. Conversations with veteran binary reverse engineering specialists, some of whom had written books on the subject, revealed that AI was now considered superior at binary analysis. This rapid advancement has led to what Stone terms the "bug apocalypse," where vendors are "completely and totally overwhelmed" by the sheer volume of vulnerabilities being discovered by AI. Crucially, these aren't just superficial web application bugs; AI is successfully identifying deep kernel vulnerabilities and complex logic bugs that require many steps to uncover—tasks previously thought to be uniquely human.

This phenomenon exacerbates an already strained defensive landscape. Even before the rise of advanced AI, defensive teams were often under-resourced, struggling to keep pace with vulnerability reports, implement timely patches, and conduct effective variant analysis. Stone’s own research, such as her 2020 year-in-review of actively exploited zero-days, revealed that one out of every four detected in-the-wild zero-days was a variant of a previously known bug, highlighting the persistent challenge of comprehensive patching. With AI now doubling the rate of vulnerability discovery (e.g., CVEs for 2024 already at double previous maximums), the traditional model of one-to-one patching for each reported vulnerability is becoming unsustainable. This escalating crisis necessitates a fundamental rethinking of how security is approached, moving beyond reactive bug fixes to more proactive, systemic solutions.

Key Findings

▶ Watch: AI outperforming human binary reverse engineering (14:30)

Maddie Stone's talk unveils several critical findings regarding the current state of cybersecurity in an AI-driven world:

• AI's Unprecedented Capabilities: AI has rapidly advanced beyond initial expectations, demonstrating proficiency in complex cybersecurity tasks previously considered human-exclusive. This includes binary reverse engineering, vulnerability research (identifying even intricate multi-step logic bugs and kernel vulnerabilities), and exploit development, with the ability to chain exploits. This rapid progression has led to a situation where AI is now "better" than seasoned human experts in some of these domains.
• The "Bug Apocalypse": The sheer volume of vulnerabilities discovered by AI has created an overwhelming challenge for vendors and defensive teams. This "bug apocalypse" is characterized by an explosion in reported bugs, with CVE counts for 2024 already doubling previous maximums, far exceeding the capacity for traditional one-to-one patching.
• Compiler-Introduced Vulnerabilities: Research by Christopher Domas, highlighted in the talk, demonstrates a novel class of vulnerabilities where C compilers can reintroduce Time-of-Check Time-of-Use (TOCTOU) bugs. Even if a developer meticulously fixes a TOCTOU in source code by snapshotting values into local variables, specific C standard rules and compiler optimizations can lead to "invented loads," causing the compiler to reload from attacker-controlled memory in the compiled binary. This results in vulnerable production code despite clean source code.
• AI as an Enabler for Scaled Discovery: While alarming, AI's ability to find such compiler-introduced vulnerabilities at scale (e.g., Claude identifying over 300 instances in security-critical open-source projects) presents a paradigm shift. It allows for the discovery and understanding of bug classes that would be "too difficult," "too manual," or "too time-intensive" for human researchers alone, offering new avenues for improving security posture.
• Inadequacy of Traditional Security Approaches: The current pace of AI-driven vulnerability discovery renders traditional, reactive security practices—like individual bug patching—obsolete. The industry can no longer rely on the same skills, processes, and best practices that have been in use for decades.
• The Evolving Human Role: As AI handles more technical tasks, the human role in cybersecurity must evolve. Stone emphasizes that creativity, adaptability, curiosity, persistence, and community building are no longer secondary "soft skills" but foundational requirements. These human attributes are crucial for identifying new problem spaces, devising novel solutions, and understanding the complex human-AI interaction dynamics.

Technical Deep Dive

▶ Watch: AI enables finding complex C standard bugs (25:00)

Maddie Stone's keynote delves into specific technical examples that underscore the need for a paradigm shift in cybersecurity. She highlights two key areas: innovative exploit mitigations and the surprising reintroduction of vulnerabilities by compilers, with AI playing a critical role in both understanding and discovering these complex issues.

Firstly, Stone points to Chrome's Miracle Pointer initiative in 2022 as a prime example of moving beyond reactive, one-to-one bug patching. Prior to this, use-after-free (UAF) vulnerabilities constituted the majority of actively exploited Chrome bugs and reported vulnerabilities, particularly within privileged processes. Instead of attempting to eradicate every UAF bug from their extensive codebase, the Chrome security team adopted a more strategic approach. Miracle Pointer was designed to render UAF bugs in the majority of the Chrome codebase unexploitable. This meant that even if a UAF existed, it could not be leveraged by an attacker, effectively neutralizing a whole class of critical vulnerabilities. This pivot from "fix every bug" to "make bugs unexploitable" represents a significant shift towards exploit mitigation strategies that change the economics of exploitation, making it far more difficult and less rewarding for adversaries.

Secondly, Stone details the fascinating and concerning research by Christopher Domas on compiler-introduced Time-of-Check Time-of-Use (TOCTOU) vulnerabilities. A classic TOCTOU occurs when a program checks a security-critical value (e.g., the length of a buffer for memcpy) from attacker-controlled memory, but then reloads that value from memory after the check, allowing an attacker to modify it in between the check and its subsequent use. This can lead to issues like buffer overflows. The standard fix for such a vulnerability is to snapshot the value into a local, non-attacker-controlled variable immediately after the check and then use that local variable for all subsequent operations within the function.

However, Domas's research revealed that due to specific decisions within the C standard and various compiler optimization settings (e.g., register use), compilers can perform what he terms "invented loads." This means that even when a developer correctly implements the snapshotting fix in the source code, the compiler might decide to optimize the code by reloading the value from the original attacker-controlled memory location instead of using the local snapshot. The result is a compiled binary that is vulnerable to TOCTOU, despite the source code appearing perfectly secure. This highlights a profound disconnect between source code security and runtime security, introducing a subtle yet dangerous attack surface.

The crucial link to AI here is that identifying these compiler-introduced TOCTOU vulnerabilities manually would be an "intensive," "manual," and "costly" endeavor, requiring meticulous analysis of compiled binaries across different versions and optimization flags. Domas, however, leveraged AI, specifically Claude, to analyze security-critical open-source projects and successfully identify over 300 instances where the source code was clean, but the compiled binary reintroduced the TOCTOU vulnerability. This demonstrates AI's unparalleled ability to scale complex, pattern-based vulnerability discovery, enabling researchers to uncover entire classes of bugs that were previously too elusive or labor-intensive for human investigation. This capability, while presenting new challenges, also offers a powerful tool for proactively improving the security posture of software by understanding and mitigating these hidden compiler-borne weaknesses.

Demo / Proof of Concept

▶ Watch: Essential human skills for the AI world (35:30)

While Maddie Stone's talk did not feature a traditional technical demonstration involving code or exploit execution, it incorporated a unique and highly effective "proof of concept" for her core message: the act of knitting on stage. Throughout the entirety of her keynote, Stone was actively engaged in knitting, demonstrating the very hobby she advocates as a means to cultivate essential human skills in an AI-dominated world.

This live, ongoing demonstration served multiple purposes. It visually reinforced the theme of the talk, illustrating how a focused, creative, and problem-solving activity, seemingly disconnected from cybersecurity, can embody the traits needed for future success. Stone humorously acknowledged the difficulty of knitting while presenting, subtly highlighting the challenge of multitasking and adapting to new situations. She also shared personal anecdotes about her knitting journey – from struggling to make a simple square (which turned into a rectangle), to debugging knitting mistakes (likening it to debugging code, watching YouTube videos and reading blogs to find solutions), and ultimately creating a sweater by embracing creativity over strict rules.

The physical act of knitting on stage, combined with her narrative, functioned as a powerful metaphor for the talk's argument: just as knitting requires patience, problem-solving, learning new techniques, and adapting when things go wrong, so too will navigating the complexities of an AI-driven security landscape demand these same human-centric attributes. She even offered the large metal knitting needles as a "prize" at the end, further solidifying the experiential aspect of her message. This unconventional "demo" effectively communicated the importance of hands-on creation, debugging, and community interaction in fostering the adaptability and critical thinking necessary for cybersecurity professionals.

Defensive Implications

▶ Watch: Q&A: Creativity is human's advantage over AI (51:00)

The rise of AI in cybersecurity, as articulated by Maddie Stone, demands a radical re-evaluation of defensive strategies. The "bug apocalypse" and AI's superior capabilities in traditional tasks mean that defenders can no longer rely on outdated practices.

1. Shift to Class-Level Exploit Mitigations: The most critical implication is to move beyond reactive, one-to-one patching of individual vulnerabilities. As exemplified by Chrome's Miracle Pointer, defensive efforts should focus on implementing broad exploit mitigations that make entire classes of vulnerabilities (e.g., use-after-free) unexploitable, even if the underlying bugs persist. This changes the economics for attackers, making exploitation significantly harder and less rewarding. Defenders should proactively identify prevalent bug classes in their systems and invest in architectural or compiler-level changes that neutralize their exploitability.

1. Rethink Security Processes and Best Practices: The established "best practices" and processes developed over decades are insufficient. Organizations must critically examine their vulnerability management, incident response, and secure development lifecycles. This includes re-evaluating how vulnerability reports are handled (given the overwhelming volume), how patches are verified (acknowledging the resource strain), and how to integrate AI into defensive tooling.

1. Leverage AI for Scaled Vulnerability Discovery and Analysis: While AI is a threat, it is also a powerful tool. Defenders should embrace AI-powered tools (like Claude in Christopher Domas's research) to find complex, subtle vulnerabilities (e.g., compiler-introduced TOCTOU bugs) at scale. This allows security teams to identify and address weaknesses that would be too time-consuming or difficult for human analysis alone, leading to a more comprehensive understanding of their attack surface. However, human expertise remains crucial for guiding AI, validating its findings, and understanding the context.

1. Cultivate Human-Centric Skills: As AI automates technical grunt work, the premium shifts to uniquely human attributes. Defensive teams must prioritize hiring and developing individuals with creativity, adaptability, curiosity, persistence, and critical thinking. These skills enable security professionals to:

• Devise novel solutions to new problems posed by AI.
• Adapt quickly to evolving threats and technologies.
• Proactively explore unknown vulnerabilities and attack vectors.
• Validate AI's output and understand its limitations.
• Identify bias or unintended consequences in AI-generated code or analysis.

1. Understand and Secure Human-AI Interaction: A critical defensive implication lies in understanding how users, particularly those with less technical sophistication, interact with and trust AI. Many users upload sensitive data to AI models, treat AI outputs as infallible truth, and may not understand the security implications of such actions. Defensive teams must:

• Educate users on safe AI interaction, data privacy, and the importance of verifying AI-generated information.
• Develop security controls that account for these new user behaviors and potential accidental data leakage or misuse by AI agents.
• Design systems that are resilient to "accidental hacking" scenarios where users unknowingly prompt AI to access unauthorized data.

1. Foster Community and Collaboration: Security professionals need to connect and collaborate across diverse backgrounds. This community aspect is vital for sharing knowledge, gaining new perspectives on AI's use and misuse, and understanding the varied threat models that emerge from different user populations. This collaborative approach can lead to more robust and inclusive security solutions.

In essence, the defensive posture in an AI world requires a strategic pivot: from a reactive, individual bug-fix mentality to a proactive, systemic approach focused on exploit mitigation, AI-assisted discovery, and the cultivation of uniquely human creative and adaptive intelligence.

Key Takeaways

• AI is fundamentally transforming cybersecurity: Artificial Intelligence now excels at binary reverse engineering, vulnerability research (including complex logic bugs), and exploit development, leading to an unprecedented "bug apocalypse" that overwhelms traditional defensive mechanisms.
• Traditional 1:1 patching is unsustainable: The sheer volume of AI-discovered vulnerabilities necessitates a shift from individually patching bugs to implementing broader, class-level exploit mitigations (e.g., Chrome's Miracle Pointer) that render entire categories of vulnerabilities unexploitable.
• Compilers can reintroduce vulnerabilities: Research shows that C compilers, through optimization and "invented loads," can reintroduce Time-of-Check Time-of-Use (TOCTOU) vulnerabilities, even when fixed in source code. AI tools like Claude are now capable of discovering these subtle, compiler-borne flaws at scale.
• Human creativity, adaptability, and curiosity are paramount: As AI automates technical tasks, the unique strengths of human intelligence—creativity, adaptability, curiosity, persistence, and critical thinking—become the most valuable assets for identifying new problems, devising novel solutions, and navigating the unknown challenges of an AI-driven security landscape.
• Hobbies and diverse communities foster essential skills: Engaging in activities outside of work, like knitting, can cultivate these critical human attributes by forcing individuals to learn new things, embrace failure, solve problems creatively, and build connections with diverse communities, offering fresh perspectives on security challenges.
• Understanding human-AI interaction is crucial: Defenders must grasp how diverse user populations, often with varying levels of technological literacy, interact with and trust AI, as these behaviors create new attack surfaces and necessitate tailored security measures and user education.

About the Speaker(s)

Maddie Stone is a highly respected security researcher with a distinguished career in identifying and mitigating critical vulnerabilities. She is best known for her tenure at Google, where she led Google Project Zero's efforts on in-the-wild zero-day exploits, making significant contributions to understanding and disclosing vulnerabilities actively used by adversaries. Her experience also includes work with Google's Threat Analysis Group and Android Security. Stone has been recognized for her impact, including being named one of Wired's 25 people who are "making things better."

After an intense period in infosec, Stone took a multi-year sabbatical, stepping away from the industry in April 2022. During this time, she embraced a diverse range of hobbies and experiences, including moving to Central America, learning to surf (and recovering from a serious surf accident), engaging in CrossFit, learning Spanish, playing paddle, skateboarding, trying pottery, journaling, and delving into personal research projects like genealogy. Her return to the security world in August 2024, particularly at Black Hat, provided her with a fresh perspective on the rapid advancements of AI and its profound implications for cybersecurity, which she shares in this keynote. She is also famously obsessed with her dog, Bug.