An Age of Experimentation
Halvar Flake (Software Engineer & Cybersecurity Researcher)
BlueHat Asia 2026 · Day 1 · Main Stage
Overview
In his keynote address at BlueHat Asia, "An Age of Experimentation," renowned cybersecurity researcher Halvar Flake delivers a thought-provoking analysis of a fundamental shift occurring in software engineering: the transition from largely deterministic systems to increasingly stochastic and random ones. Flake, a figure celebrated for his foundational contributions to exploit development (such as heap overflow exploitation on Windows) and security tooling (like Bindiff for patch diffing), argues that the rise of Large Language Models (LLMs) is a primary driver of this paradigm change. This talk challenges the deeply ingrained assumptions of determinism that have long underpinned computer science and software development, urging the industry to adapt to a new reality of empirical experimentation.

Key moments
- 7:00 Halvar Flake's impact: heap overflows and patch diffing
- 11:00 LLMs: Discovered technology requiring experimentation, like fire
- 15:00 LLMs inferring developer intent changes security research
- 17:00 The death of determinism in software engineering
- 24:00 LLMs are random and chaotic; small changes cascade
- 34:00 Tomato farmer protocol for noisy empirical experiments
- 38:00 LLM development feels like playing slot machines
- 42:00 LLM code review like fuzzing: tracking bug 'catch rates'
An Age of Experimentation
Speakers: Halvar Flake, Software Engineer & Cybersecurity Researcher
Conference: BlueHat Asia
YouTube: https://www.youtube.com/watch?v=mzsUTzzlowU
Overview
In his keynote address at BlueHat Asia, "An Age of Experimentation," renowned cybersecurity researcher Halvar Flake delivers a thought-provoking analysis of a fundamental shift occurring in software engineering: the transition from largely deterministic systems to increasingly stochastic and random ones. Flake, a figure celebrated for his foundational contributions to exploit development (such as heap overflow exploitation on Windows) and security tooling (like Bindiff for patch diffing), argues that the rise of Large Language Models (LLMs) is a primary driver of this paradigm change. This talk challenges the deeply ingrained assumptions of determinism that have long underpinned computer science and software development, urging the industry to adapt to a new reality of empirical experimentation.
Flake contends that while technological advancements from 1995 to 2022 were largely foreseeable, the emergence of LLMs represents a disruptive force that few anticipated. He frames LLMs not as engineered creations, but as "discovered" phenomena, akin to humanity's discovery of fire—powerful tools whose underlying mechanisms are still poorly understood. This lack of foundational understanding, combined with the inherent non-determinism of LLMs, necessitates a radical re-evaluation of how software is built, tested, and secured. The implications extend across all facets of the software lifecycle, demanding that developers and security professionals embrace the mindset and methodologies of empirical scientists.
The significance of Flake's message is profound, especially for the security community. For decades, security vulnerabilities have often been traced to deterministic discrepancies between a developer's mental model of software and its actual implementation. The introduction of pervasive randomness threatens to destabilize these existing models, making traditional debugging, testing, and security analysis techniques increasingly inadequate. Flake's call to action is clear: the industry must retool its processes, education, and philosophical approach to computing to navigate this new "age of experimentation" effectively, or risk being perpetually fooled by its own assumptions.
Background
▶ Watch: Halvar Flake's impact: heap overflows and patch diffing (7:00)
The context of BlueHat Asia itself provides a compelling backdrop for Flake's message. Originating in the mid-2000s, BlueHat was Microsoft's internal response to grappling with security challenges highlighted by external conferences like Black Hat. It was designed to foster intimate, often brutally honest conversations between Microsoft executives and engineers, and leading security researchers. Early BlueHat events saw researchers demystifying complex exploits, such as heap overflows on Windows, which were poorly understood internally despite stack-based overruns being common knowledge. Halvar Flake was instrumental in this era, systematically breaking down heap exploitation techniques in a way that profoundly impacted Microsoft's security posture. His pioneering work also included creating tools like Bindiff, which enabled efficient patch diffing—a technique still in use 20 years later for understanding security updates and developing exploits.
Flake sets the stage by observing that for nearly three decades (roughly 1995 to 2022), technological progress, while significant (e.g., internet, smartphones), largely followed predictable trajectories. The core functionality of computers and software remained conceptually similar. However, the last few years have introduced changes that he found "very difficult to foresee" and "frankly surprising." He draws a crucial distinction between "engineered" technologies, like the internal combustion engine (understood from conception), and "discovered" technologies, like fire (where utility precedes understanding). He posits that LLMs fall squarely into the latter category. We know how to build them and observe their capabilities, but we lack a fundamental theoretical understanding of how they achieve such complex behaviors.
This empirical discovery process is evident in their development: starting as text completion software, their chat capabilities were noticed, then the effectiveness of "think step by step" prompting, leading to unforeseen implications. This empirical, rather than abstract, progress is a hallmark of the current AI research landscape, demanding significant computational resources for experimentation. This sets the stage for his core argument: the "unreasonable effectiveness of statistically emulating a consistent inner monologue" has unveiled a new, unpredictable frontier in computing.
Key Findings
▶ Watch: LLMs inferring developer intent changes security research (15:00)
Halvar Flake's central thesis is that the traditional paradigm of deterministic behavior in software is rapidly eroding, primarily due to the integration of LLMs. He identifies several critical shifts:
- Death of Determinism: The determinism we've come to expect in software, painstakingly built by electrical and process engineers, is fragile and under attack from multiple fronts:
- Hardware Determinism: Cracks emerged with phenomena like Rowhammer, demonstrating that hardware is only "average-case deterministic" and can be made non-deterministic under stress (e.g., voltage glitching, temperature variations).
- Temporal Determinism: The "memory wall" and performance optimizations like speculative execution and caching introduce timing variations. While performance engineers are aware, these became security-critical with vulnerabilities like Spectre, where timing differences leaked secrets across trust boundaries.
- Semantic Determinism: LLMs fundamentally break semantic determinism. They are inherently random, requiring sampling for token generation. Even with fixed seeds, factors like differences in floating point arithmetic implementations across hardware and issues with concurrent inference on multiple GPUs make exact reproducibility challenging.
- LLMs as Chaotic Systems: Beyond mere randomness, LLMs exhibit characteristics of chaotic systems. Drawing an analogy to the butterfly effect from dynamical systems theory (and Jurassic Park), Flake explains that a "very small change in an initial condition... cascades out of control." He cites the Lyapunov exponent as a measure of this sensitivity. In the context of LLMs, changing just one token in a prompt can lead to entirely unpredictable outcomes, a state the formal verification community denotes with the symbol "top" (meaning "anything can happen").
- Software Engineering Enters an Age of Experimentation: Given the inherent randomness and chaotic nature of LLMs, traditional abstract reasoning and deterministic debugging are insufficient. Software development, and especially security, must transform into an empirical science. This requires developers to become amateur or professional statisticians, skilled in experimental design, hypothesis testing, and understanding confidence intervals.
- Inadequacy of Current Workflows: Existing software engineering processes, particularly CI/CD pipelines, are ill-suited for stochastic systems. They are designed for monocausal determinism, where a single test run is often sufficient. With LLMs, evaluating a change requires numerous, often costly, experimental runs to differentiate genuine improvements from random fluctuations. This leads to the uncomfortable conclusion that "most of us are playing slot machines."
- Flawed AI Benchmarking: Flake critically notes that the entire AI community is "fast and loose with their confidence intervals." Many benchmarks assume independence between problems, leading to overly optimistic and potentially misleading statistical conclusions about model performance.
Technical Deep Dive
▶ Watch: LLMs are random and chaotic; small changes cascade (24:00)
Flake delves into the technical underpinnings of why LLMs represent such a profound shift. He clarifies that while often termed "Artificial Intelligence," LLMs are more accurately described as "artificial thought." They simulate the human stream of consciousness or inner monologue, primarily through statistical emulation of text completion. This "artificial thought" has demonstrated an "unreasonable effectiveness" in solving complex problems that were considered impossible just a few years ago. Examples include performing long-horizon tasks using Bash and Python (even leading to accidental compromises, such as on Hugging Face), understanding and translating code between languages, and creating plausible solutions to millennium math problems. A significant implication for security is their ability to infer a developer's intent from code, a task that was historically difficult to automate and crucial for identifying vulnerabilities.
The core of the technical challenge lies in the departure from monocausal determinism, a bedrock principle in computer science. For decades, electrical engineers strove to create hardware that appears deterministic, enabling predictable software behavior. This illusion, however, is fragile. Flake outlines its erosion across three layers:
- Hardware Determinism: Modern chip manufacturing pushes transistors to their physical limits, leading to process variations and inherent flaws. Hardware only behaves deterministically "most of the time if you're being nice to them." Aggressive memory access patterns can trigger Rowhammer, where repeated access to one memory row flips bits in an adjacent row, demonstrating non-deterministic behavior. Similarly, taking a CPU "out of spec" (e.g., by overheating it, as seen with some SGX attacks) can invalidate security guarantees. This means attackers can exploit the "worst hardware that neither you nor the vendor can see as bad."
- Temporal Determinism: The growing disparity between CPU speed and memory access speed (the "memory wall") has led to complex hardware designs like multi-level caching, pre-fetching, and speculative execution. While these improve performance, they introduce highly non-deterministic timing characteristics. CPU manuals guarantee what an instruction does, but not how long it takes. This temporal variability became a critical security vector with Spectre, where secrets could be leaked across trust boundaries by observing differential access times to memory based on speculative execution paths.
- Semantic Determinism: This is where LLMs deliver the final blow. LLMs are, by their very nature, designed to be non-deterministic. They rely on random sampling for the next token to enable problem exploration. Even efforts to make them reproducible with fixed random seeds are thwarted by practical realities:
- Floating point arithmetic is not perfectly commutative and can be implemented differently across various hardware architectures, leading to subtle variations.
- Large models often require concurrent inference across multiple GPUs, introducing further non-determinism due to timing and ordering issues.
- The inherent chaotic systems nature of LLMs means a minuscule change (e.g., one token in a prompt) can propagate unpredictably through the recurrent state equations, leading to drastically different outputs. This sensitivity, measured by the Lyapunov exponent, means that any reasoning about the output after a small input perturbation is effectively impossible ("top" symbol).
This fundamental shift means that software incorporating LLMs can no longer be assumed to behave consistently. The deterministic models in developers' heads are now fundamentally mismatched with the reality of the systems they are building, opening new avenues for security vulnerabilities and making traditional development and debugging paradigms obsolete.
Demo / Proof of Concept
▶ Watch: Tomato farmer protocol for noisy empirical experiments (34:00)
While the introductory remarks for the BlueHat Asia conference highlighted the historical importance of live demonstrations in early BlueHat events—where security researchers would present concrete exploits to Microsoft engineers—Halvar Flake's talk itself did not include a specific live demo or proof of concept. Instead, his presentation focused on a high-level conceptual and philosophical shift in software engineering and security, supported by examples and analogies rather than direct technical demonstrations of LLM exploitation or non-deterministic behavior.
Defensive Implications
▶ Watch: LLM code review like fuzzing: tracking bug 'catch rates' (42:00)
The "age of experimentation" ushered in by LLMs demands a fundamental reorientation for defenders and software engineers alike. Flake argues that everyone involved is now, by necessity, an empirical scientist and an amateur or professional statistician.
- Embrace Empirical Science: Developers must learn experimental design and hypothesis testing. This involves rigorously defining what they want to measure, accounting for all sources of variance, and being acutely aware of self-deception (as famously warned by Richard Feynman: "The first principle as a scientist is you must not fool yourself, and you are the easiest person to fool."). Running a test multiple times and observing its natural fluctuation is crucial before attributing changes to modifications.
- Account for Costly Experiments: LLM experiments are not free; they incur significant computational costs (e.g., "$50, $100" per benchmark run, potentially "$1,000" to validate a small improvement). This means:
- Effect Size Matters: Small improvements may be too expensive to reliably measure. It might be more economical to bundle several changes and test their aggregate effect.
- Noisy Neighbors: Cloud environments introduce external variability (e.g., "noisy VM neighbor" saturating memory bandwidth). Flake recounts losing weeks on performance optimization due to 15% fluctuations in identical code runs. He suggests the "tomato farmer protocol," where two versions of a program run concurrently on the same VM to isolate the effect of a change from environmental noise.
- Rethink CI/CD and Benchmarking: Traditional CI/CD pipelines, designed for deterministic systems, are "poorly suited" for LLM-integrated software. Running just one or a few tests before merging a Pull Request (PR) is insufficient. The cost of thoroughly evaluating a PR (potentially "a couple of thousand dollars") could easily exceed the developer's daily salary, creating a perverse incentive to merge on "a hunch." Furthermore, current AI benchmarks often make overly optimistic assumptions about the independence of test cases, leading to unreliable confidence intervals. Flake estimates that establishing a 1% improvement on a benchmark with 100 known bugs and a 2% standard deviation could require "52 runs" of the full benchmark.
- LLM-based Code Review as Fuzzing: Flake asserts that LLM-based code review (e.g., using tools like Raptor or Code QL security) is more akin to fuzzing than traditional static analysis. Each LLM run, like a fuzzer run with a new random seed, may yield different bugs. The metric to track should be the "rate of useful discoveries per money spent," similar to fishery management where one tracks catch rates per effort, rather than aiming for a definitive "all bugs found" state. New models or prompts will continuously find new issues and miss old ones.
- Strategic Development and Harnessing:
- Benchmark Hierarchy: Build a hierarchy of benchmarks: small, rapid tests for quick iteration, and larger, more representative benchmarks for long-term validation.
- Harnessing and Prompting: While effective now, the value of specific harnessing and prompting techniques will diminish over time as models are trained on these improvements. Continuous innovation in prompts will be necessary to maintain an edge.
- Accelerate the OODA Loop: Making experiments cheaper and faster directly accelerates the "Observe" and "Orient" phases of the OODA loop, which is critical for adapting to rapid change.
Flake concludes that the industry must accept the reality of stochastic systems. While statistical rigor isn't always necessary for every minor change, it's vital to avoid "treading water while feeling productive." The current era of rapid, unsettling technological change demands "focus and not fear."
Key Takeaways
- Determinism is Dying: The long-held assumption of deterministic behavior in software is eroding across hardware (e.g., Rowhammer), temporal (e.g., Spectre), and semantic layers, with LLMs being a primary catalyst for the latter.
- LLMs are Stochastic and Chaotic: Large Language Models introduce inherent randomness and chaotic behavior into software systems, meaning small changes in input (like a single prompt token) can lead to unpredictable and drastically different outcomes.
- Software Engineering Must Become Empirical: Developers and security professionals must transition from relying on abstract, deterministic models to embracing empirical science, requiring a strong understanding of experimental design, statistics, variance, and hypothesis testing.
- Traditional Workflows are Obsolete: Existing CI/CD pipelines, debugging methodologies, and security analysis tools (designed for deterministic systems) are poorly suited for the stochastic nature of LLM-integrated applications, necessitating a re-evaluation of development and testing processes.
- Experiments are Costly and Complex: Reliably measuring improvements in LLM-driven systems is expensive (in compute and time) and challenging due to high variance, requiring a strategic approach to benchmarking and a keen awareness of external noise factors.
- Adaptation is Continuous: The rapid pace of technological change driven by LLMs will continue, demanding constant learning, adaptation, and a focus on building flexible, empirically validated systems rather than clinging to outdated deterministic paradigms.
About the Speaker(s)
Halvar Flake is a distinguished Software Engineer & Cybersecurity Researcher known for his groundbreaking contributions to the field. He was instrumental in educating the security community and Microsoft engineers on advanced exploit techniques, particularly regarding heap overflow exploitation on Windows, which was a poorly understood vector in the early 2000s. Flake also developed Bindiff, a widely used tool for patch diffing that enables security researchers to quickly identify changes in binaries between software versions, aiding in vulnerability discovery and exploit development. He is described as a "great collaborator" and a "really deep thinker," with a history of challenging conventional wisdom and providing critical insights into the evolving landscape of software security.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
Halvar Flake doing what Halvar Flake does: taking a phenomenon everyone's vaguely aware of and articulating it with enough precision that you can't unsee it. The core thesis—that LLMs break semantic determinism and software engineering needs to become empirical science—is correct and undersold in the industry. Not a research talk, but a strategic keynote that earns its slot.
Heather Calloway (CISO) — STRONG ACCEPT
Halvar Flake makes a compelling case that LLM integration fundamentally breaks the deterministic assumptions underlying our entire security and development apparatus. This is a strategic-level argument worth your leadership team's time—not because it hands you a checklist, but because it reframes how you should think about testing, assurance, and vendor claims for any LLM-integrated system.