fail: jmp fail (everything I got wrong in RE and security research)

Christopher Domas (Independent Security Researcher)

RE//verse 2026 · Day 1 · Coral Ballroom A

Overview

In a captivating keynote at RE//verse, independent security researcher Christopher Domas presented "fail: jmp fail," a profound reflection on the often-unseen failures that underpin successful security research. Rather than showcasing exploits and breakthroughs, Domas meticulously detailed a decade's worth of "side quests"—research tangents, improbable ideas, and ultimately unsuccessful projects—arguing that these apparent failures are, in fact, the crucial scaffolding upon which genuine knowledge and future successes are built. His talk championed a philosophy of "reckless curiosity," urging researchers to pursue even mildly interesting or seemingly pointless ideas, and crucially, to document and retain the "artifacts" generated from these endeavors.

Watch on YouTube

Visual summary for fail: jmp fail (everything I got wrong in RE and security research) by Christopher Domas
Visual summary for fail: jmp fail (everything I got wrong in RE and security research) by Christopher Domas

Key moments

  1. 0:00 Introduction: The longest x86 instruction?
  2. 2:00 Unexpected complexity of timing a simple NOP
  3. 6:26 The hidden value of reckless curiosity and failures
  4. 8:00 TSX: Hardware rollback for anti-debugging attempts
  5. 30:00 Discovering the 'BACCD00' CPUID backdoor magic number
  6. 47:00 Solving symbolic execution path explosion with prior failures
  7. 55:00 Achieving a billion-cycle instruction to bypass security
  8. 57:30 Conclusion: The invaluable scaffolding of research failures

fail: jmp fail (everything I got wrong in RE and security research)

Speakers: Christopher Domas (Independent Security Researcher)

Conference: RE//verse

YouTube: https://www.youtube.com/watch?v=iOq8O_phwbA

Overview

In a captivating keynote at RE//verse, independent security researcher Christopher Domas presented "fail: jmp fail," a profound reflection on the often-unseen failures that underpin successful security research. Rather than showcasing exploits and breakthroughs, Domas meticulously detailed a decade's worth of "side quests"—research tangents, improbable ideas, and ultimately unsuccessful projects—arguing that these apparent failures are, in fact, the crucial scaffolding upon which genuine knowledge and future successes are built. His talk championed a philosophy of "reckless curiosity," urging researchers to pursue even mildly interesting or seemingly pointless ideas, and crucially, to document and retain the "artifacts" generated from these endeavors.

Domas, known for groundbreaking past research such as Canter Dust, Mafia Scater, and Sand Sifter, challenged the prevailing narrative of linear, uninterrupted discovery in security. He posited that by exclusively sharing successes, the community inadvertently creates a false sense of continuity, obscuring the messy, non-linear, and failure-ridden reality of technical exploration. Through a series of personal anecdotes, he illustrated how seemingly useless tools or obscure pieces of knowledge, derived from failed attempts, can unexpectedly coalesce to solve entirely different, complex problems later on.

The core message of "fail: jmp fail" is a call to embrace the "fail jump fail" loop as a fundamental part of the research process. It's an invitation to view dead ends not as wasted time, but as opportunities to map the "darker corners" of systems and accumulate a personal "bag of failures"—a collection of specialized tools, niche knowledge, and unique perspectives that might one day prove invaluable. This keynote served as a powerful reminder that in the pursuit of understanding complex systems, the journey through failure is often more enriching and transformative than the destination of success.

Background

▶ Watch: Introduction: The longest x86 instruction? (0:00)

The talk began with a seemingly innocuous question that spiraled into a profound exploration of computational complexity and research methodology: "How many cycles does the longest machine instruction take on x86?" Domas initially attempted to answer a simplified version of this question: how long does a NOP (No Operation) instruction take? What appeared to be a straightforward measurement quickly devolved into a multi-day struggle.

His initial approach—reading the Timestamp Counter (TSC), executing NOP, reading TSC again, and computing the difference—yielded wildly inconsistent results (e.g., 40 cycles, 20 cycles, even negative cycles). This unexpected variability forced him into a deep dive into complex CPU behaviors he hadn't initially considered, such as out-of-order execution, fencing instructions (like LFENCE), and register dependencies. He eventually learned that accurately timing instructions requires specialized techniques, including the use of RDTSCP and LFENCE instructions to serialize the CPU pipeline. Even after laboriously determining a NOP takes "one cycle," he discovered further complexity: NOP isn't a single instruction but has multiple aliases across different CPU vendors, generations, and SKUs, each potentially with different timings and encodings.

This initial "failure" to answer a simple question highlighted Domas's central thesis: reality is surprisingly complex, and even minor curiosities can lead to extensive, unexpected side quests. He described this tendency as a "total inability to not pursue things that are even just mildly interesting," comparing it to a gamer relentlessly pursuing side quests for seemingly useless rewards. This personal research philosophy, he argued, is where the most interesting discoveries and the most valuable "artifacts" are found, even if the original goal is never met. The talk then proceeded to enumerate numerous such "failed" side quests from his past research, each yielding unique insights and tools.

Key Findings

▶ Watch: The hidden value of reckless curiosity and failures (6:26)

The overarching "key finding" of Domas's talk is the intrinsic value of research failure and the strategic accumulation of "artifacts" from these endeavors. He demonstrated that even when a primary research goal is not met, the process often yields specialized tools, obscure knowledge, and novel perspectives that can unexpectedly contribute to future, unrelated successes.

Here are some of the technical "failures" and their resulting "artifacts" highlighted in the talk:

  • TSX Experiments (Transactional Synchronization Extensions):
  • Goal: Anti-debugging (transactional aborts on debugger interrupts), opaque predicates (undocumented abort conditions), and covert channels (L1 cache evictions inducing aborts).
  • Failure: Anti-debugging was too limited, opaque predicates were brittle due to undocumented conditions changing across microcode updates, and the covert channel was too slow and of questionable utility.
  • Artifacts: An obscure obfuscator and "ineffective code to evict cache lines in some specific situations."
  • Binary Self-Symmetry Visualization:
  • Goal: Visualize repeated data in binaries to identify related tables or structures.
  • Failure: Extremely slow to process (hours for small files), difficult to integrate into a practical reverse engineering workflow, often made the problem of analysis worse.
  • Artifacts: An ability to search for repeated substrings within a file.
  • Power Analysis for Reverse Engineering (RAPL registers):
  • Goal: Use RAPL registers to perform fine-grained power analysis, potentially down to individual instruction or micro-op level, to leak microcode.
  • Failure: RAPL registers have an abysmal 1ms refresh rate with significant, unpredictable jitter, making fine-grained measurements extremely difficult.
  • Artifacts: Specialized drivers for precise power measurements across different CPU cores.
  • Transmeta Code Morphing Software (CMS) DMA Bypass:
  • Goal: Use a PCIe DMA device to locate and reverse engineer the hidden Code Morphing Software (CMS) in Transmeta processors.
  • Failure: The CMS was not found in memory via DMA, suggesting deeper hardware-level interception at the Northbridge.
  • Artifacts: A DMA card, now a "neat toy" for future experiments.
  • Branchless ARM Compiler for Glitching Hardening:
  • Goal: Create code resilient to glitching attacks by removing all conditional branches, replacing them with array indexing based on computed conditions.
  • Failure: Made code "really, really slow," introduced predictable memory access patterns exploitable by glitching, and was still vulnerable to dropping individual instructions.
  • Artifacts: An ARM global control flow graph flattening tool (the "burnt pancake corner").
  • Wrap 32 Disable Exploitation:
  • Goal: Exploit code relying on 32-bit address wraparound by toggling the wrap 32 disable bit in a control register, targeting System Management Mode (SMM) firmware.
  • Failure: Lack of suitable legacy 32-bit code in firmware that actually used segmentation and relied on wraparound behavior.
  • Artifacts: A "janky" firmware assembly search tool capable of scanning large databases for specific assembly patterns.
  • Secret CPUID Leaves:
  • Goal: Investigate a mysterious CPUID leaf (BACCD00, interpreted as "backdoor" in hex speak) found in firmware, search for other undocumented leaves, and analyze their timing behavior.
  • Failure: BACCD00 was revealed to be a signaling mechanism for debuggers under emulation, and no other secret leaves or unusual timings were found on physical hardware.
  • Artifacts: An improved instruction timing code capable of collecting statistics and analyzing anomalies, and a CPUID analysis tool for iterating and timing leaves.
  • CPU Errata Exploration (Intel Quark):
  • Goal: Reproduce and exploit documented processor errata on the simple Intel Quark architecture using an instruction fuzzer.
  • Failure: Even with clear documentation, reproducing exploitable fringe cases on modern CPUs proved extremely difficult and time-consuming.
  • Artifacts: Infrastructure for diffing CPU states and a tool for automatically generating and mutating machine code snippets.
  • PCIe Peripheral Spoofing:
  • Goal: Automatically spoof various PCIe device and vendor IDs using an FPGA-based DMA card to observe how platforms interact with different peripherals.
  • Failure: Too slow (brute-forcing 4 billion IDs, lengthy FPGA bitstream rebuilds, and power cycles).
  • Artifacts: An FPGA bitstream pipeline for much faster firmware rebuilding on DMA cards.
  • SKINIT Cache Coherency Attack:
  • Goal: Compromise a secure loader during SKINIT execution by using cache coherency instructions (WBINVD) to desync memory measurement.
  • Failure: Likely due to thread quiescing mechanisms during SKINIT, preventing concurrent cache manipulation.
  • Artifacts: Specialized cache primitives for "sniping" specific cache lines with precise timing.
  • Synthesizing an ISA:
  • Goal: Automatically derive machine encodings and generate an assembler/disassembler for an undocumented instruction set by fuzzing instructions and observing state changes.
  • Failure: The tooling was immensely complex and time-consuming to build (200 hours vs. 10 hours for manual analysis), brittle, and only worked for one specific architecture.
  • Artifacts: The concept of synthesizing ISAs through automated fuzzing and state diffing.
  • Branchless Symbolic Execution:
  • Goal: Solve the path explosion problem in symbolic execution by applying the "branchless compiler" concept to collapse multiple symbolic states into one.
  • Failure: It merely traded an "insurmountable number of states" for a "single insurmountably complex state," creating a new, equally intractable problem.
  • Artifacts: Re-validation of the "burnt pancake corner" (the ARM global control flow graph flattener) as a tool that might be unexpectedly useful.
  • NOP Computation:
  • Goal: Explore the theoretical possibility of performing computation using only NOP instructions by manipulating invisible CPU state (timing, cache, power, PMCs, speculative execution).
  • Failure: No concrete method found, "spiraled into hundreds of bad ideas."
  • Artifacts: Deep knowledge of the vast, undocumented space of NOP variations (quadrillions of them beyond standard recommendations).
  • The Longest Machine Instruction:
  • Goal: Find an instruction that runs for "10 million cycles" to bypass a core sleep timeout, building on the initial NOP timing quest.
  • Method: A complex chain involving a custom FPGA bitstream on a DMA card to slow down MMIO accesses, disabling a Northbridge watchdog timer, and an 8-byte PCI config space read carefully offset to overlap multiple MMIO registers.
  • Result: Achieved a single move instruction taking "north of a billion cycles."
  • Failure: Other subsequent defenses prevented full exploitation, but the original question was arguably answered.
  • Artifacts: A concrete technique for achieving an incredibly long-running single instruction.

Technical Deep Dive

▶ Watch: Discovering the 'BACCD00' CPUID backdoor magic number (30:00)

Christopher Domas's talk, while philosophical, was deeply rooted in technical explorations. Several examples stand out for their intricate technical details and the creative problem-solving involved, even in failure.

TSX for Anti-Debugging and Covert Channels

Transactional Synchronization Extensions (TSX), a feature introduced by Intel, allows developers to define transactional memory regions. Code within an XBEGIN and XEND block can modify CPU state, but if certain conditions (e.g., interrupts, conflicts) occur, the transaction aborts, and all changes are rolled back, branching to an abort handler.

Domas explored using this rollback capability for security purposes:

  1. Anti-Debugging: Malicious code could reside within a transactional block. If a debugger sets a breakpoint or single-steps, it triggers an interrupt, causing a transactional abort. The CPU state rolls back, and execution jumps to a benign path, showing the debugger harmless code. The challenge was its limited utility, as the debugger had to specifically hit the transactional region.
  2. Opaque Predicates: TSX specifications list some abort conditions but explicitly state that others are undocumented and can change. Domas built a tool to generate random code snippets within transactional regions and observe if they caused aborts. This allowed him to build a library of "transactional opaque predicates"—code blocks whose branching behavior (commit or abort) was known to the malware author but difficult for a reverse engineer to predict without execution, and potentially unstable across CPU generations.
  3. Covert Channels: Reverse engineering efforts had shown TSX tracks changes in the L1 data cache. Domas hypothesized that a sending thread could induce cache line evictions, which in turn could trigger transactional aborts on a receiving thread running a transactional region that continuously reads from a buffer. By counting these aborts, the receiver could infer data sent by the sender, creating a covert channel without explicit IPC. While technically intriguing, this proved "way too slow" for practical use.

Fine-Grained Power Analysis with RAPL Registers

The quest for fine-grained power analysis aimed to observe the power consumption of individual machine instructions or even micro-operations. Domas identified RAPL (Running Average Power Limit) registers as a potential software-based solution, avoiding specialized hardware. The initial idea was simple: read RAPL, execute code (e.g., AES), read RAPL again, and measure the difference.

The critical hurdle was the "abysmal" refresh rate of RAPL registers—approximately 1 millisecond. This meant that any instruction or algorithm running faster than 1ms would be lost in the noise of other system activity within that measurement window. Domas devised an ingenious technique to overcome this:

  1. Synchronized Measurement: Repeatedly read a RAPL register until its value changes. This signals the start of a new 1ms measurement window.
  2. Precise Delay: Wait for almost 1 millisecond.
  3. Algorithm Execution & Second Read: Start executing the target algorithm and immediately read the RAPL register again. This "snipes" the very beginning of the algorithm's power consumption within the new measurement window.
  4. "Dragging" the Algorithm: Repeat the entire process thousands of times, each time slightly adjusting the initial delay. This effectively "drags" the target algorithm across the second RAPL measurement point, allowing for the construction of a detailed power profile by filtering out noise.

Despite this clever methodology, Domas discovered significant, unpredictable jitter in the RAPL update intervals, adding tremendous noise. Weeks of effort to reverse engineer this jitter failed, and the ultimate goal of micro-op granularity remained elusive, potentially because RAPL registers might not even update at that level.

The Billion-Cycle MOVE Instruction

Domas's talk began with the simple question of the longest instruction and culminated in its potential answer: a single move instruction engineered to take "north of a billion cycles." This was a complex, multi-layered "MacGyver-style" attack:

  1. Target: Bypass a security check that relied on a CPU core sleeping, requiring an instruction that would keep the core busy for over 10 million cycles.
  2. Slow DMA Device: Domas reused his PCIe DMA card, specifically a PCIe Squirrel (an FPGA-based COTS device). He modified its FPGA bitstream to deliberately slow down its response time to accesses to its extended configuration space. This turned the DMA card into a controlled bottleneck.
  3. MMIO Space Discovery: On the CPU side, he used an MSR (Model-Specific Register) read to discover the Memory-Mapped I/O (MMIO) space, which is where the CPU communicates with peripherals like the DMA card.
  4. Northbridge Watchdog Disable: Often, the Northbridge (a critical component connecting the CPU to high-speed devices like PCIe) includes watchdog timers that prevent MMIO transactions from taking too long. These needed to be disabled to allow the deliberately slow DMA access to complete.
  5. Overlapping PCI Config Space Read: The core of the technique involved a specific type of PCI access. While the CPU specs advise against it, Domas performed an 8-byte read to the PCI config space. Crucially, this read was offset such that it would overlap multiple MMIO registers on the custom-slowed DMA device. Behind the scenes, a single 8-byte move instruction from the CPU would translate into a series of individual MMIO accesses to the DMA card. Because each of these underlying MMIO accesses was artificially slowed by the FPGA bitstream, the cumulative effect was an incredibly prolonged execution time for a single CPU instruction.

This intricate combination of hardware manipulation (custom FPGA firmware), low-level software control (MSR reads, Northbridge access), and undocumented CPU behavior (overlapping PCI config space reads) resulted in a single move instruction that effectively halted the CPU for over a billion cycles. While this specific bypass didn't lead to full exploitation due to other defenses, it definitively answered Domas's initial, seemingly trivial question, showcasing the power of combining disparate "failed" artifacts.

Demo / Proof of Concept

▶ Watch: Solving symbolic execution path explosion with prior failures (47:00)

While Christopher Domas's talk was rich with descriptions of various experiments, tools, and technical prototypes, it did not feature a live, interactive demonstration of any specific exploit or tool. Instead, the "Proof of Concept" for each "failure" was presented through detailed explanations of the methodology, the challenges encountered, and the resulting "artifacts"—the code snippets, conceptual frameworks, or specialized tools that were developed along the way. The narrative itself served as a compelling demonstration of his research philosophy, illustrating how each side quest, despite its ultimate failure to achieve its primary objective, contributed to a broader understanding and a growing "inventory" of useful components.

Defensive Implications

▶ Watch: Conclusion: The invaluable scaffolding of research failures (57:30)

Christopher Domas's "fail: jmp fail" offers several profound implications for defenders, even though it focuses on research failures rather than direct exploits:

  1. Embrace "Reckless Curiosity" in Defense: Just as attackers probe the "darker corners," defenders should adopt a similar mindset. Don't limit security assessments to known vulnerabilities or common attack paths. Pursue odd behaviors, undocumented features, and seemingly irrelevant details. These "side quests" can uncover obscure weaknesses or provide deeper insights into system resilience.
  2. Document and Share Internal Failures: Organizations often focus on successes, but internal "failures"—unsuccessful exploit attempts, dead-end research, or unexpected system behaviors that didn't lead to a direct finding—are invaluable. Documenting these, creating an internal "bag of failures," can foster a knowledge base where seemingly useless information from one project might be the missing piece for another. This breaks the illusion of linear progress and encourages a more realistic, collaborative understanding of complex systems.
  3. Understand Low-Level Hardware-Software Interactions: Many of Domas's side quests delve into intricate interactions between software, CPU microcode, and hardware components (e.g., TSX, RAPL, DMA, PCIe, SMM, CPUID). Defenders must recognize that robust security requires understanding these layers, as subtle misconfigurations or undocumented behaviors at the hardware level can undermine software protections.
  4. Anticipate Novel Side-Channel Attacks: Even when Domas's attempts at power analysis or cache coherency attacks failed, the underlying concepts (e.g., using RAPL registers for fine-grained timing, cache manipulation during secure boot) remain potent. Defenders should be aware that such techniques, even if difficult to perfect, can be developed and refined by persistent attackers.
  5. Beyond Documentation: Validate System Behavior: The discovery of undocumented TSX abort conditions, the wrap 32 disable bit, or the BACCD00 CPUID leaf highlights that relying solely on public documentation can be risky. Defenders should employ internal research and fuzzing (similar to Domas's instruction input fuzzer or errata exploration) to validate expected system behaviors and uncover undocumented features that might be exploited.
  6. Complexity as Both a Shield and a Weakness: The talk consistently shows that even "simple" questions unravel into immense complexity (e.g., NOP timing). This complexity can make exploitation difficult, but it also creates a vast attack surface where unexpected interactions can lead to vulnerabilities. Defenders should approach complex systems with humility, acknowledging that perfect understanding is elusive and hidden flaws are likely.
  7. The "Longest Instruction" as a Primitive: The final "billion-cycle move instruction" demonstrates that attackers can engineer incredibly long-running single instructions by chaining hardware-software interactions. This primitive could be used to bypass time-based security checks, trigger watchdog timeouts, or induce race conditions in other contexts. Defenders should consider the resilience of time-sensitive security mechanisms against such prolonged, single-instruction stalls.

Ultimately, Domas's talk encourages defenders to foster a culture of deep, persistent, and unconventional inquiry, recognizing that the most robust defenses often emerge from the lessons learned in the pursuit of the seemingly impossible.

Key Takeaways

  • Failure is a Foundation for Success: Research failures, though not yielding immediate exploits, are invaluable for expanding knowledge, developing specialized tools, and building the conceptual "scaffolding" for future breakthroughs.
  • Accumulate "Artifacts": Do not discard or forget the tools, code snippets, or obscure insights gained from failed projects. These "artifacts" (like a "burnt pancake corner" or a "broken pocket watch") can unexpectedly combine to solve new, complex problems.
  • Embrace Reckless Curiosity: Pursue ideas that are weird, improbable, or mildly interesting, even if they seem pointless. This "reckless curiosity" leads to exploration of the "darker corners" of systems, revealing unique insights that narrow, goal-oriented research might miss.
  • Security Research is Non-Linear: The illusion of continuous, linear discovery in security research is false. The reality is messy, filled with bad ideas, invisible failures, and constant backtracking. Sharing these failures provides a more accurate and beneficial understanding of the research process.
  • Deep Technical Understanding is Critical: Many "failures" stemmed from unexpected complexity at the microarchitectural level (TSX aborts, RAPL jitter, NOP aliases). A profound understanding of low-level CPU and hardware interactions is essential for both finding and defending against vulnerabilities.
  • The "Longest Instruction" is an Engineered Primitive: A single machine instruction can be engineered to run for an extraordinarily long duration (over a billion cycles) by chaining hardware-software interactions, which can be a powerful primitive for bypassing time-based security checks.

About the Speaker(s)

Christopher Domas is an independent security researcher renowned for his profound technical contributions and engaging presentations. He has been described as a "tour de force" in the security community, consistently delivering "amazing both technical contents and presentations." Domas is recognized for several highly impactful past research projects, including Canter Dust, Mafia Scater, and Sand Sifter. His work often delves into the intricate depths of processor architectures and low-level system interactions, pushing the boundaries of what is understood about hardware and software security. His keynote at RE//verse underscored his unique perspective on research methodology, emphasizing the often-overlooked value of embracing failure and pursuing unconventional paths in the quest for knowledge.

Reviews

Dr. Zero (Offensive Security Researcher) — MUST SEE

Domas strips away the polished CVE theater and shows the grueling, messy reality of real reverse engineering. This is what actual exploit development looks like—months of dead ends, weird hardware behaviors, and broken tooling. If you want to know how the best in the world actually work, you watch this.

Heather Calloway (CISO) — SOLID

This is a masterclass in hardware research methodology from one of the discipline's absolute best. It is not a talk for your board, your CFO, or your risk committee. Send your vulnerability research team to watch it to understand how elite adversaries map the darker corners of a system, but don't expect a direct policy output.

→ Top-rated talks at RE//verse 2026

All talks from RE//verse 2026