Thinking Like a Compiler: Obfuscation from the Other Side

Laurie Kirk (Researcher · Google)

RE//verse 2026 · Day 2 · Coral Ballroom A

Overview

In her keynote address at RE//verse, Google researcher Laurie Kirk presented a compelling argument for a paradigm shift in reverse engineering: instead of viewing the compiler as an adversary, reverse engineers should endeavor to "think like a compiler." This approach advocates for a deep understanding of the compilation, linking, and source code transformation processes to effectively peel back layers of obfuscation. Kirk, known for her contributions to open-source tools and her educational YouTube channel, challenged the traditional mindset of treating binaries as opaque black boxes, proposing that insight into how software is built can dramatically improve deobfuscation efforts.

Watch on YouTube

Visual summary for Thinking Like a Compiler: Obfuscation from the Other Side by Laurie Kirk
Visual summary for Thinking Like a Compiler: Obfuscation from the Other Side by Laurie Kirk

Key moments

  1. 4:28 Dramatic compiler optimization: C++ to two lines assembly
  2. 9:00 LLVM Intermediate Representation: The universal language bridge
  3. 18:00 Obfuscator-LLVM's hidden cost: De-optimization and performance hits
  4. 27:00 Beating stripped static linking and missing symbols
  5. 37:00 Demangling C++ names to reconstruct original code structure
  6. 49:00 Future challenges: Framework bloat and LLM context limits
  7. 53:10 Decompilers vs. LLMs: A hybrid future for reverse engineering

Thinking Like a Compiler: Obfuscation from the Other Side

Speakers: Laurie Kirk, Researcher, Google

Conference: RE//verse

YouTube: https://www.youtube.com/watch?v=jfqFHHsYQAs

Overview

In her keynote address at RE//verse, Google researcher Laurie Kirk presented a compelling argument for a paradigm shift in reverse engineering: instead of viewing the compiler as an adversary, reverse engineers should endeavor to "think like a compiler." This approach advocates for a deep understanding of the compilation, linking, and source code transformation processes to effectively peel back layers of obfuscation. Kirk, known for her contributions to open-source tools and her educational YouTube channel, challenged the traditional mindset of treating binaries as opaque black boxes, proposing that insight into how software is built can dramatically improve deobfuscation efforts.

The talk systematically deconstructs the journey from high-level source code to executable binary, highlighting points where obfuscation techniques are introduced, either intentionally by malware authors or inadvertently through compiler optimizations and language features. Using a custom "encrypt binary" as a running example, Kirk demonstrated how knowledge of the underlying compiler ecosystem, particularly LLVM, can transform a seemingly impenetrable piece of code into a comprehensible structure. This perspective is vital for modern reverse engineering, as advanced obfuscation often leverages the very mechanisms compilers use to optimize code.

Kirk's presentation not only offered practical strategies for tackling common obfuscation challenges but also emphasized the importance of proactive learning. She urged reverse engineers to stay ahead of the curve by understanding emerging programming languages and compiler research before they become prevalent in the threat landscape. By adopting a compiler-centric view, the talk posits that reverse engineers can move beyond reactive analysis, developing more robust and efficient deobfuscation methodologies for the increasingly complex binaries encountered today.

Background

▶ Watch: Dramatic compiler optimization: C++ to two lines assembly (4:28)

The traditional approach to reverse engineering often treats a compiled binary as an opaque entity, a "black box" whose internal workings must be deciphered without prior knowledge of its creation. This reactive stance places reverse engineers in a constant struggle against the compiler, which, from their perspective, transforms readable source code into an often convoluted and optimized machine-level representation. The core problem, as Laurie Kirk articulates, is that this perspective overlooks a crucial insight: obfuscation developers, malware authors, and even legitimate software engineers are not fighting the compiler; they are working with it. They leverage its capabilities—both intended and unintended—to produce applications that are challenging to analyze.

To illustrate this point, Kirk presented a simple C++ code snippet involving std::accumulate and a lambda function. When compiled with the lowest optimization level (-O0), the resulting assembly code closely mirrored the C++ structure, yielding numerous lines. However, when the same code was compiled with the highest optimization level (-O3), the assembly shrunk to a mere two lines. This dramatic reduction is a testament to the compiler's optimization passes, which analyze input source code for inefficiencies and redundancies, transforming it into a more efficient and smaller binary. This example underscores the compiler's profound influence on the final binary's structure, an influence that can be harnessed for both legitimate optimization and malicious obfuscation.

Understanding this transformative process requires comprehending the vast and complex compiler ecosystem. Kirk likened this ecosystem to the Andromeda galaxy, replete with diverse compilation toolchains (e.g., LLVM, GCC, MSVC), multiple programming languages (C++, Rust, Swift, Zig, Kotlin, Nim, Vlang), and various instruction set architectures. Obfuscation can be injected at any stage within this "galaxy," from source code modifications to compilation and linking. At the heart of many modern toolchains lies LLVM, a versatile compiler infrastructure that acts as a common intermediary for numerous languages. By focusing on LLVM, reverse engineers can gain insights that apply broadly across a wide array of binaries, moving beyond the black-box mentality to a more informed, compiler-aware analysis.

Key Findings

▶ Watch: Obfuscator-LLVM's hidden cost: De-optimization and performance hits (18:00)

Laurie Kirk's talk delivered several key findings that collectively advocate for a fundamental shift in the reverse engineering paradigm:

1. The Compiler as an Ally, Not an Adversary: The central finding is that reverse engineers should stop "fighting" the compiler and instead leverage an understanding of its internal processes. By "thinking like a compiler," one can anticipate how source code translates to machine code, how optimizations occur, and where obfuscation techniques are likely to be injected. This proactive stance transforms the challenge of deobfuscation into a structured, layered analysis.

2. LLVM's Pervasive Influence: A critical insight is the widespread adoption of LLVM across the modern software development landscape. Languages like Rust and Swift directly utilize LLVM modules, and even newer languages like Nim and Vlang can transpile to C, which is then often compiled by Clang, an LLVM-based compiler. This makes LLVM Intermediate Representation (IR) a crucial, almost universal, "Rosetta Stone" for understanding a vast range of binaries. Mastering LLVM IR provides a powerful common denominator for deobfuscation.

3. Obfuscation as De-optimization: Many common obfuscation techniques, particularly those implemented at the compiler level (e.g., Obfuscator-LLVM/OLLVM, Armariris, Hikari), are essentially "de-optimizations." They introduce bloat, redundant code, and complex control flow that, while hindering analysis, also degrade performance. This performance hit can be a critical weakness for malware authors, as it increases execution time and resource consumption, which can be leveraged by defenders.

4. Recovering Lost Context from Compiler Artifacts: Even when symbols are stripped, compilers leave behind valuable artifacts. C++ vtables and Run-Time Type Information (RTTI), when present, offer rich structural information about classes and methods. Furthermore, name mangling, a compiler mechanism to create unique identifiers, paradoxically encodes the original function signatures and class names in a standardized, reversible format (e.g., the _Z prefix in C++). These artifacts, when understood and demangled, can reconstruct significant portions of the original source code structure.

5. The Importance of Proactive Learning: The talk strongly emphasizes the need for reverse engineers to proactively engage with new programming languages, frameworks, and compiler research. Reacting only after a new language like Rust becomes prevalent in malware places defenders at a disadvantage. By studying the compilation process of emerging technologies (e.g., C++ 26 reflection, e-graphs), reverse engineers can anticipate future obfuscation challenges and develop appropriate tooling before they become widespread threats.

Technical Deep Dive

▶ Watch: Beating stripped static linking and missing symbols (27:00)

Laurie Kirk's technical deep dive systematically explored the layers of the compiler ecosystem, demonstrating how obfuscation is introduced and subsequently countered by "thinking like a compiler." The process was broken down into three main layers: the compiler itself, the linker, and source code modifications.

Layer 1: The Compiler and LLVM-based Obfuscation

The journey begins with the compilation process, where source code is transformed. Kirk focused on LLVM, highlighting its modular structure. The process typically involves:

  1. Parser: Takes high-level source code and converts it into LLVM Intermediate Representation (IR). This IR acts as a bridge between the source and the target machine code.
  2. Optimizer: Takes the LLVM IR and runs multiple passes to perform optimizations, making the code more efficient. Crucially, this is also where LLVM-based obfuscation techniques are inserted.
  3. Generator: Takes the optimized LLVM IR and produces the final machine code for the target architecture.

Kirk explained that many languages beyond C/C++—such as Rust, Swift, Zig, Kotlin, Nim, and Vlang—either directly use LLVM or transpile to C, which is then compiled by an LLVM-based compiler like Clang. This makes LLVM IR a common denominator for analysis.

Obfuscation at this layer, epitomized by Obfuscator-LLVM (OLLVM) and newer tools like Armariris and Hikari, works directly on the LLVM IR. These are essentially custom LLVM passes that act as "de-optimizations." Common techniques include:

  • Instruction Substitution: Replacing simple instructions with more complex, functionally equivalent sequences.
  • Control Flow Flattening: Transforming linear control flow into convoluted structures, often involving nested while (true) or while (1) loops and dispatch tables, making the execution path difficult to follow.
  • Bogus Instructions/Branches: Injecting dead code or conditional jumps that always evaluate to true/false, adding bloat without altering functionality.

The counter-strategy involves recognizing these as performance-impacting transformations. Decompilers lift the machine code into their own internal Intermediate Representation (IR) (e.g., IDA's microcode or Binary Ninja's IR). At this stage, techniques like symbolic execution can be applied to identify and remove extraneous, unreachable, or functionally redundant code, effectively "re-optimizing" the obfuscated binary back to its core functionality. Kirk demonstrated how a highly flattened function graph could be simplified to its original, streamlined form.

Layer 2: The Linker and Symbol Stripping

The second layer focuses on the linker process, which combines compiled object files and libraries into a final executable. Kirk contrasted dynamic linking and static linking.

  • Dynamic Linking: The linker inserts placeholders, such as the Procedure Linkage Table (PLT) and Global Offset Table (GOT), into the binary. Calls to library functions (e.g., printf) are resolved at runtime. The PLT contains trampoline code that jumps to an address in the GOT, which is updated by the dynamic linker with the correct library function address during execution. This keeps binary sizes small.
  • Static Linking: The linker copies all necessary library code directly into the final binary. This results in much larger executable sizes but eliminates runtime dependencies. Instead of external jumps, calls become internal to the binary.

A significant obfuscation technique at this stage is symbol stripping, achieved with a simple compiler flag like -s. This removes function and variable names, turning familiar library calls into generic, unlabeled addresses. For statically linked binaries, this makes identifying common library functions (like printf or memcpy) extremely challenging.

To combat this, reverse engineers employ heuristics based on code structure. While simple hash matching is often brittle (due to compiler versions, architectures, and flags), structural patterns within functions can often reveal their identities. Tools within decompilers attempt to automatically rename these recognized library functions.

Kirk also highlighted Link Time Optimization (LTO), a powerful optimization that merges user code and library code. While beneficial for performance, LTO makes it nearly impossible to distinguish between user-defined functions and library functions, forcing reverse engineers to analyze virtually every part of the binary. This was noted as a particular pain point when working with languages like Rust.

Layer 3: Source Code and High-Level Language Features

The final layer delves into obfuscation and information leakage inherent in high-level programming language features, specifically focusing on C++.

  • C++ Vtables and RTTI: C++ uses vtables (virtual tables) to implement polymorphism and dynamic dispatch for virtual functions. A hidden member variable in a class object points to its vtable, which contains pointers to the actual implementations of virtual methods. Additionally, Run-Time Type Information (RTTI) can be embedded in the binary, providing metadata about classes and their types. Together, these can reconstruct significant class hierarchies and method structures. However, RTTI can be easily disabled with a no RTTI compiler flag, effectively stripping this valuable information.
  • Name Mangling: To allow for features like function overloading, C++ compilers use name mangling to create unique identifiers for functions, classes, and variables. These mangled names (e.g., starting with _Z for C++) encode information about the function's class, name, return type, and arguments. While seemingly obfuscated, this follows a standardized structure, allowing for demangling tools or even simple parsing to recover the original, human-readable names and signatures (e.g., _Z4Base7method3v becomes void Base::method3()).
  • Multi-threading and Standard Library Usage: High-level constructs, such as std::atomic for concurrency, can translate into surprisingly complex assembly. The compiler might generate calls to operator overloads (e.g., operator++ instead of counter++) or special atomic instructions. Understanding these transformations is crucial. Tools like C++ insights allow reverse engineers to see how high-level C++ code is interpreted and expanded by the compiler, bridging the gap between source and compiled output. By combining name demangling with insights into standard library implementations, Kirk demonstrated how intricate multi-threading code could be mapped back to simple, high-level C++ operations.

By systematically applying these techniques, Kirk showed how the "encrypt binary" could be progressively deobfuscated, revealing its core ICryptoEngine and CryptoEngine class structure, multi-threading logic, and overall intent.

Demo / Proof of Concept

▶ Watch: Future challenges: Framework bloat and LLM context limits (49:00)

Throughout the talk, Laurie Kirk used a custom "encrypt binary" as a continuous, conceptual proof of concept. This wasn't a live coding demonstration, but rather a step-by-step visual unfolding of the binary's internal structure as different deobfuscation techniques were applied. She presented "before" and "after" views of the binary's disassembled code, function graphs, and symbol tables.

Initially, the encrypt binary appeared highly obfuscated, featuring flattened control flow graphs with nested infinite loops, a clear indicator of Obfuscator-LLVM techniques. As Kirk discussed the principles of re-optimization and symbolic execution, the complex graph was shown to simplify, revealing the underlying core functionality. Later, when addressing static linking and symbol stripping, she showed how a binary with generic, unlabeled function addresses could have known library functions automatically identified and renamed using structural heuristics. Finally, in the source code layer, the encrypt binary's mangled C++ names were demangled to expose its object-oriented structure, including an ICryptoEngine interface and a CryptoEngine child class, along with its multi-threading operations. This running example effectively demonstrated the cumulative power of "thinking like a compiler" to progressively reconstruct the original logic.

Defensive Implications

▶ Watch: Decompilers vs. LLMs: A hybrid future for reverse engineering (53:10)

The insights presented in "Thinking Like a Compiler" carry significant implications for defensive security practitioners and reverse engineers:

  1. Proactive Language and Framework Monitoring: Defenders must shift from a reactive stance to a proactive one. Instead of waiting for new languages (like Rust or Go) or compiler features (like C++ 26 reflection) to appear in malware, security researchers should actively study their compilation processes, standard library implementations, and potential obfuscation vectors before they become widespread. This foresight allows for the development of deobfuscation strategies and tooling in advance.
  1. Deep Compiler Ecosystem Knowledge: A thorough understanding of compiler toolchains, especially LLVM, is no longer a niche skill but a fundamental requirement. Since many modern languages funnel through LLVM IR, mastery of this intermediate representation provides a powerful, universal lens for analyzing diverse binaries. Investing in training and tools that operate at the IR level can enhance deobfuscation capabilities.
  1. Leveraging Performance Trade-offs: Recognizing that many compiler-level obfuscations (OLLVM, Armariris, Hikari) are essentially "de-optimizations" means that defenders can look for performance overheads. Malware authors face a trade-off between obfuscation strength and execution efficiency (e.g., slow ransomware encryption could lead to detection). This trade-off can be an Achilles' heel for attackers.
  1. Sophisticated Symbol Recovery: Simple hash matching for identifying library functions is often insufficient due to varying compiler versions, architectures, and LTO. Defenders need to employ more robust structural heuristics and pattern recognition techniques within their decompilers and analysis tools to reliably identify and label known functions, even in stripped binaries.
  1. Context Management for Automated Analysis: The increasing reliance on automated reverse engineering, including Large Language Models (LLMs), highlights the challenge of "context bloat." Heavily obfuscated binaries with excessive, irrelevant code can overwhelm LLMs with token limits. Defenders need to develop intelligent pre-processing and filtering techniques to extract relevant code segments before feeding them to automated analysis systems, ensuring efficiency and accuracy.
  1. Stay Abreast of Research: Keeping up with academic research in compiler optimization and program analysis (e.g., e-graphs for equivalence checking and optimal form selection) can provide insights into future de-obfuscation techniques. Applying these research findings to reverse engineering problems can yield innovative solutions to emerging obfuscation methods.

By embracing these defensive implications, security teams can develop more resilient and forward-thinking strategies to counter the evolving landscape of software obfuscation.

Key Takeaways

  • Shift Perspective: Instead of fighting the compiler, understand its processes (compilation, linking, source transformation) to effectively deobfuscate binaries.
  • LLVM is Key: Many modern languages and toolchains rely on LLVM; mastering LLVM Intermediate Representation (IR) provides a universal foundation for understanding and deobfuscating diverse applications.
  • Obfuscation has Costs: Compiler-level obfuscation often introduces performance overhead, which can be exploited by defenders as a weakness for malware.
  • Recover Hidden Information: Leverage compiler artifacts like vtables, RTTI, and name mangling (using demangling techniques) to reconstruct high-level class structures, method signatures, and other source-level context.
  • Proactive Learning is Essential: Stay ahead of the curve by learning new programming languages, frameworks, and compiler research (e.g., C++ 26 reflection, e-graphs) before they become prevalent in malicious binaries.
  • Combine Tools Strategically: Integrate traditional reverse engineering tools (decompilers, symbolic execution) with newer analytical approaches (like C++ insights, and potentially LLM-assisted analysis with careful context management) for comprehensive deobfuscation.

About the Speaker(s)

Laurie Kirk is a distinguished researcher at Google, where she specializes in the challenging fields of decompilation and deobfuscation. Beyond her contributions at Google, Kirk is widely recognized for her educational outreach through her popular "Lori Wired" YouTube channel. On this channel, she delves into complex topics in reverse engineering and low-level programming, presenting them in an accessible, engaging, and often light-hearted manner. She is also a prolific contributor to the open-source community, notably as the creator of Ghidra MCP and an open-source iOS decompiler, both built upon the Ghidra reverse engineering framework. Kirk is highly regarded for her exceptional ability to communicate intricate technical concepts with clarity and insight, making her a sought-after speaker at conferences like RE//verse.

Reviews

Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT

Strong methodological keynote from someone who actually writes decompilation tooling instead of just talking about it. Framing LLVM-based obfuscation like OLLVM or Hikari as simple 'de-optimization' that can be crushed by standard compiler passes is exactly how we should be teaching this. Required viewing for anyone still trying to manually unflatten control flow graphs in assembly.

Heather Calloway (CISO) — SOLID

This is deep-technical research on how reverse engineers can defeat modern malware obfuscation by understanding compiler pipelines like LLVM. While the mechanics belong strictly to the malware analysts, the implication for security leaders is clear: the baseline for a competent reverse engineering capability is shifting. If you maintain an in-house advanced IR function, this changes how you hire and tool that team.

→ Top-rated talks at RE//verse 2026

All talks from RE//verse 2026