Taxonomy and Deobfuscation of a Real World Binary Obfuscator

James McGowan (CTO · Back Engineering Labs), Bas Zweers (Compiler Engineer · Back Engineering Labs)

RE//verse 2026 · Day 2 · Coral Ballroom A

Overview

This talk, presented by James McGowan and Bas Zweers from Back Engineering Labs, delves into the intricate world of binary obfuscation by providing a comprehensive taxonomy and a detailed deobfuscation strategy for Obfuscator-X. This commercial obfuscator is a real-world implementation of techniques described in the Loki research paper and is notably deployed in modern anti-cheat systems by major gaming companies like Riot, EA, and Faceit. The speakers highlight the significant challenge Obfuscator-X poses, asserting its deobfuscation is considerably more complex than dealing with well-known obfuscators like VMProtect or Themida.

Watch on YouTube

Visual summary for Taxonomy and Deobfuscation of a Real World Binary Obfuscator by James McGowan, Bas Zweers
Visual summary for Taxonomy and Deobfuscation of a Real World Binary Obfuscator by James McGowan, Bas Zweers

Key moments

  1. 4:00 Introducing Obfuscator-X: a real-world Loki implementation
  2. 7:45 First encounter with Mixed Boolean Arithmetic (MBA) in VM
  3. 11:00 Watermelon framework and SSA for deobfuscation strategy
  4. 17:30 Attacking Mixed Boolean Arithmetic using signature vectors
  5. 26:30 Anti-emulation: abusing 80-bit FPU floating point semantics
  6. 34:00 Demonstrating deobfuscated anti-emulation function from Vanguard
  7. 44:45 Q&A: Advice for anti-cheat developers

Taxonomy and Deobfuscation of a Real World Binary Obfuscator

Speakers: James McGowan (CTO, Back Engineering Labs); Bas Zweers (Compiler Engineer, Back Engineering Labs)

Conference: RE//verse

YouTube: https://www.youtube.com/watch?v=3LtwqJM3Qjg

Overview

This talk, presented by James McGowan and Bas Zweers from Back Engineering Labs, delves into the intricate world of binary obfuscation by providing a comprehensive taxonomy and a detailed deobfuscation strategy for Obfuscator-X. This commercial obfuscator is a real-world implementation of techniques described in the Loki research paper and is notably deployed in modern anti-cheat systems by major gaming companies like Riot, EA, and Faceit. The speakers highlight the significant challenge Obfuscator-X poses, asserting its deobfuscation is considerably more complex than dealing with well-known obfuscators like VMProtect or Themida.

The motivation behind this deep dive extends beyond academic curiosity. McGowan and Zweers aim to understand the cutting-edge of software protection, particularly in the high-stakes arms race between anti-cheat developers and cheaters. Furthermore, their work serves to independently verify the behavior of kernel-level anti-cheat binaries, addressing public concerns and social media discourse regarding potential misuse of user data. By deobfuscating and analyzing these systems, Back Engineering Labs provides a layer of trust and transparency, confirming that, at least in the cases examined, these anti-cheats are solely focused on cheat detection.

The article will explore Obfuscator-X's core mechanisms: its virtualizing obfuscator architecture, the advanced handler merging techniques derived from Loki, and its pervasive use of Mixed Boolean Arithmetic (MBA). The speakers systematically break down these components, detailing their approach to statically deobfuscating the complex binary and reconstructing a human-readable representation suitable for existing reverse engineering tools like IDA Pro, Binary Ninja, and Ghidra.

Background

▶ Watch: Introducing Obfuscator-X: a real-world Loki implementation (4:00)

Software obfuscation, often described as "security through obscurity," offers no absolute guarantees against reverse engineering but serves as a critical deterrent, increasing the time and effort required for analysis. In many scenarios, such as client-side anti-cheats, Digital Rights Management (DRM) in streaming services like Google Chrome, or system security products like PatchGuard, it's the only viable protection strategy. The video game industry, in particular, has become a fierce battleground where obfuscation plays a pivotal role. The financial stakes are immense, driving a detection-versus-evasion arms race between anti-cheat developers and cheat creators. Modern anti-cheat systems are highly sophisticated, contending with advanced hardware like DMA cards and extremely determined adversaries.

Cheaters, who physically own their systems, can easily extract anti-cheat binaries from disk or memory. Conversely, anti-cheat systems continuously collect data and can dump suspicious binaries for server-side analysis. In both contexts, obfuscation is employed to protect proprietary logic and evade detection. Historically, cheaters have relied on cracked or outdated versions of commercial obfuscators like VMProtect and Themida. However, these older solutions face increasing compatibility issues, especially with modern Windows security features like Intel's Control-flow Enforcement Technology (CET), which enforces shadow stacks and control-flow integrity in the kernel. VMProtect and Themida struggle to simultaneously support both CET and Structured Exception Handling (SEH).

This limitation has prompted anti-cheat developers to seek alternatives. Some, like EasyAntiCheat, have developed their own in-house obfuscators. Others, including Riot, EA, and Faceit, have adopted third-party solutions, most notably the obfuscator referred to as Obfuscator-X. This system stands out as a significantly more challenging target for deobfuscation compared to its predecessors. It implements techniques from the publicly available Loki obfuscator project, originally highlighted by Tim Blaziko, making it a state-of-the-art example of real-world binary protection. The goal of McGowan and Zweers' work was to develop a "magic wand-like" tool capable of ingesting an obfuscated binary and producing a deobfuscated output that seamlessly integrates with existing decompilers, yielding clean, readable code.

Key Findings

▶ Watch: Watermelon framework and SSA for deobfuscation strategy (11:00)

The research into Obfuscator-X yielded several critical insights into its design and the challenges it presents for reverse engineering:

  • Threaded Virtual Machine Architecture: Obfuscator-X operates as a threaded virtual machine, distinct from traditional bytecode-based VMs. Instead of a central dispatcher interpreting bytecode, its virtual machine handlers (VM handlers) directly branch to one another via tail calls, creating a complex, interwoven control flow.
  • LLVM-Compiled Handlers: A significant discovery was that the VM handlers themselves are compiled using LLVM. This was identified through specific code generation patterns, such as the selection logic for 32-bit versus 64-bit division, which optimized for older CPUs. This characteristic allows for the handlers to be treated as regular functions, making them amenable to lifting into an Intermediate Representation (IR).
  • Pervasive Mixed Boolean Arithmetic (MBA): MBA is a cornerstone of Obfuscator-X's obfuscation, extensively used to obscure computations and, crucially, the targets of computed branches. The obfuscator primarily employs linear MBA, which involves sums of bitwise expressions. Deobfuscating these expressions requires sophisticated techniques, including the use of signature vectors (akin to truth tables) and algebraic approaches like change of basis matrices.
  • Complex Handler Merging: Building upon techniques from the Loki paper, Obfuscator-X utilizes handler merging, also known as "super operators." This involves combining two or more distinct VM handlers into a single, larger handler. This merging is controlled by a key state, which determines which of the merged handler's computations and branch targets are ultimately selected. Identifying and tracking this key state across the control flow graph is one of the most difficult aspects of deobfuscation.
  • Advanced Anti-Emulation: Obfuscator-X incorporates specific anti-emulation techniques designed to thwart dynamic analysis. A notable example involves the use of 80-bit floating-point semantics through FPU instructions like FP RAM FLD and FSTP. Emulators such as Unicorn and QEMU, which often truncate these 80-bit values to 64-bit, fail to correctly model the behavior, leading to incorrect execution and emulator crashes.
  • Custom Deobfuscation Framework: To tackle these challenges, the team developed Watermelon, a custom binary-to-binary lifting and rewriting framework. Watermelon lifts compiled binaries to a Static Single Assignment (SSA) form, allowing for powerful manipulation and deobfuscation transformations before recompiling the cleaned binary.
  • Independent Anti-Cheat Verification: Through their deobfuscation efforts, the speakers were able to analyze the inner workings of Riot Games' Vanguard anti-cheat driver (vgk.sys). Their findings provide independent confirmation that these kernel-level anti-cheats are genuinely focused on detecting cheats and are not engaging in malicious data exfiltration, directly addressing common public concerns.

Technical Deep Dive

▶ Watch: Attacking Mixed Boolean Arithmetic using signature vectors (17:30)

Obfuscator-X's complexity stems from its multi-layered obfuscation approach, combining a virtualized execution model with advanced control flow and data obfuscation techniques.

Virtual Machine Architecture

The entry point into an obfuscated function in Obfuscator-X typically begins with a canonical prologue, followed by a jump into a new section. This new section contains a call instruction that leverages a "silly little jump negative one trick" to create overlapping instructions, making initial static analysis difficult. Binary Ninja's Value Set Analysis (VSA) can, however, resolve the computed branch target. This initial sequence leads to the VM context setup, where all x86 registers are saved into a virtual machine context allocated on the stack.

Unlike traditional VMs, Obfuscator-X employs a threaded virtual machine design. Instead of a central dispatcher interpreting a bytecode stream, each virtual instruction or VM handler is a self-contained function. These handlers, identified as being compiled with LLVM (evidenced by code generation patterns such as 32-bit vs. 64-bit division selection), execute their logic and then use a computed jump RAX (a tail call) to transfer control directly to the next VM handler. This creates a dense, interwoven control flow graph where the "bytecode" is implicitly defined by these direct handler-to-handler jumps.

Certain operations that cannot be effectively obfuscated, such as moves into debug registers, are handled by boxed handlers. These handlers load values from the VM context, perform the un-obfuscatable operation, and then write any updated values back into the context. Interestingly, even no-op instructions are boxed, preventing their stripping and further complicating analysis. When the VM needs to call out to a native function, it performs a complex stack argument copying operation. Any potential arguments on the callee's stack within the VM are copied to a newly allocated stack space, ensuring correct argument placement for the native function call.

Deobfuscation Strategy with Watermelon

The deobfuscation strategy leverages Watermelon, a custom binary-to-binary lifting and rewriting framework developed by Back Engineering Labs. Watermelon takes compiled binaries, lifts them into a Static Single Assignment (SSA) form, and then recompiles them after transformations. Since Obfuscator-X's VM handlers are standard LLVM-compiled functions, Watermelon can lift them into its own IR.

The primary challenge for static deobfuscation is reconstructing the entire function's control flow graph (CFG), especially given the computed jump RAX targets obscured by MBA. The approach involves iterative CFG exploration. As new handlers are discovered, they are lifted and added to the growing function representation. A key problem with SSA construction is that it typically requires knowledge of the entire CFG to correctly place phi nodes (which merge values from different control flow paths).

To address this, the VM context, which is allocated on the stack, is conceptually divided into uniform eight-byte slots, referred to as VM registers. For every newly discovered block, phi nodes are optimistically inserted at its beginning for every VM context slot. This ensures that even if multiple incoming edges are discovered later in the exploration, the SSA form can be maintained. This technique, though seemingly aggressive, allows for the iterative construction of the CFG while preserving SSA properties. The speakers note that similar techniques can be implemented using open-source tools like LLVM and Remill from Trail of Bits.

Handler-Local Obfuscation Techniques

Bas Zweers detailed the techniques used to obscure the logic within individual VM handlers:

Mixed Boolean Arithmetic (MBA)

MBA is a well-known obfuscation technique that replaces simple arithmetic expressions with more complex, functionally equivalent forms. For instance, A + B might become A XOR B + 2 * (A AND B). Obfuscator-X predominantly uses linear MBA, which involves sums of bitwise expressions where variables only appear within the bitwise operations. While Obfuscator-X doesn't directly use polynomial MBA, multiple applications of linear MBA can lead to expressions exhibiting polynomial-like behavior.

The key to attacking linear MBA lies in signature vectors. A signature vector is derived by evaluating an expression for all combinations of 0 and 1 for its input variables. Ryckeboer et al. demonstrated that if two expressions have equivalent signature vectors, they are functionally equivalent. This property is crucial because it allows for rapid equivalence testing, which SMT solvers like Z3 often struggle with due to their lack of this heuristic.

A common MBA hardening technique is the introduction of auxiliary variables (also called dummy variables). These variables are added to an MBA expression but do not affect its final result, preventing direct signature vector matching. To counter this, the deobfuscation process computes signature vectors and then identifies and removes spurious variables. This is done by checking if flipping a variable's bit in the input combinations has any effect on the output. An efficient algorithm, similar to Quine-McCluskey reduction and leveraging the Intel Speedup reduction, can generate a bitmask of live variables, allowing the reconstruction of a simplified signature vector.

The deobfuscation strategy includes:

  1. Aggressively computing signature vectors for every SSA variable resembling an MBA expression.
  2. Having a fast path for common patterns (e.g., OR, AND, XOR, simple linear sums).
  3. Attacking specific classes of MBA:
  • Constant MBAs: If all entries in a signature vector are equivalent, the expression is a constant. Obfuscator-X heavily uses these to obscure integer constants, often employing a modular inverse trick with co-prime numbers. Pruning these first is highly effective.
  • Linear Sums (and Affine Expressions): These are common in memory computations. Deobfuscation involves a change of basis matrix. By transforming the signature vector from a minterm basis (e.g., X AND NOT Y) to a "nicer" basis (e.g., 1, X, Y, X AND Y), the expression can be trivially expanded into its simplified form (e.g., X + Y). This linear algebra approach is computationally cheap and also works for reducing bitwise MBAs (by working over GF2 instead of addition), normalizing expressions for better compiler optimization passes.

Handler Merging and Key State Identification

The concept of handler merging significantly complicates CFG exploration. Instead of independent handlers, Obfuscator-X merges two or more handlers into a single "super operator." The actual execution path within this merged handler, including its computations and the outgoing branch target, is determined by a key state. This key is set in the previous handler, making its identification challenging.

LLVM optimizations further obscure these key computations, as they are often masked with random values. The deobfuscation framework requires a known bits pass (either custom-written or LLVM's default) to deduce the values of these keys. Once the key state for a merged handler is known, if a previously visited handler is re-encountered with a different key state, the original handler is duplicated, the new key state is propagated, and constant propagation is applied. This effectively "unmerges" the handlers, restoring the original control flow.

Anti-Emulation Techniques

Obfuscator-X employs sophisticated anti-emulation techniques to thwart dynamic analysis. One notable trick involves the use of 80-bit floating-point semantics with FPU instructions like FP RAM FLD (FPU register load) and FSTP (FPU store and pop). These instructions manipulate 80-bit floating-point values in the FPU stack.

The trick works because emulators like Unicorn and QEMU often truncate these 80-bit values to 64-bit during emulation. Obfuscator-X manipulates the exponents of FPU registers (e.g., ST1 and ST0) to different values. When 80-bit semantics are correctly honored, the FP RAM instruction effectively becomes a no-op, passing the value through. However, if an emulator truncates to 64-bit, this operation produces a junk value, causing incorrect execution and potentially leading to detection.

A practical workaround for this specific anti-emulation technique, when the FPU is not used for any other legitimate computation (which is the case for Obfuscator-X, according to the speakers), is to simply skip the FP RAM instruction during emulation. This allows the emulator to continue execution without triggering the anti-emulation logic.

Demo / Proof of Concept

▶ Watch: Demonstrating deobfuscated anti-emulation function from Vanguard (34:00)

To demonstrate the efficacy of their deobfuscation framework, James McGowan presented a live example using a benign function from Riot Games' Vanguard anti-cheat driver (vgk.sys). The chosen function is a simple anti-emulation routine found within an IOCTL handler, which was not obfuscated itself, but called into obfuscated functions. The specific function highlighted was the third call within this handler.

The original obfuscated function exhibits the familiar VM entry pattern: a canonical prologue, VM context setup, and then a flow into the virtual machine. After the deobfuscation process, the vgk.sys binary is patched such that the canonical prologue now jumps directly to the newly generated, deobfuscated output. This deobfuscated code is given an "outline" attribute in IDA Pro, allowing the decompiler to present it as an inlined, clean control flow graph within the calling function.

The deobfuscated function reveals an anti-emulation check that leverages the SMSW instruction to read the CR0 register. The function then inspects the 8th bit of CR0, which is the paging bit. In a 64-bit long mode environment, the paging bit must be enabled. The anti-cheat relies on the assumption that some emulators might incorrectly return a CR0 value with the paging bit unset (e.g., zero or an incomplete value). If this condition is met, it indicates an emulator that is not accurately modeling the CPU state, thus triggering the anti-emulation. The demo showed that in a real system, the 8th bit was indeed set, as expected. The function also included stub calls that, in the obfuscated binary, appeared to allow control flow to rejoin an exit point, implying they could return. However, the deobfuscated view clarifies that one of these stubs involves a syscall, which would not typically return in a way that allows control flow to rejoin, revealing a misleading control flow path in the obfuscated version. This example effectively showcased the "magic wand" capability of their tool, transforming opaque, VM-driven code into clear, decompilable C-like output.

Defensive Implications

▶ Watch: Q&A: Advice for anti-cheat developers (44:45)

The detailed analysis and deobfuscation of Obfuscator-X carry significant implications for both developers employing such protections and the broader cybersecurity community engaged in defensive measures.

For Developers and Anti-Cheat Engineers:

  • Robustness of Obfuscator-X: The research confirms that Obfuscator-X, with its threaded VM, MBA, handler merging, and FPU-based anti-emulation, represents a highly sophisticated and effective obfuscation solution against typical reverse engineering efforts. For the vast majority of cheaters or casual analysts, statically deobfuscating such a binary is an insurmountable task without specialized tooling and deep expertise.
  • Dynamic Analysis Remains Key for Adversaries: The speakers emphasize that most cheaters rely on dynamic analysis rather than static deobfuscation. Therefore, while static protection is valuable, anti-cheat developers should continue to prioritize offloading as much detection logic as possible to the server, under the fundamental principle that the client is inherently untrusted.
  • Understanding Vulnerabilities of Obfuscators: Knowledge gained from deobfuscation can inform stronger obfuscator design. For instance, the specific FPU anti-emulation trick, once understood, can be easily bypassed by emulators. This highlights the constant need for obfuscators to evolve and integrate new, harder-to-defeat techniques.
  • Potential for Targeted Attacks: The identification of "boxed handlers" for privileged instructions (e.g., debug register checks) reveals potential points of interest for adversaries. These specific, un-obfuscatable operations could be targeted for hooking or manipulation, as they represent critical interaction points with the underlying system.

For Cybersecurity Defenders and Researchers:

  • Advanced Malware Analysis: The techniques developed for deobfuscating Obfuscator-X are directly applicable to analyzing highly obfuscated malware, spyware, and nation-state tools. Understanding how complex MBA, handler merging, and anti-emulation are implemented in real-world binaries provides a blueprint for developing automated or semi-automated deobfuscation tools.
  • Value of Compiler Frameworks: The success of this project underscores the immense value of having a robust binary-to-binary lifting and rewriting framework (like Watermelon, or LLVM+Remill) in a reverse engineer's toolkit. Such frameworks enable the transformation of opaque binaries into an analyzable SSA form, which is crucial for complex static analysis.
  • Combining Static and Dynamic Analysis: While the talk primarily focused on static deobfuscation, the speakers stressed the importance of augmenting static analysis with dynamic execution tracing. Tracing can significantly simplify the identification of critical values, such as the key states required for handler merging, by observing their runtime behavior. Tools like Mergen, which performs concolic execution and lifts to LLVM, represent a promising hybrid approach.
  • Dispelling Misinformation: The independent verification of anti-cheat driver behavior serves a broader defensive purpose by building trust and dispelling misinformation. Confirming that kernel-level anti-cheats are not engaging in malicious activities can help foster a more informed public discourse around security software.
  • FPU-based Anti-Emulation: Defenders should be aware of and account for 80-bit FPU semantics in their emulation platforms. Emulators need to accurately model these complex CPU features to avoid being detected by sophisticated anti-emulation tricks.

Overall, this research not only provides a deep technical understanding of a cutting-edge obfuscator but also equips the community with methodologies and considerations for analyzing and defending against complex binary protections in various security contexts.

Key Takeaways

  • Obfuscator-X is a highly sophisticated binary obfuscator employed by major anti-cheat systems, utilizing a threaded virtual machine, pervasive Mixed Boolean Arithmetic (MBA), and advanced handler merging techniques derived from the Loki paper.
  • Deobfuscation requires a robust binary rewriting framework and advanced static analysis. The Watermelon framework, which lifts binaries to SSA form and iteratively explores the CFG with dynamic phi node placement for VM registers, is crucial for tackling this complexity.
  • MBA can be systematically deobfuscated using signature vectors for equivalence checking, techniques to remove auxiliary variables, and change of basis matrices for simplifying linear sums, even when dealing with semi-linear expressions.
  • Handler merging is a significant challenge to CFG reconstruction, requiring the identification and propagation of key states that control execution flow, often obscured by LLVM optimizations and random masking.
  • Anti-emulation techniques leveraging 80-bit FPU semantics (e.g., FP RAM) can effectively detect emulators that incorrectly truncate floating-point values to 64-bit, but can be bypassed with targeted emulator modifications if FPU is not otherwise used.
  • Independent analysis of kernel-level anti-cheats (like Riot's Vanguard) confirms their focus on cheat detection, debunking claims of malicious data exfiltration and providing a layer of trust through transparency.
  • Combining static analysis with dynamic execution tracing (e.g., through tools like Remill, LLVM, or concolic execution projects like Mergen) significantly aids in resolving dynamic values like key states and computed branch targets, making deobfuscation more efficient.

About the Speaker(s)

James McGowan is the CTO of Back Engineering Labs. He has extensive experience in the fields of obfuscation, deobfuscation, and reverse engineering, particularly within the context of video games and anti-cheat systems. His work at Back Engineering Labs involves building software obfuscators to aid "the good guys," while also engaging in deobfuscation research to understand and counter advanced protection techniques.

Bas Zweers is a Compiler Engineer at Back Engineering Labs. His expertise lies in the intricate details of compiler engineering, which was instrumental in solving the complex challenges presented by Obfuscator-X's Mixed Boolean Arithmetic (MBA) and anti-emulation techniques.

Reviews

Dr. Zero (Offensive Security Researcher) — MUST SEE

This is what real reverse engineering looks like. McGowan and Zweers didn't just read the Loki paper; they built a custom SSA lifter to mathematically dismantle the commercial implementation protecting Riot's Vanguard. A masterclass in defeating threaded VMs and linear MBA that every serious reverser needs to watch.

Heather Calloway (CISO) — SOLID

This is a deep-technical reverse engineering talk focused on the commercial obfuscation used in gaming anti-cheats like Riot Vanguard. While the mechanics of Mixed Boolean Arithmetic are strictly for your malware analysts, the strategic value here is the independent validation of a highly privileged kernel driver. If your employees have these games installed on WFH endpoints, this research provides critical proof that the vendor isn't silently exfiltrating data.

→ Top-rated talks at RE//verse 2026

All talks from RE//verse 2026