Attacking modern software protection with Dynamic Binary Instrumentation

Attacking Modern Software Protection with Dynamic Binary Instrumentation

Speakers: Holger Unterbrink, Technical Leader, Cisco Talos

Conference: REcon 2025

YouTube: https://www.youtube.com/watch?v=4FyUTJRitrc

Overview

Modern software protections—anti-debugging routines, anti-tamper checks, VM detection, code obfuscation, and self-modifying code—were once the exclusive domain of sophisticated malware. Today they are routine. Commercial protectors like VMProtect are widely available, and even commodity malware authors routinely employ them. These defenses reliably defeat traditional static analysis and debugger-based approaches, burying critical behavioral logic behind layers of runtime trickery. In this REcon 2025 talk, Holger Unterbrink of Cisco Talos presents Dynamic Binary Instrumentation (DBI) as a powerful, often-overlooked alternative that can circumvent the majority of these protections transparently—and shows how to build practical instrumentation tools with DynamoRIO in relatively few lines of code.

Background

▶ Watch: Introduction: DynamoRIO on Windows — setup and capabilities (2:37)

What Is Binary Instrumentation?

Binary instrumentation is the process of inserting additional code into a compiled executable without modifying the source code. It can be performed statically—by patching the binary on disk—or dynamically, at runtime. Dynamic Binary Instrumentation operates while the target executes, giving the analyst access to runtime values, live control flow, and the actual behavior of obfuscated code rather than a misleading static picture.

The major DBI frameworks in active use include:

• DynamoRIO — open source, BSD-licensed, strong Windows x86/x64 performance
• Intel Pin — widely used but proprietary license concerns
• Valgrind/Callgrind — popular on Linux; strong profiling story
• Frida — excellent for mobile targets (iOS, Android)
• QBDI — another option with good cross-platform support

Unterbrink chose DynamoRIO for this work primarily because of its performance advantage on Windows x86/x64 and its BSD-like open-source license, which removes any concern about licensing conflicts in tools released publicly.

Why DBI for Malware Analysis?

Traditional debuggers announce their presence. Debugger detection via IsDebuggerPresent, hardware breakpoint scanning of debug registers, timing-based RDTSC checks, and anti-VM techniques are all routinely embedded in protected samples. DBI frameworks, particularly DynamoRIO, offer an important advantage: by running as an in-process virtual machine that manages its own code cache, DynamoRIO is almost entirely transparent to the target process. Most malware detection routines simply do not see it.

Additional practical advantages include:

• Bare-metal execution: Unlike a full VM environment, DBI can run directly on hardware, defeating hypervisor detection checks.
• Multi-threading support: DynamoRIO correctly handles multithreaded targets and, by default, follows child processes spawned by the malware.
• Automation: Repetitive tasks such as unpacking, shellcode detection, or C2 extraction can be automated with custom instrumentation clients.

Key Findings

▶ Watch: First instrumentation: Hello World DBI client working (16:50)

Unterbrink's talk demonstrates that the vast majority of anti-analysis techniques encountered in real-world malware are transparent to DynamoRIO with no special handling required:

| Technique | DynamoRIO transparent? |

|---|---|

| Shellcode execution (encrypted second stage) | Yes |

| TLS/WinHTTP communication instrumentation | Yes (with caveats on modern CPUs—see below) |

| CRC32 code validation checks | Yes |

| IsDebuggerPresent anti-debug | Yes |

| Hardware breakpoint register check (DR0–DR3) | Yes |

| RDTSC timing checks (non-aggressive) | Yes |

| Large loops | Yes |

| Self-modifying code and interleaf tricks | Yes |

| Single-step CPU exception + debug register read from exception context | No — crashes DynamoRIO silently |

The single documented failure case is notable: a technique that deliberately triggers a single-step CPU exception and then reads the debug registers from the resulting exception context causes DynamoRIO to silently terminate the target process without printing an error message. This silent exit is the "uncomfortable" part—an analyst who does not know this technique is in use may assume the sample ran to completion normally.

Unterbrink also encountered an unexpected anomaly during preparation: on very modern mobile-class CPUs, WinHTTP-based TLS communication failed under DynamoRIO instrumentation with error code 12175 (WINHTTP_ERROR_SECURE_FAILURE), while the same code ran flawlessly on older hardware (e.g., a Celeron-based machine). Whether this is a new Windows anti-tampering feature targeting modern microarchitecture features or a DynamoRIO bug on recent CPUs remains an open question slated for further investigation.

Technical Deep Dive

▶ Watch: Walkthrough: heavily annotated DBI analysis client (24:14)

How DynamoRIO Works Internally

DynamoRIO operates as a process-level virtual machine. Rather than executing the target's original code directly on the CPU, it copies the target's basic blocks into an isolated code cache that it fully controls. The workflow is:

1. The original target binary is loaded.
2. DynamoRIO intercepts execution basic block by basic block, copying each into the basic block cache.
3. At each branch, the DynamoRIO dispatcher evaluates where control flow should go next.
4. Once a sequence of basic blocks is executed repeatedly in the same order, DynamoRIO promotes them into a trace cache, inlining and optimizing them to reduce overhead.

All register state, stack pointers, and memory address patching are handled transparently by DynamoRIO so the target application behaves exactly as it would on real hardware—it simply does not know it is being instrumented.

Writing a DynamoRIO Client

DynamoRIO terminology calls user-written instrumentation modules clients. Clients are compiled as DLLs and loaded by the drrun.exe launcher, which injects the DynamoRIO core runtime into the target process. The command-line syntax is:

The development environment requires Visual Studio 2019 or 2022, CMake, and the DynamoRIO SDK. No recompilation of DynamoRIO itself is necessary for typical client development—the prebuilt release package suffices.

Key Extensions

Two DynamoRIO extensions are used most heavily:

• DR Manager (drmgr): Provides the event registration framework—callbacks for module loads, thread initialization, basic block instrumentation, process exit, and more.
• DRwrap (drwrap): Provides function-wrapping capability, allowing a client to intercept function calls and manipulate arguments or return values before and after execution.

Event Callbacks

The DynamoRIO event model is callback-driven. Unterbrink's demo client registered:

• dr_register_module_load_event: Fires whenever a DLL is loaded by the target; used here to hook specific functions with drwrap.
• dr_register_bb_event: Fires on every basic block; used to implement the instruction-level tracer.
• dr_register_exit_event: Fires on process exit; used for cleanup.

Function Wrapping in Practice

The antx.exe demo binary contains a function always_true() that always returns 1. To flip its return value at runtime, the client registers a post-function wrapper that sets RAX to 0. In practice, this maps directly to real-world use cases:

• Patching an is_debugger_present() wrapper to always return false.
• Patching a VM detection function to always report "not in a VM."
• Bypassing license checks, anti-tamper routines, or any predicate whose return value controls behavior.

Building a Code Tracer

The basic block event callback receives the program counter of each instruction being executed. Unterbrink's tracer filters instructions to a defined address range (simulating "step into everything" debugger behavior) and uses DynamoRIO's built-in disassembler (configured to Intel syntax via disassemble_set_syntax(DR_DISASM_INTEL)) to print each instruction. The result is a runtime-generated execution trace without touching a debugger.

For the VMProtect analyzer described in the talk, a version of this tracer was used to resolve indirect addressing operands—values calculated on the fly by VMProtect's virtual machine—and write the resolved values as comments into an IDA Pro IDB file, dramatically accelerating the subsequent static analysis pass.

Practical Anti-Analysis Coverage

Unterbrink's antx.exe demo program deliberately implemented a comprehensive battery of anti-analysis checks to stress-test DynamoRIO's transparency. Self-modifying code, interleaf tricks (where a conditional branch skips to the middle of what appears to be a different instruction, reinterpreting its encoding), CRC32 integrity checks over code regions, and API-based anti-debugging (IsDebuggerPresent, debug register checks via GetThreadContext) all ran without modification under DynamoRIO. Code tracing followed the self-modified control flow correctly on both the first and second call to the modified function.

Demo / Proof of Concept

▶ Watch: Anti-analysis evasion check discovered and bypassed (38:31)

The primary demonstration consisted of the antx.exe program—a purpose-built target that compresses many real-world anti-analysis techniques into a single binary. Key demonstrated scenarios:

1. Function patching: always_true() had its return value overridden to 0 at runtime via drwrap, illustrating anti-debugging bypass.
2. Code tracing: All instructions executed within a specified address range were disassembled and printed, following self-modified basic blocks correctly.
3. Anti-analysis bypass: CRC32 code integrity checks, IsDebuggerPresent, debug register checks, RDTSC timing, and shellcode execution all ran transparently.
4. C2 communication: WinHTTP TLS traffic was successfully intercepted on older hardware (the CPU-dependent failure was highlighted as a known limitation under investigation).
5. Self-modifying code with interleaf: DynamoRIO correctly tracked the reorganized control flow across two calls to the modified function.

Post-conference, Unterbrink committed to releasing a set of demo tools on the Cisco Talos GitHub account.

Defensive Implications

▶ Watch: Devirtualization result: original code branch recovered (44:47)

While the talk is framed as a reverse engineering tool, DBI has direct defensive implications:

• Automated unpacking pipelines: DynamoRIO clients can be written to detect when a process writes executable data to memory and then jumps to it, enabling automated unpacking at scale.
• Behavioral analysis at bare metal: Running DBI on actual hardware rather than inside a VM eliminates entire classes of VM-fingerprinting detection while still providing full execution visibility.
• CI/CD-integrated malware triage: Custom DynamoRIO clients can extract hardcoded C2 addresses, configuration structures, and decrypted strings from protected samples faster than manual analysis—valuable for threat intelligence pipelines.
• Limitations awareness: The single-step exception + debug register read technique and the CPU-dependent TLS anomaly on modern processors represent gaps that malware authors could exploit to detect or evade DBI-based tooling.

Key Takeaways

• DynamoRIO is open source (BSD license), performs well on Windows x86/x64, and installs by simply unpacking a zip archive.
• Building a useful DynamoRIO client—a code tracer, a function patcher, an unpacking helper—requires surprisingly little code once the callback model is understood.
• The vast majority of common anti-analysis techniques are transparent to DynamoRIO out of the box: shellcode, TLS interception, CRC32 integrity checks, IsDebuggerPresent, hardware breakpoint register checks, RDTSC, self-modifying code, and interleaf tricks all pass without requiring custom workarounds.
• The key failure case is a technique that uses a deliberate single-step CPU exception to read debug registers from the exception context. DynamoRIO silently exits the target process in this scenario—an analyst needs to know to look for this.
• A currently unexplained failure involving WinHTTP TLS calls on modern mobile-class CPUs under DynamoRIO instrumentation is under investigation.
• The trace cache can be disabled with an undocumented -disable_traces switch, useful when per-basic-block interception is needed unconditionally.
• Debugging DBI clients is best done by adding verbose dr_printf statements; the next escalation is enabling DynamoRIO's built-in logging at levels 2, 3, or 4.

About the Speaker

Holger Unterbrink is a Technical Leader and security researcher at Cisco Talos, the threat intelligence division of Cisco. His work focuses on malware analysis, reverse engineering, and the development of tools to defeat increasingly sophisticated software protections. He is active on Twitter as @h_underbr_72 and has previously published analysis of VMProtect-protected malware along with supporting instrumentation tooling on the Cisco Talos GitHub.