Bypassing Anti-Debugging: A Hybrid Real-Simulated Approach to Rootkit Analysis

Bypassing Anti-Debugging: A Hybrid Real-Simulated Approach to Rootkit Analysis

Speakers: Yong-Xu Yang (Intern, TeamT5); Heng-Ming Fan (Malware Analyst, National Institute of Cyber Security); Yu Xuan Luo (Senior Student, National Sun Yat-sen University)

Conference: CODE BLUE

YouTube: https://www.youtube.com/watch?v=Sbxnz0NHryo

Overview

Analyzing rootkits presents a formidable challenge in cybersecurity due to their privileged kernel-mode operations and sophisticated anti-analysis techniques. This talk, "Bypassing Anti-Debugging: A Hybrid Real-Simulated Approach to Rootkit Analysis," presented at CODE BLUE, introduces KDENU, a novel emulator designed to overcome these obstacles. The speakers, Yong-Xu Yang, Heng-Ming Fan, and Yu Xuan Luo, detail their approach to creating a robust analysis environment that combines the control of emulation with the realism of native execution, specifically targeting kernel drivers that employ advanced anti-debugging and anti-virtualization mechanisms.

The core innovation of KDENU lies in its hybrid semi-emulated, semi-native architecture, which leverages Unicorn's C++ bindings and kernel dumps to provide an exceptionally realistic and undetectable execution environment. This allows for deep inspection of kernel-mode malware, even those that actively attempt to detect and evade traditional debugging setups. The project not only provides a powerful new tool for malware analysts but also sheds light on the cutting-edge detection techniques employed by sophisticated anti-cheat engines, offering valuable insights for both defenders and developers alike.

Background

▶ Watch: Introduction and talk agenda overview (0:00)

Rootkits are a particularly insidious form of malware that operate in the highly privileged kernel mode of an operating system. Their deep integration into the OS core makes them exceptionally difficult to detect and analyze. Traditional methods, such as static analysis, are often thwarted by obfuscation and virtualization techniques. Dynamic analysis methods, while more effective, face their own set of challenges, primarily the rootkits' pervasive anti-debugging and anti-virtualization mechanisms.

Existing dynamic analysis approaches typically involve dual machine debugging, where a debugger like WinDbg connects from one machine to another (physical, VMware, or KVM). However, these setups are often detectable. Virtual machines like VMware can leave specific artifacts, and hypervisor-based techniques (e.g., Cheat Engine DBVM, Hyperdebugger, or custom hypervisors, often based on EP hooks in Intel CPUs) frequently expose detectable symbols, such as specific values in Model Specific Registers (MSRs). These detection vectors allow rootkits to identify and evade analysis environments.

To counter these limitations, the concept of an emulator emerges as a superior solution, offering higher control over the execution environment and thus better evasion of detection. The speakers reviewed several existing emulation solutions:

1. Speakeasy: Implemented in Python, it loads and executes PE files by simulating Windows API calls. Its drawbacks include inadequate handling of kernel-mode Structured Exception Handling (SEH), inability to intercept specific register operations, and a lack of multi-threading support.
2. Quilling: A well-known and versatile project, also Python-based and built on Unicorn. While it technically supports Windows kernel drivers, its source code reveals many functions that merely return fake values without actual execution. Crucially, its recently added SEH support does not function correctly for drivers, and it also lacks proper multi-threading capabilities.

• Common issues for Speakeasy and Quilling: Both are Unicorn-based, rely on fabricated simulated environmental parameters, fail to support SEH for drivers, and lack multi-threading.

1. Kernel Anti-Cheat Emulator (KACE): This project maps drivers to user mode for execution, offering native execution speed. It uses Vector Exception Handling (VEH) for exception handling, for example, to trap attempts to read or write special registers like CR3. KACE is notable for using ntdll function calls to obtain real environmental parameters. However, its primary drawback is running in user mode, making it detectable (e.g., by checking if the RIP register's value falls within user space addresses).

The talk emphasized two critical components often mishandled by existing emulators: Structured Exception Handling (SEH) and multi-threading. Without proper SEH emulation, errors lead to crashes or unintended execution paths instead of gracefully entering the exception handler, causing the driver to unload or the system to crash. Similarly, inadequate multi-threading support can lead to incorrect execution flows, especially when multiple threads access shared data, as threads might execute out of order, leading to stale or incorrect values. Addressing these fundamental shortcomings was a key motivation for KDENU.

Key Findings

▶ Watch: KDENU's core design: kernel dumps for realistic environment (13:20)

The primary finding of this research is the successful development and validation of KDENU, a novel hybrid semi-emulated, semi-native approach for rootkit analysis. This emulator, built using Unicorn's C++ bindings, distinguishes itself by enabling access to real environmental parameters within an emulated environment, granting the emulator the highest control over the process.

Key contributions and discoveries include:

• Realistic Environment Initialization: KDENU utilizes kernel dumps to create a highly realistic initial emulation environment, mapping necessary system parameters from a pre-snapshot of a real system. This is a significant improvement over emulators that rely on fabricated environmental values.
• Robust SEH and Multi-threading Support: Unlike its predecessors, KDENU provides strong, functional support for both Structured Exception Handling (SEH) and multi-threading, which are crucial for the correct execution and analysis of complex, modern kernel drivers.
• Advanced Hooking Mechanisms: The emulator incorporates sophisticated hooking techniques, including improved function entry hooks that avoid visible int3 instructions, one-shot hooks for capturing return values, and global hooks for monitoring a wide range of special instructions that Unicorn's default instruction hooks cannot cover.
• Hybrid Real-Simulated Interaction: KDENU pioneers the use of kernel dumps to provide real parameters for system function calls during execution, yielding more realistic return values compared to simply calling ntdll functions or returning fake data. This "mix and match" approach ensures accuracy without sacrificing control.
• LIEF Library Bug Discovery: During development, the team discovered and reported a security cookies bug in the popular LIEF library when parsing PE files, which has since been merged.
• Uncovering Advanced Anti-Analysis Techniques: Through the analysis of a real-world anti-cheat driver using KDENU, the speakers successfully identified and demonstrated several advanced anti-debugging and anti-virtualization techniques, including int 20h for PatchGuard detection, DbgPrompt for debugger presence checks, various MSR-based detections (e.g., for KVM, LBR/BTF manipulation), and the use of NtCaptureImageExceptionValue for OS-dependent offset discovery.

These findings collectively present a significant leap forward in the capability to analyze highly evasive kernel-mode malware, providing a more reliable and less detectable analysis platform.

Technical Deep Dive

▶ Watch: KDENU demo: exception handling and multi-threading (20:50)

The design and implementation of KDENU represent a sophisticated blend of emulation and native interaction, specifically tailored for the intricacies of kernel driver analysis. The core principle is to create a highly realistic, yet fully controlled, execution environment that can bypass advanced anti-analysis techniques.

Initialization and Driver Loading:

KDENU's environment initialization begins by mapping essential system parameters from a kernel dump. This dump is a pre-snapshot taken at a breakpoint set at the DriverEntry of any driver, providing a genuine baseline of the operating system's state. The target driver for analysis is then loaded and overrides the original driver within this emulated environment. For this, KDENU utilizes the LIEF library to parse the PE file, notably leading to the discovery and reporting of a security cookies bug in LIEF.

Hybrid Execution Model:

As the driver executes, KDENU continuously monitors for special instructions and memory accesses, logging all activity. When a system function call is encountered, the emulator performs a crucial check:

1. Kernel Dump Handling: It first determines if the function call can be handled using the kernel dump. This involves retrieving real parameters from the pre-snapshot, leading to more realistic return values. This approach, inspired by a project called "What The F," is a pioneering technique for emulation.
2. Hooking: If the kernel dump cannot handle the function, or if the function's return value might be detectable, KDENU resorts to hooking the call. This "mix and match" strategy provides the best of both worlds: realism when possible, and control when necessary.

Advanced Hooking Mechanisms:

KDENU employs several innovative hooking techniques:

• Function Entry Hooks: Instead of placing a visible int3 instruction at the start of a function (which is detectable), KDENU's hook triggers at the function entry, then emulates the int3 instruction after the function has been entered.
• One-Shot Hooks: To capture return values, a one-shot hook is set at the function entry. This hook then sets a second one-shot hook at the address on RSP (Return Stack Pointer). When the function returns, the second hook triggers, allowing KDENU to retrieve the function's return value.
• Data Access Monitoring: Inspired by BeeC, KDENU monitors important kernel objects, logging every read and write operation to them.
• Global Hooks for Special Instructions: Unicorn's native instruction hooks are limited to a few instructions (e.g., syscall, cpuid). For all other special instructions, KDENU implements global hooks. When the emulator is about to execute such an instruction, the global hook intercepts it, handles the operation, records the action, and then manually advances the Instruction Pointer (IP) to continue execution. This ensures a comprehensive view of the driver's low-level activities.
• Unicorn C++ Bindings: The choice of Unicorn's C++ bindings over the more common Python bindings facilitates the implementation of these global hooks and provides finer-grained control.

Structured Exception Handling (SEH) Implementation:

Initially, the team attempted a complex, full emulation of the Windows SEH process within Unicorn, a task described as requiring "about five pages." However, the introduction of kernel dumps drastically simplified this. KDENU now overwrites a critical parameter in the kernel dump: the InvertedFunctionTableList. This involves modifying the pointer of the original driver's exception table to point to the new driver's exception table address. Without this modification, the system would fail to find the correct address when an exception is triggered. After this, KDENU simply emulates a call to RtlRestoreContext to handle the SEH, making the process significantly more robust and efficient.

Multi-threading Support:

Unicorn does not natively support multi-threading, presenting a significant challenge. The initial idea of using a single thread with context switching after a fixed instruction count on a shared instance was discarded due to severe performance impacts. Ultimately, KDENU adopted a parallel approach with thread-specific memory locking. In this model:

• Memory allocation is performed by the main Unicorn instance and recorded.
• When a child thread accesses mapped memory, it first checks if the main instance has allocated it or if it exists in the kernel dump.
• If so, the data is mapped from the main instance or the kernel dump. Otherwise, an error is triggered.

This approach ensures that multiple threads can operate concurrently while maintaining data integrity and a consistent view of the memory space, crucial for emulating complex, multi-threaded kernel drivers.

Through these technical choices, KDENU achieves a unique position among emulators, offering unparalleled control and realism for rootkit analysis.

Demo / Proof of Concept

▶ Watch: KDENU evaluation, future plans, and open source release (28:00)

The practical capabilities of KDENU were effectively showcased through a live demonstration, highlighting its robust handling of Structured Exception Handling (SEH) and multi-threading, followed by a deep dive into the sophisticated detection techniques uncovered in a real-world anti-cheat driver.

The demonstration first illustrated KDENU's SEH support. The log output clearly showed an "expression is called and the driver continues," indicating that when an exception was triggered, KDENU successfully intercepted and handled it, allowing the driver to proceed without crashing or unloading, mirroring desirable native execution behavior.

Next, the multi-threading synchronization was demonstrated. The logs displayed two distinct thread IDs: "Thread A wait for thread B. Thread B does its work and signal. Then thread A continues reading the share data." This sequence confirmed that KDENU correctly emulates inter-thread communication and synchronization, ensuring that shared data access occurs in the proper order, preventing race conditions and incorrect execution flows that plague other emulators.

The most compelling part of the demo involved applying KDENU to a famous anti-cheat engine, which, for reasons of discretion, remained unnamed ("if you know, you know"). The speakers emphasized that static analysis of this particular sample was "almost impossible" due to heavy obfuscation, including the use of SEH for control flow and packing with a VM protector. This underscored the necessity of a powerful dynamic analysis tool like KDENU.

KDENU successfully uncovered a multitude of advanced detection methods employed by the anti-cheat driver:

1. Anti-Debugging Techniques:

• int 20h Instruction: The anti-cheat uses the int 20h instruction, which triggers the Kiswa interrupt, to verify the state of PatchGuard. If PatchGuard is enabled, it indicates a potentially tampered or debugged environment.
• PatchGuard Callbacks: It creates specific callbacks, identified as PatchGuardDbgCb, to directly invoke PatchGuard, serving as another means to detect modifications or debugging.
• DbgPrompt Function: This function is typically used by WinDbg to print debug messages to its console. The anti-cheat leverages DbgPrompt by purposefully triggering an error and handling it via SEH. If WinDbg is attached, the message is printed, but if no debugger is present, the SEH handler is invoked, allowing the anti-cheat to infer the presence or absence of a debugger. This technique effectively evades detection by emulators like Quilling or Speakeasy that do not properly handle SEH in this context.

1. MSR-Based Detections:

• Hypervisor Detection: The anti-cheat reads specific Model Specific Registers (MSRs). If the values correspond to known hypervisor configurations (e.g., KVM), it triggers an SEH for handling, similar to the DbgPrompt method, thus detecting the emulation environment.
• Disrupting Hypervisor Analysis: It writes to DBGCTL MSR (0x1D9), specifically setting and resetting its value, to flash Last Branch Records (LBR) and Branch Trace Flag (BTF). This action is designed to disrupt hypervisor-based analysis that relies on these features for tracing execution flow.
• LSTAR MSR Manipulation: The anti-cheat writes to LSTAR MSR (0xC0000082), which by default points to KiSystemCall64. This MSR is frequently trapped by many hypervisors. The anti-cheat writes an int 3 instruction to LSTAR and sets it to a pre-allocated address, indicating that it performs its own detection and hooking of system calls.
• Last Branch MSR (0x1DB) for VMEXIT Detection: Some hypervisors fail to restore state information correctly during VMEXIT and VMENTRY. The anti-cheat reads the Last Branch MSR (0x1DB), forces a VMEXIT (e.g., by triggering a privileged instruction), and then reads the MSR again upon VMENTRY. If the value differs, it signifies a virtualized environment.

1. OS-Dependent Offset Discovery:

• The anti-cheat uses NtCaptureImageExceptionValue to find OS-dependent offsets. This function, also seen in projects like Backbone, helps malware adapt to different OS versions, making it harder to analyze consistently.

While these advanced techniques were detailed, the speakers also briefly mentioned "easier methods" that could be implemented for driver analysis prevention, such as constantly repeating checks on C_USER_SHARED_DATA and DR7 (Debug Register 7) values, though these were not elaborated upon in the presentation. The demo conclusively proved KDENU's capability to dissect and understand even the most sophisticated anti-analysis measures.

Defensive Implications

▶ Watch: Audience Q&A: handling native DLL calls and future detections (30:00)

The insights gleaned from the development of KDENU and its application to advanced anti-cheat drivers have profound implications for both defenders (security analysts, incident responders) and developers (driver authors, anti-malware vendors).

For defenders, KDENU provides a powerful new tool to analyze evasive kernel-mode malware. Traditional dynamic analysis setups, including dual-machine debugging and many hypervisor-based solutions, are demonstrably detectable by sophisticated rootkits. Understanding the specific anti-debugging and anti-virtualization techniques uncovered (e.g., int 20h for PatchGuard, DbgPrompt for debugger presence, various MSR manipulations like DBGCTL and LSTAR, and Last Branch MSR checks for VMEXIT inconsistencies) is crucial. Defenders must recognize that their analysis environments might be compromised or detected, leading to incomplete or misleading analysis results. KDENU's hybrid real-simulated approach offers a path to bypass these detections, enabling more accurate and comprehensive analysis of highly resistant threats. This knowledge can inform the development of more robust analysis infrastructures that are less susceptible to evasion.

For developers of legitimate drivers, anti-malware solutions, or even those designing anti-cheat mechanisms, the talk serves as a blueprint for implementing advanced self-protection. The techniques demonstrated—such as integrating int 20h checks, using DbgPrompt as a debugger canary, strategically manipulating MSRs to disrupt hypervisors (e.g., flashing LBR/BTF, hooking LSTAR), and leveraging NtCaptureImageExceptionValue for dynamic offset resolution—can significantly raise the bar for reverse engineering and analysis. While the ethical implications of some of these techniques (especially those that disrupt legitimate analysis) require careful consideration, understanding their implementation can lead to more resilient software. The research also highlights the importance of robust Structured Exception Handling (SEH) and multi-threading in driver design, as their proper implementation is not only critical for stability but also for creating more complex and harder-to-analyze control flows that can further deter reverse engineering attempts.

In essence, KDENU's development and the findings presented bridge a critical gap in understanding and combating advanced kernel-mode threats. It equips defenders with a more potent weapon for analysis and provides developers with insights into building more secure and resilient kernel-mode components, fostering a more informed and capable cybersecurity landscape.

Key Takeaways

• Rootkit analysis remains a significant challenge: Modern kernel-mode malware employs sophisticated anti-debugging, obfuscation, and anti-virtualization techniques that render traditional static and dynamic analysis methods ineffective or detectable.
• Existing emulators fall short: Current emulation solutions like Speakeasy and Quilling lack robust support for critical features like Structured Exception Handling (SEH) and multi-threading, leading to inaccurate or failed driver execution. KACE, while innovative, is detectable due to its user-mode execution.
• KDENU offers a novel hybrid approach: By combining Unicorn's C++ bindings with the strategic use of kernel dumps, KDENU creates a highly realistic, yet fully controlled, semi-emulated and semi-native execution environment that can bypass advanced detection mechanisms.
• Realistic environment and robust core features are crucial: KDENU's use of kernel dumps for environment initialization and its strong, functional implementation of SEH and multi-threading are fundamental to accurately emulating complex kernel drivers.
• Advanced anti-analysis techniques were successfully uncovered: KDENU effectively identified sophisticated anti-cheat methods, including int 20h for PatchGuard detection, DbgPrompt for debugger presence, various MSR-based hypervisor detections (DBGCTL, LSTAR, Last Branch MSR checks), and dynamic OS-dependent offset resolution using NtCaptureImageExceptionValue.
• Empowering both defenders and developers: This research provides a powerful tool for security analysts to dissect highly evasive malware and offers valuable insights for developers to implement more resilient and self-protecting kernel-mode software.

About the Speaker(s)

The talk was a collaborative effort by three individuals from Taiwan's cybersecurity community.

Yong-Xu Yang, also known as Jello, is a junior majoring in Computer Science at National Sun Yat-sen University. He is currently an intern with the ATL team at TeamT5 and is a competitor for the upcoming World Skills competition in the cyber security category. He primarily hosted the presentation and discussed the core design and implementation of KDENU.

Heng-Ming Fan, known as Hobberger, is a Malware Analyst at the National Institute of Cyber Security in Taiwan. He played a crucial role in the project by assisting with debugging and fixing numerous bugs. Unfortunately, he was unable to attend the conference due to health issues.

Yu Xuan Luo, also known as Artemis, is a senior student also majoring in Computer Science at National Sun Yat-sen University. She contributed to the project by introducing the background and motivation behind the design of KDENU.