Evading in plain sight: how adversaries beat user-mode protection engines for over a decade

Evading in plain sight: how adversaries beat user-mode protection engines for over a decade

Speakers: Omri Misgav, Independent Researcher, Independent researcher

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=VK_eak13r_A

Overview

In this insightful talk, independent researcher Omri Misgav delves into the pervasive and often underestimated methods adversaries employ to bypass user-mode protection engines. Highlighting a decade of established techniques, Misgav meticulously breaks down the fundamental flaws in relying solely on user-mode hooks for security and demonstrates how attackers consistently evade detection by manipulating the very environment these engines are designed to protect. The presentation serves as a critical examination of why traditional user-mode security measures are insufficient against sophisticated threats.

Misgav's research underscores a significant gap in current security understanding and frameworks, notably the limited coverage within MITRE ATT&CK for these highly prolific post-exploit techniques. By outlining three major adversary tactics—Hook Evasion, Argument Forgery, and Engine Disarming—the talk provides defenders with a comprehensive understanding of the adversary's playbook. It emphasizes that many of these evasion methods are trivial to implement, share common primitives, and often serve as prerequisites for other advanced bypasses, such as those targeting ETW and AMSI. This detailed exploration is crucial for security professionals seeking to build more resilient defenses against modern malware.

Background

▶ Watch: Introduction: User-mode protection engines and hooking. (0:00)

User-mode protection engines are ubiquitous components in modern security software, designed to monitor and control the behavior of user applications. These engines operate by injecting code into user-mode processes, typically employing two primary monitoring approaches: instrumentation, where the software package is pre-modified by the provider, or hooking, which involves runtime modification of target functions by security vendors. The preference for user-mode implementation stems from several perceived advantages: it's considered more stable (less prone to blue screens), simpler to develop, and benefits from the absence of Windows' kernel patch protection. Crucially, user-mode provides the full context of application operations, allowing visibility into data in clear text, such as SSL/TLS sockets.

The most common hooking method is inline hooking, where the protection engine directly overwrites the initial instructions of a target function in memory, typically with control flow instructions that redirect execution to the engine's monitoring logic. While effective against naive threats, this approach introduces a fundamental vulnerability: the protection engine relies on the integrity of the very user-mode environment it seeks to protect. Adversaries, recognizing this design flaw, have developed a wide array of techniques over the past decade to circumvent these hooks, execute original code, or disable the engines entirely, often going unnoticed by standard detection mechanisms.

Key Findings

▶ Watch: Tactic 2: Binary Restoration (Unhooking) techniques. (4:48)

Omri Misgav's presentation reveals several critical findings concerning the efficacy and vulnerabilities of user-mode protection engines:

• Fundamental Design Flaw: The core issue is that user-mode protection engines operate within the same security context they are meant to defend. This inherent reliance makes them susceptible to manipulation by adversaries who can control or influence that environment.
• Three Major Adversary Tactics: Misgav categorizes evasion techniques into three overarching tactics: Hook Evasion, Argument Forgery, and Engine Disarming. These represent a comprehensive framework for how attackers circumvent user-mode defenses.
• Prolific and Trivial Implementation: Many of the discussed techniques are not overly complex; they are often trivial to implement, share common underlying primitives, and have readily available source code online. This ease of implementation contributes to their widespread adoption by malware.
• Prerequisite for Other Evasions: User-mode hook evasion is frequently a foundational step for other advanced bypasses, such as those targeting Windows' Event Tracing for Windows (ETW) or Anti-Malware Scan Interface (AMSI). An attacker must first establish control or bypass initial monitoring before proceeding with further evasions.
• Inadequate MITRE ATT&CK Coverage: The talk highlights a significant gap in current threat intelligence frameworks, specifically MITRE ATT&CK, which offers only minimal coverage—a single paragraph—on the entire topic of user-mode hook evasion, often only referencing unhooking or engine disarming. This underrepresentation suggests a lack of awareness or prioritization of these critical post-exploit techniques.
• Call Stack Spoofing: Even advanced detection mechanisms relying on call stack analysis can be circumvented, as adversaries have developed methods to spoof call stacks. This necessitates more robust defenses like CT shadow stacks, hardware-controlled environments (emulators/sandboxes), or transition-based code tracing.
• User-Mode Hooks Alone Are Insufficient: The overarching conclusion is that relying solely on user-mode hooks for security is no longer adequate. A multi-layered approach incorporating kernel-level monitoring, hardware-assisted security, and advanced integrity checks is essential.

Technical Deep Dive

▶ Watch: Tactic 3: Direct System Call invocation methods. (10:00)

Misgav meticulously breaks down the technical mechanisms adversaries leverage across three primary tactics: Hook Evasion, Argument Forgery, and Engine Disarming. Each tactic comprises several sophisticated techniques designed to bypass user-mode protection engines.

1. Hook Evasion

The goal of Hook Evasion is to prevent the protection engine from intercepting API calls by executing the original, unhooked instructions.

• Secondary DLL Mapping:
• Concept: Malware creates its own private instance of a system library (e.g., ntdll.dll) that remains unhooked. It then uses this private instance for its operations.
• Techniques:
• Reflective loading: Loading a DLL directly from memory without touching the disk.
• Clone and reload: Copying the DLL from disk with a new name and reloading it.
• Remap section: Using low-level NT APIs (NtCreateSection, NtMapViewOfSection) to map a new, unhooked section of the DLL into memory.
• Detection: Call stack analysis can reveal missing expected DLLs or unexpected duplicate system DLLs. Forensic analysis (e.g., process snapshots) can identify "floating PEs" (Portable Executables) in memory or duplicate entries in the Process Environment Block (PEB).

• Binary Restoration (Unhooking):
• Concept: After the protection engine installs its hook, the adversary restores the original instructions in the target function's memory region.
• Techniques:
• Copy from secondary DLL map: If a private, unhooked DLL instance already exists (from secondary DLL mapping), the original code can be copied from there.
• Copy from other processes:
• Create a suspended child process to obtain an unhooked copy of ntdll.dll.
• Create a debug DLL and wait for the target DLL to load, then extract its original code.
• Search existing processes: Iterate through running processes, find one not monitored by the protection engine, read its memory, and copy the original code. This often involves VirtualProtect to modify memory permissions and memcpy to write the original bytes.
• Section Refresh: Instead of copying, this technique unmaps the code section of the hooked DLL from memory using low-level APIs (NtUnmapViewOfSection) and then remaps it again from its original file on disk (NtMapViewOfSection), effectively restoring its pristine state. Data sections are captured and restored to maintain integrity.
• Short Circuiting: Protection engines often copy the original instructions to an intermediate location (a trampoline) before overwriting the target function. Adversaries can scan the process address space for memory regions with specific attributes (e.g., executable, read-only) to locate this original code. Once found, they can either overwrite the hook directly to redirect to this original stub or modify the trampoline itself to bypass the engine's logic.
• Detection: Forensics can be quite effective, with tools like specific Volatility plugins designed to detect unhooking. Runtime detection is more challenging but possible for some techniques by monitoring memory regions for unexpected changes or comparing function prologues against known good states.

• Direct System Call Invocation:
• Concept: Bypass user-mode hooks entirely by directly invoking kernel-mode system calls, rather than going through the hooked user-mode API wrappers (e.g., functions exported by ntdll.dll).
• System Call Mechanics: System calls transition from user mode to kernel mode. On Windows, the system service index is placed in the EAX register, followed by either the syscall instruction or IN2E. Notably, IN2E was the older interface but has been reintroduced as the default with Virtualization-Based Security (VBS) enabled.
• Techniques:
• Manual Parsing of ntdll.dll: Disassemble ntdll.dll (from disk or memory) to extract the specific system call index (SSN - System Service Number) for a desired function. The malware then directly uses the syscall or IN2E instruction with this index.
• Index Calculation from ntdll.dll: ntdll.dll exports its NT functions (e.g., NtCreateFile) in memory, sorted by their system call index. An attacker can count offsets to find the desired function's index or, if a function is hooked, query a neighboring function's index and adjust accordingly.
• Evansgate: A Windows-specific technique for 32-bit applications to interact with 64-bit kernel system calls. The malware leverages the WOW64 emulation layer or directly transitions its context to 64-bit within the 32-bit process, then calls the native 64-bit ntdll.dll's system calls or manually invokes them.
• Kernel Parsing (User Space): Instead of ntdll.dll, the adversary loads the kernel binary (e.g., ntoskrnl.exe) into user space. They then parse it to locate Zw functions, which hold the system call indexes. For functions where Zw variants aren't exported, the private, non-exported KI_SERVICE_TABLE within the kernel can be located by correlating exported Zw and Nt function pairs. While not all system services are covered, this provides access to most critical functionalities.
• Hardcoded Indexes: The simplest method, but least flexible. Adversaries embed specific system call indexes directly into their malware. This requires maintaining separate indexes for each Windows build or version they wish to target, as indexes are not guaranteed to be stable across OS updates.
• Dynamic Resolution: At runtime, the malware sets an exception handler and places a hardware breakpoint on the syscall instruction of the target system call service. When the function is triggered and the breakpoint hits, the system call index can be extracted from the EAX register within the exception handler context.
• Detection: The most straightforward runtime indicator is the absence of expected system DLLs in the call stack during critical operations. On systems with VBS enabled, using the syscall instruction instead of the IN2E instruction can also be a strong indicator of direct system call invocation. Forensic analysis is challenging due to potential obfuscation or packing.

• Code Splicing:
• Concept: Rebuild the entire function stub at a different, unmonitored memory location. This was historically used by packers and anti-cheat solutions.
• Technique: Use reflective loading to obtain the original code, then create a new, clean copy in memory. All subsequent calls are directed to this new copy, bypassing any hooks on the original function.
• Detection: Few direct runtime or forensic indicators. A drawback is that internal functions called by the spliced stub might still be hooked.

• Stub Reuse:
• Concept: A simpler variant of short-circuiting. If the original function instructions (stubs) can be found (e.g., copied by the hooking engine), the attacker simply calls them directly.
• Detection: Similar to code splicing, direct detection is difficult without deep memory inspection.

2. Argument Forgery

Argument Forgery represents a shift in strategy: instead of evading the hook, the adversary allows the protection engine to intercept the call but feeds it false parameters. This creates a Time-of-Check, Time-of-Use (TOCTOU) vulnerability.

• Concept: The protection engine checks benign parameters, allowing the operation to proceed. After the check, but before the actual malicious action, the adversary swaps the benign parameters for malicious ones.
• Techniques:
• Leveraging Hardware Breakpoints:

1. Set an exception handler.
2. Identify the return point of the hooked function (e.g., at a +7 offset from the function start, where original instructions resume).
3. Place a hardware breakpoint at this return point.
4. Call the target function with fake (benign) parameters.
5. The protection engine processes these fake parameters.
6. When the function returns and the hardware breakpoint is hit (after the engine's check), the exception handler swaps the parameters (either in registers or on the stack, depending on the calling convention) with the real, malicious values (e.g., a malicious URL).
7. Execution continues with the malicious parameters, bypassing the initial check.

• Vendor-Specific Exception Generation: Less reliable, this technique involves feeding an argument designed to cause an exception within the specific vendor's code. This exception can then be handled to switch parameters.
• Detection: Requires sophisticated runtime monitoring that can detect parameter changes after initial checks but before the system call. This is challenging as it involves monitoring the entire lifecycle of an API call rather than just its entry point.

3. Engine Disarming

Engine Disarming is the most direct approach: the adversary attempts to disable or unload the entire user-mode protection engine.

• Techniques:
• Call FreeLibrary: Directly call the FreeLibrary API to unload the protection engine's DLL from the process.
• Trigger Unload Function: Some vendors provide a specific unload or uninstall function within their engine, which can be leveraged if discovered.
• Unmap All DLLs: A more drastic measure, unmapping all DLLs from the process. This leaves the attacker without any standard library functionality, severely limiting their subsequent actions.
• Leverage OS Process Mitigations: Utilize OS features like Microsoft signed binaries only to prevent unsigned third-party DLLs (including security engines) from loading into a new process. This limits the payload to a signed executable or DLL.
• Preloading in New Process: When a new process is created, the adversary can inject code at an early enough stage (e.g., using debugger APIs or early injection techniques) before the protection engine has a chance to load into that process. This ensures the new process remains unmonitored.
• Vendor-Specific Issues: Exploiting specific vulnerabilities or undocumented features in a particular vendor's product, such as toggling an enable/disable flag in memory.
• Detection: Monitoring for attempts to FreeLibrary critical security DLLs, observing unusual process creation patterns (e.g., processes launched without security DLLs), or detecting early-stage injections/debugger attachments.

Demo / Proof of Concept

▶ Watch: Newer Tactic: Disabling the protection engine entirely. (22:30)

While the talk did not feature a live demonstration or explicit proof-of-concept code, Omri Misgav noted that many of the discussed techniques are "quite trivial to implement" and that "most of it has source code online" for those interested in experimenting. The focus was on the underlying principles and mechanisms rather than a specific tool or exploit. Misgav also mentioned the release of a white paper complementing the talk, which includes references to open-source projects and further details on detection.

Defensive Implications

▶ Watch: Fundamental flaw: User-mode hooks are insufficient for security. (27:30)

Omri Misgav's comprehensive overview of adversary techniques for bypassing user-mode protection engines highlights a critical need for defenders to evolve their strategies. The fundamental design flaw of user-mode hooks—that they operate within the environment they aim to protect—renders them insufficient as a sole defense.

Defenders must adopt a multi-layered approach that moves beyond simple user-mode hooking:

1. Shift to Kernel-Mode and Hardware-Assisted Security: For robust protection, security solutions should leverage kernel-mode monitoring, hypervisor-based techniques, or hardware-assisted virtualization. These approaches operate at a lower privilege level or outside the guest OS entirely, making them significantly harder for user-mode malware to subvert.
2. Enhanced Call Stack Integrity: While adversaries can spoof call stacks, defenders should still implement and improve call stack analysis. This includes leveraging modern OS features like Control-flow Enforcement Technology (CET) Shadow Stacks where available, or employing techniques like transition-based code tracing in emulated or sandboxed environments where hardware control is possible.
3. Comprehensive Runtime Monitoring:

• DLL Integrity: Monitor for unexpected DLL loads/unloads, duplicate system DLLs, or missing critical security DLLs in process memory or call stacks.
• Memory Region Monitoring: Continuously scan for unexpected memory modifications in critical code sections, changes in memory protection attributes, or the presence of "floating PEs."
• System Call Anomalies: Detect direct syscall instruction usage, especially on systems with VBS enabled where IN2E is expected. Analyze system call arguments for rapid, post-check modifications indicative of Argument Forgery (TOCTOU) attacks.
• Process Creation and Injection: Scrutinize new process creations for early-stage injection attempts, debugger API usage, or the absence of expected security DLLs, which could indicate Engine Disarming attempts.

1. Advanced Forensics: Develop and utilize robust forensic tools (like advanced Volatility plugins) capable of detecting artifacts of unhooking, secondary DLL mapping, and other memory-based evasions in process snapshots.
2. Behavioral Analysis: Focus on detecting malicious behavior rather than solely relying on API call interception. This includes monitoring for unusual process interactions, attempts to modify security software, or suspicious network communications that might indicate a successful bypass.
3. Static and Dynamic Analysis: Employ static analysis to identify hardcoded system call indexes in malware binaries and dynamic analysis in sandboxed environments to observe how malware resolves system call indexes at runtime or manipulates parameters.
4. Educate and Update Threat Intelligence: Acknowledge the critical gap in frameworks like MITRE ATT&CK regarding user-mode hook evasion. Actively contribute to and consume threat intelligence that details these techniques to better prepare against them.

By recognizing the inherent limitations of user-mode hooks and adopting a proactive, multi-faceted defense strategy, organizations can significantly improve their resilience against sophisticated adversaries.

Key Takeaways

• User-mode hooks alone are insufficient: Relying solely on user-mode protection engines creates a fundamental design flaw, as adversaries can manipulate the environment they aim to protect.
• Adversaries have a rich, mature toolkit: Techniques for hook evasion, argument forgery, and engine disarming are well-established, often simple to implement, and have been evolving for over a decade.
• Direct system calls are a major bypass: Attackers can bypass user-mode hooks entirely by invoking kernel-mode system calls directly, using various methods from parsing ntdll.dll to hardcoded indexes and dynamic resolution.
• TOCTOU attacks are effective against checks: Argument forgery exploits time-of-check, time-of-use vulnerabilities, allowing malicious parameters to be swapped in after a benign check by the security engine.
• Detection requires deep OS knowledge and layered monitoring: Effective defense demands comprehensive runtime monitoring (call stack, memory integrity, syscall anomalies), advanced forensics, and a move towards kernel-level or hardware-assisted security.
• MITRE ATT&CK needs expanded coverage: Current threat intelligence frameworks significantly underrepresent the breadth and prevalence of user-mode hook evasion techniques, leading to potential blind spots for defenders.

About the Speaker(s)

Omri Misgav is an independent researcher specializing in low-level research, particularly focusing on OS internals and malware. He has presented at various security conferences, and his talk on "Evading in plain sight" at Virus Bulletin marks his first appearance at the conference. His work aims to shed light on advanced adversary techniques and their implications for modern security solutions.