Cat & Mouse - or chess?

Fabian Mosch (Team Leader, Pentesting/Red Team · r-tec IT Security)

TROOPERS23 · Day 1 · Defense & Management · Defense & Management

Overview

In the realm of modern cybersecurity, Endpoint Detection and Response (EDR) systems stand as a critical line of defense, designed to detect and neutralize malicious activity at the endpoint. However, the cat-and-mouse game between defenders and attackers constantly pushes the boundaries of these technologies. Fabian Mosch's talk, "Cat & Mouse - or chess?", presented at TROOPERS, dives deep into this dynamic, specifically focusing on the prevalent EDR detection mechanism of userland hooking and introducing a novel technique to bypass it.

Watch on YouTube · Slides

Visual summary for Cat & Mouse - or chess? by Fabian Mosch
Visual summary for Cat & Mouse - or chess? by Fabian Mosch

Key moments

  1. 0:00 Introduction: Cat and Mouse - or Chess?
  2. 3:49 Understanding userland hooking: EDR's main detection method
  3. 12:15 Overview of existing userland hook bypass techniques
  4. 23:00 The new approach: preventing EDR DLLs in suspended processes
  5. 26:00 Key challenges implementing the Ruy Lopez bypass technique
  6. 36:00 Ruy Lopez demo: Bypassing AMSI in a Powershell process
  7. 45:00 Operational security (Opsec) and future improvements

Cat & Mouse - or chess?

Speakers: Fabian Mosch, Team Leader, Pentesting/Red Team, r-tec IT Security

Conference: TROOPERS

YouTube: https://www.youtube.com/watch?v=A2xwp_emlBY

Overview

In the realm of modern cybersecurity, Endpoint Detection and Response (EDR) systems stand as a critical line of defense, designed to detect and neutralize malicious activity at the endpoint. However, the cat-and-mouse game between defenders and attackers constantly pushes the boundaries of these technologies. Fabian Mosch's talk, "Cat & Mouse - or chess?", presented at TROOPERS, dives deep into this dynamic, specifically focusing on the prevalent EDR detection mechanism of userland hooking and introducing a novel technique to bypass it.

Fabian Mosch, a Team Leader for Pentesting/Red Team at r-tec IT Security, shares his extensive experience in evading EDRs, viewing such challenges as integral to his work. His presentation outlines how EDRs operate, summarizes existing bypass methods, and then unveils his own innovative approach, dubbed "Ruy Lopez." This technique fundamentally shifts the initiative, allowing an attacker to preemptively disable EDR's userland detection capabilities in a newly spawned process, thereby securing a significant advantage in the ongoing cybersecurity chess match.

The talk provides a comprehensive technical breakdown of EDR userland hooking, the intricacies of developing robust bypasses, and the practical challenges faced during implementation. By demonstrating how EDRs can be disarmed before they even begin their analysis, Mosch offers invaluable insights for both offensive security professionals seeking to refine their evasion tactics and defensive teams looking to understand and counter the latest adversary techniques.

Background

▶ Watch: Introduction: Cat and Mouse - or Chess? (0:00)

Endpoint Detection and Response (EDR) systems are sophisticated security solutions that continuously monitor and collect data from endpoint devices, such as workstations and servers, to detect and investigate suspicious activities. These systems employ various detection mechanisms, broadly categorized into userland and kerneland detections. Userland detections operate within the context of user processes, while kerneland detections leverage the operating system's kernel for deeper insights and control.

Fabian Mosch highlights that, from his extensive experience, userland detection, particularly userland hooking, remains a primary and highly effective method employed by EDR vendors. This technique involves EDR systems injecting their own Dynamic Link Libraries (DLLs) into user processes. Once loaded, these EDR DLLs patch specific Windows API functions in memory, replacing the legitimate function's initial instructions with a jump instruction. This jump redirects execution to the EDR's own code within its DLL.

For instance, if an attacker's malware attempts to call a critical API like NTProtectVirtualMemory (used to change memory protections) or NTCreateThreadEx (used to create new threads), the EDR's hook intercepts the call. The EDR can then analyze the input arguments passed to the API function (e.g., the start address for NTCreateThreadEx). By scrutinizing these arguments and the memory regions they point to, EDRs can perform runtime analysis, such as memory scans for known malicious patterns (e.g., Cobalt Strike, Sliver shellcode). If malicious activity is detected, the EDR can prevent the API call from completing or even terminate the offending process. While EDRs also utilize kernel callbacks and Event Tracing for Windows (ETW) for kerneland detections, Mosch emphasizes that userland hooking is the most prevalent and often the first line of behavioral defense.

Key Findings

▶ Watch: Overview of existing userland hook bypass techniques (12:15)

Fabian Mosch's talk first summarizes several well-known techniques for bypassing EDR userland hooks, setting the stage for his novel contribution. These existing methods include:

  • Unhooking: This involves retrieving a fresh, unpatched copy of ntdll.dll from disk and overwriting the hooked ntdll.dll functions in the process's memory. This effectively removes the EDR's jump instructions, restoring the original API functionality.
  • Direct Syscalls: Instead of calling high-level Windows APIs, attackers can directly invoke syscalls by embedding the raw assembly instructions (ASM stubs) for ntdll.dll functions within their own code. Techniques like Hell's Gate, Tartarus Gate, and SysWhispers (v2, v3) retrieve these syscall stubs from memory or embed them directly, bypassing any userland hooks placed on the higher-level API wrappers.
  • Hardware Breakpoints: Tools like "Tampering Syscalls" leverage hardware breakpoints to intercept API calls before the EDR's hook is reached. This allows the attacker to manipulate input arguments (e.g., replacing a malicious start address with zero) before the EDR can inspect them. After the EDR's check, the original arguments are restored, and the legitimate syscall proceeds.
  • DLL Entry Point Patching: The "SharpBlock" technique creates a new process in a suspended state with the DEBUG_ONLY_THIS_PROCESS flag. Acting as a debugger, the attacker can intercept the Load DLL Debug Event and patch the entry point of EDR DLLs (e.g., by inserting a 0xC3 instruction for RET), causing them to immediately exit upon loading without placing any hooks.

Mosch then introduces his own innovative approach, which he dubs "Ruy Lopez", drawing an analogy from chess where the white player (typically the EDR) makes the first move. His technique aims to seize this "first move" advantage by preventing EDR DLLs from loading into a new process altogether, thereby bypassing userland hooks before they are even placed. The core finding is that by creating a process in a suspended state, an attacker can inject custom shellcode that hooks DLL loading functions (like NtCreateSection) before any EDR DLLs are initialized, allowing the attacker to control which DLLs are mapped into memory.

Technical Deep Dive

▶ Watch: The new approach: preventing EDR DLLs in suspended processes (23:00)

The "Ruy Lopez" technique is built upon a fundamental observation: when a new process is created in a suspended state (CREATE_SUSPENDED), only ntdll.dll is initially loaded. This provides a critical window of opportunity for an attacker to establish control before any EDR DLLs can inject themselves and set up userland hooks.

The detailed technical flow of Ruy Lopez is as follows:

  1. Suspended Process Creation: The attacker's malware creates a new target process (e.g., powershell.exe) using the CREATE_SUSPENDED flag. This ensures the process starts with minimal initialization, specifically with only ntdll.dll loaded.
  2. Memory Allocation and Shellcode Injection: The attacker's process allocates a region of memory within the newly spawned, suspended process. Into this allocated memory, it injects custom-written position-independent shellcode (PIC).
  3. Remote Hooking of DLL Loading Functions: The injected shellcode's primary task is to remotely hook critical ntdll.dll functions responsible for loading DLLs, such as NtCreateSection. This is achieved by overwriting the initial bytes of NtCreateSection with a jump instruction that redirects execution to a specific part of the injected shellcode.
  4. Interception and Manipulation: When the suspended process is later resumed, the operating system attempts to load other necessary DLLs, including the EDR's own DLLs. Any call to NtCreateSection (or similar DLL loading functions) will now be intercepted by the attacker's shellcode.
  5. Preventing EDR DLL Loading: The shellcode's logic determines if the DLL currently being loaded is an EDR DLL (e.g., by checking its name or hash). If it is, the shellcode manipulates the return value of NtCreateSection, causing it to "fail" (e.g., returning STATUS_UNSUCCESSFUL or 0). This prevents the EDR DLL from being properly mapped into the process's memory.
  6. Process Resumption: After the shellcode has set up its hooks and is ready to intercept DLL loading, the host process resumes the suspended child process. From this point onwards, the attacker's shellcode has preemptive control over which DLLs are loaded.

Implementing this technique presented several significant challenges, primarily due to the constraints of position-independent code and the suspended process environment:

  • Writing Position-Independent Code (PIC): Mosch highlighted the difficulty of writing PIC from scratch. PIC cannot rely on global variables or static addresses. All API functions must be resolved at runtime by parsing the Process Environment Block (PEB) and walking the in-memory module lists to find ntdll.dll and then specific function exports. The standard C runtime (CRT) startup routine also had to be bypassed by replacing the main entry point with a custom jump, as the CRT performs stack and memory initializations that can interfere with PIC.
  • ntdll.dll Only: A crucial limitation is that the injected shellcode can only use functions exported by ntdll.dll. The process initialization is not complete, meaning kernel32.dll or other common system libraries are not yet available. This necessitated manually reimplementing many standard functions (e.g., strcmp, memcpy) using only ntdll.dll primitives or raw assembly. Debugging was also complicated, requiring a custom logger that writes to a file using ntdll.dll functions instead of standard debugging output.
  • Restoring Original Functionality: After the EDR DLL is blocked, legitimate DLLs still need to be loaded. The shellcode must be able to call the original NtCreateSection function. To achieve this, Mosch implemented an "egg hunter" mechanism. The host process copies the original bytes of NtCreateSection into a specific "egg" pattern within the shellcode. The shellcode then locates this egg, extracts the original bytes, and restores NtCreateSection before calling it for legitimate DLLs.
  • Stack Alignment: Standard PIC often includes assembly stubs for stack alignment (e.g., 16-byte alignment). However, Mosch found that including these in his shellcode corrupted the input parameters for NtCreateSection because they were already on the stack. The solution was to directly jump to the hook function, as the calling function for NtCreateSection already handles stack alignment.
  • Error Code Fiddling: The specific NTSTATUS return value from the hooked NtCreateSection is critical. Returning 0 (success) or a specific error code (e.g., 0xC000000D for STATUS_INVALID_PARAMETER) could lead to different behaviors across EDRs and target DLLs. For instance, blocking amsi.dll for PowerShell required a specific error code to prevent process crashes, while for other EDR DLLs, returning 0 might be more effective. This "fiddling around" highlights the nuance required for successful evasion.

The name "Ruy Lopez" perfectly encapsulates the strategy: by making the first, controlling move in the newly created process, the attacker gains a decisive advantage, much like the white player in a chess game.

Demo / Proof of Concept

▶ Watch: Ruy Lopez demo: Bypassing AMSI in a Powershell process (36:00)

Fabian Mosch demonstrated the "Ruy Lopez" proof of concept (PoC) by targeting amsi.dll within a newly spawned PowerShell process. The demonstration highlighted the effectiveness of the technique in preventing AMSI from loading, which typically scans PowerShell scripts for malicious content.

The PoC performed the following steps:

  1. It spawned a new PowerShell process in a suspended state.
  2. It injected its custom shellcode into this suspended process.
  3. The shellcode hooked NtCreateSection.
  4. Upon resumption, when PowerShell attempted to load amsi.dll, the hook intercepted the call and prevented AMSI from being mapped into memory.

To verify the bypass, Mosch showed that attempting to execute the string AmSi ScAn BuFfEr (a known AMSI test string) in a normal PowerShell session resulted in the script being blocked. However, in the PowerShell process spawned by "Ruy Lopez", the same string was simply unrecognized as a command, confirming AMSI was not active. Further verification using Process Hacker demonstrated that amsi.dll was indeed absent from the process's loaded modules.

Subsequently, Mosch showcased the post-bypass capabilities by successfully loading and executing two well-known offensive tools:

  • WinPwn: A PowerShell script for privilege escalation and post-exploitation. This loaded and ran without issue, confirming the AMSI bypass.
  • Mimikatz: A popular tool for credential dumping. While Mimikatz successfully loaded and was initially executed, it triggered a memory scanner from Windows Defender. This is a crucial distinction: the userland hook bypass for AMSI was successful, but a different EDR mechanism (behavioral memory scanning) still detected the known malicious binary in memory. Mosch noted that even this memory scan could sometimes be bypassed by providing input arguments directly, making the execution faster than the scanner's response time, but acknowledged Defender's inconsistent behavior during live demos.

Mosch reported that the "Ruy Lopez" technique was tested against multiple EDR vendors (though specific names were not disclosed beyond Defender) and successfully blocked their respective EDR DLLs from loading. In these tests, no alerts were triggered by userland hook detections. However, he also noted limitations:

  • The technique cannot block EDR DLLs that are injected into a process rather than loaded via standard DLL loading mechanisms (e.g., psi_inject.dll from one vendor).
  • Some EDR vendors' DLLs, if blocked, could cause the target process to crash if the EDR's code does not properly handle the NtCreateSection error status code.
  • Crucially, the technique does not work against Defender for Endpoint (MDE) because MDE primarily relies on ETW and kernel-mode detections, not userland hooks.

Defensive Implications

▶ Watch: Operational security (Opsec) and future improvements (45:00)

Fabian Mosch's "Ruy Lopez" technique, while demonstrating a potent offensive capability, also provides critical insights for defensive strategies. The core takeaway for defenders is that EDR userland hooks, despite their prevalence, can be preemptively bypassed.

Indicators of Compromise (IOCs) and Detection Opportunities:

Mosch acknowledges that injection and hooking techniques leave well-documented IOCs. These include:

  • Private committed memory: The injected shellcode resides in memory regions not backed by legitimate files on disk.
  • Hooked functions: Manual inspection of ntdll.dll in a process's memory would reveal the patched API functions (e.g., a jump instruction at the start of NtCreateSection).
  • Suspended process creation: The initial creation of a process in a suspended state is an unusual behavior that could be flagged.
  • Missing EDR DLLs: A legitimate process running without expected EDR DLLs loaded is a strong indicator of compromise.

However, Mosch notes a significant gap: automated detection for these specific IOCs was not observed during his testing against EDRs. EDR vendors, at the time of the talk, were not consistently checking for processes spawned in suspended mode, or for hooks placed within ntdll.dll itself. This highlights a blind spot that blue teams and threat hunters should actively address.

Defensive Recommendations:

  1. Monitor for Suspended Process Creation: Implement robust logging and alerting for processes created with the CREATE_SUSPENDED flag, especially when followed by remote memory allocation, injection, and thread resumption.
  2. Memory Scanning for Unbacked Code: Enhance memory scanning capabilities to identify private committed memory regions that contain executable code but are not backed by legitimate DLLs on disk.
  3. Integrity Checks for ntdll.dll: Develop or deploy tools that periodically verify the integrity of critical ntdll.dll functions in memory, looking for unexpected patches or jump instructions. This can be done by comparing in-memory code with a known good copy from disk.
  4. Process Module Analysis: Monitor the loaded DLLs within processes. Alert on critical system processes (like PowerShell) where expected EDR or AMSI DLLs are conspicuously absent.
  5. Leverage Kernel-Mode Telemetry: For EDRs that rely heavily on kernel callbacks and ETW (like Defender for Endpoint), these mechanisms remain viable for detecting post-bypass activity (e.g., memory scans for Mimikatz, as seen in the demo). Defenders should ensure these kernel-level protections are robust and well-configured.
  6. Embrace Advanced OpSec for Attackers (and thus, Detection for Defenders): Mosch outlined several OpSec improvements for attackers, which directly translate into advanced detection strategies for defenders:
  • Direct Syscalls for Host Process: Attackers are moving towards using direct syscalls for the host process's injection and hooking operations, reducing the chance of detection by higher-level API monitoring. Defenders need to analyze syscall traces.
  • RX Memory Regions: Attackers aim for Read-Execute (RX) memory regions for shellcode instead of Read-Write-Execute (RWX), as RWX is a common IOC. Defenders should prioritize flagging RWX memory regions with executable code.
  • API Hashing for DLL Names: Using hashes instead of cleartext DLL names (e.g., amsi.dll) prevents static signature detection. Defenders should consider behavioral analysis and dynamic string extraction.
  • Hardware Breakpoints for Evasion: If attackers use hardware breakpoints instead of memory patching to avoid hooking IOCs, defenders need to monitor for debugging activity within processes (e.g., DBG_CREATE_PROCESS_DEBUG_EVENT, DBG_RIPEVENT_DEBUG_EVENT).
  • Alternative Execution Styles: Techniques like threadless inject avoid creating new threads, reducing a common IOC. Defenders should focus on code injection into existing threads or atypical execution flows.

Ultimately, while "Ruy Lopez" provides a method to bypass userland hooks, it introduces new (or leverages existing) IOCs that, if properly monitored, can lead to detection. The evolving nature of this "chess game" requires continuous adaptation from both sides.

Key Takeaways

  • Userland Hooking is a Primary EDR Mechanism: EDRs heavily rely on injecting DLLs and patching Windows API functions in user processes to intercept and analyze malicious activity at runtime.
  • "Ruy Lopez" Offers a Novel Pre-Loading Bypass: The technique creates a process in a suspended state, injects custom shellcode, and hooks DLL loading functions (e.g., NtCreateSection) before EDR DLLs can load, effectively preventing EDR userland hooks from being placed.
  • Complex Implementation Challenges: Developing "Ruy Lopez" required extensive work with Position-Independent Code (PIC), exclusively using ntdll.dll functions, manual API resolution, and intricate handling of stack alignment and NTSTATUS return values.
  • Effective Against Userland Hooks, But Not All EDR Layers: The PoC successfully bypassed AMSI and prevented EDR DLLs from loading in tests. However, other EDR mechanisms, such as memory scanning, can still detect post-bypass malicious payloads like Mimikatz, demonstrating the multi-layered nature of EDRs.
  • Significant OpSec Considerations for Attackers, Detection Opportunities for Defenders: While automated detection for "Ruy Lopez" was not observed at the time of the talk, the technique leaves identifiable Indicators of Compromise (IOCs) like private committed memory, hooked ntdll.dll functions, and suspended process creation, which blue teams can leverage for manual or advanced behavioral detection.
  • Does Not Apply to All EDRs: The technique is ineffective against EDRs that do not rely on userland hooks (e.g., Defender for Endpoint, which uses ETW/kernel callbacks) or those that inject their DLLs rather than loading them via standard mechanisms.

About the Speaker(s)

Fabian Mosch is a Team Leader for Pentesting/Red Team at r-tec IT Security, a company based in Germany. His work primarily involves simulating real-world attackers by breaking into company environments and escalating privileges, with the ultimate goal of providing actionable security recommendations. Fabian has a deep passion for AV/EDR evasion, viewing it as a continuous challenge rather than an insurmountable barrier. Inspired by the open-source community, he actively contributes by sharing his knowledge through a personal blog, YouTube videos, and various scripts and tools on GitHub. His specialized interest in bypassing security controls forms the foundation of his research and presentations, such as "Cat & Mouse - or chess?".

Reviews

Dr. Zero (Offensive Security Researcher) — SOLID

Mosch delivered a highly competent technical dive into preempting EDR userland hooks by hijacking NtCreateSection in a suspended process. It's not a magic bullet—ETW and kernel callbacks don't care—but the PIC engineering is solid. Strong red team craft, honestly presented without the usual invincibility bullshit.

Heather Calloway (CISO) — SOLID

A deep-technical breakdown of a novel EDR evasion technique that prevents security DLLs from loading in suspended processes. While the exploit mechanics belong to the red team, the strategic takeaway for leadership is clear: userland EDR detection is fragile and cannot act as a single point of failure in your endpoint strategy.

→ Top-rated talks at TROOPERS23

All talks from TROOPERS23