EDR Introspection

EDR Introspection

Speakers: Levi Cailleret, C-Cert, Raiffeisen Schweiz

Conference: Area41

YouTube: https://www.youtube.com/watch?v=vqty37beAmc

Overview

In the realm of cybersecurity, Endpoint Detection and Response (EDR) solutions are the frontline defense against advanced threats. However, for both red and blue teams, EDRs often operate as opaque black boxes, offering limited insight into their internal workings. This lack of transparency makes it challenging to understand precisely how and when an EDR detects or misses malicious activity, hindering effective threat emulation and defense optimization. Levi Cailleret's talk, "EDR Introspection," addresses this critical gap by presenting a novel framework designed to reveal the exact actions an EDR performs at any given moment when analyzing malware.

This presentation, stemming from Cailleret’s master's thesis at ETH, focuses primarily on Microsoft Defender for Endpoint (MDE), specifically its underlying MSMP engine. The core objective was to build a monitoring system capable of tracking MDE's steps – from static analysis to emulation and runtime tracking – thereby demystifying its detection logic. By employing techniques similar to those used by EDRs themselves, Cailleret demonstrates how security professionals can gain unprecedented visibility into MDE's decision-making processes, offering invaluable insights for both offensive and defensive strategies.

Background

▶ Watch: Introduction: EDRs as black boxes (0:00)

Modern EDR solutions are sophisticated platforms designed to protect endpoints from a wide array of cyber threats. Their operation typically involves a multi-stage process to evaluate executables and running processes. Initially, when an executable is placed on disk, EDRs perform static checks to determine if it's known good, known bad, or unknown. If unknown, it may proceed to emulation, where the file is run in a sandboxed environment to observe its behavior. This emulation phase often includes a memory scan to detect in-memory threats. If the file is still deemed unknown but allowed to run, the EDR continues its analysis when the user invokes the executable, repeating scans and conducting thorough memory inspections of the live process. Throughout its runtime, the EDR continuously tracks the executable, performing additional scans if suspicious activities are detected.

EDRs gather their intelligence from various low-level system sources. A common technique is anti-DLL hooking, where the EDR injects its own DLLs into processes to intercept and monitor system calls (syscalls), such as opening files or processes. This allows the EDR to observe nearly every action an application performs. Another crucial source is Event Tracing for Windows (ETW), a powerful logging mechanism within the Windows kernel. MDE, for instance, attaches to over 130 ETW providers, many of which log kernel-level events. Additionally, kernel callbacks and mini filters are utilized by EDRs to monitor process and file system activities, though these were less central to this particular talk. The challenge for security researchers has always been to reverse-engineer these complex mechanisms to understand the EDR's internal logic, a task made difficult by the proprietary and often obfuscated nature of these solutions.

Key Findings

▶ Watch: Live demo: EDR hook back reveals Defender actions (8:00)

The EDR Introspection framework yielded several significant findings, fundamentally altering the perception of how EDRs like Microsoft Defender for Endpoint operate. Foremost among these was the empirical confirmation that the previously theorized mechanisms – static checks, emulation, memory scanning, and runtime tracking – are indeed actively employed by Defender. While their existence was known to malware developers, the framework precisely revealed when, how, and what triggers these actions, providing concrete data rather than speculation.

A pivotal discovery was the ability to observe the main phases of Defender's operation using readily available ETW traces and administrator-level permissions, circumventing the initial need for complex kernel-level bypasses. This simplified approach makes EDR introspection more accessible for blue teams and researchers. For instance, static scans and emulation activities could be correlated with stream scan and scan events within the ETW anti-malware engine stream, while memory scans were observable through Defender's OpenProcess calls to target processes, logged in Kernel Audit API Calls.

The research also highlighted the effectiveness of deconditioning techniques as a viable evasion strategy. Through controlled experiments, it was demonstrated that EDRs could be "exhausted" or desensitized by a high volume of benign or irrelevant activity before a malicious payload is deployed. This confirms prior research by Dobbin ("Super Mega") and Bamboozle EDR, suggesting that EDRs have internal limits on their scanning capabilities or event processing, which attackers can exploit. Specific behavioral triggers were also identified; for example, whoami spawning calculator was flagged as suspicious, leading to deeper EDR inspection, yet this could still be bypassed under certain conditions. The framework ultimately provides a granular view of Defender's interactions, logging specific API calls like NtOpenProcess, VirtualMemoryRead, and QueryInformation, revealing the exact data Defender attempts to access during its analysis.

Technical Deep Dive

▶ Watch: Detailed EDR analysis of process injection attack (12:00)

The core innovation of EDR Introspection lies in its ability to turn the tables on the EDR itself, using similar monitoring techniques to observe the EDR's internal processes. The initial, most comprehensive approach involved hooking the EDR's anti-DLLs, specifically targeting ntdll.dll within the MsMpEng process. By injecting a monitoring DLL into MsMpEng, the framework could record every system call made by Defender, such as NtOpenProcess when it attempts to analyze a suspicious process, or VirtualMemoryRead when it extracts memory regions for analysis. This provided a detailed log of Defender's interactions with other processes and files.

Achieving this level of control over MsMpEng required bypassing several robust EDR protections:

1. Protected Process Light (PPL) anti-malware: MDE runs as a PPL process, similar to LSASS, restricting access to only other PPL-protected processes or those signed by Microsoft's ELAM driver. The solution involved using vulnerable drivers (e.g., kernel driver utility) to elevate the monitor's process to PPL anti-malware status.
2. Kernel Callbacks: Even with PPL status, kernel callbacks could strip access rights when attempting to open the EDR process. A tool like EDR Sandbox was used to disable these callbacks, granting the monitor full access to MsMpEng via OpenProcess.
3. Code Integrity: Custom DLLs are not signed by Microsoft, preventing their direct loading into MsMpEng. This was circumvented using reflective injection, a technique commonly employed by malware authors, where a DLL is loaded directly from memory without touching disk or requiring a signed binary.

Once the hooker DLL was loaded, a function hook was injected into the ntdll.dll of MsMpEng. Specifically, a jump instruction was placed at the beginning of functions like NtOpenProcess, redirecting execution flow to a custom logging function. After recording the details (e.g., target process ID, access rights), the original function's code was executed, allowing Defender to proceed with its intended action without interruption. This "EDR hook back" provided real-time visibility into Defender's system-level interactions.

Building upon this, the full EDR Introspection Framework integrated the MsMpEng hooker DLL with a sophisticated ETW parser and filter. This parser attached to five critical ETW traces:

• Thread Intelligence: Provides context on thread activities.
• Kernel Process: Logs process creation, termination, and modifications.
• Kernel File: Records file system operations.
• Kernel Audit API Calls: Captures important system calls and API usage, crucial for observing Defender's OpenProcess calls.
• ETW anti-malware engine: This was particularly insightful, as Defender outputs extensive debug information through this stream, including details about static scans, emulation, and internal states.

A key refinement of the research demonstrated that the primary EDR phases – static scanning, emulation, and memory scanning – could be observed without the complex kernel trickery. By solely parsing administrator-level ETW traces, specifically looking for stream scan and scan events in the ETW anti-malware engine for static and emulation activity, and OpenProcess calls from Defender in Kernel Audit API Calls for memory scans, a significant portion of EDR introspection could be achieved with less privilege.

The talk presented analyses of several attack scenarios:

• Process Injection (MSF Venom calc): An attack.exe (shellcode to launch calc) was observed. Defender first opened attack.exe with VirtualMemoryRead and QueryInformation to read its Process Environment Block (PEB) and ntdll.dll loader entries. An MMOC cache lookup confirmed ntdll.dll as known good. When attack.exe later spawned whoami which then spawned calculator, Defender's behavior triggers activated. It opened whoami for vmqueryinformation and performed approximately 200 memory hits, searching for shellcode. Despite this, anti-emulation in the attack and memory obfuscation techniques prevented detection, even after Defender performed an NtReadFile to collect IOCs.
• PowerShell Execution: Any PowerShell process was heavily monitored by Defender. Commands like LS or whoami triggered scans. Invoke-Mimikatz, an obviously malicious command, was blocked, and Defender initiated an emulation run on the PowerShell process itself.
• Mimikatz.exe: Storing Mimikatz.exe on disk immediately triggered detection, leading to heuristic scans, emulation, and file deletion. Execution was outright prevented.

These detailed logs, comprising thousands of events per attack, allowed for precise mapping of Defender's actions against specific malware behaviors, providing an unprecedented level of visibility into its internal logic.

Demo / Proof of Concept

▶ Watch: Simplified EDR monitoring with administrator rights (15:00)

Levi Cailleret demonstrated the EDR Introspection framework through several live examples, showcasing both the kernel-level hooking capability and the administrator-level ETW monitoring.

The first demonstration illustrated the kernel-level EDR hook back. Using EDR Sandbox, kernel callbacks were disabled, and the monitor was run with PPL anti-malware privileges. The custom DLL was then reflectively injected into MsMpEng. The live terminal on the right side of the screen immediately began logging every NtOpenProcess call made by Defender. When the speaker opened the calculator application, the log rapidly populated with Defender's intense scanning activity across the system, confirming its deep and continuous monitoring. This visually underscored the sheer volume of operations Defender performs in the background.

The second demonstration focused on the simplified, administrator-level ETW monitoring for a process injection attack. A malicious attack.exe (MSF Venom shellcode for calc) was placed in a monitored directory. The attack.exe script, designed with anti-emulation and memory obfuscation, was executed. The monitor displayed that a heuristic scan for the file was initially skipped, indicating prior analysis. Upon execution, the monitor showed Defender opening the attack process with VirtualMemoryReadWrite and submitting scan reports. Crucially, even as the attack escalated – starting threads, decrypting the payload, injecting it into whoami, changing memory protection to read-execute (a known suspicious behavior) – the monitor consistently reported no detection from Defender. The attack successfully launched calculator via whoami, and the final Defender log confirmed, "Nothing was detected. Nothing was prevented." This demo powerfully highlighted how a well-crafted, obfuscated payload could evade even the "best" MDE P2 license.

The third demonstration explored Defender's behavior against more overt malicious actions, again using administrator-level ETW monitoring. First, a new PowerShell process was started. The monitor immediately showed Defender conducting extensive scans against the PowerShell process and its subsequent commands (LS, whoami). When Invoke-Mimikatz was attempted, Defender swiftly blocked it, and the monitor revealed that Defender initiated an emulation run on the PowerShell process, successfully detecting "PS attack tool.A." Next, the raw Mimikatz.exe binary was tested. As soon as the file was stored on disk, Defender detected it, triggering a heuristic scan, emulation, and ultimately deleting the file. Attempting to execute it was immediately prevented. This segment showcased Defender's effectiveness against known, signature-based threats, contrasting sharply with its struggles against the custom process injection demonstrated earlier.

These demos provided compelling, real-time evidence of Defender's internal decision-making, illustrating both its strengths against known threats and its blind spots against sophisticated, evasive malware.

Defensive Implications

▶ Watch: Conclusion: EDR bypass strategies and deconditioning (20:00)

The findings from EDR Introspection carry profound implications for cybersecurity defenders. The most critical takeaway is that EDRs, even leading solutions like Microsoft Defender for Endpoint, are not infallible black boxes. Their detection capabilities are not absolute and can be bypassed by sophisticated attackers employing specific evasion techniques. Defenders must understand that reliance solely on EDR alerts can create a false sense of security.

The research confirms that EDRs can be blind to malware that is:

• Fully unknown: Lacking known Indicators of Compromise (IOCs) or signatures.
• Obfuscated: Designed to confuse static analysis.
• Delayed: Using start delays to bypass initial emulation.
• Encrypted: Storing payloads in encrypted memory to evade memory scans.
• Deconditioned: Utilizing techniques to exhaust or overwhelm the EDR's scanning capabilities during runtime.

For blue teams, this means moving beyond a passive "set and forget" approach to EDR. It is crucial to:

1. Validate EDR Coverage: Use frameworks like EDR Introspection to actively test their EDR configurations against various attack techniques. Understanding why an attack is missed is as important as knowing what was detected.
2. Focus on Behavioral Anomalies: While specific behaviors like whoami spawning calculator are recognized by EDRs, the research showed these can be bypassed. Defenders should develop more robust behavioral analytics that correlate multiple, seemingly benign events to identify a malicious chain of actions.
3. Address Deconditioning: The demonstrated effectiveness of deconditioning techniques (e.g., allocating memory repeatedly to exhaust scans, or flooding ETW logs) highlights a significant vulnerability. Defenders should investigate if their EDRs have configurable limits or enhanced detection logic to identify and prevent such resource exhaustion attacks. Tuning EDRs to be more resilient to these tactics is essential.
4. Leverage Multiple Telemetry Sources: EDRs are powerful, but they are one layer of defense. Integrating EDR telemetry with other logs (e.g., SIEM, network logs, custom kernel tracing) provides a more holistic view and can help detect attacks that bypass the EDR's primary mechanisms.
5. Educate and Train: Security teams need to be aware of the limitations of their EDRs and continuously train on advanced threat emulation techniques. This knowledge empowers them to hunt for threats that might otherwise go unnoticed.

Ultimately, EDR Introspection provides defenders with the tools and insights to become more proactive, enabling them to harden their defenses, improve their threat hunting capabilities, and more accurately assess their true security posture against evolving threats.

Key Takeaways

• EDRs are Not Black Boxes Anymore: The EDR Introspection framework successfully demystifies the internal workings of Microsoft Defender for Endpoint, revealing its exact actions during malware analysis through detailed logging of system calls and ETW events.
• Multi-Stage EDR Detection Confirmed: Defender actively employs static checks, emulation, memory scanning, and runtime tracking, but the specific triggers and effectiveness of these mechanisms vary significantly based on the sophistication of the malware.
• Kernel-Level Insights vs. Admin-Level Accessibility: While initial deep insights required complex kernel-level bypasses and reflective injection to hook MsMpEng, the most crucial EDR phases (static analysis, emulation, memory scans) can be observed using administrator-level ETW traces, making introspection more accessible.
• Evasion Techniques are Effective: Advanced malware leveraging obfuscation, start delays, encrypted memory, and especially deconditioning (exhausting EDR resources) can successfully bypass even the highest-tier EDR licenses, as demonstrated by the undetected process injection attack.
• Deconditioning is a Proven Threat: Techniques like repeatedly allocating memory to exhaust EDR scans or flooding ETW logs (as seen in Super Mega and Bamboozle EDR) are confirmed methods to desensitize EDRs and allow malicious actions to proceed undetected.
• Actionable Intelligence for Both Sides: Red teams can leverage this knowledge to craft more effective and evasive payloads, while blue teams can use the framework to rigorously test their EDR configurations, understand their true security posture, and improve their threat hunting and incident response strategies.

About the Speaker(s)

Levi Cailleret is a security professional currently working at Raiffeisen Schweiz, where he is part of the C-Cert team. Prior to his current role, Levi gained valuable experience as an ex-red teamer and is also an active participant in Capture The Flag (CTF) competitions. The "EDR Introspection" project presented at Area41 was a significant part of his master's thesis completed at ETH, showcasing his deep technical expertise and commitment to advancing cybersecurity research.