Ghost on the network: Bypassing EDR

Ghost on the network: Bypassing EDR

Speakers: Jess (Piki Security)

Conference: CHCon

YouTube: https://www.youtube.com/watch?v=qMxSi7VU6AQ

Overview

In this insightful CHCon talk, Jess from Piki Security delved into the often-misunderstood capabilities and limitations of Endpoint Detection and Response (EDR) systems. Titled "Ghost on the network: Bypassing EDR," the presentation systematically demonstrated how these advanced security solutions, despite vendor claims, can be circumvented by determined attackers employing "living off the land" techniques and strategic exploitation. Using a compelling narrative of a "ghost cat" infiltrating a vampire's "haunted castle" with two layers of defenses, Jess illustrated a practical, multi-stage attack chain designed to evade EDR detection on both Linux and Windows environments.

The talk is critically important for both offensive and defensive security professionals. For attackers, it provides a playbook for stealthy operations, emphasizing the effectiveness of native tools and careful payload crafting. For defenders, it serves as a stark reminder that EDRs are not a silver bullet. Instead, they are but one layer in a comprehensive defense strategy, underscoring the enduring importance of system hardening, robust monitoring, and a multi-layered security architecture.

Jess's demonstration highlighted that while EDRs do make an attacker's job harder—blocking approximately half of the attempted attacks in this exercise—they are far from impenetrable. The talk challenges the perception of EDR as an ultimate defense, advocating for a more realistic understanding of its role and encouraging organizations to adopt a holistic security posture that accounts for potential bypasses.

Background

▶ Watch: Introduction and problem setup: Vampire castle (0:00)

The premise of the talk revolved around a simulated breach scenario targeting two distinct network segments, each protected by its own EDR system. Jess established a lab environment featuring two Metasploitable virtual machines—intentionally vulnerable systems designed for penetration testing practice—one running Linux and the other Windows. The "vampire's castle" analogy neatly encapsulated the challenge: an outer wall (Linux server) and an inner security system (Windows server), both accessible via public wireless networks and employing identical, yet vulnerable, architectures.

At the core of the discussion was the definition and perceived efficacy of EDR. As defined by Wikipedia, EDR is a cybersecurity technology that continuously monitors endpoints for threats, collects event data, alerts administrators to suspicious behavior, and performs automatic actions to mitigate risks. Vendors often market EDRs as advanced platforms capable of stopping all attacks at the endpoint. However, Jess's talk aimed to dissect this claim, demonstrating that while EDRs are designed to detect and block suspicious activities—such as ransomware-like file system traversal—their effectiveness can be significantly curtailed by an attacker who understands their detection mechanisms.

The attack methodology was structured around the MITRE ATT&CK framework, mirroring real-world adversary tactics. The "ghost cat's" mission involved a series of phases: initial reconnaissance, gaining access, escalating privileges, maintaining persistence, moving laterally, and exfiltrating data, all while meticulously bypassing the EDR. A key strategic principle adopted for EDR evasion was "living off the land"—prioritizing the use of legitimate, native system tools and processes to minimize "odd behavior" that EDRs are designed to flag. This approach contrasted with the common assumption that all advanced attacks require custom or highly obfuscated malware, proving that even well-known vulnerabilities and default system utilities can be weaponized effectively.

Key Findings

▶ Watch: Privilege escalation: Finding plain-text passwords (8:00)

Jess's presentation yielded several critical findings regarding EDR bypass techniques and the inherent limitations of these systems:

• Initial Exploits Often Undetected: A surprising discovery was that many of the initial exploits and payloads delivered via Metasploit—a tool synonymous with offensive security—did not trip the EDR. This suggests that EDRs may not always have comprehensive signatures or behavioral analytics for common, albeit malicious, network exploitation frameworks, especially when the initial compromise occurs at the network layer.
• Benign Tools for Reconnaissance and Privilege Escalation: The use of standard Linux commands like ps (process enumeration) and cat /etc/group (user group identification) for reconnaissance and privilege escalation went entirely undetected. This reinforced the "living off the land" principle, demonstrating that EDRs often permit legitimate system utilities to run without scrutiny, even when used in an attacker's context.
• Plaintext Credentials Remain a Major Weakness: The discovery of plaintext usernames and passwords within a simple PHP web application and its associated MySQL database was a significant finding. This highlights that fundamental security hygiene failures (e.g., lack of password hashing) can render even advanced EDR defenses irrelevant, as an attacker can gain legitimate access credentials.
• Password Reuse Enables Stealthy Root Access: The ability to reuse a password found in a web application to SSH into the Linux server and then use sudo su to achieve root access without tripping the EDR was a major bypass. This underscored that EDRs struggle to differentiate between legitimate administrator actions and an attacker masquerading as one, especially when using valid credentials.
• Installing Offensive Tools on Compromised Hosts: Perhaps the most astonishing finding was that installing Nmap (a port scanner) and even Metasploit itself directly onto the already compromised Linux server did not trigger EDR alerts. This suggests a significant blind spot, as the presence of such tools on an endpoint should arguably be a high-fidelity indicator of compromise.
• Exploiting Known Vulnerabilities from Compromised Hosts: The successful exploitation of EternalBlue (an old NSA exploit) from the compromised Linux server against the Windows target, again without EDR detection, further illustrated the EDR's limitations. This indicates that even well-known, high-impact vulnerabilities might not be detected when the attack originates from an already trusted (albeit compromised) internal host.
• Partial EDR Effectiveness: While many attacks succeeded, Jess noted that approximately 50% of the techniques attempted were stopped by the EDR. This confirms that EDRs do provide a significant defensive barrier, making the attacker's job "a lot harder," even if not impossible.
• Evasion via Payload Modification and Shell Type: When facing a second, likely signature-based EDR, Jess successfully bypassed it by making minor tweaks to the payload (adding a few extra characters). Additionally, using a bind shell (which opens a port on the target) instead of a reverse shell (which calls back to the attacker) proved effective where reverse shells were blocked, suggesting EDRs might have stricter outbound network traffic monitoring.
• SSH Socks Proxy Detection: Interestingly, attempting to use an SSH socks proxy to tunnel traffic through the compromised Linux server did trip the EDR. This indicates that some EDRs are capable of detecting unusual network tunneling activities, even if they miss the installation of the tools used to create them.

Technical Deep Dive

▶ Watch: Lateral movement: Scanning Windows from Linux host (12:00)

The technical execution of the EDR bypass strategy was a meticulously crafted multi-stage process, leveraging both common offensive security tools and "living off the land" techniques.

The initial phase involved reconnaissance against the target Linux server from the attacker's machine. Jess utilized Nmap, the standard port scanner, with the -A flag for aggressive service fingerprinting. This initial network scan, a fundamental step in any penetration test, did not trip the EDR, as these systems primarily focus on host-based process and behavioral monitoring rather than network-level scanning. Nmap revealed an old, vulnerable version of IRC running, which became the initial point of entry.

For initial access to the Linux server, Jess employed Metasploit, often referred to as the "Swiss Army knife" for exploiting network services. Leveraging its extensive database of exploits, a module targeting the vulnerable IRC daemon was selected. Metasploit offers various payload types, including reverse shells (where the compromised process calls back to the attacker), bind shells (where the target opens a port for the attacker to connect to), and Meterpreter shells (an advanced, feature-rich shell for post-exploitation). Crucially, most of the initial Metasploit exploits and payloads used here, surprisingly, did not trigger the EDR.

Once a shell was established on the Linux machine, the focus shifted to privilege escalation. Adhering to the "living off the land" philosophy, Jess used benign Linux commands to enumerate the system. The ps command was used to list running processes, identifying potential avenues for exploitation on the intentionally vulnerable system. Further reconnaissance involved inspecting /etc/group to identify user groups, specifically the sudo group, which contained users like "Leia," "Luke," and "Han." The goal was to find credentials for one of these privileged users.

A search for plaintext passwords in common locations like /var (often used by databases and web servers) initially yielded no immediate results. However, investigating running web applications led to the discovery of a "payroll app" – a single PHP file. Examination of this PHP file's source code revealed a critical misconfiguration: a username and password for the MySQL database stored in plain text at the top of the file. Using the mysql command from the compromised shell, Jess enumerated the database, exported the users table, and confirmed that the application was storing user passwords in plaintext. Leia's payroll app password was thus compromised.

The critical step of password reuse then allowed for a significant leap in access. Assuming Leia might reuse her password, Jess attempted to SSH directly into the Linux system using Leia's credentials. This attempt was successful, providing an interactive shell with greater flexibility than the initial Metasploit shell. Since Leia was a member of the sudo group, the command sudo su with Leia's password granted a full root shell on the Linux machine, all without triggering any EDR alerts.

To establish persistence, Jess modified the system to ensure continued access. The attacker's public key was added to Leia's authorized_keys file, enabling passwordless SSH access in the future. Additionally, a "cheeky" modification was made to the sudo configuration to allow Leia to perform privileged actions without entering her password, further solidifying control and reducing the chance of accidental detection.

With root access on the Linux server, the next objective was lateral movement to the Windows target. Jess tested the EDR's limits by installing Nmap directly on the compromised Linux server. This operation completed without detection, allowing Nmap to be used from the internal network to scan the Windows server using the -A flag for comprehensive fingerprinting. During this phase, an attempt to use an SSH socks proxy to tunnel traffic through the Linux server did trip the EDR, indicating that while individual tools might pass unnoticed, certain advanced tunneling techniques can be flagged.

The Nmap scan revealed a web server running on the Windows machine with WebDAV enabled, configured for "old school web hosting" that allowed file uploads. This became the vector for gaining access to Windows. Jess used MSFVenom (another tool from the Metasploit suite) to generate a PHP reverse shell (specifically a Meterpreter payload). This malicious PHP code was then uploaded to the WebDAV share.

In a further test of EDR efficacy, Jess installed Metasploit itself on the compromised Linux server. Astonishingly, the EDR did not trip, despite Metasploit's clear offensive purpose. The Windows machine was also found to be outdated and vulnerable to EternalBlue, an infamous NSA exploit. Exploiting EternalBlue from the compromised Linux machine, again, did not trigger the EDR. While some Java exploits did cause EDR alerts, the general trend was that attacks originating from an already compromised, internal host were less likely to be detected. Ultimately, a Metasploit module was used to upload the Meterpreter shell to the Windows host, achieving code execution and allowing for data exfiltration (access keys to the "outer walls").

Finally, the talk demonstrated that the entire attack chain was successfully replicated against a second EDR system protecting the "inner security system." Minor adjustments were needed: for a signature-based EDR, the payload was slightly tweaked (adding a few extra characters) to bypass detection. Additionally, a bind shell was used instead of a reverse shell, suggesting that the second EDR might have been more aggressive in monitoring outbound connections. These minor modifications proved sufficient to compromise the second network, leading to access to the "treasure room."

Demo / Proof of Concept

▶ Watch: Attacker takeaways: Live off the land, test in lab (16:30)

The entire presentation served as a comprehensive demonstration and proof of concept for bypassing EDR. Jess outlined a practical, step-by-step methodology within a controlled lab environment. This setup utilized two Metasploitable virtual machines, intentionally designed with vulnerabilities, to simulate a real-world corporate network with distinct Linux and Windows servers, each protected by a separate EDR solution.

The demonstration involved the live execution of various tools and techniques, including:

• Nmap for initial reconnaissance and internal network scanning.
• Metasploit for exploiting vulnerable services (IRC, EternalBlue) and delivering payloads.
• MSFVenom for crafting malicious PHP reverse shells.
• Standard Linux commands (ps, cat, sudo) for enumeration, privilege escalation, and persistence.
• Manual inspection of web application code and database contents to uncover credentials.

The core of the demo was to show that, despite the presence of EDR, these tools and methods could be chained together to achieve full system compromise and lateral movement. The talk explicitly detailed which actions tripped the EDR and which did not, providing concrete evidence of the bypasses. While not a live, real-time hacking demonstration during the conference, the detailed recounting of the lab exercise provided a robust proof of concept for the presented EDR evasion techniques.

Defensive Implications

▶ Watch: Defender takeaways: Layered defenses, configure EDR (18:30)

Jess’s talk, while focused on offensive techniques, offered crucial insights for defenders. The primary message is clear: don't ditch the EDR. While it can be bypassed, EDR still blocked roughly half of the attempted attacks, making the attacker's job significantly harder. It's an important layer, but not the only one.

The most critical defensive implication is the need to layer defenses, akin to the Swiss cheese model used in aviation safety. Each security control has "holes," and EDR is just one slice of cheese. To build a resilient defense, organizations must implement a comprehensive security architecture that includes:

• System Hardening and Updates: Many of the attack vectors, such as the vulnerable IRC daemon and the EternalBlue exploit, relied on outdated and unpatched systems. Regular patching, configuring services securely, and disabling unnecessary features are fundamental.
• Multi-Factor Authentication (MFA): The successful use of a stolen password for SSH access and sudo su could have been prevented or severely hampered by MFA, which adds another layer of verification beyond just a password.
• Intrusion Detection Systems (IDS): Both network-based and host-based IDSs can complement EDR by detecting suspicious network traffic or specific host activities that EDR might miss or allow. For example, an NIDS might flag the initial Nmap scan or unusual outbound connections, while a HIDS could detect the installation of Metasploit.
• Robust Logging and Monitoring: Even if an EDR doesn't block an attack, comprehensive logging across endpoints, network devices, and applications, coupled with proactive monitoring, can help detect the attack chain in progress. The discovery of plaintext passwords, for instance, should ideally be flagged by a code review or configuration management system.
• Appropriate EDR Configuration: Jess emphasized that EDRs have numerous settings that balance assertiveness with user impact. Defenders should ensure their EDRs are "cranked as far up as possible" without hindering legitimate business operations. This often requires careful tuning and understanding of the environment.
• Regular EDR Testing: To identify gaps and ensure optimal configuration, organizations should actively test their EDR solutions. Tools like Atomic Red Team can be used to simulate a battery of attacker techniques, revealing which are stopped and which bypass the EDR. This allows defenders to understand their true security posture and implement additional controls to plug identified gaps.

In essence, defenders must move beyond a reliance on a single security product and embrace a defense-in-depth strategy where multiple controls work in concert to detect and mitigate threats at various stages of the attack kill chain.

Key Takeaways

• EDRs are not foolproof: Despite vendor claims, EDR systems can be bypassed, particularly by determined attackers employing "living off the land" techniques and exploiting common misconfigurations.
• "Living off the land" is highly effective: Utilizing native system tools and legitimate processes (e.g., ps, cat, sudo, Nmap installed post-compromise) is a stealthy method to evade behavioral EDR detection.
• Fundamental security hygiene remains critical: Misconfigurations like plaintext passwords in application code or databases, and the use of outdated, vulnerable software (e.g., old IRC, EternalBlue), provide easy entry points that even advanced EDRs struggle to prevent once legitimate credentials or system access is obtained.
• Layered defenses are essential: EDR is just one component of a strong security posture. Organizations must adopt a "Swiss cheese model" approach, combining EDR with system hardening, MFA, IDS, comprehensive logging, and proactive monitoring to create multiple barriers against attackers.
• Payload modification can bypass signature-based EDRs: Simple tweaks to malicious payloads (e.g., adding a few characters) can be sufficient to evade signature-based detection, highlighting the need for EDRs to rely more on behavioral analysis.
• Regular testing and tuning are crucial for EDRs: Defenders should actively test their EDRs using tools like Atomic Red Team and ensure they are configured to maximum assertiveness without impacting user experience, to identify and address detection gaps.

About the Speaker(s)

Jess, a co-founder of Piki Security, brings 13 years of professional experience in penetration testing to the cybersecurity landscape. Having spoken at numerous conferences over the years, Jess is well-versed in both the offensive and defensive aspects of security. Beyond professional endeavors, Jess enjoys gaming, specifically mentioning a current deep dive into Factorio, and exploring the outdoors through drives around Wellington and bushwalks.