Threat From The Inside: Investigate eBPF Malware

Threat From The Inside: Investigate eBPF Malware

Speakers: Geri Revay, Malware Researcher and Threat Intelligence Analyst, Fortinet

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=yXUXCM9lBA4

Overview

In this compelling talk at FIRSTCON, Geri Revay, a seasoned Malware Researcher and Threat Intelligence Analyst at Fortinet, delved into the escalating threat landscape surrounding eBPF (extended Berkeley Packet Filter) technology. Drawing from his unique background transitioning from offensive security (pentesting) to defensive roles, Revay offered a comprehensive exploration of eBPF's capabilities, its emerging use in malware, and crucial techniques for investigation and reverse engineering. The presentation highlighted the dual nature of eBPF: a powerful, legitimate kernel technology that, due to its deep access and control, presents significant opportunities for malicious actors to establish stealthy rootkits, evade detection, and perform sophisticated attacks.

The core of Revay's discussion centered on demystifying eBPF for security professionals, providing both an understanding of its underlying mechanisms and practical guidance on how to combat its misuse. He meticulously detailed how eBPF programs can manipulate kernel operations to "lie" to the operating system, "steal" sensitive data, and "cheat" to gain privileges. Crucially, the talk provided a roadmap for incident responders and malware analysts facing potential eBPF-based threats, emphasizing memory forensics and specialized tooling to uncover hidden malicious activity. As eBPF adoption grows, even extending to Windows, Revay’s insights underscore the urgent need for defenders to grasp this technology to effectively secure modern computing environments.

Background

▶ Watch: Introduction to eBPF technology and its capabilities (0:00)

The journey to eBPF began with BPF, the Berkeley Packet Filter, now often referred to as classic BPF (CBPF). Originally designed in the early 1990s as a kernel extension for efficient network packet manipulation and monitoring, CBPF was instrumental in tools like tcpdump. However, its functionality was limited; it was stateless, allowing only simple logic and basic packet filtering. Recognizing the immense potential of running custom code within the kernel without the overhead and risk of traditional kernel modules, developers embarked on a significant extension.

This led to the creation of eBPF, the extended Berkeley Packet Filter. While the "packet filter" in its name no longer accurately reflects its vast scope, eBPF has evolved into a general-purpose, in-kernel virtual machine that can execute sandboxed programs. Its primary goal is to enable dynamic, programmable kernel functionality without risking a kernel panic—the Linux equivalent of a Blue Screen of Death. By providing a safe, sandboxed environment, eBPF allows users to attach custom programs to various hook points within the kernel. These hook points are extensive, encompassing system calls, function entry/exit points (Kprobes for kernel functions, Uprobes for user-space functions), tracepoints, and a wide array of networking events.

eBPF programs are written in a C-like language, compiled into eBPF bytecode, and then loaded into the kernel. Before execution, a strict verifier component ensures the program is safe, won't crash the kernel, and will terminate (e.g., no infinite loops). Once verified, the bytecode is typically compiled into native machine code by a Just-In-Time (JIT) compiler for optimal performance. This architecture makes eBPF incredibly powerful for legitimate use cases across various industries. Large companies like Google and Netflix leverage it for network observability, load balancing, and performance monitoring. In the security domain, open-source projects like Falco (an EDR for containers), Cilium (cloud-native network observability and DOS protection), and Tracy (threat detection and forensics) utilize eBPF to monitor system activity, detect anomalies, and enforce policies with unprecedented visibility and efficiency. Revay emphasized that eBPF itself is a robust and beneficial technology; however, its inherent power and deep kernel access also make it a potent tool for malicious actors, introducing unique security challenges.

Key Findings

▶ Watch: Demo: eBPF for privilege escalation (sudoad) (10:00)

Geri Revay’s talk unveiled several critical findings regarding the landscape of eBPF malware and defensive strategies:

1. Emerging Threat Vector: While classic BPF has seen use in malware like BPFdoor and Symbiote (primarily for network-level hiding), true eBPF-based malware is an emerging threat. Revay noted that the complexity of writing eBPF programs has historically been a barrier, but this is expected to change as the technology matures and tools (potentially including AI) lower the entry threshold for threat actors.

1. Unparalleled Offensive Capabilities: eBPF grants attackers extraordinary abilities to operate stealthily within the kernel. Revay categorized these capabilities into "lying, stealing, and cheating":

• Lying: eBPF rootkits can hide processes, files, and network connections by intercepting kernel calls and manipulating the data returned to user-space applications (e.g., hiding a malicious entry in crontab or altering ls output). This also extends to anti-forensics and network evasion for C2 traffic.
• Stealing: eBPF can exfiltrate sensitive data, such as the contents of encrypted files, by hooking into file access operations when the data is decrypted in memory.
• Cheating: Privilege escalation is achievable by manipulating kernel responses, such as injecting a user into the sudoers list presented to the sudo command, as demonstrated with the sudoadd tool.

1. The "Everything is Lying to You" Challenge: A central finding for defenders is that standard host-based forensic tools are inherently unreliable when eBPF malware is active. Since eBPF operates within the kernel and can intercept system calls, it can present falsified information to any user-space utility attempting to query the system state. This necessitates a shift towards memory forensics and specialized eBPF-aware tools.

1. Crucial Investigation Tools and Techniques: Revay identified the bpf tool as the defender's best friend. This utility allows listing active eBPF programs, inspecting their associated maps, load times, and originating users. By correlating this information with system logs (syslog) and process lists, defenders can often trace malicious eBPF programs back to their originating binaries, even if those binaries attempt to hide themselves. Memory dumps, analyzed with tools like Volatility (especially with the BPF-B3 plugin), offer an unadulterated view of the kernel's true state.

1. Straightforward Reverse Engineering: Despite its kernel-level nature, Revay demonstrated that reverse engineering eBPF bytecode is "not rocket science." The eBPF instruction set is well-documented and relatively simple. The primary challenge lies in extracting the eBPF object file, which is often embedded within a larger ELF executable. Once extracted, tools like Ghidra can effectively disassemble and decompile the eBPF bytecode, making analysis feasible.

1. Windows Adoption and Future Complexity: The talk highlighted Microsoft's initiative to implement eBPF on Windows, aiming to replace problematic kernel drivers with a safer alternative. While beneficial for portability, Revay expressed concern that introducing this Linux-centric kernel technology to Windows will significantly increase system complexity, potentially leading to new vulnerabilities for security researchers to discover. This expansion suggests a broader attack surface for eBPF threats in the future.

Technical Deep Dive

▶ Watch: How to investigate and detect eBPF malware (18:50)

eBPF's power stems from its ability to insert custom logic directly into the kernel's execution path without modifying the kernel source code. This is achieved through hook points such as system calls, Kprobes (kernel function entry/exit), Uprobes (user-space function entry/exit), tracepoints, and various network events. When a hooked event occurs, the eBPF program is triggered, executes its logic, and then returns control to the original kernel path.

Revay categorized the offensive applications of eBPF into three main areas:

1. Lying (Rootkits, Anti-Forensics, Network Evasion):

• File Hiding: An eBPF program can hook readdir or getdents system calls. When a program like ls requests directory contents, the eBPF program can filter out specific filenames, making them invisible to the user.
• Content Manipulation: By hooking read or write system calls, eBPF can alter the content of files on-the-fly. For instance, the textreplace example from bad-bpf demonstrated replacing "eviljob" with "apt update" in a crontab file when it's read, effectively concealing malicious scheduled tasks while the original content remains on disk and the task continues to execute.
• Network Evasion: eBPF can intercept network packets at a very early stage (before they reach the kernel's network stack). This allows for manipulating incoming C2 traffic by removing identifiable headers or adding content to outgoing packets after they have passed through security tooling, making C2 communication stealthier.

1. Stealing (Data Exfiltration):

• eBPF can monitor file access operations. If a program opens an encrypted file, the data is decrypted in memory before being processed. An eBPF program can hook the read operation at this point, capture the decrypted content, and then exfiltrate it through a covert channel (e.g., hidden network packets).

1. Cheating (Privilege Escalation):

• sudo Manipulation: The sudoadd example illustrated how an eBPF program can hook the system call responsible for reading the sudoers file. When sudo queries this file, the eBPF program injects the unprivileged user's entry into the returned data, granting them temporary sudo capabilities without modifying the actual file on disk.
• Binary Redirection: eBPF can hook execve (or related exec family calls) to redirect the execution of a legitimate binary (e.g., ssh) to a malicious, manipulated version, unbeknownst to the user.

eBPF's relevance is particularly acute in cloud environments and containers. Since containers often share the same host kernel, an eBPF program loaded on the host can potentially access or manipulate data across multiple containers, or even be used for container escapes, making it a powerful tool for lateral movement and privilege escalation in these setups.

The lifecycle of an eBPF program involves several stages:

1. Source Code: Developers write eBPF programs using C, Rust, or other languages, often leveraging libraries like BCC or libbpf.
2. eBPF Bytecode Compilation: The source code is compiled into an ELF object file containing eBPF bytecode.
3. Loading and Verification: When loaded into the kernel, the bytecode is first subjected to the eBPF verifier. The verifier performs static analysis to ensure the program is safe to run: it checks for infinite loops, out-of-bounds memory access, and ensures the program will terminate. Importantly, the verifier focuses on safety (kernel stability) rather than security (malicious intent).
4. JIT Compilation: If the verifier passes, the eBPF bytecode is typically translated by a Just-In-Time (JIT) compiler into native machine code. This optimized code is then executed directly by the CPU within the kernel context. If JIT is disabled, an interpreter executes the bytecode.

The eBPF instruction set is 64-bit with 10 general-purpose registers (R0-R9) and a frame pointer (R10) for stack access. R0 is used for return values, R1-R5 for function arguments, and R6-R9 are callee-saved registers. This architecture, though specific, is well-documented, making its bytecode relatively straightforward to understand.

For investigating a running system, the bpf tool is invaluable. To dump the eBPF bytecode of an active program (e.g., program ID 73):

sudo bpf tool prog dump id 73 xlated

This command outputs a readable pseudo-code representation of the eBPF bytecode, including references to eBPF maps (key-value stores used for state management and inter-program communication). To see the JIT-compiled native assembly:

sudo bpf tool prog dump id 73 jitted

When analyzing a suspicious file, the challenge is to extract the embedded eBPF ELF object. This involves:

1. Initial Check: Use file <binary> to see if it's a standard executable. If no eBPF mention, it's likely embedded.
2. Locate Embedded ELF: Use strings <binary> | grep "ELF" to find the signature of the embedded ELF file. This reveals its offset within the main executable.
3. Extract: Use dd to extract the embedded ELF object based on the identified offset and size:

dd if=<malicious_binary> of=<extracted_ebpf.o> bs=1 skip=<offset> count=<size>

1. Verify: file <extracted_ebpf.o> should now confirm it's an "eBPF object file."
2. Analyze: Load the extracted eBPF object file into a disassembler/decompiler like Ghidra, which has built-in support for eBPF, displaying functions and decompiled code for analysis.

Demo / Proof of Concept

▶ Watch: Reverse engineering eBPF programs: bytecode and JIT (26:40)

Geri Revay showcased two compelling demonstrations from the bad-bpf open-source project, illustrating eBPF's potent offensive capabilities: privilege escalation and stealthy file content manipulation.

Demo 1: Privilege Escalation with sudoadd

This demonstration highlighted how an unprivileged user could gain temporary root access by manipulating the sudo mechanism using an eBPF program.

Scenario: An unprivileged user named "bobby tables" attempts to access the /etc/shadow file, which requires root privileges, using sudo cat /etc/shadow. As expected, the command fails, indicating that "bobby tables" is not in the sudoers file.

Mechanism:

1. On a separate terminal, the attacker runs the sudoadd -u bobby tables program. This program loads an eBPF program into the kernel.
2. The loaded eBPF program hooks the system call that reads the /etc/sudoers file (or relevant configuration).
3. When the sudo command (executed by "bobby tables") queries the kernel to check user privileges, the eBPF program intercepts the kernel's response.
4. Before the sudo program receives the content, the eBPF program dynamically injects an entry for "bobby tables" into the sudoers file content, making it appear as if "bobby tables" has sudo privileges.

Result: After the sudoadd program is running, "bobby tables" can successfully execute sudo cat /etc/shadow, demonstrating temporary privilege escalation. Once the sudoadd program is terminated, the eBPF program is unloaded, and "bobby tables" loses sudo capabilities again, proving the transient nature of the manipulation. This illustrates how eBPF can "cheat" the operating system into granting privileges without altering files on disk.

Demo 2: Stealthy File Manipulation with textreplace

This demonstration focused on using eBPF to hide the execution of a malicious scheduled task by altering the output of file reads on the fly.

Scenario: A crontab entry exists with a malicious job: * /usr/bin/eviljob. The eviljob binary logs its execution to /var/log/evil.log, confirming it runs every minute. An investigator reading the crontab file would see this malicious entry.

Mechanism:

1. The attacker executes the textreplace program with specific parameters: textreplace -f /etc/crontab -s "eviljob" -r "apt update". This loads an eBPF program.
2. The eBPF program hooks the read system call for the /etc/crontab file.
3. Whenever /etc/crontab is read by any process (e.g., cat /etc/crontab), the eBPF program intercepts the file content.
4. It then searches for the string "eviljob" and replaces it with "apt update" before the content is returned to the requesting process.

Result: After textreplace is active, reading /etc/crontab now shows apt update, making the malicious job appear as a routine system update. However, the eviljob continues to execute in the background, as verified by new entries appearing in /var/log/evil.log. This proves that the eBPF program is only manipulating the view* of the file, effectively "lying" to the investigator, while the underlying malicious activity persists. Once textreplace is terminated, the crontab file reverts to showing "eviljob" again.

These demonstrations vividly illustrate eBPF's capacity for creating sophisticated, stealthy rootkits and anti-forensics mechanisms that bypass traditional detection methods by operating at the kernel level.

Defensive Implications

▶ Watch: Q&A: Future of eBPF malware and detection challenges (36:00)

Defending against eBPF malware presents unique challenges due to its kernel-level capabilities and potential for obfuscation. Geri Revay emphasized a critical paradigm shift: "Everything is lying to you." Standard user-space tools cannot be trusted to reflect the true state of the system. Therefore, defenders must adopt specialized strategies:

1. Memory Forensics is Paramount: The most reliable way to uncover eBPF-based deception is to capture a full memory dump of the compromised system. Analyzing this dump with tools like Volatility (specifically with the BPF-B3 plugin) allows investigators to bypass kernel-level manipulation and see the actual running eBPF programs, their relationships, and associated data structures. While powerful, Revay noted that setting up and using BPF-B3 effectively requires prior preparation and playbooks.

1. Leverage the bpf tool: This is the primary utility for interacting with and inspecting eBPF programs on a live Linux system.

• Listing Programs: sudo bpf tool prog show will list all currently loaded eBPF programs, providing their IDs, types, and attached hook points.
• Correlating Information: Crucially, bpf tool also displays the load time and the user ID that loaded each program. This information can be used to pivot to other system logs:
• syslog: Search syslog for events around the eBPF program's load time, looking for the originating process or activity.
• Process Monitoring: Correlate the load time and user ID with process creation events to identify the executable that loaded the eBPF program. This is vital for finding the malicious binary on disk for further analysis, as attackers might delete the executable after loading the eBPF program.
• Inspecting Maps: eBPF programs often use eBPF maps to store data and maintain state. sudo bpf tool map show lists these maps. Inspecting their contents (sudo bpf tool map dump id <map_id>) can reveal critical clues, such as the "eviljob" and "apt update" strings from the textreplace demo, directly indicating manipulation.

1. Proactive Monitoring with eBPF Against eBPF:

• Tracy: Tools like Tracy can be configured to monitor for the BPF_ATTACH syscall, which is used when an eBPF program is loaded. This allows for real-time detection of new eBPF program deployments.
• Baselining: In controlled environments, such as production servers or containers, establishing a baseline of expected eBPF programs is essential. Any deviation from this baseline (e.g., new, unknown eBPF programs being loaded) should trigger an alert for immediate investigation. Revay noted that this approach is more effective for proactive monitoring than reactive incident response. In particular, containers, which typically have a predictable set of eBPF programs, are ideal candidates for such baselining.

1. Awareness of Windows eBPF: With Microsoft's adoption of eBPF for Windows, defenders must prepare for a new attack surface. While currently limited in functionality compared to Linux, its growth will introduce similar complexities and potential for abuse, necessitating cross-platform eBPF detection and analysis capabilities. The increased complexity of the Windows kernel due to eBPF integration could also lead to new vulnerabilities.

In summary, defending against eBPF malware requires moving beyond traditional host forensics. It demands proficiency with specialized eBPF tooling, robust memory analysis capabilities, and a proactive approach to monitoring and baselining kernel-level activity.

Key Takeaways

• eBPF is a double-edged sword: It's a powerful, legitimate kernel technology enabling advanced functionality, but its deep access and control make it an extremely potent tool for sophisticated malware and rootkits.
• Trust nothing on a compromised system: Traditional host-based forensic tools can be easily deceived by eBPF malware, which can manipulate system call outputs to "lie" about system state. Memory forensics is crucial for uncovering the truth.
• The bpf tool is your best friend: This utility is indispensable for investigating running eBPF programs, providing details like IDs, types, hook points, load times, and associated users, which can be correlated with system logs to trace malicious activity.
• eBPF programs are reversible: Despite operating at the kernel level, eBPF bytecode is a well-documented and relatively simple instruction set, making reverse engineering straightforward once the eBPF object file is extracted from its containing binary.
• Proactive monitoring and baselining are critical: Tools like Tracy can detect new eBPF program loads, and establishing a baseline of legitimate eBPF activity in your environment is essential for identifying anomalies and emerging threats.
• Expect more eBPF malware: The power of eBPF, coupled with potentially decreasing entry barriers (e.g., through AI-assisted development), suggests that eBPF-based threats will become more prevalent across both Linux and, eventually, Windows environments.

About the Speaker(s)

Geri Revay is a Malware Researcher and Threat Intelligence Analyst at Fortinet. With a background rooted in offensive security, specifically pentesting, Revay has transitioned his expertise to the defensive side, focusing on understanding and combating malicious software. His work involves in-depth malware research and contributing to threat intelligence efforts. Based in Hungary, he currently lives in Germany, bringing a global perspective to his security insights.