Unmasking the unseen: a deep dive into modern Linux rootkits and their detection

Unmasking the unseen: a deep dive into modern Linux rootkits and their detection

Speakers: Ruben Groenewoud (Security Research Engineer, Elastic), Remco Sprooten (Security Research Engineer, Elastic)

Conference: Virus Bulletin

YouTube: https://www.youtube.com/watch?v=XmyQf0h9mmg

Overview

In "Unmasking the unseen: a deep dive into modern Linux rootkits and their detection," Ruben Groenewoud and Remco Sprooten, both Security Research Engineers at Elastic, provide a comprehensive exploration of the evolving landscape of Linux rootkits. This talk delves into the historical progression of rootkit technologies, from traditional userland and kernel modules to cutting-edge techniques leveraging eBPF and io_uring. The speakers meticulously dissect the various hooking mechanisms employed by these stealthy adversaries, offering critical insights into their operational methodologies and the challenges they pose for detection.

The presentation is particularly significant as it not only categorizes and analyzes a vast dataset of existing Linux rootkits but also uncovers critical vulnerabilities in current detection strategies. Groenewoud and Sprooten highlight how simple modifications can render many established antivirus and EDR solutions ineffective, underscoring the urgent need for more sophisticated, behavior-based detection approaches. A standout contribution is the revelation of a novel syscall hooking technique, "Flip Switch," which demonstrates how rootkits can circumvent modern kernel protections, emphasizing the ongoing arms race between attackers and defenders in the Linux security domain.

This talk is crucial for security professionals, incident responders, and system administrators operating in Linux environments. It provides a foundational understanding of rootkit mechanics, illuminates the limitations of traditional security tools, and offers actionable strategies for enhancing detection and defense. By showcasing both the breadth of current threats and the potential for future evasion techniques, Groenewoud and Sprooten equip the audience with the knowledge necessary to anticipate and counteract the most advanced forms of Linux stealth malware.

Background

▶ Watch: Introduction and Linux Rootkit Evolution Overview (0:00)

Linux rootkits are malicious software designed to achieve stealth and persistence by hiding their own presence, as well as malicious processes, files, and network activities. Unlike their Windows counterparts, which often rely on kernel drivers or bootkits, Linux rootkits employ a diverse array of techniques, ranging from userland manipulations to deep kernel-level compromises. Understanding their evolution is key to comprehending the current threat landscape.

Historically, Linux rootkits began in userland, primarily leveraging shared object injection. These rootkits manipulate the dynamic linker, often via LD_PRELOAD or dynamic linker configuration files, to ensure their malicious shared objects are loaded before any legitimate libraries. By hooking standard C library functions (e.g., libc functions), they can intercept and modify calls related to file system access, process listing, and network connections to hide their activities. While still effective for single-user or system-wide compromise depending on injection, they are easily bypassed by statically compiled binaries, making them relatively simple to detect.

As userland rootkits became easier to spot, attackers shifted to kernel space, leading to the proliferation of Loadable Kernel Modules (LKMs). LKMs operate with full privileges within the kernel, allowing them to hook syscalls, manipulate kernel structures, and intercept virtually any kernel function. This grants them unparalleled control, enabling comprehensive stealth, privilege escalation, backdoor implementation, and C2 communication. However, LKMs face limitations: they can be blocked by kernel module signing enforcement, are highly kernel-dependent (requiring recompilation or specific targeting for different kernel versions), and are prone to crashing the system during kernel updates, making them noisy and less stable.

More recently, researchers have explored advanced interfaces like eBPF (extended Berkeley Packet Filter) and io_uring for rootkit development. eBPF leverages an in-kernel virtual machine to run small, verified programs in kernel context without direct kernel modification. Attackers can attach eBPF programs to Kprobes, tracepoints, or Linux Security Modules (LSMs) to intercept and manipulate kernel behavior. While powerful, eBPF rootkits are limited by kernel version support (Linux 3.18+, mature 4.9+), are visible if no legitimate eBPF tooling is present, and are constrained by a strict eBPF verifier with limited helper functions. The io_uring asynchronous I/O system, introduced in Linux 5.1+, offers a novel stealth mechanism. By performing operations like read and write without traditional syscalls, io_uring rootkits can evade EDRs that primarily hook conventional syscalls. However, they lack the ability to mimic execution syscalls and require the liburing package, which is not available by default, making their setup potentially noisy.

Elastic's research involved analyzing 34 Linux rootkit families (comprising multiple samples), including both open-source projects and closed-source tools used by attackers. The statistical breakdown revealed that approximately 60% of identified rootkits are kernel-based, emphasizing the attacker's preference for deep system control. Userland rootkits still constitute a significant 35%, while eBPF-based rootkits account for a smaller 6%, primarily originating from research rather than widespread in-the-wild use. The speakers anticipate an increase in eBPF rootkits in the future.

The analysis of hooking techniques employed across these rootkits provided further insights:

• IDT (Interrupt Descriptor Table) hooking: Obsolete in modern kernels (post-2.x) as the int8 interrupt call is no longer used for syscall entry.
• Syscall table hooking: Traditionally involved finding the syscall table in memory and replacing handlers. This technique became largely obsolete in kernel 6.9+ due to a fundamental change in how syscalls are dispatched (from an array to a switch statement).
• Inline hooking: Modifying the first few bytes of a function to jump to attacker-controlled code. Possible but faces defenses against executable code modification.
• VFS (Virtual File System) hooking: Modifying read/write operations for specific files, offering limited control.
• Kprobes, Tracepoints, Perf events, eBPF: Debugging and tracing tools abused by rootkits. Their legitimate purpose and inherent limitations often make rootkits using them more visible.

Analysis of frequently hooked syscalls and kernel functions revealed consistent patterns: getdents64 and kill were top syscalls for hiding and communication, respectively. lookup_name was a top hooked function, used by almost all rootkits to locate other kernel functions. Userland rootkits predominantly hooked functions related to file reading and opening, aligning with their goal of hiding files.

A particularly surprising finding from their detection statistics was the fragility of traditional, hash-based detection. When tested against the top 10 most used rootkits from VirusTotal, simply stripping debugging symbols dramatically reduced detection rates. For example, one rootkit went from ~30 detections to just one. Even more alarmingly, adding a single null byte to the end of a file, which has no impact on functionality, also significantly decreased detection, indicating that many engines rely purely on file hashes—a technique from the 1990s that is inadequate for modern threats.

Key Findings

▶ Watch: Shocking detection statistics: hash-based failures (12:00)

The Elastic research team uncovered several critical findings that challenge conventional wisdom in Linux security and highlight the evolving threat landscape:

1. Correlation Between Hooking Extent and Maliciousness: A direct and logical correlation exists between the number of functions or syscalls a rootkit hooks and its overall malicious functionality. Proof-of-concept rootkits often hook only one or two functions, whereas in-the-wild rootkits designed for comprehensive system takeover typically hook 20, 30, or even 50 functions. This emphasizes that real-world threats are far more complex and pervasive than academic demonstrations.

1. The Failure of Hash-Based Detection: One of the most striking revelations was the severe inadequacy of hash-based detection mechanisms. The team demonstrated that for known, in-the-wild rootkits, simply stripping debugging symbols from the binary or adding a single null byte to the end of the file could drastically reduce or even eliminate detections across multiple antivirus engines. This indicates a widespread reliance on static signatures that are easily circumvented, leaving systems vulnerable to trivial modifications of known malware.

1. Pumakit Case Study: Extensive Capabilities and Evasion: The analysis of Pumakit, a sophisticated rootkit found in the wild, showcased advanced capabilities beyond mere hiding. Pumakit included its own loader disguised as a legitimate cron binary, performed connection concealment, disarmed SELinux, tracked program launches, hid files and directories, and incorporated extensive credential theft functionality (targeting password prompts, private keys). Its communication channel between userland and kernel space utilized the rmdir command combined with specific arguments (e.g., rmdir zarya/c.0 to retrieve configuration), demonstrating creative abuse of legitimate syscalls. Pumakit also employed Ftrace and syscall table overrides for hooking, notably searching for the E8 byte (the start of a CALL instruction) to skip fentry calls injected by tracing tools, thereby finding the true beginning of kernel functions for more reliable hooking.

1. The "Flip Switch" Technique: Resurrecting Syscall Hooking in Modern Kernels: Despite syscall table hooking being considered obsolete in Linux kernels 6.9 and above due to a change from an array-based dispatch to a switch statement, Remco Sprooten developed a novel technique called "Flip Switch." This technique identifies the E8 byte (CALL instruction) within the compiled switch statement of the syscall dispatch function. By locating and replacing the offset following this E8 byte, "Flip Switch" allows an attacker to redirect execution to their own malicious function, effectively re-enabling syscall table hooking on modern kernels (demonstrated on kernel 6.14). This finding critically undermines a significant kernel security improvement and highlights the continuous innovation of attackers.

1. Shocking Detection Rates for Novel Rootkits: The "Flip Switch" proof-of-concept rootkit, when uploaded to VirusTotal, achieved a dismal one detection, which was self-generated by the speaker. This stark result underscores the profound gap in current security product capabilities against novel, even relatively simple, kernel-level evasion techniques. It serves as a powerful call to action for the security community to invest more heavily in behavioral detection and advanced kernel monitoring.

Technical Deep Dive

▶ Watch: Introducing 'Flip Switch': a new syscall hooking technique (19:30)

Linux rootkits leverage a variety of sophisticated techniques to achieve stealth and persistence, operating at different layers of the operating system.

Userland Rootkits (LD_PRELOAD)

These rootkits exploit the dynamic linker's behavior. By setting the LD_PRELOAD environment variable or modifying dynamic linker configuration files (e.g., /etc/ld.so.preload), an attacker can force the system to load a malicious shared object (.so file) before any other libraries. This malicious .so then intercepts calls to standard C library functions (e.g., open, read, getdents, stat, kill). For instance, to hide a file, the rootkit's hooked open function might check if the requested file path matches a hidden pattern. If it does, it returns an error or simply pretends the file doesn't exist. This technique is effective because many userland applications rely on these standard library functions. However, statically compiled binaries bypass the dynamic linker, rendering LD_PRELOAD rootkits ineffective against them.

Loadable Kernel Modules (LKMs)

LKMs operate directly within kernel space, granting them unparalleled control. They can hook syscalls by modifying the syscall table (or its modern equivalent). In kernels prior to 6.9, the syscall table was an array of function pointers. An LKM could locate this table in kernel memory, disable write protection, and replace legitimate syscall function pointers (e.g., sys_read, sys_getdents64) with pointers to its own malicious functions. These malicious functions would perform their hiding logic (e.g., filtering out specific processes or files from getdents64 results) and then optionally call the original syscall to maintain legitimate system behavior. LKMs can also hook other kernel functions not directly exposed as syscalls, manipulate kernel structures (e.g., task_struct to hide processes), and even directly modify kernel memory. Their limitations include kernel version dependency and the risk of system instability.

eBPF Rootkits

eBPF offers a new paradigm for kernel-level functionality. It allows users to load small, sandboxed programs into the kernel, which are then attached to various hook points like Kprobes (kernel function entry/exit), tracepoints (predefined kernel events), or LSMs. The eBPF verifier ensures program safety and prevents arbitrary code execution in the kernel. An eBPF rootkit can, for example, attach to sys_open Kprobe, inspect the file path, and if it matches a hidden file, return an error before the original sys_open is called. It can also manipulate network traffic by attaching to network-related hook points. While powerful, the verifier restricts what eBPF programs can do, and their presence can be detected by monitoring eBPF program loading and attachment. The bpf() syscall is central to loading and managing eBPF programs and maps.

io_uring Rootkits

Introduced in Linux 5.1+, io_uring provides an asynchronous I/O interface that can bypass traditional syscalls for certain operations. Instead of making direct read() or write() syscalls, applications submit I/O requests to an io_uring submission queue, which are then processed by the kernel. This provides a stealth advantage because many EDRs focus on hooking traditional syscalls. An io_uring rootkit can intercept or manipulate these requests within the kernel, for example, to hide file contents or network communications. However, io_uring currently lacks mechanisms for executing programs, limiting its scope compared to full LKM rootkits. The io_uring_setup, io_uring_enter, and io_uring_register syscalls are used to manage the io_uring subsystem.

Hooking Techniques in Detail

• Syscall Table Hooking (Pre-6.9 Kernels): As mentioned, this involved directly modifying the array of function pointers.
• Ftrace Hooking: Pumakit notably used Ftrace, a dynamic tracing mechanism within the kernel. Ftrace allows functions to be "traced" by inserting a jump instruction at their entry point. Pumakit would leverage this by finding the true start of a function, specifically by looking for the E8 byte (the opcode for a CALL instruction in x86-64) that would typically follow an fentry trampoline (an injected call by tracing tools). By targeting the instruction after this fentry call, Pumakit ensured it hooked the actual function logic without interfering with existing tracing infrastructure.
• "Flip Switch" (Post-6.9 Kernels): The most revolutionary technical finding addresses the obsolescence of traditional syscall table hooking in kernels 6.9+. In these newer kernels, the syscall dispatch mechanism changed from a direct array lookup to a switch statement. When a switch statement is compiled in C, it often translates into a series of CALL instructions (opcode E8) followed by offsets to the target functions within the compiled binary. The "Flip Switch" technique involves:

1. Locating the sys_call_table (which is now effectively the switch statement's implementation).
2. Identifying the specific E8 (CALL) instruction within the switch statement that corresponds to the target syscall number.
3. Modifying the offset immediately following this E8 byte. This offset dictates where the CALL instruction jumps. By replacing it with an offset pointing to the attacker's malicious function, the rootkit effectively re-establishes syscall hooking for modern kernels. This bypasses the protective measures introduced in kernel 6.9 and above, demonstrating a significant vulnerability. Remco's proof of concept successfully implemented this on kernel 6.14.

Demo / Proof of Concept

▶ Watch: Live demo: elevating privileges with 'Flip Switch' rootkit (20:30)

The most impactful demonstration in the talk centered around the novel "Flip Switch" syscall hooking technique. Remco Sprooten presented a live proof of concept to validate its efficacy on modern Linux kernels, specifically kernel 6.14, where traditional syscall table hooking was previously considered dead.

The demonstration proceeded as follows:

1. A debug version of the "Flip Switch" rootkit, developed by Remco, was compiled.
2. This rootkit was then inserted into a running Linux system with a 6.14 kernel.
3. The speaker showed that his current user was not root but possessed a variety of groups.
4. To demonstrate the rootkit's capability, the kill command was used with a specific signal number (in this case, kill -64). This kill syscall was one of the functions hooked by the "Flip Switch" rootkit. The rootkit intercepted this call, recognized the specific signal as its trigger for privilege escalation, and granted the user root privileges.
5. Immediately after executing the kill -64 command, the user's shell session was elevated to root, showcasing the successful compromise and privilege escalation facilitated by the "Flip Switch" technique.

Following the live demonstration, the compiled "Flip Switch" rootkit was uploaded to VirusTotal to assess its detection rate against a wide array of security products. The results were stark: the rootkit was detected by only one engine, which, as Remco humorously admitted, was his own signature specifically crafted for this rootkit. This nearly zero detection rate for a fully functional, novel kernel-level rootkit on a modern kernel underscored the critical gap in existing security solutions. The code for the "Flip Switch" technique was also made publicly available on GitHub during the talk, enabling other researchers and defenders to study and develop countermeasures.

Defensive Implications

▶ Watch: Minimizing attack surface and kernel hardening recommendations (30:00)

The detailed analysis of Linux rootkits and their evasion techniques provides crucial insights for defenders. Given the demonstrated failure of static, hash-based detection, a strong emphasis on behavioral detection and proactive system hardening is paramount.

Behavioral Detection Strategies:

• Shared Object Userland Rootkits:
• Monitor for the creation of shared object files (.so) in unusual or unexpected directories.
• Detect modifications or creations in dynamic linker configuration files (e.g., /etc/ld.so.preload, /etc/ld.so.conf.d/*) or the LD_PRELOAD environment variable.
• Loadable Kernel Module (LKM) Rootkits:
• Loading Phase: Monitor for init_module or finit_module syscalls, which are essential for loading any kernel module, regardless of the loader used.
• Kernel Tainting: Alert on kernel logs (e.g., dmesg, /var/log/kern.log) indicating "loading out of tree module" or "tainting the kernel" from unsigned modules. This often signifies a suspicious LKM.
• Unusual Kill Signals: Monitor for kill syscalls using signals between 32 and 64, as these are generally unused by benign Linux functionalities and are commonly repurposed by rootkits for C2 or actions.
• Kernel Stack Faults: Track kernel stack faults, which can indicate an unstable or malicious LKM causing system crashes.
• Persistence Mechanisms: LKMs do not persist across reboots by default. Look for file creations or modifications in etc/modules, modprobe configuration files, or modules-load.d directories, as these are common methods for LKM persistence.
• UDEV Rules: Monitor for suspicious udev rules (e.g., in /etc/udev/rules.d/) that trigger the execution of malicious loaders upon device addition (e.g., as seen with Reptile rootkit abusing /dev/random).
• eBPF Rootkits:
• BPF Syscalls: Monitor for the bpf() syscall, specifically for BPF_MAP_CREATE, BPF_PROG_LOAD, and BPF_PROG_ATTACH commands. Baseline legitimate eBPF usage in your environment to detect anomalies.
• BPF_PROBE_WRITE_USER Helper: Alert on kernel log messages indicating the use of bpf_probe_write_user, as this helper allows eBPF programs to write to user space and is considered a potentially dangerous capability.
• io_uring Rootkits:
• io_uring Syscalls: Monitor for io_uring_setup, io_uring_enter, and io_uring_register syscalls. io_uring is not widely used by common applications, so its presence might be highly suspicious in many environments.
• General Persistence and Evasion Techniques:
• Systemd Units/Timers, Cron Jobs, Init/RC Scripts: Monitor for new or modified entries in these common persistence locations.
• Fake Kernel Worker Threads: Detect processes parented by kworker or kthread that are not legitimate kernel processes and are executing shell commands, as seen with Pumakit.
• Log File Tampering: Alert on attempts to delete, clear, or overwrite dmesg, kernel logs, syslog, boot logs, or shell history. Tools like journalctl can also be used for log clearing.
• Time Stomping: Monitor for file modifications that alter timestamps (e.g., using touch) to hide the true creation/modification time of malicious files.

Minimizing Attack Surface and Hardening:

• Keep Kernel Up-to-Date: While not a silver bullet, updated kernels often include patches for known vulnerabilities and may introduce new security features.
• Kernel Hardening: Implement kernel hardening measures such as disabling unneeded kernel modules, enabling kernel address space layout randomization (KASLR), and restricting direct kernel memory access.
• Secure Boot and Signature Enforcement: Enforce Secure Boot and kernel module signature enforcement. This is highly effective against unsigned LKMs, preventing them from being loaded.
• Kernel Protection Tools: Utilize tools like Linux Kernel Runtime Guard (LKRG), which monitors kernel integrity and detects runtime attacks, including many rootkit activities.
• Behavioral Monitoring & EDR: Deploy robust EDR (Endpoint Detection and Response) solutions that focus on syscall monitoring, process ancestry analysis, file integrity monitoring, and network behavior analysis rather than just static signatures.
• Access Control Frameworks: Implement and enforce SELinux or AppArmor. While a root user can disable these, they provide strong preventative controls and can restrict even root-level actions if configured correctly.
• Periodical Active Scanning: Conduct regular, active scans of the system for anomalies, hidden files, and unauthorized processes.
• Minimize Initial Access Vectors: The most critical defense is to prevent initial compromise. Secure SSH configurations, strong passwords, multi-factor authentication, and reducing internet-facing services significantly lower the risk of rootkit installation.

By combining these strategies, defenders can build a multi-layered defense that is more resilient to the sophisticated and evolving threats posed by modern Linux rootkits.

Key Takeaways

• Traditional hash-based detection is critically flawed: Simple modifications like stripping debug symbols or adding a null byte can bypass many AV/EDR solutions, highlighting the need for more advanced behavioral analytics.
• Linux rootkits are evolving rapidly: From userland and LKMs to eBPF and io_uring, attackers continuously adopt new kernel interfaces to achieve stealth and persistence, requiring defenders to stay abreast of cutting-edge techniques.
• Syscall hooking is not dead in modern kernels: The "Flip Switch" technique demonstrates that syscall table hooking can be re-enabled on Linux kernels 6.9+ by exploiting the compiled structure of the switch statement, undermining recent kernel security improvements.
• In-the-wild rootkits are highly sophisticated: Cases like Pumakit showcase extensive capabilities beyond mere hiding, including credential theft, SELinux disarming, and creative communication channels, emphasizing the comprehensive nature of real-world threats.
• Behavioral detection is paramount: Given the ease of bypassing static signatures and the continuous innovation in evasion, monitoring for anomalous process execution, syscall sequences, kernel module loading, and file system changes is crucial for effective rootkit detection.
• Proactive hardening and attack surface reduction are essential: Keeping kernels updated, enforcing secure boot/module signing, using kernel protection tools, and minimizing initial access vectors are fundamental to preventing rootkit compromise.

About the Speaker(s)

Ruben Groenewoud is a Security Research Engineer at Elastic, where he contributes to the threat research and detection engineering team. His work primarily involves malware analysis, dynamic analysis, and general research into Linux malware.

Remco Sprooten is also a Security Research Engineer at Elastic, working within the Elastic Security Labs team. He specializes in Linux malware reverse engineering and is responsible for maintaining Elastic's security blog, often sharing his team's research findings.