Two Fat Men, One Filesystem

Two Fat Men, One Filesystem

Speakers: Signedness

Conference: 44CON

YouTube: https://www.youtube.com/watch?v=NuxCUMIH5M8

Overview

This talk, "Two Fat Men, One Filesystem," presented by Signedness, delves into a series of "unbelievable" and long-standing server-side vulnerabilities within the Network File System (NFS) protocol, specifically focusing on versions 2 and 3. The speaker highlights fundamental design flaws that enable remote attackers to gain full filesystem access and, in many cases, achieve remote code execution on vulnerable systems. The research primarily targets BSD-derived operating systems but also touches upon older Linux and Apple implementations, revealing bugs that the speaker contends should have been discovered and patched decades ago.

The presentation underscores the critical implications of these vulnerabilities, which stem from a combination of insecure protocol design, lax credential handling, and unexpected behavior in core NFS procedures. Beyond theoretical exploitation, the talk demonstrates practical attacks against common setups, including a remote root shell on FreeBSD and full access to various Network Attached Storage (NAS) devices. The findings serve as a stark reminder that even decades-old, seemingly well-understood protocols can harbor severe, easily exploitable flaws with widespread real-world impact.

Background

▶ Watch: Introduction, talk scope, and bug excitement (0:00)

NFS, a distributed filesystem protocol developed by Sun Microsystems in the 1980s, is widely recognized for its age and inherent insecurities. Its pervasive adoption across various operating systems, largely due to the permissive BSD license allowing code propagation, means that vulnerabilities often manifest across a broad spectrum of environments. The speaker notes observing the same bugs in "maybe 10-15 different operating systems" during their research. This particular investigation into NFS began serendipitously when the researchers, while pursuing a different project, encountered frequent remote kernel crashes on multiple BSD systems, including NetBSD, OpenBSD, and FreeBSD. The intrigue of remote kernel bugs quickly shifted their focus to NFS.

The talk primarily concerns NFSv2 and NFSv3, the most prevalent versions found on internal networks, and notably supported even on Windows. The underlying mechanism for NFS is Remote Procedure Call (RPC), another Sun technology from the 80s. RPC allows clients to execute functions on remote servers by serializing arguments into XDR (External Data Representation) format. Authentication in these NFS versions is notoriously weak, often relying on UNIX authentication, where clients merely declare their User ID (UID) and Group ID (GID), and the server trusts these claims implicitly. The speaker humorously equates this to calling "vegan meatballs meat bolts," highlighting the lack of genuine verification. While BSD systems offer options like map_all or map_root to map client-claimed root UIDs to a low-privileged user (e.g., nobody), and Linux has similar "root squashing," these are often misconfigured or bypassed. A critical, often overlooked vulnerability mentioned is the server's failure to enforce requests originating from a privileged port, allowing any user to spoof root credentials.

The process of mounting an NFS share involves two key RPC services: RPCbind (a portmapper that tracks RPC programs and their endpoints) and MountD (responsible for handing out share information and enforcing IP-based access restrictions). Once a client obtains a list of available shares from MountD and requests a mount, MountD returns a file handle – an opaque, unique identifier for filesystem content. Crucially, while MountD enforces IP restrictions, the NFSd (NFS daemon), which handles subsequent file operations using these handles, often "does not really care where you're coming from." This architectural separation means that if an attacker can acquire a valid file handle, they can use it regardless of their network location or whether they would have passed MountD's initial checks. File handles themselves typically consist of a filesystem ID and a file ID. Their generation varies by OS and filesystem, with some implementations (e.g., HP-UX, or BSD's use of i-node numbers) being significantly easier to guess, a weakness that previous research has also explored.

Key Findings

▶ Watch: Fuzzing tools and historical NFS vulnerabilities (4:00)

The central and most "unbelievable" discovery presented is a directory traversal vulnerability inherent in the NFSv2 and NFSv3 protocols, specifically exploiting the LOOKUP procedure with the parent directory identifier ...

1. Directory Traversal via LOOKUP with ..:

• The LOOKUP procedure takes a file handle for a directory and a filename within that directory, returning a new file handle for the specified file.
• On many vulnerable systems, when LOOKUP is invoked with .. as the filename, it returns a valid file handle for the parent directory, even if that parent directory is outside the originally exported share.
• This allows an attacker to repeatedly call LOOKUP with .. to traverse upwards from an exported subdirectory all the way to the root directory of the server's underlying filesystem.
• This vulnerability affects all major BSD variants (FreeBSD, OpenBSD, NetBSD, DragonFly BSD), as well as older versions of Apple's macOS (e.g., 10.6) and some legacy Linux kernels (e.g., Ubuntu 16.04). Modern Linux and recent macOS versions typically prevent this traversal, either by returning the same file handle or zeroing out the relevant parts.
• Once the root file handle is obtained, an attacker can then perform any NFS operation (read, write, create, delete) on any file or directory on the entire system, limited only by the permissions associated with the (often spoofed) client credentials.

1. Weak and Long-Lived File Handles:

• While some BSD implementations (like FreeBSD's FFS) include a 32-bit random number in the file ID component of the handle, others rely solely on i-node numbers or easily guessable identifiers (e.g., HP-UX using major/minor device IDs).
• The 32-bit randomness, while present, is insufficient for robust security. The speaker demonstrated a brute-forcing tool that can uncover such handles, estimating it takes roughly 75 hours to iterate through all possibilities for a single handle.
• Crucially, NFS file handles often persist as long as the NFS server lives, meaning an attacker has ample time to brute-force and reuse them.

1. Bypassing MountD IP Restrictions:

• MountD, the service responsible for initial share access, typically enforces IP-based restrictions. However, NFS often operates over UDP.
• The researchers developed a tool capable of IP spoofing, allowing them to forge requests originating from an authorized IP address. This bypasses MountD's checks, enabling the attacker to obtain initial file handles for exported shares even if their true IP is unauthorized.
• Once a file handle is acquired, NFSd, which handles all subsequent operations, does not re-verify the client's source IP, rendering MountD's restrictions moot for ongoing access.

1. mknod for Kernel Memory Access:

• The NFS MKNOD procedure allows clients to create special device files. A particularly potent capability demonstrated is the creation of /dev/kmem or /dev/mem.
• Historically, /dev/kmem provided read/write access to kernel memory, acting as a powerful rootkit API. While modern OSes have severely limited its access, the ability to create such files remotely over NFS, especially if the server is configured to trust root-spoofing clients, still presents significant security risks.

1. Remote Kernel Crashes:

• Beyond logical vulnerabilities, extensive fuzzing of NFS implementations revealed multiple remote kernel bugs, leading to system crashes.
• The speaker specifically mentioned providing proof-of-concept code at their workshop to crash OpenBSD systems in three different ways: one client-side and two server-side. This highlights the general instability and fragility of many NFS codebases under adversarial input.

Technical Deep Dive

▶ Watch: Core topic: "unbelievable" NFSv2/v3 server bugs (6:00)

The core of the "unbelievable" vulnerability lies in the NFS LOOKUP procedure and its interaction with directory traversal. In NFSv2 and NFSv3, LOOKUP is designed to take a file handle representing a directory and a filename string. It then returns a new file handle for the specified file or directory within the context of the initial handle. The critical flaw emerges when the filename provided is .., the standard Unix notation for the parent directory.

On a properly secured NFS implementation (such as modern Linux kernels or Apple systems post-10.6), a LOOKUP call with .. on an exported file handle would either return the same file handle (preventing upward traversal) or, if it detects an attempt to go above the exported share, return an error or a modified handle that prevents further access (e.g., zeroing out the file handle part while returning the filesystem ID, as seen in Ubuntu).

However, on vulnerable BSD systems (OpenBSD 7.4/7.5, FreeBSD, NetBSD, DragonFly BSD), the LOOKUP procedure with .. genuinely returns a new, valid file handle for the actual parent directory in the underlying filesystem hierarchy. This means if an attacker mounts /NFS/data, they receive a file handle for /NFS/data. A subsequent LOOKUP with .. on this handle would yield a handle for /NFS, and another LOOKUP .. would provide a handle for / (the root directory). Once the root file handle is obtained, the attacker effectively has full access to the entire server's filesystem, limited only by the claimed client credentials.

File handles themselves are typically opaque, 32-byte data structures. They are composed of two main parts: a filesystem ID (FSID), which identifies the specific filesystem (e.g., FFS), and a file ID (FID), which uniquely identifies a file, directory, or other object within that filesystem (often derived from an i-node number). The security of these handles varies significantly:

• On BSD systems using FFS, the FSID is generated from the filesystem name and a sequential number, then hashed. The FID is based on the i-node number plus a 32-bit random number.
• Other implementations, like HP-UX, might use major and minor device IDs, which are much easier to guess.
• Some operating systems rely solely on i-node numbers for FIDs, making brute-forcing trivial.

Even with the 32-bit random component, the speaker demonstrated that it is not sufficiently robust. A custom file handle brute-forcing tool was developed, which takes a hexadecimal string template and allows specifying nibbles to brute-force with question marks (e.g., ????????12345678). This tool can run in flood mode, sending packets and listening for responses. The speaker estimated that iterating through a 32-bit unknown value takes approximately 75 hours. Given that NFS file handles are long-lived (often persisting until the NFS server is reinstalled), this duration is well within practical attack windows.

Another crucial technical aspect is IP spoofing. NFSv2 and NFSv3 often utilize UDP, a connectionless protocol. This characteristic, combined with MountD's IP-based access controls and NFSd's subsequent lack of IP verification, creates a significant window for attack. The custom tool developed by the researchers integrates IP spoofing capabilities, allowing an attacker to forge the source IP address of their NFS requests. This enables them to bypass MountD's initial network restrictions and obtain a valid file handle for an exported share. Once this handle is acquired, all subsequent operations—including the LOOKUP .. traversal—can be performed using the legitimate, but now compromised, file handle, regardless of the attacker's true IP.

The talk also touched upon the NFS procedures available through NFSd, which closely mirror standard Unix filesystem operations: ACCESS, READLINK, READ, WRITE, CREATE, MKDIR, SYMLINK, MKNOD, REMOVE, RENAME, and LOOKUP. The READDIRPLUS procedure, which provides extended information similar to an ls -l command (including file system ID, file ID, and the complete file handle), was highlighted. While Ubuntu's READDIRPLUS zeros out the file handle for the parent directory, OpenBSD's implementation happily provides it, further aiding the traversal attack.

Demo / Proof of Concept

▶ Watch: "NFSdoom.com" idea and focus on v2/v3 (8:00)

The talk featured several compelling demonstrations and real-world scenarios illustrating the practical exploitation of these NFS vulnerabilities.

The primary demonstration showcased a full remote root compromise of a FreeBSD system. The scenario involved a FreeBSD server exporting a share named /share/private to a specific host (e.g., 20.x.x.20). The attacker, originating from an unauthorized IP, initially attempted to mount the share but received a "permission denied" error from MountD. To bypass this, the attacker utilized their custom tool with IP spoofing enabled, forging the source IP address to match the authorized host. This successfully allowed them to obtain a valid file handle for /share/private.

With the initial file handle in hand, the attacker then proceeded to exploit the LOOKUP .. vulnerability. They repeatedly invoked the LOOKUP procedure with .. as the filename, traversing upwards through the filesystem hierarchy: from /share/private to /share, and finally to / (the root directory). Once the root file handle was acquired, the attacker demonstrated full control over the system. Specifically, they navigated to /var/at/jobs, a directory where the at daemon processes scheduled tasks based on filenames. They then computed a future timestamp for a file name and uploaded a simple shell script (bin/sh) into this directory, along with a bin/sh binary. By setting appropriate permissions and ownership (spoofing root credentials), the at daemon eventually executed the uploaded script, providing the attacker with a remote root shell on the FreeBSD box.

Beyond the FreeBSD example, the speaker presented findings from real-world Network Attached Storage (NAS) devices from unnamed manufacturers. These devices commonly export subdirectories (e.g., /volume1/backup, /usr/share/web/archive) rather than the root filesystem directly. However, the LOOKUP .. traversal attack proved equally effective. By mounting an exported subdirectory and repeatedly traversing upwards, the researchers gained full access to the NAS device's entire underlying filesystem. This exposed sensitive areas like /system folders, /homes directories (where authorized_keys could be modified for SSH access), and /web directories (where PHP webshells could be uploaded for remote code execution). The speaker explicitly stated that these attacks resulted in remote shells on the target NAS devices.

The researchers also introduced their custom explore command-line tool, designed to automate many of these attack steps. Given an NFS server, the explore command automatically resolves all available shares, attempts to traverse them upwards to find the root directory, and then looks for specific sensitive files. The demo showed it successfully dumping the contents of /etc/shadow (if root access is obtained), listing the home directory of the root user, and identifying the authorized_keys file for potential modification.

Finally, the talk briefly mentioned the capability to use the MKNOD procedure to create device files. While not fully demonstrated in the main talk, the speaker encouraged attendees to experiment with creating /dev/kmem or /dev/mem over NFS, highlighting its potential (though now limited) for gaining powerful kernel-level access. The fuzzing efforts, which uncovered remote kernel crashes in OpenBSD, were also presented as a proof of concept of the general fragility of NFS implementations.

Defensive Implications

▶ Watch: RPC fundamentals: Sun technology, XDR, "authentication" (10:00)

The vulnerabilities highlighted in this talk underscore the critical need for a robust defensive posture when deploying and managing NFS, particularly versions 2 and 3. Defenders must be aware of several key implications and implement specific mitigations:

1. Avoid Exporting Root-Level or Sensitive Directories: The most fundamental takeaway is to never export a directory that, if traversed upwards, could expose sensitive parts of the server's root filesystem. If NFS is necessary, export only dedicated, isolated filesystems or subdirectories that contain no sensitive parent directories. The speaker suggested that operating systems should issue warnings if an exports configuration allows .. traversal above the intended share.

1. Implement Strong Credential Mapping: Always use features like map_all or map_root on BSD systems, or root_squashing on Linux, to ensure that client requests claiming root (UID 0) are mapped to an unprivileged user (e.g., nobody) on the server. This prevents an attacker, even if they spoof root, from gaining actual root privileges on the NFS server.

1. Strict IP-Based Access Control: While MountD's IP restrictions can be bypassed via IP spoofing (especially over UDP), they still provide an important layer of defense. Configure NFS exports to be accessible only from a very limited set of trusted IP addresses or subnets. Deploy NFS only on internal, segmented networks, and never directly expose NFS services (ports 2049, 111, and others via RPCbind) to the internet. The Shodan statistic of over 150,000 internet-facing NFS servers is a significant concern.

1. Prioritize Patches and Updates: The identified vulnerabilities affect all major BSD operating systems and older Linux/Apple versions. It is imperative to keep NFS server software fully patched. While some vendors (like OpenBSD) might consider certain behaviors "expected," patches for remote kernel crashes and other logical flaws are still crucial. Consider upgrading to modern NFS implementations (e.g., NFSv4) if possible, as they often include stronger authentication mechanisms (like Kerberos) and better security features, though even NFSv4 has had its share of vulnerabilities.

1. Disable NFSv2/v3 if not Essential: If an environment does not strictly require NFSv2 or NFSv3, these older, more vulnerable protocol versions should be disabled in favor of NFSv4 or alternative file sharing protocols that offer better security.

1. Monitor NFS Traffic: Implement network monitoring to detect unusual NFS activity, such as repeated LOOKUP requests with .. or attempts to create device files (MKNOD). While this may not prevent the initial exploit, it can aid in early detection and incident response.

1. Consider Filesystem Permissions and ACLs: Even with the LOOKUP .. traversal, an attacker's ability to read or write files will ultimately be limited by the effective permissions on the target files. Implement strict filesystem permissions and Access Control Lists (ACLs) on all shared data and critical system files.

Key Takeaways

• NFSv2 and NFSv3 harbor fundamental, decades-old design flaws that allow for widespread and severe remote exploitation.
• The LOOKUP .. directory traversal vulnerability enables attackers to ascend the filesystem hierarchy from an exported subdirectory to the server's root, gaining full filesystem access.
• IP spoofing can effectively bypass MountD's network-based access restrictions, making initial file handle acquisition possible even from unauthorized networks.
• BSD-derived operating systems (FreeBSD, OpenBSD, NetBSD, DragonFly BSD) are particularly susceptible to the LOOKUP .. traversal, as are older Linux and Apple versions.
• Successful exploitation can lead to remote code execution (e.g., via at jobs, modifying authorized_keys, or uploading web shells on NAS devices) and access to sensitive system files like /etc/shadow.
• NFS servers exposed to the internet (over 150,000 identified by Shodan) represent a significant, easily exploitable attack surface.

About the Speaker(s)

The talk was delivered by Signedness, a security researcher with a deep passion for BSD operating systems, which they claim to use "pretty much everywhere." Despite humbly admitting to not being "very good at presenting," their content-rich research and insights clearly speak for themselves. Signedness has a history of impactful security work, including a notable talk about "owning VMS at Defcon" approximately 15 years prior, which earned them the moniker "HP Enemy Number One" within the VMS community. While their usual research focus is on memory corruption vulnerabilities, this presentation ventured into the realm of logical flaws within established protocols, demonstrating a versatile and impactful research capability. The speaker also candidly shared their experience recovering from a stroke, which caused a delay in the presentation and affected their ability to develop some of the tools initially envisioned.