NVMe: New Vulnerabilities Made Easy

NVMe: New Vulnerabilities Made Easy

Speakers: Tal Lossos, Vulnerability Researcher, CyberArk

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=FvsvY-iI2uA

Overview

In this insightful presentation from HITB Phuket, Tal Lossos, a Vulnerability Researcher at CyberArk, demystifies the process of finding critical vulnerabilities within the Linux kernel, specifically highlighting a significant flaw discovered in the NVMe driver. Titled "NVMe: New Vulnerabilities Made Easy," Lossos's talk outlines a practical, open-source vulnerability research methodology that emphasizes static code analysis as a powerful, yet often overlooked, tool for uncovering "low-hanging fruit" bugs in even the most scrutinized codebases. The core of his presentation revolves around demonstrating how straightforward techniques can yield high-impact discoveries, culminating in a remote Denial of Service (DoS) vulnerability and an authorization bypass in the NVMe over TCP (NVMe/TCP) implementation.

Lossos challenges the perception that finding impactful kernel vulnerabilities requires highly complex reverse engineering or advanced fuzzing techniques. Instead, he advocates for a disciplined approach combining open-source code review with readily available static analysis tools. His work not only exposed critical issues in the NVMe driver but also showcased similar vulnerabilities in other prominent Linux kernel modules, including NVIDIA's open-sourced drivers and the NTFS file system implementation. This talk serves as a compelling guide for security researchers, illustrating how a focused methodology can efficiently identify exploitable weaknesses in fundamental system components.

The implications of Lossos's findings are substantial, impacting a wide array of systems from cloud infrastructure (Amazon, Azure) to on-premise servers and local Linux machines, all of which leverage NVMe technology. The identified vulnerabilities, particularly the remote DoS and authentication bypass, underscore the critical need for continuous security auditing of core kernel components, even those recently introduced or perceived as mature. By sharing his methodology and findings, Lossos empowers defenders and researchers to adopt more accessible and effective strategies for bolstering the security posture of modern operating systems.

Background

▶ Watch: Introduction and talk agenda (0:00)

Lossos begins by outlining his pragmatic approach to open-source vulnerability research, stressing the inherent advantages of having immediate access to source code. This eliminates the initial need for reverse engineering, allowing researchers to directly inspect code for potential flaws. He highlights that even a superficial understanding of the codebase can reveal "low-hanging fruit" bugs. Furthermore, the collaborative nature of open-source projects means that unclear code snippets can often be clarified through online searches or AI assistants.

Static code analysis emerges as a cornerstone of Lossos's methodology. He champions the use of custom or existing tools to automatically scan for common vulnerability patterns, noting that writing such tools for open-source projects is significantly simpler than developing scripts for closed-source binaries. Debugging is also streamlined, as researchers can recompile the kernel with custom print statements to trace execution paths and confirm potential bugs.

However, open-source research isn't without its challenges. Lossos points out the lack of bug bounties for Linux kernel vulnerabilities, making it less financially rewarding for some researchers. He also shares a personal observation: the sheer volume of code in open-source projects can be overwhelming, leading to time wasted on understanding irrelevant sections. To counter this, he emphasizes the importance of focused research.

Despite the availability of source code, Lossos argues for the continued relevance of reverse engineering compiled binaries. He illustrates this with a personal anecdote about a kernel driver vulnerability he found. A seemingly clear memcpy overflow in the source code, where user-controlled start_column and stop_column variables dictated copy size, was rendered unexploitable in the compiled binary. This was due to the Fortify Source mitigation, which inserted an if statement to check buffer boundaries, effectively preventing exploitation beyond a denial of service. The only way to detect this compiler-introduced change was by examining the compiled binary in a tool like IDA, reinforcing the adage that "the code that the programmer writes is only what the programmer intended the machine to do, but if you want to actually know what the machine will run, the only way you can do it is opening up in a reverse engineering program."

To efficiently find memory corruption bugs, Lossos introduces dump fuzzing, a technique involving sending random bytes to a driver interface. In Linux, several methods exist for user-space programs to communicate with kernel drivers:

• File System Exposure: Drivers can expose files (e.g., character devices) that user-space can interact with using standard read() and write() system calls.
• ioctl API: A common interface where each functionality is distinguished by a unique command parameter, allowing for diverse operations.
• Sysfs/Procfs: Pseudo file systems where functionalities are exposed as dedicated files, making interaction straightforward via read() and write(), often used in shell scripts for convenience.
• mmap: For high-speed interaction, shared memory regions can be mapped between kernel and user space.
• Netlink Sockets: A socket-based API for kernel-user space communication, useful for those who prefer network-like interfaces over file-based ones.

Lossos demonstrates a simple dump fuzzer he wrote (around 20-50 lines of code) that targeted a driver implemented via sysfs, successfully crashing it by sending random data to writable files. This experience led him to explore whether static code analysis could have expedited the discovery of such bugs.

He defines static code analysis as tools that identify potential vulnerabilities by analyzing source code without execution. While custom tools can be written, many existing options are available: Clang Static Analyzer, CPPCheck for C/C++, Pylint/Mypy for Python, ESLint/JSLint for JavaScript, and CodeQL for more query-language-focused analysis. For his kernel research, Lossos aimed for simplicity, selecting CPPCheck due to its ease of use. He primarily sought buffer overflows/underflows, integer overflows/underflows, null pointer references, and uninitialized variables. CPPCheck allows outputting results to XML, which can then be parsed into an HTML report for easy review and filtering.

Key Findings

▶ Watch: Static code analysis using CPP Check (10:00)

Lossos's research methodology, centered on simple static code analysis with CPPCheck, proved remarkably effective in uncovering several significant vulnerabilities across various Linux kernel components. His initial attempt to find the aforementioned memcpy overflow with CPPCheck was unsuccessful, likely because the tool couldn't infer that start_column and stop_column were user-controlled. However, this did not deter him.

He then turned his attention to NVIDIA's newly open-sourced kernel drivers, driven by the hypothesis that even major vendors might introduce vulnerabilities when porting closed-source code. His intuition paid off:

• He discovered a null pointer dereference where a pointer was declared NULL, but then dereferenced just one line later without checking the return value of acpi_bus_get_device(). This function could legitimately return NULL, leading to a crash. This was a clear example of a "low-hanging fruit" bug that CPPCheck successfully identified.

Following this, Lossos and his colleague Alon Zvi investigated the NTFS driver implementation in the Linux kernel, noting its lack of recent maintenance. This neglect often signals potential vulnerabilities.

• They uncovered another null pointer dereference within the NTFS code. The bug involved two pointers, atter (initialized to NULL) and atter_b (properly initialized). The code consistently worked with atter_b throughout a block of logic, only to mistakenly use atter for a dereference at the very end. This subtle programming error resulted in an exploitable bug, detailed further in a blog post by Alon Zvi.

The most impactful finding, and the main subject of the talk, was a critical vulnerability in the NVMe driver itself. Lossos specifically focused on newly introduced features, reasoning that fresh code often contains more bugs. He examined the patch notes for Linux kernel version 6 release candidates and identified a new feature: NVMe in-band authentication support.

• Using CPPCheck against this new code, he quickly pinpointed a null pointer dereference within the nvme_setup_o() function. The vulnerability stemmed from the ctrl_key member of the ctrl struct. If an error occurred during key extraction, ctrl_key was explicitly set to NULL and then immediately dereferenced in a logging statement. This seemingly innocuous logging operation, when triggered with an invalid key, led directly to a kernel crash.

These findings collectively demonstrate that even with basic static analysis tools, significant vulnerabilities can be found in widely used and critical open-source codebases, including the Linux kernel. The ease of discovery for these bugs, many of which are straightforward null pointer dereferences, highlights the ongoing need for robust code review and automated analysis, even in well-maintained projects.

Technical Deep Dive

▶ Watch: Finding the NVMe driver null pointer vulnerability (14:30)

The main vulnerability discussed by Lossos lies within the NVMe driver's in-band authentication support, specifically a null pointer dereference in the nvme_setup_o() function. To understand this, it's crucial to grasp the underlying technology.

NVMe (Non-Volatile Memory Express) is a high-performance transfer protocol designed for accessing non-volatile storage media, predominantly SSDs (Solid State Drives), over a PCIe (Peripheral Component Interconnect Express) bus. It's the standard for modern, high-speed storage.

• NVMe-oF (NVMe over Fabrics) is an extension that allows NVMe commands to be transferred over network fabrics like Fibre Channel, InfiniBand (RDMA), or Ethernet (TCP/IP), rather than being limited to direct PCIe connections.
• NVMe/TCP is a specific implementation of NVMe-oF that uses the TCP/IP protocol for transport, enabling NVMe storage to be accessed remotely over standard Ethernet networks.

NVMe is ubiquitous: it's the default disk technology for many virtual machines in cloud providers like Amazon and Azure, common in on-premise servers, and integrated into the Linux kernel itself. This widespread adoption means any vulnerability in its driver has a significant attack surface.

The core of the NVMe vulnerability is a null pointer dereference within the nvme_setup_o() function. Lossos identified the following code pattern:

1. A ctrl_key member within a ctrl struct is involved in key extraction.
2. If an error occurs during this key extraction process, the ctrl_key is explicitly set to NULL.
3. Immediately after, this NULL ctrl_key is dereferenced, typically in a logging statement. This dereference of a NULL pointer in kernel space leads directly to a kernel panic or an "oops."

To trigger this, an attacker needs to provide an invalid key. The function's internal logic defines an invalid key as one that is not base64 encoded or whose length is not one of the expected values: 36, 52, or 68 characters. This makes it remarkably easy for an attacker to craft a malicious key.

Exploiting null pointer dereferences in the Linux kernel is often challenging due to several mitigations:

• map_min_address: This kernel parameter prevents mapping low memory addresses, including page zero (address 0x0), where NULL typically points. Bypassing this is a prerequisite for directly interacting with a dereferenced NULL pointer.
• SMAP (Supervisor Mode Access Prevention) and SMEP (Supervisor Mode Execution Prevention): These mitigations prevent the kernel from accessing or executing user-space memory, respectively. Even if an attacker could map page zero, SMAP/SMEP would typically prevent the kernel from interacting with attacker-controlled data or code placed there.
• Oops and panic_on_oops: An "oops" signifies a kernel internal error, potentially allowing the system to continue running in a degraded state. However, many production kernels are configured with panic_on_oops, which forces an immediate kernel panic and system crash upon an oops, effectively turning any oops into a Denial of Service.

Lossos's key insight was recognizing that the NVMe in-band authentication feature, being tied to NVMe/TCP, suggested a remote primitive. By analyzing the call stack, he traced nvme_setup_o() upwards:

• nvme_setup_o() is called by functions responsible for parsing incoming requests.
• These parsing functions are, in turn, called by the NVMe-oF implementations (TCP, RDMA, Fibre Channel).
• Specifically, the NVMe/TCP implementation calls the generic kernel receive message function (generic_kernel_recvmsg).

This chain of calls strongly indicated that the vulnerability could be triggered remotely by sending a specially crafted NVMe/TCP connection request containing an invalid authentication key. The remote nature elevates a local DoS to a remote DoS, significantly increasing its impact. The topology involves a host (client) attempting to access storage from a target (server), with the vulnerable code residing on the target.

Setting up an NVMe/TCP environment for testing wasn't straightforward for the authentication feature due to lack of documentation for the newly introduced code. Lossos found that the BLK test framework (a testing framework for block devices in the kernel) contained a recently committed test specifically for the authentication feature, which provided the necessary configuration details. This highlights his key takeaway: when documentation is scarce, test code can be an invaluable resource.

Demo / Proof of Concept

▶ Watch: Live demo: Remote NVMe denial of service (26:30)

Tal Lossos presented two compelling live demonstrations to showcase the impact of his findings.

The first demonstration illustrated the remote Denial of Service (DoS) against an NVMe/TCP target.

1. Setup: Lossos configured an NVMe/TCP server (the "target") on a virtual machine (left side of the screen). This server was set up to allow connections from a specific client based on its Host NQN (NVMe Qualified Name), a unique identifier for NVMe hosts.
2. Attacker Machine: On another virtual machine (right side), designated as the "attacker," Lossos displayed its Host NQN. Crucially, this NQN was allowed by the server, indicating an authorized connection attempt.
3. Triggering the DoS: Using the nvme command utility, Lossos initiated a connection request from the attacker machine to the remote NVMe/TCP server. Within this request, he injected an invalid authentication key (as defined by the vulnerability: not base64 encoded or of an incorrect length).
4. Result: Upon sending the malformed connection request, the NVMe/TCP server immediately crashed, demonstrating a clear remote Denial of Service. The kernel panicked due to the null pointer dereference, confirming the exploitability of the bug.

The second demonstration focused on an authorization bypass stemming from the cleartext transmission of the Host NQN.

1. Initial Attempt: Lossos first attempted to connect to the NVMe/TCP server from an unallowed attacker host (different Host NQN). As expected, the connection was rejected with an "not allowed to connect" error message.
2. Network Sniffing Insight: He then explained that by observing the connection request on the network (e.g., with Wireshark), it becomes apparent that the Host NQN is transmitted in cleartext within the NVMe/TCP connection packet. This critical piece of information means that the server's authorization mechanism relies solely on the integrity of this cleartext identifier.
3. Bypass Execution: For the demo, instead of performing a network attack like ARP spoofing to capture a valid Host NQN, Lossos manually copied the Host NQN of an allowed client.
4. Spoofing: He then used the nvme command utility's --host-nqn flag to spoof the allowed Host NQN from the attacker machine.
5. Result: By presenting the spoofed Host NQN, the attacker machine was able to successfully connect to the NVMe/TCP server, completely bypassing the intended authorization mechanism. This demonstrated that an attacker on the same network segment could easily impersonate an authorized host, potentially gaining unauthorized access to storage resources or performing further attacks.

Both demonstrations effectively showcased the severity of the identified vulnerabilities, moving from a theoretical understanding of static analysis findings to practical, high-impact exploits in a live environment.

Defensive Implications

▶ Watch: Bypassing NVMe authorization via cleartext host NQN (34:00)

The vulnerabilities uncovered by Tal Lossos in the NVMe driver, particularly the remote Denial of Service and authorization bypass, carry significant defensive implications for systems administrators, cloud providers, and security teams.

1. Patching and Timely Updates: The most immediate and crucial defensive action is to apply kernel patches promptly. Lossos mentioned that the kernel developers merged his fix quickly, leading to the issuance of "a few CVEs." Organizations must ensure their Linux systems, especially those acting as NVMe/TCP targets or hosts, are running the latest stable kernel versions that include these fixes. Automated patch management is vital for large deployments.

1. Network Segmentation and Access Control: The authorization bypass vulnerability highlights the dangers of relying on cleartext identifiers for access control. While the NVMe/TCP protocol itself might have security features, the cleartext transmission of Host NQN makes it susceptible to network-level attacks. Defenders should implement strong network segmentation to isolate NVMe/TCP traffic. This includes:

• Placing NVMe/TCP storage networks on dedicated, isolated VLANs.
• Implementing firewall rules to strictly limit which hosts can initiate NVMe/TCP connections to storage targets.
• Employing IPsec or other secure tunneling protocols if NVMe/TCP traffic must traverse untrusted networks, although the talk implies the issue is with the Host NQN itself being cleartext.

1. Secure Configuration Practices: Default configurations often prioritize functionality over security. System administrators should review and harden NVMe/TCP configurations, ensuring that only necessary hosts are explicitly allowed to connect. If the Host NQN authorization mechanism is deemed insufficient due to its cleartext nature, additional layers of authentication (e.g., using CHAP - Challenge-Handshake Authentication Protocol, if supported and properly configured) should be explored, or the reliance on Host NQN for critical access decisions should be re-evaluated.

1. Continuous Security Auditing and Code Review: Lossos's success with simple static analysis tools like CPPCheck underscores the importance of integrating such tools into continuous integration/continuous deployment (CI/CD) pipelines for kernel modules and drivers. Even mature or widely used open-source projects can harbor "low-hanging fruit" vulnerabilities. Developers and maintainers should:

• Regularly run static analysis tools against their codebases.
• Prioritize security-focused code reviews, especially for new features or significant code changes.
• Pay close attention to error handling paths, as NULL pointer dereferences often occur when unexpected errors lead to unhandled states.

1. Understanding Kernel Panic Behavior: While a Denial of Service might seem less severe than Remote Code Execution (RCE), a remote kernel panic on a production server can be devastating, leading to service outages and data unavailability. Defenders should understand how their systems are configured regarding panic_on_oops and ensure that critical services have robust failover and recovery mechanisms in place.

1. Awareness of New Features: New features, like the in-band authentication in NVMe, are often ripe for vulnerabilities due to less scrutiny during initial development. Security teams should pay extra attention to recently introduced functionalities in critical software components and prioritize their audit.

By adopting these defensive strategies, organizations can significantly reduce their exposure to similar vulnerabilities in NVMe and other critical kernel components, enhancing the overall resilience and security of their infrastructure.

Key Takeaways

• Static Code Analysis is Powerful: Simple, readily available static analysis tools like CPPCheck can effectively uncover "low-hanging fruit" vulnerabilities, including critical null pointer dereferences, in complex and widely used codebases like the Linux kernel.
• Focus is Key in Open-Source Research: The vastness of open-source projects can be overwhelming. A focused approach, targeting new features or specific code patterns, is crucial for efficient vulnerability discovery.
• Test Code as Documentation: When formal documentation is lacking, especially for new features, test frameworks and existing test code can be an invaluable resource for understanding functionality and configuration.
• Don't Underestimate Cleartext Protocols: Relying on cleartext identifiers (like NVMe Host NQN) for authorization in network protocols creates a significant attack surface for impersonation and bypasses through simple network sniffing or spoofing.
• Kernel Vulnerabilities are Ubiquitous: Even core kernel components and drivers from major vendors (like NVIDIA or NVMe) can contain easily discoverable, high-impact vulnerabilities, highlighting the ongoing need for continuous security auditing.
• Remote DoS is a Critical Threat: A remote kernel panic, even if not leading to code execution, can cause significant disruption and service outages in production environments, underscoring the importance of addressing such vulnerabilities promptly.

About the Speaker(s)

Tal Lossos is a Vulnerability Researcher at CyberArk, where his work primarily focuses on operating system internals. While his expertise spans various OS components, he specializes in Linux kernel research and has recently expanded his focus to include macOS. Lossos is dedicated to discovering and analyzing vulnerabilities, particularly those that can be found through efficient, open-source research methodologies. His presentation at HITB Phuket exemplifies his commitment to sharing practical techniques and significant findings with the security community.