Ubuntu Shiftfs: Unbalanced Unlock Exploitation Attempt (CVE-2023-2612)

Ubuntu Shiftfs: Unbalanced Unlock Exploitation Attempt (CVE-2023-2612)

Speakers: Jean-Baptiste Cayrou, Security Researcher, Synacktiv

Conference: GreHack

YouTube: https://www.youtube.com/watch?v=KyRCJ05A_2s

Overview

In this GreHack talk, Jean-Baptiste Cayrou, a security researcher at Synacktiv, delves into a critical kernel vulnerability (CVE-2023-2612) he discovered in Ubuntu's Shiftfs filesystem. Titled "Ubuntu Shiftfs: Unbalanced Unlock Exploitation Attempt," the presentation meticulously details the journey from initial bug discovery through to the complex process of achieving a Local Privilege Escalation (LPE). Cayrou’s research highlights the inherent risks associated with race conditions in kernel code and the challenges of exploiting them, particularly in the context of file system operations within unprivileged namespaces.

The talk is a deep dive into Linux kernel internals, offering valuable insights into methodology for auditing kernel source code, identifying attack surfaces, and developing sophisticated exploitation techniques. Cayrou's work not only exposes a specific flaw but also serves as a case study for understanding the subtle yet powerful vulnerabilities that can arise from concurrency issues and improper locking mechanisms within the operating system's core. His persistence, even after initially abandoning the exploit due to perceived impossibility, underscores a crucial lesson in vulnerability research.

This vulnerability is particularly significant because it targeted a component unique to Ubuntu, Shiftfs, which is designed to modify Unix permissions during file access. By leveraging an unbalanced unlock primitive, an unprivileged user could trigger a use-after-free (UAF) condition, ultimately leading to root access. The presentation’s technical rigor and the speaker’s candid discussion of both successes and setbacks make it an essential resource for kernel developers, security researchers, and anyone interested in advanced Linux exploitation.

Background

▶ Watch: Introduction and methodology for kernel bug hunting (0:00)

Jean-Baptiste Cayrou's journey into kernel vulnerability research began in early 2022 with a clear objective: to find an LPE bug in the Ubuntu kernel, primarily to participate in the Pwn2Own Vancouver contest and gain root privileges. Beyond the immediate goal, his motivations included a desire to deepen his understanding of Linux kernel internals and refine his skills in exploitation techniques.

His initial methodology was informed by the prior success of a colleague at Synacktiv, who had successfully exploited an Ubuntu kernel vulnerability the previous year. This colleague's work, documented in a Synacktiv blog post, provided a strategic starting point: focus on uncommon attack surfaces within the kernel, as these are likely less audited than more central or widely used components. A prime candidate for such a surface is functionality accessible through unprivileged namespaces, which, on Ubuntu, are often enabled by default for non-root users. Namespaces are a Linux kernel feature providing process isolation, creating separate sets of resources for processes (e.g., Docker relies heavily on them). The talk specifically focused on mount namespaces, which isolate filesystem mount points.

To identify potential targets within this less-audited surface, Cayrou employed a straightforward but effective technique: grepping the kernel source code for the SF_USER_NS_MOUNT flag. This flag indicates filesystems that can be mounted by unprivileged users within a namespace. The search yielded a list of candidates, among which Shiftfs stood out as it is specific to Ubuntu and not part of the upstream Linux kernel, making it a potentially less scrutinized component.

Before diving into code review, Cayrou emphasized the importance of foundational knowledge. This involved reading the official kernel documentation for the Virtual File System (VFS) layer, which outlines the structures and APIs required for implementing filesystems. Additionally, studying past CVEs affecting various filesystems provided insights into common bug patterns and exploitation vectors. Finally, familiarizing oneself with existing kernel exploitation blog posts offered practical ideas on how to transform a primitive bug into a functional LPE exploit. This comprehensive preparation laid the groundwork for a targeted and efficient vulnerability discovery process, even though it initially led him away from the eventual target.

Key Findings

▶ Watch: Triggering the unbalanced unlock with MQ filesystem (10:00)

Cayrou's initial audit of namespace-accessible filesystems, lasting three weeks, surprisingly yielded no bugs. He had even initially skipped Shiftfs, believing it to be thoroughly audited by his colleague. However, after this dry spell, he returned to Shiftfs, and it was there he discovered the critical vulnerability, which he aptly termed an "unbalanced unlock."

The core of the bug resided within the shiftfs_create_object function, a helper in the Shiftfs kernel module responsible for creating various filesystem objects like files, folders, or symbolic links. This function's role is to retrieve the appropriate operation function pointer (e.g., mkdir for creating a directory) from the underlying filesystem on which Shiftfs is mounted. The critical flaw emerged when the underlying filesystem did not implement a required operation.

Specifically, if shiftfs_create_object attempts to retrieve a function pointer (e.g., for mkdir), and the underlying filesystem (such as the MQ filesystem, which lacks a mkdir operation) returns NULL, the subsequent logic is flawed. Instead of handling this NULL pointer gracefully and ensuring proper synchronization, the code immediately jumps to an unlock operation via a goto statement. This results in an unbalanced unlock: a directory lock is released without having been acquired first.

Triggering this bug required a specific sequence of actions, all performable by an unprivileged user within a mount namespace:

1. Initialize a new mount namespace using unshare -m.
2. Create two directories, D1 and D2.
3. Mount an MQ filesystem (e.g., mount -t mqfs mqfs D1) on D1. The MQ filesystem is crucial because it intentionally lacks certain VFS operations like mkdir.
4. Mount Shiftfs on D2, pointing to D1 as its underlying filesystem (e.g., mount -t shiftfs shiftfs D2 D1).
5. Attempt to create a directory within D2 (e.g., mkdir D2/test_dir).

When mkdir D2/test_dir is executed, Shiftfs attempts to proxy the mkdir call to the MQ filesystem mounted at D1. Since MQFS does not implement mkdir, shiftfs_create_object receives a NULL function pointer. This triggers the goto to the unlock call without a preceding lock, leading to the unbalanced unlock. The immediate effect observed was a kernel trace, indicating that a process was blocked for over 120 seconds, with stack traces showing semaphore manipulation and locking mechanism issues, confirming the presence of a synchronization primitive malfunction. This bug was later assigned CVE-2023-2612.

Technical Deep Dive

▶ Watch: Techniques for winning the narrow race window (18:00)

The ultimate goal of a kernel LPE is to change the current process's UID to 0, effectively gaining root privileges. Achieving this requires several complex steps: obtaining kernel code execution (or at least arbitrary read/write primitives), bypassing KASLR (Kernel Address Space Layout Randomization) through a kernel leak, and then patching the process's credential structure in kernel memory. The unbalanced unlock primitive, while not directly providing arbitrary read/write, creates a powerful foundation for building such an exploit.

The core issue of an unbalanced unlock is that it disrupts the integrity of kernel locking mechanisms. Normally, when a process wants to modify a directory (e.g., create, remove, rename files), it first acquires a lock on that directory. This lock prevents race conditions by ensuring only one process can perform structural modifications at a time. By releasing a lock that wasn't taken, the bug allows multiple processes to concurrently operate on a directory, leading to unpredictable and potentially exploitable states.

Cayrou illustrated this with a typical unlink operation. In a normal scenario, if Process A calls unlink on file_one, it first acquires the directory lock. If Process B simultaneously tries to open file_one, it will block, waiting for the lock. Process A then proceeds to remove the link and decrements the inode usage count (i_count) associated with file_one. Once i_count reaches zero, the kernel frees the associated resources. After completing its operation, Process A releases the lock, allowing Process B to resume, only to find file_one no longer exists.

The unbalanced unlock primitive fundamentally alters this dynamic. With the bug, a third process (Process C) can trigger the unlock while Process A holds the lock and Process B is waiting. This prematurely releases the lock, allowing Process B to resume its operation concurrently with Process A. If both Process A and Process B are performing unlink operations on the same file, the i_count could be decremented twice, potentially reaching zero while other references to the file (e.g., open file descriptors) still exist. This scenario directly leads to a Use-After-Free (UAF) vulnerability.

The conceptual exploitation strategy was clear: synchronize two processes (A and B) to call unlink on the same file, then use Process C to trigger the Shiftfs bug and unlock the directory, creating the race window. However, the practical implementation presented significant challenges. The race window for unlink is typically very narrow, making precise synchronization extremely difficult.

To overcome these challenges, Cayrou employed several techniques for race condition exploitation:

• Minimal Kernel Configuration: Building a custom kernel with only essential modules significantly reduced compilation and testing times, allowing for rapid iteration.
• Timing Measurements: Using RDTSC (Read Time-Stamp Counter) on x86 architectures allowed for precise measurement of race window durations, crucial for understanding and optimizing synchronization.
• CPU Affinity and Priority: Assigning processes to specific CPUs and manipulating their task priorities helped in controlling execution flow and increasing the likelihood of winning the race.
• Interrupt-based Delays: Drawing inspiration from academic papers like "Racing against the Clock," Cayrou explored using kernel interrupts to temporarily halt operations within the race window, effectively prolonging it.

A crucial breakthrough came with the discovery that the inotify feature could be leveraged to significantly increase the unlink race window. inotify allows processes to register for filesystem events, such as file deletion. By having a process register an inotify watch on the target directory, the unlink operation becomes longer due to the overhead of event processing, providing a wider window for the race condition to occur. With this technique, Cayrou achieved a success rate of approximately 1 win in 100 attempts, which, while not perfect, was sufficient to demonstrate exploitability within a few seconds.

The detailed UAF sequence is as follows:

1. A file, file_one, is created with an initial i_count of 1.
2. Process A opens file_one, obtaining FD1. The i_count increments to 2.
3. Process B opens file_one, obtaining FD2. The i_count increments to 3.
4. Process C registers an inotify watch on the directory containing file_one, blocking itself until a deletion event occurs.
5. Processes A and B are then synchronized to call unlink on file_one simultaneously. Process A acquires the directory lock first, while Process B waits.
6. Process A proceeds with its unlink operation, removing the link and decrementing i_count (e.g., from 3 to 2). This action triggers the inotify event, waking up Process C.
7. Process C immediately triggers the Shiftfs unbalanced unlock bug by calling mkdir on the Shiftfs mount point. This releases the directory lock.
8. Process B, which was waiting, now resumes its unlink operation, decrementing i_count again (e.g., from 2 to 1).
9. At this point, both unlink operations have completed, and the i_count is 1. Crucially, the kernel resources for file_one are not yet freed because FD1 and FD2 are still open.
10. The attacker then closes FD1. This decrements i_count to 0. The kernel, seeing i_count reach zero, frees the resources associated with file_one.
11. Later, when the attacker attempts to access FD2 (which is still open but points to freed memory), a Use-After-Free occurs. The kernel attempts to read or write from memory that has been deallocated and potentially reallocated for other purposes.

From this UAF primitive, the standard kernel exploitation chain can proceed:

1. Memory Reoccupation: The freed memory region is deliberately reoccupied with attacker-controlled data, often by allocating specific kernel objects of the same size.
2. Kernel Read/Write Primitives: By carefully crafting the reoccupied structure, the UAF can be transformed into arbitrary kernel read/write primitives. This might involve corrupting pointers or function tables.
3. KASLR Bypass: Using the read primitive, the attacker can leak kernel pointers to defeat KASLR, determining the base address of the kernel in memory.
4. Credential Patching: Finally, the write primitive is used to locate and modify the current process's cred structure (specifically, setting uid, gid, euid, egid to 0), granting root privileges.

Cayrou demonstrated a screenshot of his exploit successfully executing these initial steps, showing that "two unlink operation return zero" (indicating the race was won) and the subsequent "reuse the internal memory object with control control value."

A significant hurdle arose when testing the exploit on updated Ubuntu versions. The exploit unexpectedly failed, causing the CPU to block. This was attributed to a patch in the kernel's internal locking system, which effectively restricted the race condition to a single CPU thread, making exploitation seem impossible. Initially, Cayrou abandoned this path, reporting the bug to Ubuntu, which subsequently patched it by ensuring that the shiftfs_create_object function always performs a balanced lock/unlock. However, while preparing for the GreHack talk, Cayrou revisited the issue and realized that the inotify trick could be scaled. By registering up to 128 processes with inotify for deletion events, he could drastically increase the race window, achieving a success rate of over 50% even on updated kernels with the locking system patch. This breakthrough validated the exploitability of the bug on modern Ubuntu systems.

Demo / Proof of Concept

▶ Watch: Animated walkthrough of the Use-After-Free exploit (22:00)

While a live demonstration was not performed during the talk, Jean-Baptiste Cayrou presented compelling evidence of his exploit's functionality through a screenshot of its output. This visual proof-of-concept confirmed the successful execution of the critical initial stages of the LPE chain.

The screenshot displayed console output indicating that both unlink operations, central to triggering the race condition, had successfully returned zero. This message, "two inlink operation return zero," explicitly verified that the delicate timing required to win the race condition and achieve concurrent unlink calls was met. Following this, the output showed "normal value" for specific attributes of the file descriptors, which then transitioned to "reuse the internal memory object with control control value" after the controlled allocation phase. This demonstrated that:

1. The unbalanced unlock bug was successfully triggered.
2. The subsequent race condition was won, leading to the desired state where the i_count was decremented in an exploitable manner.
3. The Use-After-Free (UAF) condition was successfully created by closing one file descriptor, freeing the target kernel memory.
4. The freed memory was then reallocated with attacker-controlled data, proving the capability to manipulate kernel objects.

This screenshot served as a concrete demonstration that the fundamental primitives for achieving arbitrary kernel read/write and ultimately privilege escalation had been established. It validated the complex interplay of the Shiftfs bug, the MQ filesystem, mount namespaces, and advanced race condition techniques, including the innovative use of multiple inotify watchers.

Defensive Implications

▶ Watch: Conclusion: Lessons learned and future of namespace access (30:00)

The primary defensive implication arising from CVE-2023-2612 is the critical importance of keeping operating systems updated. Ubuntu promptly patched this vulnerability by modifying the shiftfs_create_object function to ensure that a directory lock is always acquired before any unlock operation, thus preventing the unbalanced unlock primitive. Users of affected Ubuntu versions should ensure their systems are fully patched to mitigate this specific LPE vulnerability.

Beyond immediate patching, this research highlights several broader defensive considerations:

• Auditing Uncommon Code Paths: The vulnerability resided in Shiftfs, an Ubuntu-specific kernel component that was likely less scrutinized than upstream Linux kernel code. Defenders, and especially distribution maintainers, should prioritize thorough auditing of distribution-specific or less-common kernel modules, as these often present a fertile ground for novel vulnerabilities.
• Unprivileged Namespaces as an Attack Surface: The entire attack chain began from an unprivileged user creating a mount namespace. While namespaces are crucial for containerization and isolation, they also introduce new attack surfaces if not carefully secured. Ubuntu has recognized this and plans to restrict the use of unshare (the command used to create namespaces) for unprivileged users in future versions, which would significantly raise the bar for exploits leveraging this vector.
• Robustness of Locking Mechanisms: The bug's root cause was a subtle flaw in the kernel's locking logic. This underscores the need for extreme vigilance in implementing and reviewing synchronization primitives within kernel code. Even seemingly minor missteps in lock acquisition and release can lead to severe race conditions and UAF vulnerabilities. Automated static analysis and rigorous testing, especially with concurrency-focused tools, are essential.
• Complexity of Race Condition Exploitation: While race conditions are notoriously difficult to find and exploit, this talk demonstrates that determined attackers can overcome these challenges using sophisticated timing and synchronization techniques (e.g., inotify scaling). Defenders should assume that if a race condition exists, it is potentially exploitable, even if it appears to have a very narrow window.
• Continuous Learning and Re-evaluation: Cayrou's personal lesson, "do not give up," applies equally to defenders. Even after a bug is patched, understanding the underlying principles and the exploitation techniques can inform better defensive strategies against future, similar vulnerabilities. The evolution of the exploit, adapting to kernel locking changes, illustrates the ongoing cat-and-mouse game between attackers and defenders.

In summary, while CVE-2023-2612 is patched, the insights gained from its discovery and exploitation serve as a valuable reminder for developers to write robust, secure kernel code, and for system administrators to maintain up-to-date systems and understand the implications of powerful kernel features like unprivileged namespaces.

Key Takeaways

• Unbalanced Locking Leads to Critical Flaws: A seemingly minor error in kernel code—an unlock operation without a preceding lock—can create a powerful primitive for exploiting race conditions, leading directly to Use-After-Free vulnerabilities.
• Unprivileged Namespaces are a Significant Attack Surface: Features like mount namespaces, while beneficial for isolation, can expose less-audited kernel components (like Ubuntu's Shiftfs) to unprivileged users, making them prime targets for LPE exploits.
• Race Condition Exploitation is Achievable with Persistence: Despite the inherent difficulty of timing and synchronizing kernel races, sophisticated techniques (e.g., using inotify to expand the race window, or scaling multiple inotify watchers) can overcome these challenges, even on hardened systems.
• Kernel Patching Changes Exploitability: Updates to core kernel mechanisms, such as the locking system, can profoundly impact existing exploit techniques, requiring attackers to adapt their strategies or abandon exploits.
• Distribution-Specific Components Need Extra Scrutiny: Vulnerabilities in unique components like Shiftfs, which are not part of the upstream Linux kernel, highlight the need for rigorous auditing of distribution-specific codebases.
• "Do Not Give Up" is a Key Research Principle: The speaker's success in revisiting and ultimately exploiting the bug, even after initially deeming it impossible, underscores the value of persistence and creative problem-solving in security research.

About the Speaker(s)

Jean-Baptiste Cayrou, known as "Ibou," is a Security Researcher at Synacktiv, specializing in the reverse engineering team. His work involves in-depth analysis of software and systems to identify vulnerabilities. Cayrou's motivation for engaging in the research presented at GreHack stemmed from a desire to deepen his knowledge of Linux kernel internals and to participate in prestigious hacking contests like Pwn2Own, with the ultimate goal of achieving root privilege escalation. His presentation reflects a strong analytical approach and a detailed understanding of complex kernel mechanisms and exploitation techniques.