Containers / Security / A Fun Time -- Pick Two
Aleksa
purplecon 2024 · Day 1 · Main Room
Overview
In "Containers / Security / A Fun Time -- Pick Two," Aleksa, a seasoned container runtime developer, takes the audience on a journey through the fundamental security mechanisms underlying modern containerization. The talk deconstructs the common perception of containers as simple, isolated environments, revealing them instead as intricate constructs built upon a "Rube Goldberg machine" of Linux kernel primitives. Aleksa emphasizes that true container security is not about magical isolation but about carefully orchestrated "lies" told by the kernel to the running processes, creating an illusion of a restricted environment.

Key moments
- 4:00 Path traversal vulnerability in basic file access
- 6:40 Using chroot to restrict file system access
- 8:45 Chroot's limitations: network, processes, resources
- 12:30 Comprehensive overview of Linux container security primitives
- 15:40 Container manager vulnerabilities via symbolic links
- 19:00 Real-world symlink attack: Docker vulnerability (2018)
- 20:50 Introducing libpathrs for safe path operations
Containers / Security / A Fun Time -- Pick Two
Speakers: Aleksa, Container Runtime Developer
Conference: purplecon
YouTube: https://www.youtube.com/watch?v=cY4ko-KhDGU
Overview
In "Containers / Security / A Fun Time -- Pick Two," Aleksa, a seasoned container runtime developer, takes the audience on a journey through the fundamental security mechanisms underlying modern containerization. The talk deconstructs the common perception of containers as simple, isolated environments, revealing them instead as intricate constructs built upon a "Rube Goldberg machine" of Linux kernel primitives. Aleksa emphasizes that true container security is not about magical isolation but about carefully orchestrated "lies" told by the kernel to the running processes, creating an illusion of a restricted environment.
The core of the presentation delves into the evolution of these kernel-level deceptions, starting from basic filesystem isolation with chroot and progressing through various namespaces, cgroups, and capabilities. While these mechanisms provide significant isolation, Aleksa powerfully demonstrates that even with this complex layering, a pervasive class of vulnerabilities related to filesystem path handling, specifically symlink race conditions (TOCTOU), continues to plague container runtimes and other privileged applications. The talk culminates in a discussion of libpathrs, a Rust library designed to mitigate these subtle yet critical flaws using advanced kernel APIs.
This talk is crucial for anyone involved in container development, deployment, or security. It demystifies the low-level workings of container isolation and exposes a category of vulnerabilities that are frequently overlooked yet have severe implications, potentially leading to host compromise. Aleksa’s engaging presentation style, combined with practical examples and a historical perspective on these bugs, provides invaluable insights into the ongoing challenges of building truly secure containerized environments.
Background
▶ Watch: Path traversal vulnerability in basic file access (4:00)
The journey into container security begins with understanding how programs interact with the operating system kernel. When a program requests a file, the kernel acts as an intermediary, providing an abstraction of the filesystem. This abstraction is a "lie" in the sense that the program doesn't see the raw disk bytes but a structured, navigable hierarchy. Initially, this abstraction is flat, allowing a program to access any file on the system, including sensitive ones like /etc/shadow or navigate upwards using ../. This fundamental lack of isolation poses an immediate security risk, as a compromised application could easily read or even modify critical system files.
To address this, early forms of isolation emerged. The chroot system call, introduced in BSD in 1982, was one of the first attempts to "lie" to a program about its root directory. By changing the effective root of a process to a subdirectory, chroot prevented access to files outside this designated boundary, effectively sandboxing the filesystem view. While revolutionary for its time (and initially conceived for testing new system versions), chroot proved to be insufficient. A chrooted process could still interact with the host's network, see all system processes, exhaust resources with a fork bomb, or elevate privileges.
The limitations of chroot led to the development of a more comprehensive suite of Linux kernel primitives, collectively known as namespaces. These mechanisms extend the "lying" principle beyond the filesystem to other aspects of the operating system:
- Network namespaces isolate network interfaces, routing tables, and firewall rules.
- PID namespaces remap process IDs, making processes outside the namespace invisible and preventing attacks like fork bombs from affecting the entire system.
- Mount namespaces (which have largely superseded
chrootwith mechanisms likepivot_root) provide a truly independent view of the filesystem mount points. - User namespaces remap user and group IDs, allowing a root user inside a container to be an unprivileged user on the host.
Beyond namespaces, other crucial security mechanisms include cgroups (control groups) for resource limiting (CPU, memory, I/O, number of processes), capabilities for fine-grained control over superuser privileges, seccomp (secure computing mode) for filtering system calls, and SELinux and AppArmor for mandatory access control. The combination of these primitives forms the complex foundation of modern container runtimes. This intricate layering, while powerful, also creates a challenging environment for developers, increasing the surface area for subtle bugs, especially when a privileged container manager must interact with an untrusted container's environment.
Key Findings
▶ Watch: Chroot's limitations: network, processes, resources (8:45)
The talk highlights several critical findings regarding container security and general Unix system robustness:
- Container Security is a "Rube Goldberg Machine": Far from being simple, lightweight virtual machines, containers rely on a complex, layered aggregation of Linux kernel primitives. These include
pivot_root(a more secure alternative tochroot), mount namespaces, network namespaces, PID namespaces, user namespaces, cgroups, capabilities, seccomp, SELinux/AppArmor, andno_new_privs. Getting this intricate setup correct is exceptionally challenging, and a misconfiguration or bug in any layer can compromise the entire system.
- Filesystem Path Handling is a Persistent Achilles' Heel: Despite the sophisticated isolation mechanisms, the fundamental way privileged processes interact with filesystem paths provided by untrusted sources remains a major vulnerability. Aleksa emphasizes that this is "the most common kind of security issues with have containers over the years."
- Symlink Race Conditions (TOCTOU) are Pervasive and Dangerous: The core vulnerability discussed is the time-of-check-to-time-of-use (TOCTOU) race condition, particularly involving symbolic links. A privileged process (like a container manager) might check a path to ensure it's safe (e.g., not a symlink to a sensitive host file), but before it uses that path (e.g., opens it), an attacker can swap the original file with a malicious symlink. This allows the attacker to trick the privileged process into operating on an arbitrary file on the host system.
- Traditional Path Sanitization is Insufficient: The common approach of "sanitizing" paths (e.g., resolving symlinks and
../sequences before opening) is fundamentally flawed when dealing with untrusted input in a concurrent environment. As demonstrated, an attacker can exploit the time window between the sanitization check and the actual file operation to introduce a malicious symlink, rendering the sanitization ineffective.
- This Problem Extends Far Beyond Containers: While the talk focuses on container runtimes, Aleksa points out that this class of TOCTOU bugs is a long-standing issue in Unix systems. A recent review of CVEs revealed "five or six CVEs" of this exact bug, with only two directly related to containers. This underscores that any program handling untrusted paths, especially with elevated privileges, is susceptible.
- Modern Kernel APIs Offer a Solution: The solution lies in using atomic kernel primitives that combine path resolution and file operation into a single, non-racy step. Specifically,
openat2andO_PATHare highlighted as the correct tools to safely handle paths by effectively resolving them without actually opening the target file until a secure context is established.
Technical Deep Dive
▶ Watch: Comprehensive overview of Linux container security primitives (12:30)
The technical foundation of container security lies in the Linux kernel's ability to provide isolated environments through a series of "lies" or abstractions. At its core, when a program attempts to open a file, it interacts with the kernel, which then resolves the path. This resolution process involves traversing directories from a defined root.
Initially, the most straightforward form of filesystem isolation was chroot. A chroot environment alters the perceived root directory for a process and its children. If a process inside a chroot tries to access /etc/shadow, the kernel resolves this path relative to the chroot's boundary. If /etc/shadow doesn't exist within the chroot, the operation fails, preventing access to the host's sensitive files. However, chroot has limitations: it only affects the filesystem view, leaving other system resources exposed.
Modern container runtimes employ a more robust mechanism called pivot_root in conjunction with mount namespaces. Unlike chroot, pivot_root changes the root filesystem of the current process and moves the old root to another mount point, allowing for more complete isolation and preventing easy "breakout" attempts that were sometimes possible with chroot. Mount namespaces ensure that a process has its own isolated view of the filesystem hierarchy, including its own set of mount points, separate from the host.
Beyond the filesystem, other namespaces provide critical isolation:
- Network namespaces provide a completely isolated network stack. A container process within its own network namespace sees only its own network interfaces (often virtual ones), routing tables, and firewall rules. To enable external connectivity, a virtual bridge is typically established between the container's network namespace and the host's, allowing for controlled communication and the application of granular firewall rules.
- PID namespaces isolate process trees. A process inside a PID namespace will see its own process as PID 1 and other processes within its namespace with remapped PIDs. Processes outside its namespace are invisible. This prevents a container from listing or terminating arbitrary host processes and effectively mitigates fork bomb attacks, where a process rapidly creates child processes to consume system resources.
- User namespaces remap user and group IDs. A user with root privileges (UID 0) inside a container can be mapped to an unprivileged user ID on the host. This significantly reduces the impact of a container breakout, as even if an attacker gains root within the container, those privileges are severely curtailed on the host.
Resource control is managed by cgroups (control groups). These allow the container manager to define limits on CPU, memory, I/O, and the number of processes a container can create. This prevents denial-of-service attacks from a runaway or malicious container. Further layers of defense include capabilities, which break down the monolithic root privilege into smaller, more specific permissions (e.g., CAP_NET_ADMIN for network administration), and seccomp, which allows filtering and restricting the set of system calls a process can make, effectively hardening the attack surface. SELinux and AppArmor provide mandatory access control policies for even stricter enforcement of resource access. The no_new_privs flag is also crucial for sandboxing, preventing a process from gaining new privileges.
The complexity arises when the privileged container manager needs to interact with files inside the container, or with files whose paths are provided by the container. This is where the symlink race condition (TOCTOU) becomes a critical vulnerability. An attacker inside a container can create a symbolic link (a file that points to another file or directory) to a sensitive host path, such as /etc/shadow. If the container manager tries to perform an operation like copying a file from the container to the host, it might first "sanitize" the path by resolving any symlinks to ensure it's not pointing outside the container. However, if the attacker can swap the original file with a malicious symlink after the check but before the file is opened, the container manager will inadvertently operate on the host's /etc/shadow file. This is a classic TOCTOU bug, where the state of the filesystem changes between the time it's checked and the time it's used. This vulnerability bypasses many of the aforementioned kernel protections because the interaction happens at a higher, logical level that the kernel's atomic operations don't inherently protect against.
Demo / Proof of Concept
▶ Watch: Real-world symlink attack: Docker vulnerability (2018) (19:00)
Aleksa illustrates the critical nature of symlink race conditions with two concrete examples, one from Docker in 2018 and a more recent variant identified this year. These examples demonstrate how an attacker can leverage the trusted operations of a container manager to compromise the host system.
The first example describes a Docker bug discovered in 2018. In this scenario, an attacker inside a container crafts a malicious setup: they create a directory and a symbolic link pointing to a sensitive host file, such as /etc/shadow. The attacker then continuously swaps these two elements, creating a race condition. When a privileged operation, such as copying a file from the container to the host, is initiated (e.g., docker cp), the container manager attempts to resolve the path. During the brief window between the path being checked for safety (e.g., ensuring it's not a symlink to an arbitrary host file) and the actual open system call, the attacker swaps the benign file with the malicious symlink. The container manager, having already performed its safety check, proceeds to open the now-malicious symlink, which resolves to /etc/shadow on the host. This allows the attacker to either read the host's /etc/shadow or, even worse, overwrite it, potentially granting them arbitrary user access to the host system.
A more complex, but fundamentally similar, attack was identified this year, involving volume mounts between containers. In this scenario, two containers share a volume, which acts as a shared directory. The attacker, within one container, again creates a symlink and continuously swaps it. The attack specifically targets the operation of creating a mount point. On Linux, creating a mount point often requires creating a file or directory at the target location. The attacker requests the container manager to set up a new container with a mount on a path like /fubar/newfile. During the process of preparing this mount, the container manager attempts to create the necessary file or directory. The attacker exploits the TOCTOU race condition by swapping their benign file with a symlink pointing to an arbitrary location on the host filesystem. As a result, the privileged container manager ends up creating or writing a file anywhere on the host, a significantly severe vulnerability that could lead to arbitrary file creation and potential privilege escalation.
These examples vividly demonstrate that traditional path sanitization techniques are inherently vulnerable to race conditions in a multi-threaded or concurrent environment. The attacker's ability to manipulate the filesystem between a "check" and a "use" operation exposes a fundamental flaw in how many privileged programs interact with untrusted paths, making these bugs a persistent threat in containerization and beyond.
Defensive Implications
▶ Watch: Introducing libpathrs for safe path operations (20:50)
The detailed analysis of container security, particularly the pervasive issue of symlink race conditions, offers several critical defensive implications for developers, system administrators, and security professionals.
Firstly, the most crucial takeaway is that simple path sanitization is insufficient and inherently insecure when dealing with untrusted input in a privileged context. Developers must move beyond naive string manipulation and path traversal checks. The time-of-check-to-time-of-use (TOCTOU) vulnerability makes any multi-step path validation and operation inherently risky. The attacker can always introduce a malicious symlink in the window between validation and execution.
Instead, defenders should leverage modern Linux kernel APIs designed for atomic and safe path resolution. Aleksa specifically highlights openat2 and the O_PATH flag as the correct primitives for this task. openat2 allows for more robust and atomic file operations, preventing race conditions by combining path resolution and the intended action into a single, indivisible kernel operation. The O_PATH flag allows a process to obtain a file descriptor that refers to a directory or file without actually opening it for read/write access. This descriptor can then be used with fstatat or openat to perform further checks or operations safely, ensuring that the target object cannot be swapped out from underneath the privileged process.
To facilitate the adoption of these safer practices, developers should consider using libraries like libpathrs. This Rust-based library encapsulates the complexities of using openat2 and O_PATH, providing a higher-level, safer API for path handling. By abstracting away the low-level details, libpathrs aims to make it easier for developers to write secure code that is resilient to symlink race conditions, minimizing the chances of introducing these subtle bugs.
Furthermore, the principle of least privilege remains paramount. Container runtimes and other privileged processes should minimize their own privileges when interacting with paths controlled by potentially untrusted users or containers. This includes carefully managing user and mount namespaces to ensure that the effective UID/GID and filesystem view of the interacting process are as restricted as possible.
System administrators and security teams must also stay vigilant and apply patches promptly. Given the complexity of container runtimes and the kernel, new vulnerabilities are continuously discovered. Regular updates to container engines, orchestration platforms, and the underlying Linux kernel are essential to mitigate known exploits, including those related to path handling.
Finally, a deeper understanding of the "Rube Goldberg machine" of Linux security primitives is vital. Developers and operators should not treat containers as black boxes but appreciate the layered security model. This knowledge helps in configuring containers securely, identifying potential weak points, and performing threat modeling that accounts for these low-level interactions. This understanding also extends beyond containers, as these symlink TOCTOU vulnerabilities affect a wide array of Unix programs that handle untrusted paths, necessitating a re-evaluation of path handling practices across the entire software ecosystem.
Key Takeaways
- Container security is fundamentally complex: It relies on a "Rube Goldberg machine" of Linux kernel primitives like namespaces, cgroups, capabilities, and seccomp, not simple, monolithic isolation.
- Filesystem path handling is a persistent vulnerability: Despite advanced isolation, the way privileged processes (like container managers) interact with paths provided by untrusted sources remains a primary source of security bugs.
- Symlink race conditions (TOCTOU) are a critical and pervasive threat: Traditional path sanitization is insufficient as attackers can exploit the time window between a path check and its use to swap in a malicious symlink, leading to host compromise.
- Modern kernel APIs are essential for robust security:
openat2andO_PATHprovide atomic, race-free mechanisms for safely resolving and operating on filesystem paths, preventing TOCTOU vulnerabilities. - Specialized libraries like
libpathrsoffer practical solutions: This Rust library helps developers correctly implement safe path handling using the appropriate kernel primitives, reducing the risk of introducing subtle security flaws. - These vulnerabilities extend beyond containers: The problem of insecure path handling affects many Unix programs that deal with untrusted input, highlighting a broader need for improved security practices across the software industry.
About the Speaker(s)
Aleksa is a dedicated and experienced developer with a primary focus on container runtimes. With nearly a decade of experience in the field, Aleksa possesses deep expertise in the lower levels of the container stack, understanding the intricate interactions between container technologies and the Linux kernel. Their work involves grappling with the complexities of container isolation, resource management, and the subtle security challenges inherent in these systems. Aleksa's insights are drawn from hands-on experience in identifying and mitigating critical vulnerabilities, particularly those related to filesystem interactions and kernel primitives.
Reviews
Dr. Zero (Offensive Security Researcher) — STRONG ACCEPT
A genuinely useful deep-dive into why container isolation keeps breaking in the same way. Aleksa nails the TOCTOU symlink problem with real CVEs and a clear path to fixing it via openat2/libpathrs. Not bleeding-edge novel, but executed well enough that practitioners will leave knowing something they didn't.
Heather Calloway (CISO) — SOLID
Solid technical exposition of why container isolation is more fragile than most operators assume. The TOCTOU/symlink race class is real and has produced exploitable CVEs. Worth your platform security lead's time, but doesn't fundamentally change how most enterprises should be thinking about container risk.