Sandboxes, Seccomp, and Syscalls: Chasing Isolation in Kubernetes

Sandboxes, Seccomp, and Syscalls: Chasing Isolation in Kubernetes

Speakers: Mark Manning, Offensive Security Engineer, Chain Guard

Conference: BSides SF

YouTube: https://www.youtube.com/watch?v=AKimmv-OYgE

Overview

In this insightful talk from BSides SF, Mark Manning, an offensive security engineer at Chain Guard and creator of SECMC compare, delves into the complex landscape of container isolation within Kubernetes, particularly for "hard mode" scenarios where arbitrary or untrusted code is executed. Manning challenges the commonly held belief that custom Seccomp profiles are the panacea for Kubernetes security, meticulously dissecting the operational burdens, inherent difficulties in consistent profile generation, and subtle bypass techniques that render many Seccomp implementations less effective than presumed.

The presentation serves two primary purposes: to validate the significant effort undertaken by organizations successfully implementing robust custom Seccomp profiles, and to caution newcomers against underestimating the challenge. Manning argues that while Seccomp offers a powerful mechanism for system call filtering, its practical application in dynamic Kubernetes environments is fraught with perils, often leading to profiles that are either too permissive or prone to crashing workloads. He introduces his open-source tools, sec_compare and sec_compute, as a means to objectively measure the effectiveness of Seccomp profiles against default baselines. Ultimately, Manning advocates for a multi-layered security approach, emphasizing the potential of microVMs and kernel emulation technologies like G Visor as more robust isolation boundaries for high-risk workloads, especially in the evolving threat landscape driven by AI and sophisticated rootkits.

Background

▶ Watch: Introduction and talk overview (0:00)

The foundational premise for this talk stems from a long-standing industry adage: "containers are not a security boundary." While widely accepted, this statement often leaves organizations pondering "what's next?" if their primary compute abstraction offers insufficient isolation. The problem becomes acute for organizations operating Kubernetes clusters in what Manning terms "hard mode"—environments designed to run arbitrary, untrusted, or potentially malicious code. This could include build servers compiling untrusted code, platforms akin to Google App Engine, or increasingly, AI inference engines processing non-deterministic "slop code."

Manning illustrates the historical context with a story, albeit one passed down through Google lore, from late 2016. Researchers reportedly compromised Google App Engine's Java runtime, breaking out of its "virtual machine" and compromising the underlying operating system. Google's response was to harden Java and Python runtimes by meticulously removing "dangerous bits" and patching every version to ensure only "good stuff" was possible—a "filter out the bad" approach. This historical event underscores the critical need for robust isolation when running untrusted code.

The speaker frames the isolation challenge into two schools of thought:

1. Filter out the bad: This approach, exemplified by Google's initial App Engine hardening and the use of Seccomp, aims to restrict dangerous operations by explicitly denying system calls that could lead to privilege escalation or container breakouts.
2. Fake the good: This involves virtualizing or emulating the underlying system, presenting a controlled environment to the workload without granting direct access to the host kernel. Technologies like microVMs and G Visor fall into this category.

At the heart of container security lies the concept of system calls. These are the fundamental API interfaces between a user process and the Linux kernel, allowing programs to request services like opening files (openat), reading data (read), or establishing network connections (accept, bind). Understanding and controlling these system calls is paramount for achieving isolation. Seccomp (Secure Computing mode), introduced in 2005, was designed precisely for this purpose. It allows processes to define a strict whitelist or blacklist of system calls they are permitted to make. Docker later simplified its use by introducing JSON-based Seccomp profiles, and Kubernetes further abstracted this with YAML definitions, making it accessible for container orchestration. The promise was clear: by filtering dangerous system calls, one could effectively mitigate container breakout risks.

Key Findings

▶ Watch: Kubernetes 'hard mode' and isolation challenges (1:00)

Manning's research reveals significant challenges and inconsistencies in achieving reliable container isolation using custom Seccomp profiles in Kubernetes. His key findings highlight that while the concept of Seccomp is sound, its practical implementation is fraught with peril, making it difficult to generate profiles that are both secure and stable.

Firstly, he demonstrates that even for a static "Hello World" program, the number of system calls can vary significantly depending on the execution environment. A simple strace on a binary might show around 170 system calls, but placing the same binary in a Chain Guard Tiny container and tracing it can yield over 1,800 system calls. This variability makes static profile generation inherently unreliable.

Secondly, Manning identifies "three perils of system call filtering" that undermine the effectiveness of custom Seccomp profiles:

1. Risk of a missing system call: Inconsistent tracing or dynamic runtime behavior can lead to a production workload attempting a blocked system call, resulting in a crash.
2. Unrelated system calls: Tracing tools can inadvertently capture system calls made by other processes interacting with the container, such as security agents like CrowdStrike, Falco, or Sysdig, or even sidecars. This can lead to profiles that include dangerous system calls (e.g., BPF, IO_uring, keyctl) that the actual workload doesn't need, effectively creating a backdoor.
3. Unapproved system calls from internal fallback mechanisms: Some applications, particularly databases like MongoDB, attempt various system calls (e.g., IO_uring) and fall back to alternatives if they fail. Tracing these attempts results in dangerous syscalls being included in the profile, even if the application would have functioned without them.

Crucially, Manning demonstrates that custom Seccomp profiles often fail to improve security significantly over Kubernetes' default Seccomp profile, and in some cases, can even degrade it. His sec_compare and sec_compute tools were developed to address this, providing a multi-dimensional analysis to measure whether a custom profile is genuinely more restrictive and less dangerous than the default. He uses a Dungeons & Dragons analogy to categorize profiles from "lawful good" (more restrictive, less dangerous) to "chaotic evil" (less restrictive, more dangerous), noting that many custom profiles inadvertently fall into the latter categories.

Finally, Manning exposes critical Seccomp bypass scenarios. He highlights that simply granting all capabilities (cap_add: all) to a container while still applying a Seccomp profile can lead to unexpected vulnerabilities. More specifically, he details how IO_uring, a Linux kernel interface designed to multiplex system calls, can bypass Seccomp restrictions. If a Seccomp profile blocks individual system calls like socket, bind, and connect, but allows IO_uring, an attacker can leverage IO_uring to perform network operations without triggering the Seccomp filter, effectively nullifying the intended network restriction. This demonstrates that Seccomp, while powerful, requires an exhaustive understanding of kernel interfaces and potential interactions to be truly secure.

Technical Deep Dive

▶ Watch: Containers are not a security boundary: What's next? (2:50)

The core of Seccomp's operation lies in its interaction with system calls, which are the programmatic interface to the Linux kernel. Every operation a user-space program performs that requires kernel resources—from reading a file to sending a network packet—is facilitated by a system call. Examples include openat (open a file), read (read from a file descriptor), accept (accept a network connection), and clone (create a new process).

Seccomp, or Secure Computing mode, was introduced in the Linux kernel in 2005. Its initial implementation was somewhat cumbersome, requiring direct kernel interaction. Docker significantly simplified its adoption by providing a JSON-based interface for defining Seccomp profiles. These profiles specify a default action (e.g., SCMP_ACT_ERRNO to return an error, SCMP_ACT_KILL to terminate the process, or SCMP_ACT_ALLOW to permit) and then a list of system calls to explicitly allow or deny, potentially with argument-based conditions. Kubernetes further abstracted this, allowing Seccomp profiles to be defined and applied via YAML manifests.

The process of generating custom Seccomp profiles typically involves tracing a workload's execution to identify all system calls it makes. Tools like strace are fundamental for this, dumping system calls for a given process. For Kubernetes, more sophisticated, context-aware tools have emerged, leveraging BPF (Berkeley Packet Filter) and kernel tracepoints:

• tracy: A BPF-based tool for tracing system calls.
• Inspector Gadget: A powerful tool suite that uses BPF to hook Linux kernel tracepoints, capturing system calls specific to a Kubernetes pod.
• kube-cuddle trace: A Kubernetes-native tracing utility.
• Security Profiles Operator: Designed to automate the generation and application of Seccomp (and AppArmor/SELinux) profiles in Kubernetes.

However, Manning's research exposes critical flaws in the reliability of these tracing methods:

1. Inconsistent System Call Capture: The number of system calls observed can vary even for identical, static code due to runtime environment differences, compiler optimizations (e.g., fork being converted to clone), or different kernel versions. This dynamic behavior makes it difficult to create a truly exhaustive and stable whitelist.
2. Tooling Differences: Different tracing tools might hook different kernel tracepoints (ptrace vs. BPF kprobes), leading to variations in the captured system call lists.
3. Ring Buffer Overflows: BPF-based tracing tools often rely on ring buffers. If the rate of system calls is too high, these buffers can overflow, causing system calls to be missed, which can lead to application crashes in production if the missing syscall is later blocked by the profile.
4. Inclusion of Unrelated System Calls: A significant security risk arises when tracing tools capture system calls not made by the target application itself, but by other processes interacting with it. For instance, if a security agent like CrowdStrike, Falco, or Sysdig is monitoring a container, its BPF programs or other instrumentation might generate system calls that are then erroneously included in the container's Seccomp profile. This can inadvertently allow dangerous capabilities like BPF or keyctl, which could be exploited for container breakouts or rootkit installation (as seen with the link-pro rootkit using BPF).
5. Database-Specific Behaviors: Databases like MongoDB often attempt various system calls, including advanced interfaces like IO_uring, and then gracefully fall back to older mechanisms if the primary attempt fails. When tracing such an application, the profile might capture these "failed" attempts, leading to the inclusion of potentially dangerous syscalls that are not strictly necessary for the application's operation.

The most critical technical vulnerability Manning highlights is the IO_uring bypass. IO_uring is a modern Linux kernel interface designed for high-performance asynchronous I/O, allowing multiple I/O operations (including network operations like socket, bind, connect, send) to be submitted to the kernel with a single system call. The problem arises because Seccomp profiles are typically defined to block individual system calls like socket or connect. If a profile blocks these individual calls but does not explicitly block IO_uring, an attacker can rewrite their network communication code to use IO_uring. This multiplexed system call will then bypass the Seccomp filter, allowing network connections to be established despite the intended restrictions. This demonstrates a fundamental challenge: Seccomp must be aware of and explicitly block all potential kernel interfaces that can achieve a forbidden action, not just the common ones.

To address the measurement problem, Manning developed sec_compare (a web service) and sec_compute (a CLI tool). These tools perform a multi-dimensional analysis of Seccomp profiles, comparing a custom profile against the Kubernetes default. They identify:

• The total number of system calls allowed.
• The set of "dangerous" system calls (e.g., ptrace, bpf, init_module) that are allowed.
• The delta between the custom profile and the default.

This allows defenders to objectively quantify whether their custom profile actually improves security or inadvertently introduces new risks, moving beyond subjective assessments.

Demo / Proof of Concept

▶ Watch: Google App Engine exploit: A lesson in isolation (3:20)

Mark Manning presented two clear demonstrations to illustrate the capabilities and limitations of Seccomp.

The first demo showcased the fundamental principle of Seccomp-based system call filtering. A standard container was used to establish a network connection to a specific port (4444), where a "secret" was retrieved. This represented a normal, unrestricted network operation. The exact same container and code were then run, but this time with a custom Seccomp profile applied. This profile was specifically designed to block network-related system calls such as socket, bind, and listen. When the container with the restricted profile attempted the same network connection, it failed with an "Operation not permitted" error. This straightforward example effectively demonstrated how Seccomp can enforce network isolation by preventing a process from making specific system calls required for socket creation and connection.

The second, more advanced demo, highlighted a critical bypass scenario involving IO_uring. Manning explained that while the previous Seccomp profile successfully blocked individual network system calls, it did not explicitly block the IO_uring system call. He then demonstrated how a custom client, rewritten to leverage IO_uring for its network operations instead of traditional socket/connect calls, could completely bypass the Seccomp restrictions. Despite the Seccomp profile being in place, the IO_uring-based client successfully established a connection and retrieved the "secret" from port 4444. This proof of concept starkly illustrated that a Seccomp profile needs to be comprehensively aware of all kernel interfaces, including modern ones like IO_uring, that can achieve a forbidden action. Failing to account for such alternative pathways renders the intended security boundary ineffective, even if individual system calls are blocked.

Defensive Implications

▶ Watch: Defining different Kubernetes operational modes (5:20)

The insights from this talk provide crucial guidance for organizations aiming to harden their Kubernetes environments, especially those operating in "hard mode" with untrusted workloads.

For organizations already investing in custom Seccomp profiles, Manning offers validation and practical advice:

• Validate and Iterate: Acknowledge the significant effort required. Use tools like sec_compare or sec_compute to objectively measure if your custom profiles are genuinely improving security over the default Kubernetes Seccomp profile. Are you reducing the overall system call surface and, more importantly, blocking dangerous calls?
• Focus on Atomic Workloads: Prioritize creating profiles for the most critical, dangerous, and static components (e.g., specific parsers, renderers, or image processing services like ImageMagick). Trying to profile every container in a large cluster is unsustainable.
• Mitigate Tracing Perils: Be aware of the risks of missing system calls (leading to crashes) and including unrelated/unapproved system calls (leading to false sense of security or actual backdoors). Carefully configure tracing tools and validate generated profiles to exclude syscalls introduced by sidecars or security agents.

For organizations considering custom Seccomp profiles, Manning urges caution:

• Understand the Operational Burden: Generating and maintaining robust, production-ready Seccomp profiles is a complex, ongoing task. It's not a "weekend project" and requires deep understanding of both application behavior and kernel interfaces.
• Beyond the Default: Recognize that Kubernetes provides a reasonable default Seccomp profile. Custom profiles must go above and beyond this baseline to be truly valuable, and this effort must be justified by demonstrable risk reduction.
• Layered Defense: Seccomp should be viewed as one layer in a multi-layered defense strategy. As Manning states, "one is none, two is one"—meaning that multiple, independently verified security controls are necessary, and each layer must be thoroughly tested.

Beyond Seccomp, Manning proposes several forward-looking defensive strategies:

• Improved Registry Support for Profiles: Imagine container registries that could host curated Seccomp profiles alongside container images (e.g., an Nginx image bundled with a verified Nginx Seccomp profile), simplifying adoption and reducing the burden on individual teams.
• Standardized Behavior Profiles: Develop common Seccomp profiles for known application archetypes (e.g., "web server profile," "shell profile") that define a baseline of expected system call behavior, even without direct profiling of specific environments.
• Prioritize Hardening Privileged Workloads: Instead of trying to harden already somewhat restricted workloads, focus efforts on the most dangerous components in a cluster: those running in privileged mode or with extensive capabilities (e.g., cap_sys_admin, cap_net_admin). The goal should be to reduce their capabilities to the absolute minimum required and then apply Seccomp restrictions.
• Embrace Stronger Isolation with MicroVMs and Kernel Emulation: For truly untrusted or "hard mode" workloads, Manning strongly recommends technologies that provide a more robust security boundary than standard containers:
• Kata Containers and Firecracker are examples of microVMs that run each container inside a lightweight virtual machine, providing hardware-level isolation similar to traditional VMs but with container-like orchestration.
• G Visor is a user-space kernel emulator developed by Google (incidentally, as a direct response to the Google App Engine exploit Manning described). G Visor intercepts system calls from the container and emulates them in user space, presenting a virtualized kernel to the application without direct access to the host kernel. This offers a strong security boundary without requiring nested virtualization or bare metal. While it incurs a performance cost, recent advancements have improved its efficiency, making it a viable option for critical isolation needs.

Manning's final takeaway about "practical complexity as a peacetime disease" serves as a potent reminder: security controls must be rigorously tested and proven in adversarial conditions, not just theoretically applied. Assuming a Seccomp profile is secure without continuous verification and testing against bypasses is a dangerous oversight. The lesson from Google App Engine's compromise and the subsequent development of G Visor underscores that proactive, robust isolation is a non-negotiable requirement when running untrusted code.

Key Takeaways

• Containers are not a security boundary, especially in "Kubernetes hard mode": Running untrusted or arbitrary code in Kubernetes demands isolation beyond standard containerization.
• Custom Seccomp profiles are deceptively complex: While theoretically powerful, generating and maintaining reliable Seccomp profiles is fraught with challenges like inconsistent system call tracing, inclusion of dangerous unrelated syscalls, and subtle bypasses (e.g., IO_uring).
• Measure your Seccomp profile's effectiveness: Tools like sec_compare and sec_compute are essential to objectively assess if a custom profile genuinely improves security over the Kubernetes default, rather than just adding complexity.
• Prioritize hardening privileged workloads: Focus on reducing capabilities and applying Seccomp to the most dangerous components in your cluster first, as these present the largest attack surface.
• Embrace stronger isolation for high-risk scenarios: For truly untrusted code, consider microVMs (Kata Containers, Firecracker) or kernel emulation (G Visor) as more robust security boundaries.
• Verify and layer your security controls: Do not assume security based on theoretical application. Rigorously test and validate each layer of defense against potential bypasses, embodying the "one is none, two is one" principle.

About the Speaker(s)

Mark Manning, who goes by the handle "antitree," is an offensive security engineer currently working for Chain Guard, a company focused on supply chain security. His career has primarily involved offensive security roles within containerized environments. Prior to Chain Guard, he worked at Snowflake, where he contributed to their Java UDFs and sandbox implementations. He also conducts side consulting, working with technologies like G Visor and Kubernetes. Manning is the creator and maintainer of SECMC compare (sec_compare.com), a service and open-source tool (sec_compute) designed to collect, compare, and analyze Seccomp profiles used in Kubernetes. His expertise lies in understanding container isolation models and building secure software supply chains.