Container Runtimes... on Lockdown: The Hidden Costs of Multi... Lewis Denham-Parry & Caleb Woodbine

Container Runtimes... on Lockdown: The Hidden Costs of Multi... Lewis Denham-Parry & Caleb Woodbine

Speakers: Lewis Denham-Parry, Senior Developer Advocate, Adera; Caleb Woodbine, Senior Developer Advocate, Adera

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=I9t7qfOjgbo

Overview

In an era where cloud-native architectures dominate enterprise infrastructure, the talk "Container Runtimes... on Lockdown: The Hidden Costs of Multi..." by Lewis Denham-Parry and Caleb Woodbine delves into the critical, yet often overlooked, challenge of workload isolation within Kubernetes environments. The speakers highlight that while Kubernetes provides powerful abstractions for managing compute resources, the underlying reality is that containers are merely processes running side-by-side on a shared host kernel. This inherent sharing introduces significant security risks, particularly in multi-tenant scenarios where diverse and potentially untrusted workloads coexist.

The presentation serves as a crucial call to awareness for developers, architects, and security professionals, urging them to move beyond the assumption that basic containerization offers sufficient isolation. Denham-Parry and Woodbine meticulously unpack the historical evolution of isolation techniques, dissect various container runtime implementations, and expose the trade-offs between security, performance, and operational complexity. Their core message emphasizes that understanding and consciously selecting the right container runtime is paramount to mitigating the risks of data exfiltration, lateral movement, and resource contention, ultimately leading to a more secure and resilient cloud infrastructure.

Background

▶ Watch: Interactive demo: Kubernetes node, pods, and containers (0:50)

The journey into container isolation begins with a fundamental understanding of what containers and container runtimes are. At its core, a container is an isolated process or set of processes that run on a host operating system. Kubernetes, while orchestrating these containers, abstracts away the underlying infrastructure, making it easy to deploy applications across various environments. However, this abstraction can obscure the fact that multiple containers often share the same host kernel. The Open Container Initiative (OCI) runtime specification defines the interface for container runtimes, enabling them to create, start, kill, and manage containers consistently. Popular OCI-compliant runtimes include runc, containerd, and cri-o.

Traditional container isolation relies heavily on Linux namespaces (e.g., PID, mount, network, user) and cgroups. Namespaces provide a degree of process isolation, making it "feel like" each container has its own system resources, while cgroups limit resource usage (CPU, memory, I/O) to prevent the "noisy neighbor" problem. Additionally, seccomp (secure computing mode) and mandatory access controls like AppArmor or SELinux help restrict the system calls a container can make, further enhancing its security posture. However, as the speakers underscore, sharing a single host kernel remains a significant attack surface. Any vulnerability in the kernel or a container breakout can compromise all workloads on that node. The concept of host access, particularly by a malicious or compromised system administrator, represents a "god-like sysadmin mode" that can bypass container-level isolation entirely.

The talk categorizes containerized workloads into three risk profiles:

1. Untrusted Code/Image: Random images from the internet with little to no attestation, posing an unknown risk. The speakers recommend checking Salsa (Supply-chain Levels for Software Artifacts) for guidance on supply chain security.
2. Remote Code Execution (RCE) as a Service: This emerging category describes developer platforms (e.g., Coder, Gitpod) that offer compute with GPUs and IDEs, allowing users to run arbitrary containers. Securing these multi-tenant environments with access to sensitive hardware is critical.
3. Sensitive Applications: Beyond just financial details, this includes applications where data breaches could put lives at risk. Even with robust security tools, the implementation and underlying isolation mechanisms are key to protecting these workloads.

Historically, isolation has evolved from completely separate physical machines to chroot environments (useful for package building, but not production security), and then to virtual machines (VMs), which offer rich isolation by providing each workload with its own kernel. The advent of cgroups and runc marked a shift towards lighter-weight containerization. More recently, innovations like Gvisor, Kata Containers, Firecracker (microVMs), and Confidential Computing projects (requiring specialized hardware) have pushed the boundaries of what's possible in container isolation, each with its own approach and trade-offs.

Key Findings

▶ Watch: Overview of container runtimes, risks, and awareness (2:55)

The talk reveals several pivotal insights into the state of container runtime isolation:

• No Universal Definition of Isolation: A significant finding is the lack of a consistent, industry-wide definition for "isolation." Different runtimes and approaches offer varying degrees and types of isolation, leading to confusion and potential misconfigurations. The speakers advocate for a "Salsa for Isolation" standard to grade and assess isolation levels objectively.
• Diverse Runtime Landscape: The industry has moved beyond just runc. There's a spectrum of runtimes, each employing distinct strategies for isolation:
• Kernel-type: runc, containerd, cri-o (rely on host kernel, Linux namespaces, cgroups).
• Sandboxing: Gvisor (user-space kernel emulation), Kata Containers (microVMs), Firecracker (lightweight VMs).
• Miscellaneous: Wasm (WebAssembly-based, strong isolation but requires application refactoring), Sysbox (enhanced user namespaces), and Adera Protect (Type-1 hypervisor based zones).
• Inherent Trade-offs: Achieving stronger isolation almost always comes with a cost. This includes performance overhead (CPU, memory, I/O), increased complexity in deployment and management, and sometimes specific hardware requirements or even a need to rewrite applications (e.g., for Wasm).
• Kubernetes Namespaces are Insufficient for Security Isolation: While useful for organizational and resource management, Kubernetes namespaces alone do not provide a strong security boundary, especially against sophisticated attackers or kernel vulnerabilities.
• The "Am I Isolated?" Initiative: The speakers introduced an open-source project designed to help users assess their actual level of isolation within their infrastructure. This tool aims to demystify the "Inception-like" problem of not knowing if one is truly isolated or still "spinning in a dream state" of shared resources.
• Continuous Patching is Non-Negotiable: Vulnerabilities like Leaky Vessels (a set of four CVEs, including one in runc discovered by Snyk in January 2023) demonstrate that even fundamental components like runc can have critical flaws allowing container breakouts and lateral movement. Timely updates are crucial.

Technical Deep Dive

▶ Watch: Core definitions: What is a container runtime? (4:10)

The core of the talk provides a detailed examination of various container runtimes, categorizing them by their isolation mechanisms and discussing their technical implications.

1. Kernel-Type Runtimes (Standard)

These are the default choices for most Kubernetes clusters today:

• runc: This is the reference OCI runtime, directly interfacing with the Linux kernel. It utilizes Linux namespaces (PID, NET, UTS, IPC, MNT, USER) and cgroups to provide basic isolation and resource limits. runc is critical for understanding container isolation and breakout risks, as it's a frequent target for security hardening and updates. It supports user namespaces for rootless containers, though the speakers caution that this can reduce kernel self-protection.
• crun: A faster, leaner OCI runtime written in C. It's designed for modern workloads, offering fast start times, low memory consumption, and robust rootless support. crun is a strong choice for optimizing performance while still leveraging kernel features.

2. Sandboxing Runtimes (Advanced Isolation)

These runtimes introduce more sophisticated mechanisms to enhance isolation, particularly for multi-tenant environments:

• Gvisor: Developed by Google, Gvisor intercepts system calls from containers and emulates the Linux kernel behavior in user space. Instead of directly calling the host kernel, containers communicate with a "fake kernel" provided by Gvisor, which then decides what to allow. This drastically shrinks the attack surface by reducing direct interaction with the host kernel. While providing stronger multi-tenant isolation than runc or crun, it cannot protect against risks posed by privileged containers (where default security defaults are removed). The caveat is a measurable performance cost compared to kernel-type runtimes.
• Kata Containers: This project uses Kernel-based Virtual Machines (KVM) under the hood to provide VM-level isolation for containers. It runs each container (or pod) inside a lightweight microVM (using options like QEMU or Firecracker). Each microVM has its own dedicated kernel, effectively eliminating the shared kernel problem. This was historically a default strong isolation option for Kubernetes. While offering robust security, Kata Containers introduces higher overhead than Gvisor or kernel-type runtimes. Implementation can also be complex, though tools like Telos Linux Image Factory can simplify deployment by pre-packaging the OS with Kata support.
• Firecracker: A specific implementation of a microVM, open-sourced by AWS. It's designed for serverless functions and container workloads, focusing on minimal overhead, fast startup times, and strong isolation. Firecracker is a key component often used within Kata Containers.

3. Miscellaneous/Emerging Runtimes

This category includes alternative approaches with varying levels of maturity and adoption:

• Wasm (WebAssembly): The speakers touch upon WebAssembly as an emerging technology for secure, sandboxed execution. While offering powerful isolation and efficiency, running Wasm workloads (e.g., with SpinKube) typically requires applications to be rebuilt to target the Wasm runtime. This presents a significant "cost" in terms of developer time and refactoring efforts.
• Sysbox: This runtime focuses on enhancing isolation by running all containers with Linux user namespaces. Unlike Kata or KubeVirt, it avoids using full VMs, making it easier to use and compatible with cloud environments without requiring nested virtualization. However, its own README acknowledges that it "does not provide the level of isolation that VM-based runtimes do," highlighting the nuanced definition of isolation.
• Adera Protect 1.0: Introduced as a new offering, Adera Protect builds on concepts of Xen (a Type-1 hypervisor) but is rewritten in Rust. It creates "zones" that run directly on a Type-1 hypervisor, avoiding nested virtualization. Each zone has its own kernel, distributed via OCI images. By operating so close to the bare metal, Adera Protect aims to achieve efficiency comparable to kernel-type runtimes while providing strong isolation. It also uniquely monitors actual energy usage per zone via a Prometheus endpoint and is hardware-agnostic, running on public cloud or bare metal.

Multi-Tenancy Considerations:

The speakers critically assess common multi-tenancy strategies:

• Kubernetes Namespaces: While logical for organizing resources, they are insufficient for security isolation between untrusted tenants.
• Linux Namespaces (Root Required): Provide some isolation but require root access, which container runtimes inherently possess.
• Rootless Containers (User Namespaces): Can offer enhanced security by running containers without root privileges, but this "violates kernel self-protection guidelines" in some contexts, reducing overall machine security.
• Different Node Pools: Allocating dedicated node pools to specific customers or tenants can provide strong isolation at the infrastructure level, but at a higher cost.
• Network Policies: Useful for restricting communication between pods but do not address kernel-level isolation concerns.
• Virtual Kubelet: Can provision VMs for pods, effectively giving each pod VM-level isolation.
• vcluster: Offers API isolation, providing a "virtual cluster" experience, but typically still runs workloads on shared underlying nodes.

Demo / Proof of Concept

▶ Watch: Understanding Linux namespaces, cgroups, and seccomp (5:00)

The speakers introduced and briefly demonstrated an open-source project called "Am I Isolated?" This tool is presented as a "love letter" to Jesse Frazelle's "Am I Contained" from a decade ago, which helped determine if a workload was running in a VM or a container. "Am I Isolated?" aims to address the current ambiguity around container isolation levels.

The tool provides simple terminal output, indicating the perceived level of isolation. The goal is not to be a definitive security audit tool but rather an educational tool to help users understand their environment's isolation characteristics. By running it within their infrastructure, users can get immediate feedback on whether their workloads are truly isolated, or if they are still within a "dream state" of shared resources, akin to the movie Inception. The speakers actively encouraged community contributions and feedback to evolve this project, emphasizing that fostering discussion around isolation definitions is its primary purpose.

Defensive Implications

▶ Watch: Classifying container types by security context (6:00)

The detailed exploration of container runtimes and isolation mechanisms provides several critical defensive implications for organizations operating in cloud-native environments:

1. Prioritize Runtime Choice Based on Workload Sensitivity: Not all workloads require the same level of isolation. For highly sensitive applications (e.g., financial, healthcare, RCE-as-a-service platforms) or untrusted code, organizations should move beyond default runc and consider stronger sandboxing runtimes like Gvisor, Kata Containers, or Adera Protect. Simple web applications or trusted internal services might tolerate the efficiency of runc or crun.
2. Implement Robust Patching and Update Strategies: The Leaky Vessels vulnerabilities (CVEs in runc and other components discovered in January 2023) serve as a stark reminder that even fundamental container runtime components are susceptible to critical flaws that can lead to container breakouts and lateral movement. Organizations must have automated, timely patching processes in place for their container runtimes and host kernels.
3. Leverage Kubernetes RuntimeClass: Kubernetes provides the RuntimeClass API object to configure and select different container runtimes for pods. Defenders should define RuntimeClass objects for various isolation levels and enforce their use through admission controllers or policy engines (e.g., OPA Gatekeeper) to ensure appropriate runtimes are used for specific workloads.
4. Understand the Costs and Trade-offs: Stronger isolation often comes with performance overhead (increased CPU, memory, I/O consumption) and operational complexity. Defenders need to evaluate these costs against the security benefits and business requirements. For example, Kata Containers might offer superior isolation but with higher resource utilization and a steeper learning curve than Gvisor.
5. Address the Shared Kernel and Host Access Risks: Recognize that runc-based containers share the host kernel, making them vulnerable to kernel exploits. Even with advanced runtimes, a compromised host or a malicious system administrator with "god-like sysadmin mode" can bypass container isolation. Implement robust host security, least privilege, and auditing for host access.
6. Seek Industry Standards for Isolation Assessment: The call for a "Salsa for Isolation" highlights a critical gap. Defenders should advocate for and contribute to efforts that standardize how isolation levels are defined, measured, and attested. Tools like "Am I Isolated?" can serve as a starting point for internal assessments and foster better understanding.
7. Consider Rootless Containers with Caution: While user namespaces for rootless containers can enhance security, the speakers note they can "violate kernel self-protection guidelines." A thorough understanding of these trade-offs is necessary before widespread adoption.
8. Evaluate Emerging Runtimes: Keep an eye on newer technologies like Sysbox for its user namespace enhancements without nested virtualization, or Adera Protect for its Type-1 hypervisor approach. These could offer improved security-to-performance ratios in the future. For those willing to refactor applications, Wasm presents a powerful, highly isolated execution model.

Key Takeaways

• Container isolation is a complex and nuanced challenge in multi-tenant Kubernetes environments, with no single, universally agreed-upon definition.
• Default container runtimes like runc provide basic isolation through Linux namespaces and cgroups but are susceptible to kernel vulnerabilities and container breakouts, exemplified by the Leaky Vessels CVEs.
• Advanced runtimes like Gvisor (user-space kernel emulation), Kata Containers (microVMs), and Adera Protect (Type-1 hypervisor zones) offer significantly stronger isolation by either intercepting syscalls or providing dedicated kernels per workload.
• Choosing the right container runtime requires a careful assessment of workload sensitivity, performance requirements, operational complexity, and the specific security threats you aim to mitigate.
• Tools like the open-source "Am I Isolated?" project are crucial for understanding and validating the actual level of isolation within your infrastructure, moving towards a "Salsa for Isolation" standard.
• Continuous patching of runtimes and kernels, strategic use of Kubernetes RuntimeClass, and a deep understanding of host-level security are fundamental for effective defense in containerized environments.

About the Speaker(s)

Lewis Denham-Parry is a Senior Developer Advocate at Adera and an organizer for KCD Edinburgh. He is passionate about cloud-native technologies and security, with a particular interest in low-level runtime mechanics. Lewis humorously shared his experience of wearing a kilt for the first time during the presentation, drawing parallels between the often-overlooked importance of good underwear and the critical, low-lying role of container runtimes. He is proud of Adera's diverse team, highlighting the company's three founding women, including CPO Ariadne and CTO Alexander, who spearheaded projects like Wolfie and Adera Protect.

Caleb Woodbine is also a Senior Developer Advocate at Adera, hailing from New Zealand. He contributed significantly to the historical context of isolation and the technical deep dive into various runtimes. Caleb expressed his personal appreciation for projects like Telos Linux, which simplifies the implementation of advanced runtimes like Kata Containers. His insights reinforced the practical challenges and solutions in deploying robust isolation in real-world Kubernetes setups.