gVisor: Modern Linux Sandboxing Technology

gVisor: Modern Linux Sandboxing Technology

Speakers: Li Qiang, Platform Security Engineer, Ant Group

Conference: HITB Phuket

YouTube: https://www.youtube.com/watch?v=bPvxMWydTko

Overview

In this insightful talk from HITB Phuket, Li Qiang, a Platform Security Engineer at Ant Group, delves into the critical challenges of modern Linux sandboxing and presents Ant Group's innovative solution built upon Google's gVisor technology. The presentation, titled "gVisor: Modern Linux Sandboxing Technology," meticulously dissects the limitations of traditional Linux sandboxing mechanisms, which often fall short in providing robust isolation against sophisticated threats. Qiang articulates Ant Group's internal security requirements, highlighting the need for a sandbox capable of both strong vertical isolation (preventing escape to the host kernel) and comprehensive horizontal isolation (restricting process, file system, and network interactions within the sandbox).

The core of the talk revolves around Ant Group's development of WebMCL (WebM Container Language), a customized process-level sandbox built on gVisor. This solution addresses the inherent weaknesses of conventional sandboxes, particularly their reliance on a shared host kernel and their often rudimentary network policy enforcement. By leveraging gVisor's unique architecture, which provides an application kernel written in Go that intercepts system calls, WebMCL achieves a superior isolation boundary. This article will explore the technical intricacies of WebMCL, its design philosophy, the specific security policies it enforces, and the broader implications for enterprise security.

This discussion is particularly pertinent for organizations grappling with the security of untrusted code, third-party applications, and vulnerable open-source software within their production environments. As cloud-native architectures and containerization become ubiquitous, the demand for highly secure, performant, and easily manageable sandboxing solutions has never been greater. Ant Group's work with gVisor offers a compelling blueprint for achieving advanced process-level isolation, demonstrating how a robust application kernel can be tailored to meet stringent enterprise security demands where traditional methods prove inadequate.

Background

▶ Watch: Introduction: Traditional Linux sandboxing limitations (0:00)

The concept of a sandbox is fundamental to modern computer security, serving as a mechanism to isolate untrusted programs or code within a restricted environment. Its primary purpose is to prevent malicious or buggy software from compromising the host system or other applications. Sandboxes achieve this by imposing strict policies on a program's access to system resources, including processes, file systems, network, CPU, memory, I/O, and devices. The speaker categorizes these restrictions into two crucial dimensions: horizontal restrictions, which prevent lateral movement or unauthorized interactions between sandboxed components, and vertical restrictions, which prevent the sandboxed process from escaping to the underlying host system or kernel.

Common use cases for sandboxes in a large enterprise like Ant Group include running untrusted or user-controlled code, isolating third-party programs (e.g., open-source software), mitigating risks from frequently updated and potentially vulnerable software (such as ffmpeg or ImageMagick, which are prone to vulnerabilities), and performing malware analysis. For a sandbox to be truly effective in such environments, it must be widely usable across diverse infrastructure (VMs, containers), provide robust defense against both horizontal movement and vertical escapes, and maintain system security policies.

Traditional Linux sandboxing mechanisms, while foundational, often present significant limitations. The talk reviews several of these:

• SetUID: Allows unprivileged users to execute binaries with the privileges of the file owner, often root. While useful for specific tasks, misconfigurations can lead to privilege escalation.
• Ptrace: A system call allowing one process to observe and control the execution of another, commonly used by debuggers. It can be leveraged for sandboxing but introduces overhead and complexity.
• Seccomp (Secure Computing Mode): Restricts the system calls a process can make, effectively reducing the attack surface. However, maintaining a precise whitelist of allowed system calls for complex applications can be an extremely difficult and error-prone task.
• Capabilities: A finer-grained permission model that breaks down the traditional root privilege into distinct capabilities (e.g., CAP_SYS_ADMIN, CAP_NET_ADMIN). Misconfigured capabilities are a common vector for container escapes.
• Chroot: Changes the apparent root directory for a running process, isolating its file system view. It's a foundational isolation tool but not a complete security solution on its own.
• Namespaces: Isolate various kernel resources (PID, mount, UTS, network, user) for groups of processes, creating a more isolated environment. This is a cornerstone of containerization.
• Cgroups (Control Groups): Restrict and manage resource consumption (CPU, memory, disk I/O) for processes, preventing denial-of-service attacks from within a sandbox.
• Netfilter: A kernel subsystem for packet filtering, NAT, and mangling, often used to control network access for sandboxed processes.
• MAC (Mandatory Access Control): Security models like SELinux or AppArmor enforce policies on how subjects (processes) can access objects (files, devices). While powerful, defining and managing MAC policies is notoriously complex and labor-intensive.

Several solutions compound these mechanisms:

• SetUID Sandbox: Combines setuid, namespaces, and chroot for basic isolation.
• Systemd Sandbox: systemd offers various sandboxing options for services, leveraging namespaces and seccomp to limit access to system resources (e.g., ProtectSystem=yes, ProtectNetwork=yes).
• SGL (Google's SGL Sandbox): Utilizes Linux namespaces and seccomp-BPF to create isolated environments for untrusted applications.
• Firejail: A setuid program that uses namespaces, seccomp-BPF, and capabilities to sandbox processes, including graphical applications.

Despite these efforts, the speaker identifies two critical disadvantages shared by almost all traditional Linux sandboxing solutions:

1. Shared Kernel: All these solutions operate on the same host kernel. If a vulnerability exists in the Linux kernel (which is increasingly complex and frequently updated), an attacker can easily achieve a vertical escape from the sandbox to the host. Updating the kernel in large production environments is often slow and impractical.
2. Lack of Fine-Grained Network Policy: Most traditional solutions offer only coarse-grained network control (e.g., "network on" or "network off"). They lack the ability to define specific IP addresses, ports, or domains that a sandboxed process can connect to, making horizontal movement difficult to prevent.

These limitations necessitate a more robust approach, one that decouples the sandboxed environment from the host kernel and provides granular control over network interactions.

Key Findings

▶ Watch: gVisor: An application kernel with two layers of defense (20:20)

The central finding presented in the talk is the inherent inadequacy of traditional Linux sandboxing approaches for enterprise-grade security, primarily due to their reliance on a shared host kernel and their lack of fine-grained network policies. This shared kernel creates an unavoidable vertical escape risk, as kernel vulnerabilities can be exploited to break out of the sandbox. Furthermore, the inability to precisely control network access within the sandbox leaves systems vulnerable to horizontal movement by malicious actors.

While gVisor emerges as a promising technology that addresses the vertical isolation problem by providing an application kernel written in Go, it was not initially designed for process-level sandboxing. Its primary use case is within the container ecosystem, where it provides strong isolation for containerized workloads. Consequently, gVisor, in its default state, lacks the necessary horizontal security policies for process-level sandboxing, particularly regarding granular control over process execution, file system access, and network interactions. It implements aspects of the OCI (Open Container Initiative) spec, but the OCI spec itself does not fully define these critical horizontal policies.

Ant Group's key contribution, therefore, is the development of WebMCL (WebM Container Language). This custom solution bridges the gap between gVisor's strong vertical isolation and the enterprise's need for comprehensive horizontal security. WebMCL effectively transforms gVisor from a container-centric isolation tool into a powerful, process-level sandbox by:

1. Customizing gVisor's source code: Modifying the core gVisor components (specifically Sentry, its syscall interception layer) to enforce specific process, file system, and network policies.
2. Implementing a policy translation layer: Converting Ant Group's custom security policies into a format that gVisor can understand and enforce, often by extending or adapting the OCI runtime specification.
3. Providing a setuid binary: This allows unprivileged users to launch sandboxed applications with the necessary isolation, setting up the gVisor environment.

In essence, WebMCL's key finding is that by strategically customizing an application kernel like gVisor, it is possible to overcome the fundamental flaws of traditional sandboxing and build a robust, two-layered defense mechanism that offers both strong vertical and horizontal security for untrusted applications in production environments.

Technical Deep Dive

▶ Watch: webmgl architecture: Building a gVisor-based sandbox (26:00)

Ant Group's solution, WebMCL, is built upon gVisor, an application kernel developed by Google, written entirely in Go. gVisor distinguishes itself by implementing a significant portion of the Linux system call interface in user space, thereby acting as a lightweight, memory-safe guest kernel for applications. This design provides a critical layer of defense: when an application within a gVisor-managed container makes a system call, it is intercepted by gVisor's Sentry (the guest kernel) rather than directly by the host Linux kernel. This interception can occur via ptrace, KVM (Kernel-based Virtual Machine), or seccomp mechanisms.

gVisor's architecture provides a two-layered defense:

1. First Layer (Sentry): The Sentry handles most system call requests. Being written in Go and memory-safe, it significantly reduces the attack surface for kernel vulnerabilities (e.g., no use-after-free, no buffer overflows). This layer provides the strong vertical isolation, preventing direct interaction with the host kernel.
2. Second Layer (Gofer & Traditional Mechanisms): For operations that require interaction with the host (e.g., file system access), gVisor uses a separate component called Gofer. Both Sentry and Gofer leverage traditional Linux security primitives like namespaces, chroot, cgroups, and capabilities when interacting with the host, forming a secondary layer of defense.

While gVisor offers excellent vertical isolation, it was primarily designed for container ecosystems (like Kubernetes) and adheres to the OCI (Open Container Initiative) runtime specification. The OCI spec, however, primarily defines how containers should be run and includes certain security aspects but lacks fine-grained policies for process-level sandboxing, particularly concerning network, process execution, and detailed file system access controls. This meant gVisor, out-of-the-box, provided strong vertical but insufficient horizontal isolation for Ant Group's specific requirements.

WebMCL Architecture and Policy Enforcement:

To address these gaps, WebMCL was developed as a process-level sandbox based on a customized gVisor. Its architecture involves:

1. setuid Binary: WebMCL itself is a setuid binary, allowing unprivileged users to set up the necessary sandboxing environment.
2. Security Policy Input: WebMCL takes a comprehensive security policy as input, defining restrictions for processes, file systems, networking, memory, and CPU.
3. OCI Spec Generation: This policy is translated into an OCI runtime specification, which is then used to launch the gVisor environment (Sentry and Gofer processes).
4. Customized gVisor: Crucially, Ant Group customized gVisor's source code to enforce additional security policies that are not covered by the standard OCI spec. These customizations are primarily implemented within the Sentry layer.

Detailed Policy Implementation:

• Filesystem Restrictions:
• rootfs-readonly: Makes the entire root file system read-only. This is mapped directly to the OCI spec's root.readonly field.
• allow-paths: Defines specific directories or files that the sandboxed process is permitted to read or write. This is implemented via OCI mounts, allowing specific paths to be writable.
• mask-paths: Specifies paths that should be inaccessible to the sandboxed process. OCI's maskedPaths can be used here.

WebMCL leverages OCI's mounts and maskedPaths features for these policies, ensuring granular control over file system interactions.

• Process Restrictions:
• exec-before: This is a critical whitelist policy defining only the specific programs that the sandboxed process is allowed to execute. The OCI spec has no direct equivalent for this. To implement this, Ant Group modified gVisor's Sentry code. When a sandboxed application attempts to execute a new program, the Sentry intercepts the system call and checks it against the predefined whitelist. Any attempt to execute an unauthorized program (e.g., a reverse shell or an unexpected utility) is blocked. This is a crucial defense against horizontal movement.

• Network Restrictions:
• no-networking: Completely disables network access for the sandboxed process.
• allow-outgoing-ip-port: Permits connections only to specified IP addresses and ports.
• allow-outgoing-domain: Allows outgoing connections only to specified domain names (requiring DNS resolution within the sandbox).
• limit-local-listening: Restricts which local ports the sandboxed process can listen on.

The OCI spec provides limited network policy support, mainly for container network interfaces (CNI), which is too heavy for process-level sandboxing. WebMCL implements these fine-grained network policies by modifying gVisor's Sentry source code. When an application triggers a network action, Sentry intercepts it and verifies against the defined policy. If the action is not explicitly allowed, it is denied. This allows for precise control over network egress and ingress, severely limiting an attacker's ability to exfiltrate data or establish command-and-control channels.

• Other Restrictions (CPU, Memory, Devices):

For resource limitations like CPU, memory, and device access, the standard OCI runtime specification provides sufficient capabilities. WebMCL directly translates its policies for these resources into the OCI spec, which gVisor then enforces via cgroups and other host mechanisms.

Challenges Encountered:

During development, Ant Group faced several challenges:

• System Call Compatibility: Some system calls have compatibility issues or are not fully supported by gVisor, requiring workarounds or acknowledging limitations.
• Cgroup Delays: cgroup resource limits, particularly for CPU, can sometimes exhibit delays (e.g., 5 seconds) in enforcement, which is unacceptable for latency-sensitive process-level sandboxing.
• Unprivileged User Execution: Ensuring that unprivileged users can effectively use WebMCL and that sandboxed processes run with appropriate user IDs within the gVisor environment.
• Kernel Fragmentation: Dealing with a diverse range of Linux kernel versions (3.x, 4.x, 5.x) in production environments. WebMCL includes an option to revert to the original command execution if gVisor is not supported on a particular kernel version.

To address performance-critical scenarios where gVisor's overhead might be too high, Ant Group also developed a traditional container runtime (based on namespaces and cgroups) called Long-LC. This provides a performance-focused alternative, while WebMCL (VM-based) remains the security-focused option. For future network policy enforcement, they are exploring user-space network stacks and packet filtering (similar to CNI but at a lower level) to achieve robust network isolation without heavy kernel modifications.

Demo / Proof of Concept

▶ Watch: Implementing fine-grained network policies in webmgl (30:20)

The talk included several live demonstrations showcasing WebMCL's capabilities and policy enforcement. These examples effectively illustrated how the customized gVisor environment intercepts and blocks unauthorized actions, providing concrete evidence of the sandbox's effectiveness.

1. GVisor Kernel Output (dmesg):

The first demonstration involved executing dmesg within the WebMCL sandbox. Instead of displaying the host kernel's messages, the output showed messages from gVisor's internal guest kernel. This visually confirmed that the application was indeed running within the gVisor environment, interacting with the application kernel rather than the host's. This is a foundational proof point that the vertical isolation is active.

1. Read-Only Root Filesystem (rootfs-readonly):

A core security policy is to make the root file system read-only. The speaker demonstrated this by attempting to create a new file (e.g., using touch) in a restricted directory within the sandboxed environment. As expected, the operation failed, indicating that the rootfs-readonly policy was successfully enforced and prevented any unauthorized writes to the file system.

1. Allowed and Masked Paths (allow-paths & mask-paths):

This demo highlighted the granularity of file system control.

• The speaker showed that attempts to touch or modify files in general directories outside of explicitly allowed paths would fail.
• Conversely, if a specific path was defined in the allow-paths policy, the sandboxed process could successfully create or modify files within that designated directory.
• For paths defined as mask-paths, any attempt to read or write to them would result in an error or an empty output, demonstrating their complete inaccessibility. This illustrates how WebMCL can create a tightly controlled file system view for untrusted applications.

1. Network Access Restrictions (allow-outgoing-domain & no-networking):

Network policies were demonstrated using the curl command.

• When attempting to curl a domain that was not explicitly whitelisted in the allow-outgoing-domain policy, the command failed with a "connection refused" or "cannot connect" error message. The error output often included specific messages from WebMCL or gVisor, confirming the interception and denial by the sandbox's network policy enforcement.
• Conversely, when curl was directed to a whitelisted domain (e.g., npm.taobao.org), the connection succeeded, and data could be retrieved. This demonstrated the fine-grained control over outgoing network connections.
• A final network demo showed the no-networking policy in action: attempting any curl command within a sandbox configured with no-networking resulted in immediate failure, confirming complete network isolation.

These demonstrations provided compelling evidence that WebMCL successfully implements and enforces its custom process, file system, and network policies atop gVisor's robust vertical isolation. The ability to intercept and block actions that violate these policies, even at the granular level of specific domains or execution paths, validates the solution's effectiveness in securing untrusted applications.

Defensive Implications

▶ Watch: Future: Balancing security, performance, and malware analysis (40:40)

The insights presented in this talk carry significant defensive implications for any organization dealing with untrusted code, third-party software, or legacy applications in production environments. The fundamental takeaway is that traditional Linux sandboxing mechanisms are often insufficient for modern threat landscapes, primarily due to their shared kernel architecture and lack of granular network controls.

Here's how defenders can leverage this information:

1. Re-evaluate Existing Sandbox Solutions: Defenders should critically assess their current sandboxing strategies. If these rely solely on seccomp, namespaces, chroot, or cgroups without an underlying application kernel like gVisor, they are inherently vulnerable to kernel exploits, leading to vertical escapes. The ease with which new kernel vulnerabilities are discovered means a shared kernel is a ticking time bomb.

1. Prioritize Vertical Isolation: The talk underscores the paramount importance of strong vertical isolation. Solutions that introduce a robust isolation layer between the untrusted application and the host kernel (like gVisor) should be strongly considered. This significantly reduces the attack surface for kernel-level exploits.

1. Demand Fine-Grained Horizontal Controls: Beyond vertical isolation, defenders must ensure their sandboxes offer granular control over process execution, file system access, and especially network interactions. The ability to whitelist specific executables, define read/write paths, and restrict network connections to specific IPs, ports, or domains is crucial for preventing lateral movement, data exfiltration, and command-and-control communications from a compromised sandbox. Standard container runtimes (even OCI-compliant ones) may not provide this level of detail out-of-the-box.

1. Consider Customization for Specific Needs: As demonstrated by Ant Group's WebMCL, off-the-shelf solutions might not meet all enterprise security requirements. Defenders should be prepared to customize open-source technologies like gVisor to implement specific, fine-grained policies that are critical to their threat model. This involves understanding the underlying architecture (e.g., gVisor's Sentry) and potentially modifying its source code or extending its policy capabilities.

1. Adopt a Hybrid Runtime Strategy: The concept of having both a security-focused (VM-based like WebMCL) and a performance-focused (traditional container like Long-LC) runtime is a pragmatic defensive strategy. This allows organizations to apply the highest level of security to the most critical or untrusted workloads, while using more performant but less isolated solutions for trusted or less sensitive applications.

1. Leverage for Malware Analysis: gVisor's ability to intercept and log every system call, file system access, and network action makes it an excellent platform for malware analysis. Defenders can use a gVisor-based sandbox to observe the behavior of suspicious binaries in a highly controlled and isolated environment, gaining insights into their functionality without risking host compromise. This can significantly enhance incident response and threat intelligence capabilities.

1. Address Operational Challenges: Defenders should also account for the operational challenges highlighted, such as managing different kernel versions, ensuring compatibility, and handling potential performance overheads. A successful implementation requires balancing security posture with operational realities.

In summary, the talk serves as a call to action for defenders to move beyond traditional, kernel-sharing sandboxing approaches. By embracing application kernels like gVisor and customizing them for specific horizontal policy enforcement, organizations can build a far more resilient and secure execution environment for untrusted code, significantly raising the bar for attackers.

Key Takeaways

• Traditional Linux sandboxing is insufficient: Existing solutions often share the host kernel, creating a critical vulnerability for vertical escapes, and lack fine-grained network policies, making horizontal movement difficult to prevent.
• gVisor offers strong vertical isolation: Google's gVisor, an application kernel written in Go, provides excellent isolation by intercepting system calls in user space, thereby protecting the host kernel from compromised applications.
• gVisor needs customization for process-level sandboxing: While gVisor excels at container isolation, its default OCI spec implementation lacks granular policies for process execution, specific file system access, and fine-grained network control required for robust process-level sandboxing.
• WebMCL provides comprehensive horizontal security: Ant Group's WebMCL customizes gVisor by modifying its Sentry component to enforce precise policies for whitelisting executables, defining read/write file paths, and restricting network connections to specific IPs, ports, or domains.
• A hybrid approach balances security and performance: Organizations can deploy a security-focused, VM-based runtime (like WebMCL) for untrusted or critical workloads and a performance-focused, traditional container runtime for less sensitive applications.
• gVisor is valuable for malware analysis: Its capability to intercept and log all system calls, file system, and network activities makes gVisor an ideal platform for safe and detailed behavioral analysis of malicious software.

About the Speaker(s)

Li Qiang is a Platform Security Engineer at Ant Group, a major financial technology company. His professional focus revolves around security research and development, particularly in low-level systems. His areas of interest include kernel internals, virtualization technologies, container security, and cloud-related security. Before joining Ant Group, Li Qiang engaged in offensive security work, specializing in vulnerability discovery and exploitation. Since transitioning to Ant Group, his work has shifted to defensive security, focusing on protecting the company's internal network and infrastructure. He has a history of presenting at security conferences, including a previous talk at HITB in 2017.