eBPF and Wasm: Unifying Userspace Extensions With Bpftime - Yusheng Zheng, eunomia-bpf

eBPF and Wasm: Unifying Userspace Extensions With Bpftime

Speakers: Yusheng Zheng, eunomia-bpf

Conference: KubeCon EU

YouTube: https://www.youtube.com/watch?v=W5C0O7vk78o

Overview

In this KubeCon EU talk, Yusheng Zheng, a PhD student and maintainer of several eBPF-related open-source projects under the eunomia-bpf organization, introduced Bpftime, an innovative userspace eBPF runtime designed to address long-standing challenges in software extensions. The presentation delved into the fundamental trade-offs between flexibility, isolation, and performance that plague traditional extension frameworks, proposing a novel approach that leverages the strengths of both eBPF (extended Berkeley Packet Filter) and WebAssembly (Wasm).

The core problem Bpftime seeks to solve revolves around the inherent difficulty in allowing software to be extended and customized (interconnectivity) without compromising the host application's safety or introducing significant performance overhead. Zheng highlighted that while extensions offer immense adaptability, they also introduce security vulnerabilities, performance degradations, and system instability if not properly isolated and managed. Bpftime aims to provide a robust framework that enables high-performance, safe, and flexible userspace extensions by unifying eBPF's powerful static verification capabilities with Wasm's flexible, capability-based interface model. This research, recently accepted into OSDI, represents a significant step towards more secure and efficient software extensibility in modern cloud-native environments.

Background

▶ Watch: Introduction to Bpftime and the challenge of extensions (0:00)

Software extensions have been a cornerstone of application design for decades, ranging from web server modules and database plugins to modern cloud-native components and integrated development environment extensions like VS Code plugins. The primary drivers for adopting extensions are flexibility and customization, allowing users and administrators to tailor software to specific needs without requiring core developers to implement every conceivable feature. However, this flexibility comes with a significant caveat: isolation. Extensions, often developed by third parties or external teams, can introduce bugs, performance regressions, or critical security vulnerabilities that compromise the entire host application.

Zheng underscored this risk with compelling examples. A popular video streaming service once suffered a serious production outage due to an engine extension getting stuck in an infinite loop. Apache HTTP server has seen similar issues, with buffer overflow bugs in Lua-based modules leading to crashes and security compromises. To quantify this, a study of CVE reports from various open-source projects revealed that over 1,000 CVEs related to extensions existed among 17,000 total CVEs, representing approximately 7% of all vulnerabilities. A significant portion of these could lead to system crashes or data leakage, emphasizing the critical need for robust isolation and safety mechanisms.

Any effective extension runtime framework must balance three core requirements: interconnectivity, safety, and efficiency. Interconnectivity refers to the extent to which an extension can interact with the host application – reading data, modifying state, or calling host functions. Safety defines the limits on an extension's ability to harm the main application, ensuring that a bug in an extension doesn't crash the entire system. Efficiency, finally, concerns the performance overhead introduced by the extension framework. The fundamental challenge lies in the inverse relationship between interconnectivity and safety: increasing one often compromises the other. Achieving a balance while maintaining high efficiency is the holy grail of extension framework design.

Existing extension frameworks exhibit various limitations in addressing this trade-off:

• Native Execution (e.g., LD_PRELOAD, dynamic modules): Offers excellent performance and simple integration but provides virtually no isolation. A single bug can crash the entire application.
• Software/Hardware Isolation (e.g., WebAssembly, RLU): Provides better isolation through runtime checks but introduces performance overhead. Often relies on the host application to correctly implement security boundaries, which can be buggy.
• Subprocess/RPC-based Approaches: Offers strong isolation by running extensions in separate processes, but suffers from significant context switching overhead, making them unsuitable for performance-critical applications.
• Verifier-based Approaches (e.g., Kernel eBPF uprobe): Provides fine-grained control and is tightly coupled with the kernel's security model. However, each extension call requires a costly kernel context switch, making it inefficient for high-frequency hooks in userspace.

These frameworks either allow too much interconnectivity without sufficient safety or provide strong safety through heavy isolation at the cost of performance and flexibility. The talk posits that the key to managing this trade-off effectively lies in the interface chosen for extensions. A well-defined interface can precisely control what resources, functions, and capabilities an extension can access, adhering to the principle of least privilege.

Key Findings

▶ Watch: The need for flexibility and isolation in software extensions (2:00)

The central insight presented in the talk is that the optimal way to manage the inherent trade-off between interconnectivity and safety in software extensions, while maintaining high efficiency, is to combine the strengths of eBPF's static verification with WebAssembly's flexible, capability-based interface model. This forms the foundation of Bpftime, a novel userspace eBPF runtime.

Bpftime's core contributions and findings include:

1. Unified Approach: It proposes treating all extension-host interactions as "split capabilities," inspired by the WebAssembly Component Model. This allows for precise, fine-grained control over what an extension can do, adhering strictly to the principle of least privilege.
2. Verifier-Enforced Safety: Unlike Wasm's reliance on runtime checks and manual host-side security implementations, Bpftime leverages the eBPF verifier. This means that security and safety properties are checked before the extension runs (at load time), significantly minimizing runtime overhead and eliminating entire classes of bugs (e.g., memory safety, infinite loops).
3. High Performance: By adopting eBPF's verifier-based approach, Bpftime achieves performance comparable to native execution for many tasks, avoiding the overheads associated with runtime checks, data copying across boundaries, or context switches inherent in other isolation methods. For instance, microbenchmarks showed approximately a 10x improvement in uprobe performance compared to kernel eBPF for userspace tracing.
4. Flexible Interfaces: While using eBPF's verification, Bpftime aims to overcome eBPF's traditional expressiveness limitations by incorporating advanced interface specifications. These specifications can encode not just simple capability types (state, function, entry points) but also binary relationships between arguments and return values, high-level semantic facts, and bitwise operation constraints, all checked by the eBPF verifier.
5. eBPF Ecosystem Compatibility: Bpftime is designed to be compatible with the existing eBPF ecosystem, including libbpf, various eBPF map types, and helper functions. This allows developers to leverage familiar tools like BCC and BPFtrace in userspace, extending their utility beyond kernel tracing.
6. Addressing Userspace Tracing Needs: Bpftime provides a highly efficient and flexible solution for userspace tracing and observability, offering significant advantages over traditional uprobe mechanisms by eliminating kernel context switch overhead and providing faster userspace memory access.

In essence, Bpftime's key finding is that a carefully designed interface, enforced by a powerful static verifier, can simultaneously deliver the high performance and low overhead of eBPF with the fine-grained, flexible security controls inspired by Wasm's component model, thereby resolving the long-standing tension between interconnectivity, safety, and efficiency in software extensions.

Technical Deep Dive

▶ Watch: Real-world examples of extension-related bugs and security risks (3:00)

The technical core of Bpftime lies in its innovative approach to defining and enforcing extension interfaces, drawing inspiration from both the WebAssembly Component Model and the advanced eBPF interface mechanisms found in the Linux kernel.

WebAssembly's Interface Approach:

The WebAssembly Component Model addresses the challenge of software extensibility by defining a specification for how Wasm modules can be composed into components and interact with their host environments. Key goals include portable interfaces across languages, capability safety, and virtualization. It introduces WIT (WebAssembly Interface Types) to define well-structured interfaces between components and the host. Security is primarily achieved through capability-based security using resource handlers, which are unforgeable references granting access to specific resources. These handlers can be passed between components, enabling fine-grained access control based on the principle of least privilege. For example, direct interface definitions might specify how components interact via request and response resources, which can only be accessed through defined functions. However, Wasm's approach typically relies on runtime checks at interface boundaries and often involves data copying, leading to performance overhead, especially for extensions that frequently interact with the host.

eBPF's Interface Approach in the Kernel:

eBPF, by contrast, prioritizes performance, achieving safety primarily through load-time verification rather than runtime checks. The eBPF verifier statically analyzes all possible execution paths to guarantee memory safety, termination (preventing infinite loops), and adherence to security policies.

Initially, eBPF's kernel interface was somewhat rigid, based on a limited set of helper functions, program types, and attach types. These provided a form of capability control but were hardcoded and lacked extensibility. As eBPF evolved, the kernel community introduced more flexible interface mechanisms:

• struct_ops: Allows users to register new eBPF program types that can be called by the kernel. This is similar to Wasm's approach of defining host functions that components can invoke.
• kfuncs: Enables eBPF programs to call specific, whitelisted kernel functions. This is akin to an "import" mechanism, allowing eBPF programs to leverage existing kernel functionality without requiring kernel modifications for each new helper.
• BTF (BPF Type Format): Provides rich type information for eBPF programs, maps, and kernel data structures. This is crucial for the verifier to perform precise type checking and understand memory layouts, similar to how WIT functions for Wasm.
• Annotations and Flags: Kernel kfuncs can be registered with specific annotations (e.g., __acquire, __release) and flags (e.g., sleepable, destructive) that provide semantic information to the verifier, allowing it to enforce more complex resource management and safety policies.

While eBPF offers strong verifier-based security and minimal runtime overhead, its expressiveness has traditionally been limited, making it challenging to define highly fine-grained, semantic-rich safety policies directly within the eBPF instruction set. It is also tightly coupled with the kernel.

Bpftime's Extension Interface Model:

Bpftime's innovation is to bridge this gap. It proposes an "Extension Interface Model" that combines the best of both worlds:

1. Split Capabilities: Like Wasm, Bpftime treats all extension-host interactions as "split capabilities." These include:

• State Capabilities: For reading/modifying host application variables.
• Function Capabilities: For calling host application functions.
• Extension Entries: Entry points for the host to invoke extension logic.

1. Verifier-Enforced Constraints: Instead of Wasm's runtime checks, Bpftime leverages the eBPF verifier to enforce these capabilities at load time. This is achieved by generating eBPF interface specifications through static analysis of the host application's code. These specifications can encode advanced constraints, such as:

• Binary relationships between arguments and return values.
• High-level semantic facts about function behavior.
• Bitwise operations over constraints.

This allows for much richer and more precise safety policies than traditional eBPF helper functions, all verified statically.

Bpftime Architecture:

Bpftime itself is a userspace eBPF runtime designed for compatibility with the kernel eBPF ecosystem. It's not merely a VM but a comprehensive runtime environment.

• libbpf Compatibility: Bpftime supports a wide range of existing eBPF features, including various eBPF map types (shared memory maps, userspace/kernel shared maps) and helper functions. This means existing eBPF tools and programs designed for the kernel can often run with minimal modifications in Bpftime.
• Userspace Attach Points: It supports userspace tracing types like uprobe and tracepoints (for userspace functions), as well as xdp and even GPU features recently added by the community.
• Dual Verifier Support: Bpftime can utilize both the kernel eBPF verifier (by running a minimal kernel in a VM, allowing it to verify programs against a specific kernel version like 6.10+) or its own userspace verifier. The userspace verifier offers greater flexibility, allowing Bpftime to operate independently of the host kernel's eBPF support or version.
• Modular Design: The Bpftime design diagram illustrates its modularity. White components represent standard eBPF elements, orange components are new additions by Bpftime (e.g., loader, specific VM features), and arrows depict execution and interaction flows. The Bpftime loader is central to loading and operating eBPF programs, interacting with a control plane application.
• VM Implementation: The Bpftime VM is built upon llvm-bpf, providing a robust and performant execution environment for eBPF bytecode in userspace.

By integrating these elements, Bpftime offers a powerful paradigm shift: providing the security and performance benefits of eBPF's static verification while simultaneously offering the interface flexibility and fine-grained control traditionally associated with more abstract models like the WebAssembly Component Model.

Demo / Proof of Concept

▶ Watch: Defining the three core requirements: interconnect, safety, efficiency (5:20)

While the talk did not feature a live, interactive demonstration, Yusheng Zheng provided several compelling examples and use cases to illustrate Bpftime's capabilities and performance advantages. The core mechanism for running eBPF programs with Bpftime involves a simple command-line interface: bpftime load <control_plane_application> <bpf_program>.

One of the primary demonstrated use cases was userspace observability and tracing, particularly with uprobe functions for monitoring userspace micro-functions. Bpftime offers significant benefits here:

• Speed and Flexibility: It's considerably faster than kernel-based uprobe for userspace functions, primarily because it avoids the costly kernel context switches. This results in much faster userspace memory access.
• No Overhead on Untraced Processes: Bpftime's userspace tracing mechanisms ensure that only the targeted processes incur overhead, leaving untraced applications unaffected.
• Tool Compatibility: Existing eBPF tools like BCC and BPFtrace can be run within the Bpftime environment with minimal fixes. Examples given included retest_counter and ssl_sniff tools, which can now operate in userspace with reduced overhead compared to their kernel eBPF counterparts.
• Microbenchmarks: Preliminary benchmarks indicated approximately a 10x improvement in uprobe performance when run with Bpftime compared to traditional kernel uprobe. Improvements were also observed in hash map operations and userspace memory access.

Beyond observability, other use cases highlighted Bpftime's versatility:

• Userspace Networking: Bpftime can be combined with technologies like DPDK and XDP (Extended Berkeley Packet Filter) to build high-performance userspace network applications, leveraging the existing eBPF ecosystem for enhanced data plane programmability and efficiency.
• Application Plugins: Examples such as Nginx plugins were mentioned, where Bpftime could provide a secure and performant way to extend web server functionality without the risks associated with native modules or the overhead of Wasm runtime checks.
• FUSE Caches and Redis Durability Tuning: These illustrate Bpftime's potential in improving performance and reliability for critical infrastructure components by allowing custom logic to be injected safely and efficiently.
• GPU Tracing and Error Injection: Advanced use cases like tracing GPU operations and injecting errors for fault tolerance testing further demonstrate the broad applicability of Bpftime's flexible and performant extension framework.

The speaker acknowledged that Bpftime is "not production ready yet" and requires further work on stability, bug fixes, and ease of use. However, the presented examples and performance figures strongly validate its potential as a powerful framework for future userspace extension development.

Defensive Implications

▶ Watch: Overview of current extension framework limitations and tradeoffs (7:10)

Bpftime presents several significant defensive implications for organizations and developers grappling with the security and performance challenges of software extensions:

• Enhanced Security Posture for Extensions: By enforcing capability-based security through a rigorous pre-runtime verifier, Bpftime drastically reduces the attack surface associated with third-party or untrusted extensions. Malicious or buggy code is prevented from executing operations outside its defined capabilities, mitigating risks like arbitrary memory access or unauthorized function calls.
• Improved System Stability and Uptime: The robust isolation provided by Bpftime ensures that a fault or bug within an extension is far less likely to crash the entire host application. This translates directly to increased system stability, reduced production outages, and higher availability for critical services.
• Prevention of Data Leakage: Fine-grained control over resource access, enforced by the verifier, means that extensions can only access the data they are explicitly permitted to. This significantly reduces the risk of sensitive internal data being exposed to attackers through compromised or poorly written plugins.
• Performance Without Compromise: Defenders no longer have to make a difficult choice between robust security and high performance for their extension frameworks. Bpftime offers both, allowing organizations to adopt powerful, flexible extensions without incurring the performance penalties often associated with strong isolation mechanisms like subprocesses or extensive runtime checks.
• Safer Customization and Innovation: Bpftime provides a secure sandbox for innovation. Developers can rapidly integrate new functionalities and customizations into their applications, knowing that the core system integrity is protected by strong, verifiable boundaries. This fosters agility while maintaining a strong security posture.
• Leveraging Existing eBPF Expertise: Organizations already utilizing kernel eBPF for security, observability, or networking can easily extend their expertise and toolchains to userspace with Bpftime. This provides a familiar and powerful paradigm for securing and optimizing userspace applications, expanding the reach of eBPF's proven security model.
• Reduced Manual Security Burden: Unlike frameworks that rely heavily on the host application to implement correct security checks (e.g., some Wasm scenarios), Bpftime's verifier-based approach offloads much of this burden. The security properties are enforced by the runtime itself, reducing the likelihood of human error in implementing security boundaries.
• Future-Proofing Extension Architectures: As software becomes more modular and distributed, Bpftime offers a forward-looking architecture for managing extensions that can adapt to evolving threats and performance demands, providing a robust foundation for building resilient and secure applications.

In essence, Bpftime empowers defenders to safely harness the power of software extensions, transforming them from potential liabilities into secure, high-performance assets.

Key Takeaways

• Unified Approach: Bpftime introduces a novel approach to userspace extensions by unifying the strengths of eBPF's static verification and WebAssembly's flexible, capability-based interface model.
• Balancing Trade-offs: It effectively addresses the long-standing challenge of balancing interconnectivity, safety, and efficiency in software extensions, a problem that traditional frameworks often fail to resolve comprehensively.
• Verifier-Enforced Safety: Bpftime leverages the eBPF verifier to enforce fine-grained, capability-based security policies at load time, minimizing runtime overhead and preventing entire classes of bugs and vulnerabilities before execution.
• High Performance: By avoiding costly runtime checks and kernel context switches, Bpftime enables high-performance userspace extensions, demonstrating up to a 10x improvement in uprobe performance for userspace tracing compared to kernel eBPF.
• eBPF Ecosystem Compatibility: The runtime is compatible with the existing eBPF ecosystem, allowing developers to adapt and utilize familiar tools like BCC and BPFtrace for userspace applications.
• Secure & Flexible Customization: Bpftime offers a robust and secure framework for extending applications, allowing for flexible customization without compromising the host system's stability or exposing it to data leakage risks.

About the Speaker(s)

Yusheng Zheng is a PhD student at Yoshi Santa Claus (as per the transcript) and plays a significant role in the open-source community. He is actively involved in maintaining several eBPF-related open-source projects under the eunomia-bpf organization. His research focuses on advancing the capabilities and applications of eBPF, particularly in areas like userspace extensibility, as demonstrated by the Bpftime project.