HIVE: A Hardware-assisted Isolated Execution Environment for eBPF on AArch64

HIVE: A Hardware-assisted Isolated Execution Environment for eBPF on AArch64

Speakers: Peihua Zhang, Chenggang Wu, Xiangyu Meng, Yinqian Zhang, Mingfan Peng, Mengyao Xie, Yuanming Lai, Yan Kang, Zhe Wang

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=8jhFEv_wK0A

Overview

The talk "HIVE: A Hardware-assisted Isolated Execution Environment for eBPF on AArch64" presented at USENIX Security '24, introduces a novel approach to enhance the security and capability of extended Berkeley Packet Filter (eBPF) programs. eBPF is a powerful in-kernel virtual machine that allows users to extend kernel functionality without modifying kernel source code or loading kernel modules, enabling dynamic and programmable kernel behavior for tasks like networking, tracing, and security. However, its tight integration with the kernel necessitates stringent security measures, primarily enforced by a software-based verifier. This talk, delivered by Peihua Zhang and his team, addresses the inherent limitations of this verification-based security model.

The core challenge lies in a dilemma: the verifier struggles to correctly identify all unsafe programs while simultaneously rejecting safe but overly complex programs, hindering eBPF's full potential. HIVE, or Hardware-assisted Isolated Execution Environment, offers a paradigm shift by leveraging AArch64 hardware features to create a robust, hardware-enforced isolation boundary for eBPF programs. This approach aims to provide strong security guarantees comparable to or exceeding the verifier's while dramatically expanding the complexity and functionality that eBPF programs can achieve, ultimately paving the way for eBPF to replace traditional kernel modules.

Background

▶ Watch: Introduction to eBPF and current security model (0:00)

eBPF operates by allowing user processes to load BPF programs into the kernel via a BPF syscall. Before execution, these programs undergo a rigorous verification process by the eBPF verifier, a critical security component designed to ensure the program causes no harm to the kernel. Once approved, the kernel's Just-In-Time (JIT) compiler translates the BPF bytecode into native machine instructions, which are then hooked to specified kernel locations and run in an event-driven manner.

The eBPF execution environment within the kernel is carefully controlled. It provides 11 registers mapped to native registers, a program context object, and supports map structures for key-value memory regions. Helper functions expand program functionality, and each program has its own stack residing on top of the kernel stack during execution. The verifier acts as the primary security barrier, performing checks based on instruction semantics and operand states, maintaining a state machine to track register types and ranges across all possible execution paths. These checks are designed to uphold 20 specific security properties, which, at a design level, translate to three core security goals:

1. Integrity: eBPF programs should only access their own memory with correct permissions.
2. Confidentiality: Information leakage (kernel layout, uninitialized data) and Spectre-type attacks must be prevented.
3. Availability: Denial-of-Service (DoS) attacks, such as infinite loops or kernel crashes due to exceptions, must be prevented, especially since preemption is disabled during eBPF execution and exception handling is not natively supported.

Despite its sophistication, the verification-based method has become a significant bottleneck for eBPF, leading to two fundamental dilemmas:

1. Correctness Dilemma: Unsafe programs can sometimes bypass verification. Analysis of eBPF-related CVEs over the last decade reveals that verifier bugs are a primary contributor to these vulnerabilities. This indicates that the software-based verifier, despite its complexity, is not infallible.
2. Capability Dilemma: Safe but complex programs are frequently rejected by the verifier. This occurs because the verifier must traverse all possible execution paths, leading to a "state exploring problem" for programs with numerous branches or unmounted loops. This severely limits the complexity and functionality eBPF developers can implement, preventing eBPF from truly replacing kernel modules.

HIVE's goal is to resolve these dilemmas by providing the same security guarantees as the verifier but through a more robust, hardware-assisted mechanism, thereby unlocking eBPF's full potential.

Key Findings

▶ Watch: HIVE's key insight: Deep privileging eBPF programs (4:05)

The central insight driving HIVE's design is to treat eBPF programs conceptually as kernel-mode applications. By drawing parallels to how user-mode applications are isolated from the kernel using hardware mechanisms, HIVE aims to apply similar principles to eBPF programs running within the kernel context.

The key findings and contributions of the HIVE project are:

• Deprivileging eBPF Programs: HIVE leverages the unprivileged load store (LSU) feature available in AArch64 architectures. These instructions are treated as if executed at Exception Level 0 (EL0), regardless of the actual Exception Level (EL1 for kernel) they are running at, effectively restricting memory access to only unprivileged pages. This forms the foundation for memory isolation.
• Independent BPF Space: To eliminate kernel layout information leakage and provide robust data isolation, HIVE introduces an independent BPF space. This dedicated memory region houses BPF data, separate from the main kernel address space.
• Exception Rollback Mechanism: To enhance availability and prevent kernel crashes from eBPF exceptions, HIVE implements an exception rollback mechanism, ensuring that program failures do not compromise kernel stability.
• Addressing Tight Coupling Challenges: Recognizing that eBPF programs are highly coupled with the kernel (e.g., accessing kernel objects, sharing stack space), HIVE provides specific solutions:
• Stack Compartmentation: The BPF stack is separated from the kernel stack.
• Object-Granular Isolation: Metadata within BPF objects is separated from accessible data.
• Secure Kernel Object Access: Specific fields of kernel objects can be securely accessed by BPF programs without compromising fine-grained protection.
• Secure Kernel Pointer Handling: For eBPF programs that legitimately need to interact with kernel pointers, HIVE utilizes ARM Pointer Authentication (PA). This hardware feature ensures pointer integrity and type safety, mitigating risks associated with pointer manipulation.
• Comprehensive Security Guarantees: HIVE is designed to meet all three security goals (integrity, confidentiality, availability) without relying solely on the verifier's complex path exploration. It inherently prevents information leakage, mitigates Spectre attacks through hardware features (leveraging CSV-3 patches), and ensures DoS resilience.
• Low Performance Overhead: Extensive evaluation demonstrates that HIVE introduces a minimal performance impact, with less than a 4% throughput slowdown when running real-world applications like Nginx and Redis, compared to vanilla eBPF.
• Enhanced Capability: HIVE significantly boosts the capability of eBPF programs. The researchers successfully compiled and ran 10 complex kernel modules as eBPF programs, all of which were previously rejected by the traditional verifier due to issues like unmounted loops or excessive branches. This illustrates HIVE's potential to enable eBPF to replace kernel modules for complex tasks.

Technical Deep Dive

▶ Watch: Core design: AArch64 unprivileged load/store for isolation (4:45)

HIVE's technical design revolves around deprivileging eBPF programs and establishing robust hardware-enforced isolation. The core idea is to treat eBPF programs as unprivileged entities, even though they run within the kernel's Exception Level 1 (EL1).

The first step is to leverage the unprivileged load store (LSU) feature on AArch64. These instructions, when executed, are treated by the Memory Management Unit (MMU) as if they originated from EL0, regardless of the actual current EL. This means any memory access attempted via an LSU instruction will only succeed if the target memory page has unprivileged permissions. This mechanism forms a strong hardware-enforced barrier, preventing direct, unauthorized access to privileged kernel memory by the eBPF program. During the JIT compilation stage, all memory access instructions within the eBPF program are transformed into their LSU equivalents.

To further enhance isolation, HIVE introduces an independent BPF space. This is a dedicated address space specifically for BPF data, distinct from the kernel's main address space. This design eliminates the possibility of kernel layout information leakage to eBPF programs. The BPF space is initialized during program loading, and all BPF-used registers are cleared upon return from helper functions, further reducing information exposure. To prevent unauthorized access between user space and the BPF space, HIVE utilizes the eODP (Enhanced Open Data Path) feature, ensuring strict isolation between these domains.

Addressing the tight coupling between eBPF programs and the kernel, HIVE implements several crucial modifications:

• Stack Compartmentation: The eBPF program's stack, which traditionally resides on top of the kernel stack and contains sensitive data like local variables and return addresses, is moved. HIVE switches the BPF stack pointer to a specific, separate memory region, effectively segmenting the BPF stack from the kernel stack. This prevents eBPF programs from directly manipulating or leaking information from the kernel stack. Similar separation techniques are applied to the metadata of other BPF objects.
• Object-Granular Isolation: BPF objects often embed metadata, such as pointers, alongside accessible data. HIVE ensures that this metadata is separated and inaccessible to the BPF program, providing a more granular form of self-page protection that basic memory isolation cannot achieve.
• Secure Kernel Object Access: While deprivileging, eBPF programs still need legitimate access to specific fields within kernel objects (e.g., network packet headers). HIVE's design accommodates these legal accesses by transforming memory access instructions to ensure they comply with permissions, effectively providing fine-grained protection without blocking necessary interactions.

HIVE also directly addresses the three security goals:

• Confidentiality: The independent BPF space inherently prevents kernel layout information leakage. Clearing registers upon helper function returns further mitigates information exposure.
• Spectre Attacks: By transforming all memory access instructions to LSUs, HIVE effectively converts potential Spectre attacks into Meltdown-style attacks. Crucially, it leverages the CSV-3 hardware patch (Cache Speculation Vulnerability 3), which forbids speculatively loaded data that results in a permission fault from being used to form an address. This hardware-level mitigation means HIVE gains Spectre protection "without any efforts" from its core design.
• Availability: HIVE implements exception capture to prevent eBPF program exceptions from causing kernel crashes. Additionally, execution timing mechanisms are in place to prevent non-terminating or excessively long program executions, thus mitigating DoS attacks caused by infinite loops.

A significant challenge arises when eBPF programs need to handle kernel pointers, as these pointers reference privileged kernel memory and thus require regular (privileged) load/store instructions, bypassing the LSU-based deprivileging. HIVE's insight here is that these kernel pointers, once passed to an eBPF program, should not be modified, and their dereferenced targets must be exclusively for the eBPF program's intended functionality. To ensure the integrity and type safety of these pointers, HIVE employs ARM Pointer Authentication (PA). PA uses cryptographic hashes (pointers are "signed") to detect unauthorized modifications or fabrications of pointers. While the talk acknowledged time limitations and did not delve into the specifics of identifying memory access to PA-protected objects, preventing attacks against PA (like replace attacks or Spectre against signed pointers), or preventing signed pointer leakage, it highlights the importance of this hardware feature for secure kernel interaction.

Demo / Proof of Concept

▶ Watch: Securing kernel pointers with ARM Pointer Authentication (8:20)

While the talk did not feature a live, interactive demonstration of HIVE, the evaluation section serves as a comprehensive proof-of-concept, showcasing its capabilities and performance. The researchers conducted a three-part evaluation:

1. Security Property Attacks and Mitigation: The team specifically conducted attacks targeting the 20 security properties that the traditional eBPF verifier aims to enforce. HIVE successfully mitigated all these attacks, demonstrating its robust security posture. This implies a rigorous testing framework where known vulnerabilities or attack patterns against eBPF were used to validate HIVE's defensive mechanisms.
2. Performance Evaluation: Over 100 existing eBPF programs were loaded into the kernel, and real-world applications like Nginx (a web server) and Redis (an in-memory data store) were run to measure the throughput slowdown. The results were highly positive, showing that HIVE introduces less than a 4% throughput slowdown compared to vanilla eBPF. This indicates that HIVE's hardware-assisted isolation comes with a negligible performance overhead, making it practical for production environments.
3. Complexity Evaluation (Capability Enhancement): This was a crucial demonstration of HIVE's ability to overcome the verifier's "capability dilemma." The researchers compiled 10 complex kernel modules into eBPF programs. All these modules were previously rejected by the standard eBPF verifier due to their inherent complexity, such as containing unmounted loops or too many branches. HIVE successfully ran all 10 of these complex programs, proving its capacity to handle code that is as intricate as traditional kernel modules. This result directly supports the claim that HIVE can enable eBPF to replace kernel modules, significantly expanding its utility.

These evaluations collectively serve as a strong proof-of-concept, illustrating HIVE's effectiveness in enhancing both the security and capability of eBPF programs with minimal performance cost.

Defensive Implications

▶ Watch: Evaluation: Security properties, throughput, and performance (9:00)

HIVE presents significant defensive implications for kernel developers, system administrators, and security architects. By shifting eBPF security from a purely software-based verification model to a hardware-assisted isolation environment, HIVE offers a more robust and scalable defense strategy.

For kernel developers, HIVE provides a pathway to safely extend eBPF's capabilities without being constrained by the verifier's limitations. This means they can write more complex, feature-rich eBPF programs, potentially allowing eBPF to genuinely replace traditional kernel modules for many tasks. This could lead to a more dynamic, secure, and maintainable kernel ecosystem, as eBPF programs are easier to update and deploy than full kernel modules. The ability to run complex logic within eBPF, previously only possible in kernel modules, opens doors for advanced in-kernel security monitoring, network filtering, and performance tracing tools.

For system administrators and security engineers, HIVE instills greater confidence in deploying and utilizing complex eBPF programs. The hardware-enforced isolation provides stronger guarantees against misbehaving or malicious eBPF code, reducing the risk of kernel crashes, information leakage, or privilege escalation. The mitigation of Spectre attacks through hardware patches (CSV-3) integrated with HIVE's design adds another layer of robust defense against sophisticated side-channel attacks. The exception capture and execution timing mechanisms also improve system stability by preventing DoS scenarios originating from eBPF programs. This allows organizations to leverage the power of eBPF for critical infrastructure without compromising the integrity or availability of their systems.

Ultimately, HIVE demonstrates a crucial architectural shift in securing in-kernel extensibility. It highlights the increasing importance of leveraging specific hardware features like AArch64's LSU, ARM Pointer Authentication, and specific CPU vulnerability patches (CSV-3) to build fundamentally more secure execution environments. This approach moves beyond simply "detecting bad code" to "preventing bad code from doing harm" through architectural isolation, offering a more resilient and future-proof defense against evolving threats.

Key Takeaways

• The traditional eBPF verifier faces a "correctness dilemma" (unsafe programs bypassing checks) and a "capability dilemma" (safe but complex programs being rejected).
• HIVE introduces a hardware-assisted isolated execution environment for eBPF on AArch64, drawing inspiration from user-mode application isolation.
• It leverages specific AArch64 hardware features, including unprivileged load store (LSU) instructions for memory isolation and ARM Pointer Authentication (PA) for secure kernel pointer handling.
• HIVE establishes an independent BPF space and implements stack compartmentation to eliminate kernel layout information leakage and provide robust data isolation.
• The system inherently mitigates Spectre attacks by transforming them into Meltdown-style attacks and utilizing CSV-3 hardware patches, while also preventing DoS attacks via exception capture and execution timing.
• Evaluation shows HIVE achieves strong security guarantees with a minimal performance overhead of less than 4% throughput slowdown and significantly enhances eBPF program capability, allowing it to run complex code previously only possible in kernel modules.

About the Speaker(s)

The talk "HIVE: A Hardware-assisted Isolated Execution Environment for eBPF on AArch64" was presented by Peihua Zhang, alongside co-authors Chenggang Wu, Xiangyu Meng, Yinqian Zhang, Mingfan Peng, Mengyao Xie, Yuanming Lai, Yan Kang, and Zhe Wang. While specific titles and affiliations were not detailed in the transcript, the depth and innovation of the research suggest their expertise in systems security, operating systems, and hardware-assisted security mechanisms, particularly within the context of AArch64 architectures and kernel technologies like eBPF. Their collective work on HIVE demonstrates a significant contribution to advancing the security and capabilities of in-kernel programmability.