CAGE: Complementing Arm CCA with GPU Extensions

CAGE: Complementing Arm CCA with GPU Extensions

Speakers: Chenxu Wang

Conference: NDSS Symposium

YouTube: (no public video)

Overview

Confidential computing stands as a cornerstone in modern data security, promising isolated and transparent execution environments that shield sensitive data from a spectrum of threats, including malicious applications, untrusted clients, and even cloud infrastructure providers. While major industry players like Intel, AMD, and IBM have introduced their own hardware primitives—such as Intel TDX, AMD SEV, and IBM PEF—Arm, a dominant force in the semiconductor industry, has responded with its Confidential Compute Architecture (CCA) and Realm Management Extensions (RME). Arm CCA introduces "realms" as the fundamental unit of a confidential environment, isolated by a lightweight hypervisor known as the Realm Management Monitor (RMM). However, a significant gap persists in Arm's confidential computing vision: the lack of robust, hardware-backed support for GPUs, which are indispensable for high-performance computing and AI workloads.

Currently, Arm CCA largely characterizes GPUs as untrusted peripherals, meaning data processed on them is not guaranteed to be secure. If confidential realms utilize GPUs, sensitive data could be compromised by an adversary controlling the GPU via a malicious driver or other software. Existing solutions often fall short by relying on hardware primitives incompatible with next-generation Arm devices, introducing overly large trusted computing bases (TCBs), or lacking compatibility with CCA's realm-style architecture. This critical challenge—how to extend secure GPU acceleration to Arm CCA without requiring hardware modifications—is precisely what the CAGE (Complementing Arm CCA with GPU Extensions) project addresses.

CAGE proposes a novel system to enable confidential GPU computing for Arm CCA. Its core objectives are to leverage existing Arm CCA security features to ensure data security on unified-memory GPUs, which are prevalent in Arm devices. CAGE introduces a unique shadow task mechanism to seamlessly integrate the GPU workflow into CCA's realm-style architecture. Furthermore, it harnesses Arm CCA's powerful Granule Protection Check (GPC) for memory isolation, safeguarding data confidentiality and integrity against even the most potent adversaries. By optimizing security operations, CAGE mitigates performance overhead, achieving an impressive average overhead of just 2.45% in evaluations, all while requiring no hardware changes to next-generation Arm devices.

Background

[▶ Watch: Problem: Untrusted GPU in Arm CCA; CAGE's solution (0:00)]()

To appreciate CAGE's innovations, it's essential to understand the foundational Arm security architectures and the landscape of GPU Trusted Execution Environments (TEEs). Arm's journey into hardware-based security began with TrustZone, which established a dichotomy between a "normal world" for conventional operating systems and applications, and a "secure world" for sensitive components. Resources were partitioned, and a Secure Monitor facilitated world switching. However, TrustZone's model proved vulnerable, as attackers could compromise secure world components or even the Secure Monitor itself.

The Armv9 architecture marked a significant evolution, introducing the Confidential Compute Architecture (CCA) and Realm Management Extensions (RME). CCA maintains the normal and secure worlds but adds a new "realm world" for confidential realms, isolated by a lightweight Realm Management Monitor (RMM). Crucially, Armv9 also introduced a "root world" to host the highest-privilege Monitor code, securely stored in firmware. A cornerstone of CCA's security is the Granule Protection Check (GPC), a novel hardware-assisted memory isolation mechanism. GPC provides flexible, fine-grained access control across the entire main memory. When any software component attempts to access a physical address, GPC consults the Granule Protection Table (GPT) to validate the access based on the memory region's security attributes. For robust security, CCA mandates that the GPT resides in the root Physical Address Space (PAS) and only permits the Monitor to configure GPC registers. CAGE leverages the inherent security of this hardware-isolated root world and the GPC mechanism to deploy its security modules within the Monitor, thereby controlling access to GPU memory and registers.

Mainstream Arm GPUs typically share a unified memory with the CPU and other peripherals, a design that differs from discrete GPU architectures. The GPU software stack, encompassing kernel-layer drivers and user-layer runtimes, manages the GPU computation environment, handling tasks like memory allocation, buffer creation, and GPU register configuration for Direct Memory Access (DMA). To manage DMA-capable peripherals like GPUs, Arm developed the System Memory Management Unit (SMMU). Similar to a CPU's Memory Management Unit (MMU), the SMMU performs address translations (Stage-1 and Stage-2) to control peripheral access to physical memory. Critically, SMMU also supports GPC in Arm CCA, with dedicated SMMU MMIO registers accessible only to the root world Monitor, enabling GPC configurations, fault handling, and TLB invalidation for peripherals. CAGE exploits the fact that a peripheral's memory access is subject to GPC, allowing it to enforce GPU environment isolation. Furthermore, the observation that most GPU software functions are data-independent allows CAGE to keep these components in the untrusted host while performing data-dependent functions within a minimal Trusted Computing Base (TCB).

CAGE operates under a strong threat model, consistent with Arm CCA's assumptions. The adversary is presumed to control the entire software stack in both the normal and secure worlds, including the GPU software, untrusted operating system, hypervisor, and any same-layer software in the secure world. This adversary's goals include leaking or tampering with sensitive data of confidential GPU tasks, either by direct access to unified memory or by controlling DMA-capable peripherals. The adversary may also attempt to break the isolated execution environment by compromising GPU software, altering task execution order, or modifying GPU registers. CAGE also considers malicious GPU tasks attempting to access data belonging to other realms. While cold-boot attacks can be defended against using Arm CCA's memory encryption, side-channel attacks and Denial-of-Service (DoS) are explicitly out of scope for CAGE. Hardware assumptions include next-generation Arm devices with RME, a hardware root of trust for secure boot and remote attestation, and a trusted GPU whose memory access is subject to GPC (e.g., an SMMU with RME support). On the software front, the Monitor is trusted, as its firmware is securely verified and loaded during secure boot.

Key Findings

[▶ Watch: Arm CCA, RME, and Granule Protection Check explained (2:00)]()

CAGE makes several significant contributions to the field of confidential computing on Arm platforms, primarily by bridging the gap in secure GPU acceleration for Arm CCA. Its key findings and contributions include:

• Hardware-Free Confidential GPU Computing: CAGE successfully extends GPU acceleration support to Arm CCA without requiring any modifications to existing or next-generation Arm hardware. This ensures wide compatibility and immediate applicability.
• Novel Shadow Task Mechanism: The introduction of a shadow task mechanism is a core innovation. This mechanism allows CAGE to adapt the complex GPU workflow to CCA's realm-style architecture by segregating data-independent GPU functions (managed by the untrusted host) from data-dependent, sensitive operations (managed securely within the Monitor and realms). This design avoids encapsulating the entire GPU software stack within the TCB, significantly reducing its size.
• Robust Two-Way Memory Isolation: CAGE leverages Arm CCA's Granule Protection Check (GPC) across the CPU, GPU, and untrusted peripherals to establish a strong, two-way isolation mechanism. This confines the GPU's memory access to its designated realm, protecting sensitive data from both external adversaries and malicious GPU tasks.
• Optimized Granule Protection Table (GPT) Maintenance: Recognizing the potential performance overhead of managing multiple GPTs, CAGE introduces two key optimization techniques: a hierarchical GPT structure for synchronizing CPU and peripheral GPTs, and a template-based approach for initializing GPU GPTs. These optimizations dramatically reduce latency associated with memory protection operations.
• Minimal Trusted Computing Base (TCB) Increment: CAGE achieves its security goals with a remarkably small increase to the TCB. Its implementation adds only 1,301 lines of code (LoC) to the Arm Trusted Firmware-A (TF-A) Monitor, which is comparable to or smaller than other state-of-the-art solutions and significantly smaller than including a full GPU software stack.
• High Performance with Minimal Overhead: Rigorous evaluation using the Rodinia GPU benchmark suite and neural network models demonstrates CAGE's efficiency. It incurs an average performance overhead of just 2.45% compared to native GPU execution, making it practical for real-world confidential GPU workloads.
• Comprehensive Defense Against Privileged Adversaries: CAGE effectively defends against a wide array of attacks from a strong adversary controlling the entire normal and secure world software stack, encompassing unauthorized memory access, illegal GPU memory management, malicious task scheduling, and circumvention attempts against GPC.

Technical Deep Dive

[▶ Watch: Arm GPU, SMMU, and CAGE's threat model (4:00)]()

CAGE's design is meticulously crafted around four critical goals: G1: Compatibility with CCA, G2: Strong Data Security, G3: Optimized Performance, and G4: No Hardware Modification. To achieve these, CAGE deploys three security components within the highest-privilege Monitor, residing in the hardware-isolated root world, which configure hardware primitives to enforce security.

The core of CAGE's approach to achieving G1 (Compatibility with CCA) is its novel shadow task mechanism. A naive approach of encapsulating the entire GPU software stack within each realm would lead to an unacceptably large TCB and attack surface. CAGE avoids this by delegating heavyweight, data-independent GPU functions to the untrusted normal world, while preserving sensitive data within realms. This addresses Challenge 1 (C1): how to adapt host GPU software, not designed for realms, without incurring significant performance overhead from frequent hypervisor and world switching.

The shadow task mechanism creates a pair of GPU tasks: a stub GPU task and a real GPU task. The stub task, managed by the untrusted GPU software on the host, mirrors the structure of a normal GPU task (including buffers, metadata, page tables) but contains no sensitive data. The real GPU task, conversely, holds the sensitive data. The workflow unfolds in three stages:

1. Initialization and Stub Task Creation: The realm user provides GPU task code, data buffer descriptions, and crucial signatures for integrity verification to the untrusted host. This sensitive data and signature are transferred securely to the realm. The GPU software then allocates memory, prepares the stub task, and stages sensitive code and data descriptions into corresponding data buffers. To mitigate performance overhead, CAGE asynchronously replays operations on the stub GPU page table (Stub GPU PTE) to the real one, leveraging the observation that GPU software typically doesn't update the GPU page table during execution.
2. Real Task Creation and Execution: When the stub task is prepared, the GPU software inserts it into the GPU task queue. For stub tasks, the GPU software temporarily blocks other task submissions, waits for current GPU computation to finish, and then routes the stub task to the Monitor. Upon reception, the Monitor creates a corresponding real task inside the realm. It first protects the GPU environment by locking three critical regions using GPC: the GPU MMIO (including the base address of the GPU page table, TTBR), the metadata (pointers to GPU buffers), and the code buffers. This prevents adversaries from leaking or modifying sensitive data. After protection, the Monitor replaces the stub task with the real task by changing the stub GPU page table to the real one and modifying buffer pointers to the new buffers within the realm. It verifies the integrity of descriptions via the provided signature and checks the current GPU status to ensure no malicious tasks are hidden. Once verified, the Monitor submits the real task by writing the start command to GPU registers.
3. Environment Restoration: After GPU computation, the Monitor restores and cleans the execution environment. It restores GPU MMIO registers and metadata values, cleans the previous GPU environment (flushing caches and TLB entries), and then restores access to GPU MMIO, metadata, and code buffers. Execution results are securely stored in the realm.

To achieve G2 (Strong Data Security), CAGE implements a robust GPU environment protection mechanism addressing Challenge 2 (C2): Arm CCA currently treats GPUs as untrusted peripherals, allowing manipulation by privileged software from any world, and the RMM lacks the privilege to prevent malicious access. CAGE's solution is a two-way isolation mechanism leveraging Arm CCA's GPC, employing two types of Granule Protection Tables (GPTs):

1. CPU and Untrusted Peripheral GPTs: These control access from untrusted components (normal and secure world software, other peripherals) to main memory. They prevent illegal access to protected realm and root regions. Specifically, GPU MMIO is configured as root in these GPTs, making it a fixed and unmodifiable region. Metadata and code buffers are configured as realm to prevent illegal access from other realms. To defend against TLB attacks, TLB entries for CPU GPC and untrusted peripheral GPCs are flushed.
2. GPU GPT for Realm Isolation: Each realm receives a unique GPU GPT. When the GPU executes a confidential task, the Monitor fetches the corresponding GPU GPT, configures GPU GPC registers (e.g., setting the base address of the GPU GPT to that of the realm), and flushes TLB entries. Since GPU GPTs and GPC configurations are exclusively accessible to the root world Monitor, the executing task cannot bypass GPU GPC. This ensures the GPU's memory access is confined to its designated page table, metadata, and buffers for computation, achieving a crucial two-way isolation between the GPU execution environment and other components.

G3 (Optimized Performance) is addressed by CAGE's GPT maintenance optimization, tackling Challenge 3 (C3): the non-trivial performance overhead associated with synchronizing CPU and untrusted peripheral GPTs and creating unique GPU GPTs for each realm. CAGE proposes two optimization mechanisms:

1. CPU and Peripheral GPT Synchronization: Instead of a unified GPT that might conflict with customized access control, CAGE leverages the hierarchical structure of GPTs. It configures a sub-level GPT, shared by the CPU GPT and untrusted peripheral GPTs, specifically to protect the reused regions (metadata and code) on GPU memory. The table descriptors of various GPTs point to this same sub-level GPT. The Monitor can then flexibly modify this sub-level table to protect or unprotect metadata and code without affecting other peripherals.
2. GPU GPT Initialization: To reduce latency during initialization, CAGE derives GPU GPTs from a template. This template configures the entire main memory as root (inaccessible). During realm creation, the realm's region is set to normal (accessible). During confidential GPU computation, the reused GPU memory (code and metadata) is additionally configured as normal to permit GPU access. These optimizations significantly mitigate overhead.

Trust Establishment in CAGE follows Arm CCA's established mechanisms:

• Secure Boot: CAGE integrates with CCA's secure boot process, ensuring that the Monitor firmware image, metadata, and payload—including CAGE's security modules—are verified for integrity and authenticity before initialization.
• Remote Attestation and Key Management: CAGE assists realms in attesting their execution environment by leveraging a hardware Root of Trust (e.g., a hard-coded private key in ROM) to attest the initial state of the realm and the Arm device, including GPU hardware. CAGE combines these to generate an attestation report. For key management, CAGE supports existing Diffie-Hellman protocols for key exchange, with a Public Key Infrastructure (PKI) in the Monitor to authenticate the realm and encrypt exchanged keys, preventing man-in-the-middle attacks and establishing a secure communication channel for sensitive data and signature transfer.

Demo / Proof of Concept

[▶ Watch: Introduction to CAGE's design and goals (6:00)]()

To thoroughly verify CAGE's functionality, security, and performance, two distinct prototypes were developed:

Functionality Prototype

The functionality prototype was designed to validate CAGE's core functions and overall system security. It was implemented on the Arm FVP Base RevC-2xAEMVA simulator, which provides support for RME and accurately simulates the latest Armv9 hardware features. Given that the FVP does not include an authentic unified-memory GPU model, a connected test engine was employed. This engine faithfully performed memory accesses as a DMA-capable peripheral, working in conjunction with an SMMU that supported RME. A 1GB memory region was reserved specifically for this test engine.

The software stack comprised Linux v5.3.0 acting as the normal world Host, and Trusted Firmware-A (TF-A) v2.8 serving as the Monitor. To verify memory isolation, GPTs were initialized on the FVP. As the existing TF-A firmware typically assigns the same GPT for all components, CAGE additionally prepared dedicated peripheral GPTs within a 32MB reserved memory region. System registers crucial for GPC enablement were configured, including GPTBR_EL3, GPCCR_EL3 for CPU GPC, and SMMU_ROOT_GPT_BASE, SMMU_ROOT_GPT_BASE_CFG for peripheral GPC. Furthermore, comprehensive TLB invalidation mechanisms were implemented for both MMU and SMMUs. The entire CAGE implementation for this prototype introduced a lean 1,301 lines of code (LoC) additions to the TF-A Monitor, demonstrating a minimal TCB footprint.

Performance Prototype

Since the FVP simulator is not cycle-accurate, a physical prototype was essential for evaluating real-world performance overhead. This prototype was deployed on an Arm Juno R2 development board, which features a Mali-T624 GPU and 8GB of DRAM. As the Juno R2 board is based on an Armv8 architecture, CAGE emulated key Armv9 CCA features to accurately model the performance characteristics of future devices:

• CCA-related instruction emulation: Armv9 instructions for GPT system registers and CPU GPC TLB invalidation were replaced with their Armv8 equivalents, carefully chosen to have similar latencies.
• MMIO operation emulation: MMIO operations targeting peripheral GPCs were emulated using memory access operations on other memory regions, coupled with TLB invalidation on the target memory region.
• GPT configuration emulation: Direct operations on the table entries of GPTs were performed, primarily involving read/write operations to GPT memory, which accurately reflects the underlying overhead.

For interaction with the Mali GPU, specific steps were added to the prototype. These included verifying GPU status via the JS_STATUS register, submitting tasks using the JS_COMMAND_NEXT register, and temporarily configuring the GPU interrupt as a Monitor interrupt to facilitate environment restoration. These detailed emulation techniques allowed CAGE to accurately model the latencies associated with Armv9 CCA operations on readily available Armv8 hardware, providing a realistic performance assessment.

Defensive Implications

CAGE offers significant defensive implications for various stakeholders in the confidential computing ecosystem on Arm devices:

• For Cloud Providers and Device Manufacturers: CAGE provides a concrete, software-only pathway to enable secure GPU acceleration within Arm CCA realms. This is critical for offering confidential AI/ML services, secure data analytics, and high-performance computing to tenants, ensuring that sensitive workloads remain protected even if the underlying cloud hypervisor or host OS is compromised. By leveraging existing hardware features, manufacturers can accelerate the adoption of confidential GPU capabilities without extensive re-engineering.
• For Realm Developers and Users: CAGE empowers developers and users to securely execute GPU-accelerated workloads within Arm realms. This means sensitive data, such as machine learning models, training data, input datasets, and intermediate results, can be processed on the GPU without fear of leakage or tampering by privileged software, including a compromised GPU driver. Users can confidently leverage the performance benefits of GPUs for sensitive applications, enhancing the utility of confidential computing.
• Robust Data Path Security: CAGE's implementation of two-way isolation using Granule Protection Check (GPC) on the CPU, GPU, and untrusted peripherals creates a formidable defense against direct memory access (DMA) attacks. This ensures that the GPU can only access memory regions explicitly permitted by the Monitor, preventing unauthorized reads or writes. Defenders should prioritize correct GPC configuration and ensure the integrity of the root world Monitor, which controls these critical access policies.
• Minimal Attack Surface: The remarkably small TCB increment of 1,301 lines of code in the TF-A Monitor significantly reduces the attack surface for the security modules themselves. This makes CAGE's security mechanisms more resilient to vulnerabilities compared to solutions that require a large, complex GPU software stack to be included within the TEE. A smaller TCB is easier to audit and verify, enhancing trust.
• Integrity Verification: The use of cryptographic signatures for GPU task code and data buffer descriptions is a crucial defensive measure against code injection and data tampering. Defenders should ensure that strong, industry-standard cryptographic primitives are used for signature generation, verification, and key management to maintain the integrity of confidential GPU tasks.
• Addressing Strong Adversaries: CAGE is designed to defend against a strong adversary controlling the entire normal and secure world software stack. This comprehensive threat model means that typical software-based attacks, including those targeting hypervisors or GPU drivers, are mitigated. This provides a high level of assurance for sensitive workloads.
• Future-Proofing and Hardware Suggestions: While CAGE operates without hardware changes, its "Future Work" section provides valuable suggestions to Arm for enhancing future CCA designs. Introducing a StreamID to distinguish SMMU GPCs for multiple peripherals or allowing SMMUs to provide each peripheral with a unique Root control page would further strengthen hardware-assisted isolation, simplify software solutions, and potentially reduce overhead. Defenders should advocate for such enhancements in future Arm architectures.
• Limitations and Further Research: It's important to acknowledge CAGE's stated limitations regarding side-channel attacks and Denial-of-Service. While CAGE addresses direct data leakage and tampering, a comprehensive defensive strategy would need to integrate additional mitigations for these classes of attacks, potentially involving architectural changes or advanced software-based countermeasures. For physical threats like cold-boot or rollback attacks, CAGE points to future CCA features like the Memory Protection Engine (MPE) and monotonic counters, which defenders should monitor and integrate as they become available.

Key Takeaways

• CAGE successfully enables confidential GPU computing for Arm CCA without requiring any hardware modifications, addressing a critical gap in Arm's confidential computing capabilities for high-performance and AI workloads.
• A novel shadow task mechanism is central to CAGE, adapting the GPU workflow to CCA's realm-style architecture by delegating data-independent functions to the untrusted host while securing sensitive data within realms, thus maintaining a minimal Trusted Computing Base (TCB).
• Strong data security is achieved through two-way memory isolation, leveraging Arm CCA's Granule Protection Check (GPC) on the CPU, GPU, and untrusted peripherals, effectively confining GPU memory access and protecting sensitive data from privileged adversaries.
• CAGE significantly optimizes Granule Protection Table (GPT) maintenance through hierarchical GPTs for synchronization and a template-based approach for GPU GPT initialization, drastically reducing performance overhead associated with memory protection.
• The system demonstrates impressive efficiency, introducing an average performance overhead of just 2.45% across diverse GPU benchmarks and neural network models, making it practical for real-world confidential applications.
• CAGE adds a minimal 1,301 lines of code (LoC) to the Trusted Firmware-A Monitor, representing a small and auditable TCB increment, enhancing its overall security posture.

About the Speaker(s)

The talk "CAGE: Complementing Arm CCA with GPU Extensions" was presented by Chenxu Wang at the NDSS Symposium. No further biographical details, such as title or company affiliation, were provided in the talk metadata or transcript.