Shared-GPU Security Learnings from Fly.io

Shared-GPU Security Learnings from Fly.io

Speakers: Matthew Braun

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=_W4NHw-4P8A

Overview

Matthew Braun, a security practitioner at Fly.io, presented a rare behind-the-scenes look at the security challenges of offering shared GPU compute to customers on a public cloud built on bare-metal infrastructure. Unlike hyperscalers who design custom hardware and silicon for GPU isolation, Fly.io had to solve GPU multi-tenancy security using commodity data center hardware, open-source hypervisors, and careful attention to PCIe, IOMMU, and firmware-level attack surfaces. The talk covers the threat model of giving untrusted users direct access to data-center-grade GPUs, the hypervisor selection journey from Firecracker to Cloud Hypervisor, the decision to use VFIO pass-through instead of Nvidia's MIG or vGPU, and the deep hardware security considerations around PCIe peer-to-peer attacks, ACS/ATS configurations, NVLink, and GPU firmware persistence.

Background

▶ Watch: GPUs as secondary computer systems — the Thunderstrike threat model (2:00)

Fly.io is a developer-focused public cloud that runs its own global fleet of servers across multiple data centers, jurisdictions, and colocation providers. The platform leverages Firecracker (the open-source VMM that powers AWS Lambda and Fargate) for its primary compute isolation. In 2023-2024, market demand for GPU compute prompted Fly.io to offer GPU instances to customers.

The fundamental challenge is that a data-center-grade GPU (such as an Nvidia A10 or L40S) is effectively a secondary computer system with its own firmware, non-volatile storage, and the ability to perform Direct Memory Access (DMA) across the host system. Braun compared the threat model to the Thunderstrike family of attacks against MacBooks via Thunderbolt — except delivered over the internet to multi-tenant infrastructure. Without proper isolation, a GPU tenant could compromise the host, attack other tenants, move laterally through the network, and persist across VM resets through firmware modification. This violates the physical security guarantees that organizations pay for with ISO 27001-certified data centers and mantrap access controls.

Key Findings

▶ Watch: Nvidia MIG and vGPU rejected — per-provision licensing kills ephemeral compute (5:30)

Firecracker does not support PCIe pass-through. This was the immediate blocker, as Fly.io's entire compute isolation model was built on Firecracker. Rather than attempt the enormous engineering effort of adding PCIe support to Firecracker, the team adopted Cloud Hypervisor, which shares the same Rust VMM core components as Firecracker but supports the expanded functionality required for GPU pass-through. This created the operational challenge of running a dual-hypervisor fleet.

Nvidia's virtualization options were unsuitable. MIG (Multi-Instance GPU) provides hardware-enforced GPU partitioning but requires Red Hat or VMware — not Cloud Hypervisor. vGPU provides software-level partitioning with broader hypervisor support but still not Cloud Hypervisor, and critically, Nvidia charges a licensing fee every time a vGPU is provisioned. For Fly.io's ephemeral compute model where VMs spin up and down in response to HTTP requests, per-provision licensing was economically impossible.

VFIO pass-through with IOMMU isolation was the solution. Each physical Nvidia card contains multiple discrete GPUs with independent memory. These are packaged as virtual functions via VFIO (Virtual Function I/O), a Linux kernel framework that safely passes PCIe devices into user space. The IOMMU (Input-Output Memory Management Unit) provides the actual security boundary by remapping device virtual addresses to physical memory and preventing unauthorized DMA. Without IOMMU enforcement, a passed-through GPU could DMA to anywhere in host memory.

PCIe peer-to-peer attacks are a real risk. Modern PCIe is hierarchical with switches that can route traffic between devices without going through the root complex where IOMMU checks are enforced. Access Control Services (ACS) can mitigate this by forcing requests through the root complex, but ACS is not available on all server-grade hardware. Key ACS settings include Source Validation (prevents request ID spoofing), Request Redirect (forces all requests to root complex for IOMMU validation), Egress Control (restricts downstream device communication), and Direct Translation (disables peer-to-peer entirely).

ATS (Address Translation Service) is another risk vector. ATS allows devices to cache virtual-to-physical address translations locally, and a malicious device could inject arbitrary physical addresses into TLP (Transaction Layer Packet) headers. Most workloads do not need ATS and it should be disabled.

NVLink bypasses all PCIe safety nets. NVLink provides direct GPU-to-GPU communication that entirely bypasses PCIe switches and IOMMU. Braun discovered that remote hands at one of their data centers had installed NVLink cables without being asked to, simply because the cables shipped with the hardware. An attacker at sufficient privilege level could have enabled NVLink and used it for cross-card DMA.

Technical Deep Dive

▶ Watch: BAR layouts and memory-mapped I/O security monitoring (10:00)

The VFIO isolation model works through IOMMU groups. Each device passed through must be in its own IOMMU group — if multiple devices share a group, they can communicate freely. The lspci -kvn command reveals IOMMU group assignments, and the PCIe BDF (Bus:Device.Function) identifier maps devices to their physical topology.

Base Address Registers (BARs) define memory-mapped I/O regions where the CPU or guest VM controls the device. A GPU may have up to six BARs — smaller regions (e.g., 16 MB) typically expose control registers, while larger regions (256 MB+) expose frame buffer memory. BARs can be monitored and protected using eBPF programs or by shimming the VFIO PCI driver to intercept and validate access against BAR definitions.

MMIO (Memory-Mapped I/O) is how the guest VM triggers GPU operations — writing to specific memory addresses that are backed by device registers. These addresses are mapped through the BAR definitions, and monitoring MMIO access patterns is essential for detecting unauthorized device manipulation.

The GPU System Processor (GSP) handles boot logic for the GPU, and its driver is loaded from the guest VM rather than the host. While the driver is signed, it is an ELF file that must be parsed, introducing parsing vulnerability surface and downgrade attack potential. The VB BIOS (firmware blobs loaded at card startup) and InfoROM (metadata storage for logging and temperature data) are not all signed and can be modified for persistence. VB BIOS signatures can be dumped and verified, though a compromised card could return a different BIOS than the one actually running.

Fly.io engaged security consultants from Atredis and Tettro who validated the ACS/ATS configurations, recommended continued fuzzing of the GPU driver stack, and helped establish monitoring practices for BAR access patterns.

Demo / Proof of Concept

▶ Watch: PCIe peer-to-peer attacks and ACS configuration for mitigation (13:30)

There was no live demo. Braun showed terminal output from test hosts including lspci -kvn output revealing IOMMU group assignments, BAR layouts, and PCIe topology information for both host-side and guest-side views of the passed-through GPU.

Defensive Implications

▶ Watch: GPU firmware persistence: VB BIOS, InfoROM, and GSP driver risks (17:30)

Organizations operating shared GPU infrastructure or evaluating GPU-as-a-service providers should ask pointed questions about isolation mechanisms. Key considerations:

Verify that IOMMU is properly configured with one device per IOMMU group. Check ACS configuration on PCIe switches (Source Validation, Request Redirect, Egress Control, Direct Translation). Disable ATS unless specifically required. Physically verify NVLink cable status on all GPU hosts. Monitor BAR access patterns via eBPF or VFIO driver shims. Establish firmware integrity baselines by dumping and verifying VB BIOS signatures.

For customers of GPU cloud providers, ask whether you can get dedicated hardware. The hyperscalers build custom GPU isolation into their platforms, but smaller providers may have varying levels of hardware-level isolation. If you are using Intel or AMD GPUs rather than Nvidia, the history of vulnerabilities may warrant additional scrutiny.

Key Takeaways

• Data-center GPUs are secondary computer systems: They have firmware, NV storage, and DMA capability that make them equivalent to remote physical access to the host if not properly isolated.
• IOMMU is the primary security boundary: Without proper IOMMU configuration, a passed-through GPU can DMA anywhere in host memory.
• PCIe peer-to-peer attacks bypass IOMMU: ACS must be enabled on PCIe switches to force all requests through the root complex, but not all server hardware supports ACS.
• NVLink bypasses everything: Direct GPU-to-GPU communication that circumvents all PCIe and IOMMU protections; physical cable presence must be verified.
• GPU firmware can persist across VM resets: VB BIOS and InfoROM modifications survive tenant turnover; not all firmware blobs are signed.
• Nvidia's licensing model blocks ephemeral compute: Per-provision vGPU licensing fees are incompatible with serverless or ephemeral VM models.

About the Speaker(s)

Matthew Braun is a security practitioner at Fly.io who wears many hats including compliance, penetration testing, and legal. He is the parent of twin seven-year-olds. He can be reached on BlueSky, Mastodon, and LinkedIn. His boss authored a blog post titled "We Were Wrong About GPUs" documenting the lessons learned about GPU market demand.