How to Secure Unique Ecosystem Shipping 1 Billion+ Cores?

How to Secure a Unique Ecosystem Shipping 1 Billion+ Cores

Speakers: Adam Zabrocki, Director of Offensive Security, NVIDIA; Marco Mittik, System Software Manager, NVIDIA

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=JmAXnQJZbWg

Reading time: ~10 minutes

Type: Briefing

TL;DR

NVIDIA is shipping over one billion RISC-V cores across its GPU, SoC, and data center product lines, having replaced its proprietary Falcon architecture with a custom RISC-V implementation called NVRISCV inside a hardware subsystem called Peregrine. At Black Hat 2025, Adam Zabrocki and Marco Mittik detailed the full security stack NVIDIA built atop this foundation — including pointer masking, hardware control flow integrity, memory tagging, a formally verified separation kernel written in Ada SPARK, and hardware glitch protections — along with the hard lessons learned about RISC-V fragmentation, custom extension costs, and the inescapable need for hardware-software co-design at scale.

Introduction

When most people think of NVIDIA security, they think of GPU driver vulnerabilities or CUDA bugs. What almost no one outside NVIDIA has seen is the security architecture underpinning the microcontrollers that run inside every NVIDIA chip — the dozens of embedded processors handling memory control, power management, resource arbitration, and security enforcement on each die.

For years, those microcontrollers ran on Falcon, NVIDIA's proprietary RISC instruction set architecture. As NVIDIA's product portfolio expanded from gaming GPUs to data center AI platforms, the limitations of Falcon became untenable. Zabrocki and Mittik present the full story of how NVIDIA replaced Falcon with a custom RISC-V implementation and built a layered security architecture designed to scale across a billion-plus cores — and what that actually required in practice.

From Falcon to RISC-V: The Architecture Transition

Each NVIDIA chipset may contain between ten and fifty microcontrollers operating at different privilege levels and trust boundaries: functional controllers (memory controller, chip-to-chip interfaces), system-level controllers (resource manager, PMU security), and data processing engines. All of these previously ran on Falcon, NVIDIA's internal proprietary RISC ISA.

▶ Watch: Why NVIDIA Chose RISC-V (06:00)

The decision to move to RISC-V came down to four factors: better baseline performance without legacy overhead, an open standard enabling hardware-software co-design through custom extensions, layered security isolation primitives, and the ability to support a configurable common platform across all microcontrollers for all products without paying the area and power cost of physically separate cores for each function.

NVIDIA introduced RISC-V in the Turing GPU architecture and has since deployed it across every product line. The custom implementation — NVRISCV — sits inside a hardware subsystem called Peregrine, which includes the processor cores and peripheral devices. A single die may contain up to fifty Peregrine instances. Internally, NVIDIA calls this the "one core strategy": one foundational hardware-software architecture for all microcontrollers across all products, so that security investments made in one area automatically benefit the entire ecosystem.

RISC-V's Drawbacks: Fragmentation, Custom Extension Costs, and Software Burden

Zabrocki is candid about RISC-V's challenges at production scale.

▶ Watch: RISC-V Fragmentation and the Custom Extension Problem (10:00)

Fragmentation is the first problem. Because RISC-V is an open specification with optional extensions, software cannot assume which features any given hardware implementation provides. The RISC-V Foundation's profiles (RVA22, RVA23, RVB) address this by mandating baseline capabilities — but the profiles themselves are not mutually compatible (RVA and RVB have different mandatory and optional extension sets), and even within a profile family, extension requirements change between versions (RVA22 makes vector extensions optional; RVA23 makes them mandatory).

Custom extensions are expensive. When no standard extension exists for a required capability, organizations build their own — but then carry the entire software toolchain burden alone. Worse, a ratified RISC-V extension may later solve the same problem better or differently, leaving the custom extension stranded. Zabrocki's explicit advice: contribute problems to RISC-V International early, because the community will standardize them, and a standard extension is always better than a proprietary one.

Custom hardware requires custom software. Every RISC-V custom extension needs compiler support, OS patches, library updates, and toolchain integration. At NVIDIA's scale, this is a significant and ongoing investment.

The Security Stack: Pointer Masking, CFI, and Memory Tagging

At the time NVIDIA began building its fuzzing infrastructure for RISC-V, neither address sanitizer (ASAN) nor code coverage instrumentation existed for the architecture. Zabrocki and his team needed them to make fuzzing effective.

▶ Watch: Pointer Masking and Hardware ASAN (16:01)

Pointer Masking / Hardware ASAN. Zabrocki developed a custom pointer masking extension and brought it to RISC-V International's TE group (now renamed Runtime Integrity). Independently, engineers at Google were working on their own version for the same reason. Rather than competing, the teams merged their use cases into a single standard — the RISC-V Pointer Masking extension, ratified as a framework for any feature requiring pointer masking, including hardware ASAN, memory tagging, pointer authentication, and hardware memory sandboxing. NVIDIA now uses hardware ASAN to fuzz GSP firmware, the resource manager, and is preparing for bare-metal microcode and RTOS fuzzing.

Hardware Control Flow Integrity (CFI). RISC-V CFI consists of two sub-extensions. The shadow stack enforces backward-edge CFI — protecting return addresses by storing them in a separate memory region inaccessible to normal code, introducing a shadow stack pointer register and shadow stack management instructions. Landing pads enforce forward-edge CFI — any indirect branch must terminate at a specific LPAD instruction with a 20-bit label hash, enabling fine-grained (not just coarse-grained) control flow integrity. The 20-bit label is better than comparable implementations on other architectures in terms of reducing hash collision probability.

NVIDIA has worked to include CFI in RISC-V profiles (some extensions mandatory, some optional) and is committed to shipping hardware CFI support on the Rubin architecture (GR20X).

Memory Tagging. Still in development, memory tagging addresses the performance overhead of hardware ASAN while providing broader bug detection capabilities. Each memory allocation receives a tag stored in the upper bits of the pointer. At every memory access, the hardware compares the tag in the pointer against the tag on the memory region. A mismatch raises a hardware exception before the bug manifests — detecting use-after-free, use-after-return, and out-of-bounds accesses pre-corruption rather than post-corruption.

Two related extensions are also in progress: extending the shadow stack to M-mode (machine mode, which currently lacks shadow stack protection despite having landing pads), and hardware fault isolation (HFI) — an in-process memory sandbox that introduces x86-segment-like memory regions with configurable base, length, and security attributes.

Peregrine Software Architecture: Separation Kernel and Formal Verification

On the software side, NVIDIA's Peregrine ecosystem implements a multi-partition architecture inspired by MILS (Multiple Independent Levels of Security), common in defense and avionics systems.

▶ Watch: Separation Kernel and Ada SPARK Formal Verification (24:01)

Multi-partition design. Partitions are isolated execution environments defined by a manifest and partition policies — signed static configuration sets specifying fine-grained hardware access control. Boot ROM parses the manifest and configures hardware limits before any application software runs. A separation kernel then parses partition policies and creates the actual partitions. Manifests and policies are signed separately from partition images, so a software change within a partition cannot affect the security configuration. Only one partition is active per core at any time; the separation kernel clears all prior partition state on a switch.

The separation kernel is small, written in Ada SPARK, and formally verified to be free of runtime errors (AORTE — Absence of Runtime Errors). It is not a hypervisor: it controls only what hardware is exposed to partitions and provides no abstraction layer. Partition software has direct access to hardware.

Boot ROM is immutable, embedded in silicon, and also written in SPARK. Additional protections include: ROM sections that hardware disables after each boot phase (so redirecting execution to already-processed sections fails), execute-only memory (preventing code dumps), all-leaf-function structure (eliminating return address spills to the stack), and external access lockdown during core software execution. A reference to Zabrocki's DEF CON talk covers a specific flaw found in the RISC-V ISA itself that makes glitch attacks more effective.

Why SPARK? Zabrocki explains the fundamental difference using finite state machine theory. C tends to expand the set of attainable machine states beyond what correct execution requires. Ada and Rust are stricter. SPARK — a formally defined, ambiguity-free subset of Ada with no undefined behavior — has an attainable state set that provably intersects only with correct execution states. Using the GNAT Prove tool, developers can produce formally verified implementations through an iterative process: if a property cannot be proven, the tool provides counter-examples explaining why.

Notable Quotes

"Software is the biggest attack surface, so how do we effectively find the software vulnerabilities, especially in the ecosystem like that?" — Adam Zabrocki ▶ 14:01

"You cannot solve the problem either only in the hardware or only in the software. You must co-design hardware together with the software to create a secure ecosystem." — Adam Zabrocki ▶ 34:03

"Something which is not a problem today might be a very critical problem tomorrow. Examples of the side channel attacks, Spectre, Meltdown — people did not think it's a problem in the past. Now it's one of the most difficult problems to solve." — Adam Zabrocki ▶ 36:03

"Please contribute to RISC-V. Because at some point, someone else might hit the same problem as you, and they will work with RISC-V to make an extension which is officially ratified, and then your extension might not be compatible." — Adam Zabrocki ▶ 12:01

Key Takeaways

• NVIDIA ships RISC-V at a scale unprecedented in the security research community — up to fifty microcontrollers per die, across GPUs, SoCs, and data center platforms — making its security architecture choices consequential for the entire industry.
• Fuzzing at scale requires hardware instrumentation. The absence of ASAN and code coverage for RISC-V forced NVIDIA to develop hardware pointer masking, which became the RISC-V standard — enabling hardware ASAN for GSP firmware, resource manager, and RTOS fuzzing.
• Hardware CFI with 20-bit label hashes enables fine-grained control flow enforcement across RISC-V, with shadow stacks protecting return addresses and landing pads protecting indirect branches.
• Formally verified software (Ada SPARK) is not optional for high-assurance microcontroller code. NVIDIA's separation kernel is formally proven free of runtime errors; its boot ROM is immutable silicon also written in SPARK.
• Design for adaptability from the start. Threat models change — Spectre and Meltdown were unknown unknowns. An ecosystem of a billion-plus cores must be flexible enough to accommodate security extensions that do not yet exist.

No slides PDF was listed for this talk.