Accelerate AI and HPC with Math Libraries on NVIDIA Blackwell GPUs | NVIDIA GTC 2025

Accelerate AI and HPC with Math Libraries on NVIDIA Blackwell GPUs

Speakers: Ozie, Principal Product Manager, Math Libraries Team, NVIDIA; Bobak, Engineering Team, NVIDIA

Conference: NVIDIA GTC 2025

YouTube: https://www.youtube.com/watch?v=Kx9Z-NCF8J4

Overview

This talk, presented by Ozie and Bobak from NVIDIA's math libraries team, provides a comprehensive overview of how NVIDIA's CUDA X math libraries are optimized to leverage the advanced capabilities of the newly introduced Blackwell GPU architecture. The core message emphasizes the critical role these libraries play in accelerating both AI and High-Performance Computing (HPC) workloads by abstracting hardware complexities, ensuring peak performance, and offering seamless hardware portability. With the introduction of Blackwell, the libraries are strategically updated to support new features like FP4 precision, enhanced memory bandwidth, and the Grace CPU as a host, enabling users to migrate their code effortlessly and achieve significant performance gains from day one. This presentation highlights specific library enhancements and benchmarks across various applications, demonstrating Blackwell's power in tackling compute-intensive, memory-bandwidth-bound, and large-scale distributed problems.

The speakers underscore that NVIDIA's math libraries serve as a vital bridge between complex GPU hardware and application developers. By encapsulating deep hardware knowledge and performance tuning, they provide an accessible programming model while delivering best-in-class performance. A key focus of the talk is hardware portability, showcasing how libraries rapidly integrate support for new architectures like Blackwell, thereby reducing migration friction for users and accelerating their time to market. Furthermore, the extensive range of functionalities offered by these libraries supports a diverse array of applications, from large language model (LLM) training and inference to scientific simulations and data processing, positioning Blackwell as a versatile platform for the future of accelerated computing.

Background

▶ Watch: Introduction to Blackwell math libraries (0:00)

NVIDIA's CUDA X math libraries comprise a vast collection of over 400 specialized libraries, forming the bedrock for high-performance computing on NVIDIA GPUs. These libraries are broadly categorized into core primitives and solvers (dense and sparse operations) and extension libraries that provide additional functionalities such as Pythonic interfaces or multi-GPU/multi-node support. The talk specifically focuses on cuBLAS, cuSOLVER, cuSPARSE, cuFFT, cuFFTMP, cuDSS, NVCOMP, and NVJPEG, demonstrating their optimization for the Blackwell platform.

The introduction of the Blackwell architecture marks a significant leap in GPU technology, bringing several critical hardware advancements. The speakers detail two primary Blackwell system configurations: the GB200 NVL72, featuring a Grace CPU as the host and a pool of 72 GPUs connected via high-bandwidth NVLink, and the DGX B200 NVL8, which uses x86 CPUs and connects eight GPUs per node via NVLink, with inter-node communication relying on InfiniBand. These systems are frequently benchmarked against Hopper-based DGX systems (H100/H200), which typically also employ x86 CPUs and an 8-GPU NVLink domain.

Blackwell introduces FP4 precision for AI workloads, further expanding the range of low-precision formats. Crucially, while FP64 and FP32 are still supported, their native hardware throughput increase has slowed. This shift necessitates novel approaches, leading to the adoption of emulation techniques within the libraries to leverage the higher throughput of AI precisions (like BF16 and INT8 tensor cores) for these higher-precision computations. Beyond compute, Blackwell significantly boosts memory capacity and bandwidth, a direct response to the escalating demands of large LLMs. Other notable features include the continued integration of Grace CPUs (first introduced with Hopper) and a new hardware decompression engine, both of which the math libraries are designed to exploit for maximum performance.

Key Findings

▶ Watch: Key Blackwell system configurations explained (3:00)

The presentation showcases substantial performance gains across a wide range of applications and numerical precisions, primarily driven by the optimization of math libraries for NVIDIA Blackwell GPUs.

1. High-Precision Emulation:

• FP32 Emulation using BF16 tensor cores in cuBLAS on B200 achieves up to 3x speedup over native FP32 throughput on Blackwell and 4x speedup over Hopper.
• FP64 Emulation using INT8 tensor cores in cuBLAS on B200 delivers up to 2.8x speedup over native FP64 throughput on Blackwell and 1.5x speedup over Hopper H200. This emulation maintains accuracy comparable to native precision and is being enabled in cuSOLVER DN for operations like QR factorization (up to 2.3x speedup for FP32 emulation).
• End-to-end application speedups were observed, such as 1.4x for ectrans (weather simulation) using FP32 emulation and 1.8x for Berkeley GW (material simulator) using FP64 emulation.

1. Low-Precision AI Acceleration:

• Blackwell extends FP8 support with block scaling and introduces FP4 with block scaling, offering finer-grained control over scaling factors for improved accuracy in training and inference.
• cuBLAS for BF16 and FP8 matmuls achieves approximately 2x speedup on B200/GB200 compared to Hopper, aligning closely with hardware specifications. FP4 shows even greater speedups.
• For key LLM training and inference workloads (e.g., Llama 3.1), the matmul components see about 2x acceleration using BF16 or FP8.

1. Memory Bandwidth-Bound Workloads:

• cuSPARSE for sparse matrix operations (SPMM, SPMV) demonstrates 1.4x to 2.4x speedup on B200 compared to H100, effectively leveraging Blackwell's increased memory bandwidth, often reaching the nominal hardware ratio of 2.4x.
• cuFFT for Fast Fourier Transforms (FFTs) consistently achieves speedups very close to the nominal hardware memory bandwidth ratio of 1.7x (B200 vs. H200) across a range of 2D FFT sizes.

1. Large-Scale Distributed Computing:

• cuFFTMP (multi-GPU multi-node FFT) on GB200 NVL72 systems exhibits excellent linear strong scaling for a 123^3 FFT problem across up to 64 GPUs. This configuration, with its high-speed NVLink connection across 72 GPUs, significantly reduces communication overhead.
• Compared to DGX B200 NVL8 systems (which rely on InfiniBand for inter-node communication), the GB200 NVL72 can achieve the same solution time with an 8x reduction in the number of GPUs for certain performance targets, illustrating the profound impact of the NVL72 fabric on scalability.

1. Heterogeneous Computing with Grace:

• cuDSS (CUDA Direct Sparse Solver) leverages the Grace CPU in GB systems through a hybrid memory mode, resulting in an average of 2.1x speedup for factorization and 2.5x for solve compared to x86-based systems (both using Blackwell GPUs). This is due to Grace's high-bandwidth C2C interconnect.
• In its hybrid execution mode, cuDSS intelligently offloads small or highly sequential computations to the Grace CPU, yielding an average of 1.8x speedup for smaller matrices, demonstrating an effective heterogeneous paradigm.

1. Hardware-Accelerated Data Processing:

• NVCOMP (data compression/decompression library) utilizes Blackwell's new hardware decompression engine, achieving significant speedups of 4.4x to 9x compared to software-only decompression on Blackwell.
• NVJPEG (JPEG decode library) leverages the dedicated JPEG engine (introduced in Hopper and continued in Blackwell), providing a 1.3x speedup for the decode phase on Blackwell compared to Hopper.

Technical Deep Dive

▶ Watch: Overview of new Blackwell hardware features (4:40)

The core of Blackwell's advancements, as implemented through NVIDIA's math libraries, lies in sophisticated techniques for numerical precision handling, enhanced data movement, and scalable system integration.

Numerical Precision: Emulation and Block Scaling

Emulation for High-Precision:

With the native hardware throughput growth for FP32 and FP64 lagging behind that of lower-precision tensor cores, NVIDIA has implemented emulation techniques within libraries like cuBLAS and cuSOLVER. The strategy is to leverage the massively parallel throughput of BF16 and INT8 tensor cores to synthesize higher precision results.

• FP32 Emulation: This is achieved by performing multiple (typically nine) BF16 inner matrix multiplications (sub-matmuls) and then summing their results. This approach exploits the high throughput of BF16 tensor cores to effectively "emulate" an FP32 matmul, achieving significant speedups while maintaining accuracy.
• FP64 Emulation: This is more involved, utilizing INT8 tensor cores. It incorporates a "degree of freedom" parameter, s, which controls the number of bits of accuracy retained and the number of sub-matmuls required. For example, s=7 corresponds to 56 bits of accuracy. The choice of s allows expert users to balance performance and precision requirements, with the library ensuring accuracy on par with native FP64. This method is crucial for HPC applications that traditionally rely on FP64.

Block Scaling for Low-Precision:

Blackwell introduces enhanced support for FP8 and FP4 precisions with block scaling. Unlike Hopper's FP8, which supported a single scaling factor for an entire tensor, Blackwell's block scaling allows sub-blocks (e.g., 32-element blocks) within a matrix to have their own unique scaling factors. This fine-grained control is critical for maintaining numerical stability and accuracy during LLM training and inference where dynamic ranges can vary significantly across different parts of a tensor. The ability to apply localized scaling factors helps prevent underflow or overflow, leading to more robust and accurate low-precision computations.

Memory and Communication Optimization

Leveraging Increased Memory Bandwidth:

Blackwell GPUs feature significantly increased memory capacity and bandwidth. Libraries like cuSPARSE and cuFFT, which are typically memory bandwidth-bound, are heavily optimized to exploit this. Their kernels are meticulously tuned to maximize data transfer rates between global memory and the GPU's processing units, ensuring that the increased hardware bandwidth translates directly into application-level performance gains. This optimization involves careful memory access patterns, cache utilization, and kernel scheduling.

Multi-GPU/Multi-Node Scaling with cuFFTMP:

cuFFTMP (Multi-GPU Multi-Node cuFFT) is designed to tackle large-scale distributed FFT problems. The talk highlights the critical role of the interconnect fabric:

• GB200 NVL72: This system represents a paradigm shift, where 72 GPUs are interconnected via a high-bandwidth NVLink fabric, effectively creating a single, large GPU domain. This architecture drastically reduces communication overhead for distributed operations, enabling near-perfect linear strong scaling. For a fixed problem size (e.g., a 123^3 FFT), throwing more GPUs into an NVL72 system directly translates to a proportional reduction in solution time, making it ideal for large-scale HPC simulations where communication is often the bottleneck.
• DGX B200 NVL8 + InfiniBand: In contrast, DGX systems with 8 GPUs per node connected by NVLink rely on InfiniBand for inter-node communication. While NVLink provides high bandwidth within a node, InfiniBand's comparatively lower bandwidth and higher latency become the dominant factor for scaling beyond the 8-GPU NVLink domain. This results in an "inflection point" where performance scaling plateaus or degrades as communication overheads begin to outweigh the benefits of adding more GPUs. The talk visually demonstrates this plateauing effect, emphasizing the architectural advantage of NVL72 for massively distributed workloads.

Heterogeneous Computing with Grace and Hardware Engines

cuDSS Hybrid Memory and Execution:

The CUDA Direct Sparse Solver (cuDSS) library exemplifies how to effectively utilize Grace CPUs in a heterogeneous system.

• Hybrid Memory Mode: Grace CPUs are connected to GPUs via a high-bandwidth C2C interconnect. This enables cuDSS to use Grace's memory as a shared memory pool, particularly beneficial for large sparse problems that might exceed the GPU's memory or require frequent data movement between CPU and GPU. The high bandwidth of the C2C link minimizes latency for these transfers, leading to significant speedups for factorization and solve phases.
• Hybrid Execution Mode: cuDSS intelligently identifies computational tasks that are more suited for CPU execution. This includes highly sequential operations or computations involving very small matrices that might not fully saturate the GPU. By offloading these tasks to the Grace CPU, the GPU is freed to focus on its strengths (massively parallel operations), leading to overall system efficiency and speedup.

Dedicated Hardware Engines:

Blackwell integrates specialized hardware engines to accelerate common data processing tasks:

• Hardware Decompression Engine: A new feature in Blackwell, this engine is leveraged by the NVCOMP library. By offloading data decompression to dedicated hardware, NVCOMP achieves substantial speedups compared to software-only solutions, critical for data-intensive AI and HPC applications that deal with compressed datasets.
• JPEG Engine: The NVJPEG library utilizes the dedicated JPEG engine (present since Hopper) for accelerated JPEG encoding and decoding. This hardware acceleration is vital for applications involving image processing, computer vision, and multimedia.

These detailed technical optimizations across numerical methods, data movement, and heterogeneous computing paradigms underscore NVIDIA's comprehensive strategy to maximize the performance of Blackwell GPUs through its robust math libraries.

Experimental Setup & Results

▶ Watch: Introduction to emulation for higher precision (6:30)

The presentation provides a wealth of experimental results, comparing Blackwell (B200, GB200 NVL72, DGX B200 NVL8) against Hopper (H100, H200, DGX H100) systems. The benchmarks cover a diverse set of applications and computational patterns, demonstrating the efficacy of the optimized math libraries.

High-Precision Emulation Benchmarks

• cuBLAS FP32 Emulation: Benchmarked on B200 against native FP32 throughput on Blackwell and Hopper. For matrix sizes above 512, cuBLAS consistently showed speedups, reaching up to 3x over native B200 FP32 and 4x over Hopper. These "roofline-like" plots demonstrated saturation at peak performance for larger matrices.
• cuBLAS FP64 Emulation: Also on B200, using INT8 tensor cores with an s=7 (56-bit accuracy) parameter. Achieved up to 2.8x speedup over native B200 FP64 and 1.5x speedup over H200.
• End-to-End Applications:
• Ectrans (weather simulation): The SGM part of this FP32 application, subject to emulation, saw a 2.4x speedup, leading to a 1.4x end-to-end application speedup.
• Berkeley GW (material simulator): For an FP64 workflow, the emulated portion achieved 2.1x speedup, resulting in a 1.8x end-to-end application speedup.
• cuSOLVER DN: QR factorization and LU decomposition, leveraging FP32 emulation, reached up to 2.3x speedup. The talk stated that FP64 emulation would be supported next.

Low-Precision AI Matmul Benchmarks

• cuBLAS BF16, FP8, FP4 Matmuls:
• Compute-bound matmuls: On B200 and GB200, speedups were close to hardware specifications, with BF16 and FP8 showing around 2x speedup over Hopper, and FP4 demonstrating significantly higher gains. Hopper FP8 was used as the baseline for block-scaled formats.
• General matmuls (1000 random sizes): To represent a broader range of workloads (including bandwidth-bound problems), average speedups were slightly lower but still substantial, indicating sustained performance across various matrix dimensions.
• LLM Workloads: Matmul components within important LLM training and inference models (e.g., Llama 3.1) showed approximately 2x acceleration using BF16 or FP8 on Blackwell.

Memory Bandwidth-Bound Workloads

• cuSPARSE: Benchmarked SPMM and SPMV operations on B200 against H100. Across a set of benchmarks, speedups ranged from 1.4x to 2.4x, aligning closely with the nominal hardware memory bandwidth ratio of 2.4x (B200 vs. H100). The results were noted to vary with data type, matrix layout, and sparsity format.
• cuFFT: Compared 2D FFT performance on B200 against H200. For a wide range of FFT sizes, cuFFT consistently achieved speedups very close to the nominal hardware memory bandwidth ratio of 1.7x (B200 vs. H200).

Distributed Scaling with cuFFTMP

• Strong Scaling of 123^3 FFT:
• GB200 NVL72: Demonstrated excellent linear scaling up to 64 GPUs, with a nearly perfect linear curve, highlighting minimal communication overhead due to the NVLink fabric.
• DGX B200 NVL8: Showed an inflection point around 8 GPUs, where the curve started to break and performance degraded due to InfiniBand becoming the communication bottleneck.
• DGX H100: Exhibited similar behavior to DGX B200 NVL8, with an inflection point at 8 GPUs and subsequent performance being largely controlled by InfiniBand bandwidth. The initial performance difference between DGX B200 and DGX H100 for fewer GPUs was attributed to the chip-level memory bandwidth ratio (approx. 1.6-1.7x).
• A key result: To achieve a solution time of 2.35 (arbitrary example), the NVL72 system required only 8 GPUs, while the DGX system required 64 GPUs, an 8x reduction in GPU count for the same performance. For very low Service Level Objectives (SLOs), like 72 (arbitrary unit), the NVL72 system could achieve it with 32 GPUs, whereas DGX systems plateaued and could not reach that target regardless of the number of GPUs.

Heterogeneous Computing with cuDSS on Grace

• cuDSS Hybrid Memory Mode: Compared cuDSS on GB systems (Blackwell with Grace) against x86 systems (Blackwell with x86 CPU). Averaged 2.1x speedup for factorization and 2.5x for solve due to Grace's high-bandwidth C2C interconnect.
• cuDSS Hybrid Execution Mode: Showed an average of 1.8x speedup for smaller matrices by offloading computations to the Grace host CPU.

Hardware-Accelerated Data Compression

• NVCOMP: Benchmarked on Blackwell, comparing software-only mode against leveraging the new hardware decompression engine. Achieved significant speedups ranging from 4.4x to 9x.
• NVJPEG: Compared decode phase performance on Blackwell against Hopper. Demonstrated a 1.3x speedup on Blackwell, utilizing the dedicated JPEG engine.

Overall, the experimental results consistently validate the performance advantages of Blackwell GPUs, particularly when paired with NVIDIA's optimized math libraries, across a spectrum of computational challenges.

Practical Implications

▶ Watch: How FP32 and FP64 emulation works (7:40)

The advancements presented in this talk have profound practical implications for a wide range of practitioners, from AI model builders and infrastructure teams to HPC researchers and system deployers.

First and foremost, the hardware portability offered by NVIDIA's math libraries significantly reduces the migration burden for users adopting Blackwell. Instead of rewriting CUDA kernels to exploit new hardware features, developers can largely rely on updated library versions to automatically leverage Blackwell's capabilities from day one. This accelerates time to market for new applications and simplifies the upgrade path for existing ones.

For AI model builders and deployers, the introduction of FP4 and block scaling for FP8/FP4 provides new avenues for model compression and faster inference with improved accuracy control. The 2x speedup for LLM matmuls directly translates to faster training iterations and lower latency for inference, which is critical for the rapidly evolving field of large language models. The ability to achieve high performance with reduced precision formats means more efficient utilization of GPU resources, enabling larger models or higher throughput within existing hardware constraints.

HPC practitioners and scientific researchers heavily reliant on FP32 and FP64 precision will benefit immensely from the emulation techniques. These allow compute-intensive scientific simulations (e.g., weather modeling with Ectrans, material science with Berkeley GW) to leverage the high throughput of tensor cores without sacrificing the required numerical accuracy. This means faster solution times for complex problems, potentially enabling higher-fidelity simulations or larger problem sizes that were previously intractable. The cuSOLVER enhancements further extend these benefits to direct sparse solvers, crucial for many engineering and physics applications.

Infrastructure teams and system architects deploying large-scale AI and HPC clusters will find the GB200 NVL72 system configuration particularly compelling. The demonstrated linear strong scaling with cuFFTMP and the potential for an 8x reduction in GPU count for certain performance targets highlights a significant shift in how distributed workloads can be handled. For communication-intensive applications, the NVL72 fabric offers a path to unprecedented scalability, avoiding the bottlenecks typically encountered with InfiniBand-bound systems. This implies more efficient resource allocation and potentially lower operational costs for achieving target performance levels in supercomputing environments.

For applications that are memory bandwidth-bound, such as sparse matrix operations (cuSPARSE) and FFTs (cuFFT), Blackwell's increased memory bandwidth, effectively harnessed by the libraries, will lead to direct performance improvements. This is crucial for data-intensive analytics, signal processing, and certain machine learning algorithms where data movement, rather than raw compute, is the bottleneck.

The integration of Grace CPUs with Blackwell GPUs offers a powerful heterogeneous computing paradigm. As demonstrated by cuDSS, leveraging Grace's high-bandwidth C2C interconnect for shared memory and offloading sequential or small tasks to the CPU can yield substantial speedups. This means developers can design more balanced applications that intelligently distribute work between CPU and GPU, optimizing overall system throughput and efficiency, especially for workloads with mixed computational characteristics.

Finally, the new hardware decompression engine and the existing JPEG engine, supported by NVCOMP and NVJPEG respectively, are vital for applications dealing with large volumes of compressed data or images. Faster data loading and processing directly contribute to improved end-to-end application performance, whether in data analytics pipelines, computer vision, or multimedia processing.

While the benefits are substantial, practitioners should be mindful of potential tradeoffs. For emulation, the s parameter in FP64 emulation allows for a balance between speed and accuracy, requiring careful consideration based on application requirements. For distributed systems, the choice between NVL72 and InfiniBand-based architectures involves significant cost and complexity considerations, with NVL72 offering superior scaling for communication-heavy strong-scaling problems but potentially at a higher initial investment. The key is to understand the specific workload characteristics (compute-bound, bandwidth-bound, communication-bound, precision requirements) to best leverage Blackwell's diverse features and the optimized math libraries.

Key Takeaways

• Seamless Hardware Portability: NVIDIA math libraries abstract Blackwell's new features (FP4, increased bandwidth, Grace, decompression engine), enabling day-one performance gains and accelerating code migration for users.
• High-Precision Workloads Accelerated: FP32 and FP64 emulation using BF16 and INT8 tensor cores deliver significant speedups (up to 4x for FP32, 1.5x for FP64) compared to Hopper, preserving accuracy for HPC and scientific computing.
• AI/LLM Performance Boost: Blackwell's FP4 and block-scaled FP8/FP4, combined with optimized cuBLAS, provide ~2x speedup for LLM matmuls, crucial for faster training and inference.
• Memory Bandwidth-Bound Tasks Excel: Libraries like cuSPARSE and cuFFT effectively leverage Blackwell's increased memory bandwidth, showing speedups up to 2.4x for sparse operations and 1.7x for FFTs.
• Unprecedented Distributed Scaling with NVL72: The GB200 NVL72 system, with its expansive NVLink fabric, enables excellent linear strong scaling for multi-GPU/multi-node applications like cuFFTMP, drastically reducing communication overhead and GPU count requirements (e.g., 8x reduction).
• Grace CPU for Heterogeneous Efficiency: cuDSS demonstrates how Grace's high-bandwidth C2C interconnect and hybrid execution modes can provide 1.8x to 2.5x speedups for sparse direct solvers by intelligently using CPU memory and offloading tasks.

About the Speaker(s)

Ozie is the Principal Product Manager with NVIDIA's math libraries team. In this role, Ozie is responsible for guiding the strategy and development of NVIDIA's extensive suite of math libraries, ensuring they effectively leverage new hardware architectures like Blackwell to deliver high performance and ease of use for a wide array of AI and HPC applications.

Bobak is a member of the engineering team at NVIDIA, also focusing on math libraries. Bobak's expertise is in the technical implementation and optimization of these libraries, particularly in areas like numerical emulation and leveraging the advanced capabilities of NVIDIA's GPU architectures. His work directly contributes to the performance gains demonstrated across various benchmarks on Blackwell.