CUDA: New Features and Beyond | NVIDIA GTC 2025

CUDA: New Features and Beyond | NVIDIA GTC 2025

Speakers: Stephen Jones, CUDA Architect, NVIDIA

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=6o_Wme-FdCU

Overview

In this comprehensive talk from NVIDIA GTC 2025, Stephen Jones, a CUDA Architect at NVIDIA, delves into the past, present, and future of the CUDA platform. The presentation, aptly titled "CUDA: New Features and Beyond," highlights the platform's evolution from a nascent C-based GPU programming model to a sprawling, multi-layered ecosystem essential for accelerated computing across diverse domains. Jones emphasizes that CUDA is far more than just CUDA C++, encompassing a vast array of SDKs, libraries, runtimes, and developer tools that collectively enable high-performance computing on NVIDIA GPUs.

The talk addresses critical trends shaping modern accelerated computing, particularly the escalating complexity of software stacks, the pervasive role of compilers, and the growing demand for Python-native GPU programming. Jones introduces significant advancements, including a "Pythonic reimagining" of the CUDA runtime, new high-level programming models like Coupile for tile-based GPU acceleration, and NVIDIA's early vision for CUDA DTX to tackle data center scale computing. This presentation is crucial for anyone involved in AI/ML, scientific computing, or systems development seeking to understand how NVIDIA is evolving CUDA to meet the demands of next-generation applications and hardware.

Jones's insights are not merely a retrospective but a forward-looking exposition, revealing the strategic investments NVIDIA is making to ensure CUDA remains the foundational platform for accelerated computing. By showcasing both immediate feature releases and conceptual future directions, the talk provides a holistic view of the challenges and opportunities in optimizing performance, enhancing developer productivity, and scaling GPU workloads from single nodes to hyperscale data centers. The underlying message is clear: CUDA's continued innovation lies in its adaptability, its comprehensive toolset, and its commitment to abstracting hardware complexity while delivering uncompromised performance.

Background

▶ Watch: Broad definition of CUDA beyond C++ (0:00)

The journey of CUDA began in 2008 as a relatively small, C-centric programming model. Stephen Jones, one of the early architects, recalls a time when developers largely wrote their own GPU kernels in CUDA C. This initial direct-to-hardware approach, while powerful, was not sustainable for building a broad, accessible platform. Over the past 17 years, CUDA has transformed into an immense and intricate stack, comprising hundreds of NVIDIA-produced SDKs and thousands of community-built tools and frameworks. This evolution reflects a fundamental shift in accelerated computing, where applications like PyTorch leverage a deep hierarchy of optimized libraries such as cuDNN, cuBLAS, and Cutlass, all ultimately resting on the CUDA C++ foundation and runtime.

A significant trend observed by Jones in recent years is the "compilers everywhere" phenomenon. Historically, compilation was an offline, build-system activity. However, modern deep learning frameworks and kernel generators like OpenAI's Triton now integrate compilation at nearly every layer of the software stack. This proliferation of compilers is driven by the paramount importance of performance optimization. Different applications—such as memory-bandwidth-limited recommender systems, communication-limited graph neural networks, or compute-limited language models—require distinct optimization strategies. Optimization is inherently hierarchical; higher layers understand the global model for distribution and data placement, while lower layers handle instruction scheduling. Each layer, with its unique visibility, must perform specific optimizations, leading to pervasive code generation and just-in-time (JIT) compilation throughout the stack. This means compilation is no longer a backend process but often resides in the critical path of program execution, making compile-time performance a crucial metric.

Concurrently, Python has emerged as the dominant language for AI/ML and is rapidly gaining traction in HPC. However, integrating Python seamlessly with high-performance GPU acceleration presents unique challenges, particularly around dependency management and maintaining performance parity with C++. NVIDIA's strategic response has been to invest heavily in making Python a first-class citizen within the CUDA ecosystem, ensuring that Python developers can leverage the full power of the GPU without sacrificing productivity or performance. This necessitates a "Pythonic reimagining" of CUDA, extending beyond mere language bindings to encompass native execution flows, robust tooling, and high-level, intuitive programming models tailored for Python developers.

Key Findings

▶ Watch: CUDA's evolution from C to a vast platform (2:00)

The talk unveils several pivotal advancements and strategic directions for the CUDA platform, addressing the twin challenges of increasing software stack complexity and the demand for data center-scale performance with developer productivity.

1. Pythonic CUDA as a First-Class Citizen: NVIDIA has significantly advanced its commitment to Python, introducing CUDA.Core, a Pythonic reimagining of the CUDA runtime for native, in-process execution. This initiative aims to reduce Python dependency complexity and provide seamless interoperability across the CUDA stack. Complementing this are tools like Ki Numeric, a direct NumPy replacement that accelerates code on GPUs and scales to thousands of GPUs with a simple import change, and NVMath Python, which provides unified, JIT-compiled Python interfaces to highly optimized CUDA libraries (e.g., cuBLAS, cuFFT), offering C++-level performance from Python. CUDA.Parallel and CUDA.Cooperative extend this further, offering Pythonic access to fundamental parallel algorithms and device-side library calls for kernel authors.

1. Coupile: A New Tile-Based Programming Model: A major highlight is Coupile, a novel CUDA language extension for "tile programming." This model shifts GPU programming from fine-grained, element-wise thread operations to higher-level, array or tensor-based operations at the block level. Coupile simplifies expressing parallel algorithms, making them more intuitive, easier to debug, and often more performant by enabling compilers to map problems efficiently to GPU hardware, including Tensor Cores. It has demonstrated remarkable performance, achieving Llama 3.1 inference within 10% of hand-tuned cuDNN performance in a matter of weeks, and proving its generality by accelerating HPC mini-apps. Coupile is integrated as a core part of CUDA, ensuring portability and stability across architectures.

1. Compilation in the Critical Path & Enhanced Developer Tools: Recognizing that JIT compilation is now integral to performance, NVIDIA has invested in tools that give developers control over optimization levels, support caching for compiled artifacts, and provide deep insights into compilation times. Compile Time Advisor and heatmaps within tools like NVIDIA Nsight Systems and NVIDIA Nsight Compute help identify and resolve compilation bottlenecks. For Python, comprehensive profiling and tracing tools, including NVTX integration and PyTorch layer support, are being developed to ensure full visibility into GPU execution.

1. Vision for Data Center Scale Computing (CUDA DTX): Looking two years ahead, Jones outlines CUDA DTX, NVIDIA's early thinking on extending CUDA to span entire data centers. This future platform will require a distributed machine model that unifies diverse hardware (e.g., Hopper, Blackwell) and a distributed runtime capable of resilient, asynchronous, and dynamically scheduled operations across potentially hundreds of thousands of nodes. While early, this vision underscores NVIDIA's commitment to addressing the complexities of hyperscale distributed GPU computing.

1. Data Center Scale Tooling: To support current and future distributed environments, NVIDIA is enhancing its developer tools. NVIDIA Nsight Cloud offers data center-scale debugging by deploying tracing agents within containers, eliminating the need for special debug containers. CUDA Checkpoint, based on the Linux CRIU utility, extends GPU-aware checkpointing to allow pausing, snapshotting, and migrating GPU workloads across machines, enabling dynamic load balancing and improved system administration without code instrumentation.

These findings collectively illustrate NVIDIA's multi-pronged strategy: democratizing GPU access through Python, simplifying complex parallel programming with higher-level abstractions like Coupile, optimizing the entire software stack through advanced compilation techniques, and laying the groundwork for truly data center-native accelerated computing.

Technical Deep Dive

▶ Watch: Deep software stack: PyTorch leveraging CUDA components (3:00)

The CUDA platform, as Stephen Jones meticulously explains, is a vast, interconnected ecosystem designed to facilitate accelerated computing on NVIDIA GPUs. It begins with low-level drivers and runtimes, extending through a multitude of SDKs (over 250 produced by NVIDIA alone) and thousands of community-developed frameworks and libraries. A prime example is PyTorch, which, while appearing high-level, orchestrates calls to cuDNN for deep neural network operations, cuBLAS for linear algebra, and ultimately relies on kernels implemented in Cutlass, which are built on CUDA C++, compiled by NVCC, and executed by the CUDA runtime. This layered dependency highlights the platform's depth and the necessity for seamless interoperability.

A defining characteristic of the modern CUDA stack is the "compilers everywhere" paradigm. Unlike traditional software development, today's accelerated applications feature compilation at virtually every layer. High-level frameworks might generate intermediate representations or specialized code, kernel generators like OpenAI's Triton synthesize highly optimized kernels, and domain-specific languages (DSLs) target various layers. This pervasive JIT compilation is a direct response to the need for problem-dependent optimization. For instance, a recommender system might be memory bandwidth-limited, a graph neural network communication-limited, and a language model compute-limited. Each requires distinct optimization strategies, best applied by compilers with specific visibility into the problem at hand, from high-level model parallelism down to low-level instruction scheduling. This shift places compilation directly in the critical path, necessitating tools that provide control over optimization aggressiveness, support pre-compiled headers, and offer cache mechanisms for previously compiled work, dramatically reducing overhead.

NVIDIA's investment in Pythonic CUDA is a cornerstone of current developments. The CUDA.Core library represents a significant step, reimagining the CUDA runtime with native Python interfaces and an in-process execution flow. This design minimizes external dependencies, simplifying the Python dependency tree for GPU acceleration. For numerical computing, Ki Numeric offers a drop-in replacement for NumPy. By simply changing an import statement, NumPy code can execute on a single GPU or transparently scale across thousands of GPUs in a cluster. The NVMath Python package provides Pythonic and unified interfaces to highly optimized CUDA libraries. Crucially, it leverages JIT compilation to fuse library calls and optimize performance based on known data types and sizes, and supports dispatching to either GPU (via NVIDIA libraries) or CPU (via NVIDIA Performance Libraries for ARM or Intel MKL for x86), all while maintaining interoperability with existing tensor libraries and multi-GPU scaling. For more granular control, CUDA.Parallel provides Python interfaces to fundamental parallel algorithms (e.g., sorting, histogramming), while CUDA.Cooperative allows direct device-side calls to these optimized C++ libraries from within Python kernels, ensuring near-native performance.

Kernel authoring in Python is evolving to embrace multiple levels of parallelism. Traditionally, CUDA C++ and low-level Python wrappers like Numba focused on thread-level parallelism, where each thread processes a single data element. While powerful, this can be complex. NVIDIA is championing Coupile, a novel extension for tile programming, which operates at the block level. Instead of element-wise operations, Coupile allows developers to express computations as array-plus-array operations on structured data tiles. The compiler then efficiently maps these tile operations to the GPU, often outperforming manual thread mapping. This abstraction is particularly intuitive for Python developers familiar with array-based libraries like NumPy and aligns well with the architecture of Tensor Cores. Coupile is not merely a library; it's a fundamental extension of the CUDA language and platform, ensuring portability, stability, and integration with existing CUDA tools like CUDA Graphs and debuggers. Its underlying architecture includes a Tile IR (Intermediate Representation) that serves as a compiler target, analogous to PTX for thread-based execution, enabling compositional optimization of mixed tile and thread code. Frameworks like NVIDIA Warp have also adopted tile models, allowing tighter integration of neural network layers and kernel fusion within their auto-differentiating simulation environments. Even Cutlass 4.0 now offers a Python compilation layer, providing full control over Tensor Cores with C++-matching performance but significantly faster JIT compilation times due to Python's nature, offering a powerful trade-off for productivity without sacrificing efficiency.

Looking ahead, CUDA DTX represents NVIDIA's early conceptualization for data center-scale CUDA. This ambitious undertaking envisions a distributed machine model that abstracts the heterogeneity of hardware (GeForce, mobile, data center GPUs like Hopper and Blackwell) into a unified view, ensuring applications can run without modification. Complementing this is a distributed runtime designed for resilience, dynamic scheduling, and asynchronous operations across potentially hundreds of thousands of nodes, acknowledging that failures are a daily occurrence at such scales. While a future vision, current efforts like NVIDIA Nsight Cloud for containerized, data center-scale debugging and CUDA Checkpoint are laying the groundwork. CUDA Checkpoint, built on Linux's CRIU, enables GPU-aware checkpointing, allowing GPU workloads to be snapshotted, saved to disk, and migrated between compatible GPUs. This capability facilitates dynamic load balancing, efficient resource utilization, and enhanced fault tolerance at the system administration level, without requiring application-level instrumentation.

Experimental Setup & Results

▶ Watch: Every layer of the stack now includes compilation (4:00)

The talk highlights several compelling experimental results that underscore the effectiveness and performance parity of the new CUDA features, particularly those targeting Python and the Coupile programming model.

1. Llama 3.1 Inference with Coupile:

• Setup: A Llama 3.1 inference workload was implemented using Coupile within a PyTorch backend. This setup allowed for leveraging the familiar PyTorch framework while injecting Coupile-generated kernels for critical operations. The performance was benchmarked against the standard cuDNN backend, which represents highly optimized, hand-tuned C++ implementations.
• Results: The Coupile implementation achieved performance within 10% of the hand-tuned C++ cuDNN backend. This result is particularly remarkable given that the Coupile implementation was developed in a matter of weeks and leverages a brand-new compiler and optimization stack. It powerfully demonstrates Coupile's ability to deliver near-native C++ performance for complex AI workloads while offering the productivity benefits of a higher-level, Python-friendly programming model.

1. HPC Mini-app (2D Fluid Simulation) with Coupile:

• Setup: To validate Coupile's generality beyond AI, a standard HPC mini-application simulating 2D fluid dynamics was ported to Coupile.
• Results: The successful porting and execution of this scientific simulation confirmed that Coupile is a versatile tool applicable across a broad range of accelerated computing problems, not just deep learning. This showcases its potential to simplify GPU programming for traditional HPC workloads while maintaining high performance.

1. Duke University Supernova Analysis with NVMath Python:

• Setup: Researchers at Duke University, working on a supernova analysis program for a space telescope, initially accelerated their Python code using CuPy. They then integrated NVMath Python libraries. The key aspect here is that these scientists, primarily Python users familiar with NumPy, directly adopted and integrated the NVMath Python interfaces without assistance from NVIDIA engineers.
• Results: The integration of NVMath Python further accelerated their application. This serves as a powerful testament to the user-friendliness and immediate utility of the Pythonic interfaces provided by NVMath Python, enabling non-NVIDIA experts to achieve significant GPU acceleration with minimal effort.

1. Cutlass Python Performance:

• Setup: The new Python compilation layer for Cutlass 4.0 was developed to provide full control over Tensor Cores directly from Python. Its performance was compared against the established C++ Cutlass.
• Results: The Pythonic Cutlass implementation was shown to match the performance of its C++ counterpart. This is a crucial finding, as it demonstrates that developers can gain the productivity and faster JIT compilation benefits of Python without any sacrifice in the raw performance achieved by highly optimized, low-level C++ code for tensor operations.

These experimental outcomes collectively validate NVIDIA's strategy of providing higher-level, more "Pythonic" abstractions and programming models without compromising the core promise of GPU acceleration: unparalleled performance. The ability to achieve near C++ performance from Python-centric tools like Coupile and NVMath Python significantly lowers the barrier to entry for GPU computing, making it accessible to a wider developer base.

Practical Implications

▶ Watch: Optimization is problem-specific for achieving performance (5:00)

The advancements in CUDA outlined by Stephen Jones carry significant practical implications for practitioners, infrastructure teams, model builders, and deployers across the AI/ML and HPC landscapes.

For practitioners and model builders, the thrust towards Pythonic CUDA is a game-changer. CUDA.Core, Ki Numeric, and NVMath Python allow developers to stay within their familiar Python environment, leveraging high-level abstractions for GPU acceleration. This dramatically reduces the cognitive load and boilerplate associated with low-level CUDA C++ programming. A simple import change with Ki Numeric can transform CPU-bound NumPy code into GPU-accelerated, multi-GPU scalable solutions. NVMath Python provides access to highly optimized libraries like cuBLAS and cuFFT with native Python syntax and JIT-level performance fusion, meaning developers can achieve C++-level performance without leaving Python. This boosts productivity, accelerates prototyping, and makes GPU programming accessible to a broader audience, including data scientists and researchers without deep C++ expertise.

The introduction of Coupile and tile programming offers a new paradigm for model builders and HPC developers. By operating on structured arrays and tensors at the block level, Coupile simplifies the expression of complex parallel algorithms. This leads to code that is more intuitive, easier to debug, and potentially more performant as the compiler can intelligently map these operations to Tensor Cores and other GPU hardware features. The demonstrated performance for Llama 3.1 inference and HPC mini-apps indicates that Coupile can be a powerful tool for developing both AI models and scientific simulations, bridging the gap between high-level frameworks and low-level optimization. The ability to mix and match tile and thread-level kernels within a single application or even a single kernel will provide unprecedented flexibility in optimizing diverse computational patterns.

For infrastructure teams and deployers, the focus on compilation in the critical path and enhanced developer tools is crucial. As JIT compilation becomes ubiquitous, understanding and optimizing compile times directly impacts application latency and resource utilization. New tools like Compile Time Advisor, compile-time tracing, and caching mechanisms help diagnose and alleviate these bottlenecks. Furthermore, the vision for CUDA DTX addresses the looming challenge of data center-scale computing. While still nascent, the concepts of a distributed machine model and a distributed runtime promise to simplify the management and programming of massive GPU clusters. Current tools like NVIDIA Nsight Cloud for containerized debugging and CUDA Checkpoint for GPU-aware workload migration offer immediate benefits. CUDA Checkpoint, in particular, enables dynamic load balancing, fault tolerance, and efficient resource allocation across a data center, allowing system administrators to manage GPU workloads without requiring application-level changes or awareness.

Tradeoffs and Limitations: While the new features offer immense benefits, developers must still make informed choices. The "one size does not fit all" philosophy remains central. While Pythonic CUDA offers high productivity, scenarios demanding ultimate, bare-metal control might still benefit from direct CUDA C++ or highly specialized Cutlass implementations. However, with Cutlass Python matching C++ performance and offering faster JIT compilation, the choice becomes more about development workflow and iteration speed than raw execution speed. The CUDA DTX vision is a long-term goal, and its full realization will evolve over time, meaning infrastructure teams will continue to manage distributed systems with current tools for the foreseeable future. Similarly, Coupile's full interoperability within a single kernel is a future enhancement. Nevertheless, the trend is clear: NVIDIA is providing a rich menu of options, allowing developers to select the right tool for the job, balancing productivity, performance, and control.

Experimental Setup & Results

▶ Watch: Optimization is a hierarchical problem across stack layers (6:00)

The talk provided concrete examples and benchmark results that validate the efficacy of the new CUDA features, particularly in the realm of Pythonic GPU programming and the novel Coupile tile-based model. These results demonstrate that NVIDIA is achieving its goal of delivering C++-level performance through higher-level, more accessible abstractions.

One of the most compelling demonstrations involved Llama 3.1 inference using Coupile.

• Setup: The Llama 3.1 inference workload was implemented within a PyTorch backend, leveraging Coupile for core computational components. This allowed the integration of Coupile's tile-based kernels into a popular deep learning framework. The performance was then rigorously compared against the highly optimized, hand-tuned cuDNN backend, which is typically written in C++ and represents the gold standard for deep learning primitive performance on NVIDIA GPUs.
• Results: Despite being a relatively new compiler and optimization stack, the Coupile implementation achieved performance within 10% of the hand-tuned C++ cuDNN backend. This was accomplished in a remarkably short development time, measured in a "couple of weeks." This result is significant because it shows that Coupile can deliver near-native performance for complex, state-of-the-art AI models while offering a more intuitive and productive programming experience compared to traditional low-level C++ kernel development.

To demonstrate the generality of Coupile beyond AI workloads, a standard HPC mini-app was also ported.

• Setup: A 2D fluid simulation, a common benchmark in high-performance computing, was reimplemented using Coupile.
• Results: The successful porting and efficient execution of this mini-app confirmed that Coupile is a versatile language extension, capable of accelerating diverse scientific and engineering applications, not just deep learning. This broad applicability ensures that the benefits of simplified GPU programming are not confined to a single domain.

The talk also highlighted a real-world application of NVMath Python by external users.

• Setup: Researchers at Duke University were developing a supernova analysis program for a space telescope. They initially accelerated their Python code with CuPy and subsequently integrated NVMath Python libraries. Crucially, this was an independent effort by scientists who primarily use Python and NumPy, without direct intervention from NVIDIA engineers.
• Results: The integration of NVMath Python further enhanced the acceleration of their program. This example underscores the ease of use and immediate value proposition of NVMath Python, enabling domain experts to leverage highly optimized GPU libraries directly from their Python code, without requiring deep knowledge of CUDA C++ or low-level GPU programming.

Finally, the performance of the new Python compilation layer for Cutlass 4.0 was discussed.

• Setup: Cutlass, known for providing maximum control and performance for Tensor Core operations, received a new Python interface. The goal was to match the performance of its C++ counterpart.
• Results: The Pythonic Cutlass was shown to deliver performance identical to the C++ version. This is a critical achievement, as it means developers can benefit from the productivity and faster JIT compilation times of Python for fine-grained tensor operations without any performance degradation. This offers a compelling choice for users who need ultimate control over GPU hardware but prefer a Python development workflow.

Collectively, these results paint a clear picture: NVIDIA's strategy to make CUDA more accessible through Pythonic interfaces and higher-level programming models like Coupile is not coming at the expense of performance. The benchmarks consistently show that these new tools can achieve performance levels comparable to, or very close to, highly optimized C++ implementations, thereby empowering a wider range of developers to harness the full power of NVIDIA GPUs.

Practical Implications

The new features and future directions for CUDA discussed in this talk have profound practical implications for a wide spectrum of users, from individual developers to large-scale infrastructure teams.

For practitioners and model builders, the most immediate benefit is a significant boost in productivity and accessibility. The "Pythonic reimagining" of CUDA, spearheaded by CUDA.Core, Ki Numeric, and NVMath Python, allows developers to stay within their familiar Python ecosystem. This means less context switching, fewer dependencies to manage, and more intuitive interfaces for GPU acceleration. Data scientists and ML engineers can now leverage the full power of NVIDIA GPUs, from single-node acceleration with Ki Numeric (a NumPy drop-in replacement that can scale to thousands of GPUs) to advanced library calls via NVMath Python, all while maintaining a Python-native workflow. This dramatically lowers the barrier to entry for GPU computing, enabling faster prototyping and iteration cycles. The ability of NVMath Python to fuse library calls through JIT compilation ensures that this productivity does not come at the cost of performance, often matching or exceeding manual C++ implementations.

The introduction of Coupile is a game-changer for writing GPU kernels. By abstracting away fine-grained thread management in favor of tile-based, array operations, Coupile makes parallel programming more intuitive and less error-prone. This is particularly beneficial for complex algorithms in AI (e.g., Llama 3.1 inference within 10% of cuDNN) and HPC (e.g., 2D fluid simulations). Developers can focus on the logical structure of their problem rather than low-level GPU specifics, letting the compiler optimize the execution for Tensor Cores and other hardware. This simplifies debugging and maintenance, ultimately accelerating the development of highly optimized applications. The future goal of intermixing tile and thread kernels within a single function will further enhance flexibility, allowing developers to use the optimal programming model for different parts of their workload.

For infrastructure teams and deployers, the talk highlights crucial advancements in tooling and scalability. The emphasis on compilation in the critical path means that tools providing control over JIT optimization levels, caching, and compile-time analysis (e.g., heatmaps, Compile Time Advisor) are essential for optimizing application startup times and overall efficiency. At the data center scale, NVIDIA Nsight Cloud offers invaluable capabilities for debugging distributed GPU workloads by seamlessly injecting tracing agents into containers. Perhaps most impactful is CUDA Checkpoint, which enables GPU-aware checkpointing and live migration of workloads. This capability allows infrastructure teams to dynamically rebalance loads, consolidate workloads onto fewer machines, perform maintenance, or recover from failures without interrupting running applications or requiring developers to instrument their code. This significantly improves resource utilization, system resilience, and operational flexibility in large-scale GPU clusters.

Tradeoffs and Limitations: While these innovations offer immense advantages, developers still face choices. The "one size does not fit all" mantra is a core principle. While Pythonic CUDA and Coupile offer high productivity and excellent performance, applications requiring the absolute maximum, granular control over every GPU nuance might still necessitate direct CUDA C++ or highly specialized Cutlass implementations. However, the fact that Cutlass Python can match C++ performance suggests that these tradeoffs are increasingly about development experience and iteration speed rather than raw execution speed. The CUDA DTX vision for data center-scale CUDA is still in its early conceptual stages, meaning production environments will rely on current distributed computing paradigms and tools for some time. Nonetheless, the direction is clear: NVIDIA is committed to providing a comprehensive, flexible platform that allows users to choose the right level of abstraction for their needs, ensuring both high performance and developer satisfaction.

Key Takeaways

• CUDA is an Expansive Platform, Not Just CUDA C++: The CUDA ecosystem has evolved into a vast, multi-layered stack encompassing drivers, runtimes, hundreds of SDKs, and thousands of community-built tools, all designed to enable accelerated computing on NVIDIA GPUs across diverse applications.
• Python is Now a First-Class Citizen for GPU Acceleration: NVIDIA has made significant investments in "Pythonic CUDA" with tools like CUDA.Core (native runtime), Ki Numeric (NumPy replacement for GPUs), and NVMath Python (JIT-optimized library interfaces). This empowers Python developers to achieve C++-level performance and multi-GPU scaling without leaving their native environment.
• "Compilers Everywhere" Drive Performance and New Tools: Modern accelerated computing features JIT compilation at nearly every layer of the software stack, driven by the need for domain-specific optimization. This necessitates new developer tools for compile-time analysis, optimization control, and caching to manage compilation overhead in the critical path.
• Coupile Simplifies GPU Programming with Tile-Based Abstraction: The new Coupile language extension introduces an intuitive, block-level tile programming model, allowing developers to express computations on arrays and tensors more naturally. This approach simplifies kernel authoring, enhances debuggability, and achieves near-native performance for both AI (e.g., Llama 3.1 inference) and HPC workloads.
• NVIDIA is Pioneering Data Center Scale CUDA: The conceptual CUDA DTX initiative outlines a future where CUDA natively spans entire data centers, with distributed machine models and runtimes designed for resilience, dynamic scheduling, and asynchronous operations across massive node counts. Current tools like CUDA Checkpoint enable GPU-aware workload migration and dynamic load balancing.
• Performance Without Compromise for Productivity: A core tenet of CUDA's evolution is to offer increased developer productivity and higher-level abstractions without sacrificing performance. New Pythonic interfaces and programming models like Coupile consistently demonstrate performance parity with highly optimized C++ code, providing developers with powerful choices.

About the Speaker(s)

Stephen Jones is a distinguished CUDA Architect at NVIDIA, where he plays a pivotal role in shaping the overall CUDA platform, from low-level drivers to high-level programming models. With over 17 years of experience at NVIDIA, starting in 2008 when CUDA was in its nascent stages, he has witnessed and significantly contributed to its evolution from a small C-centric framework to the expansive, multi-faceted ecosystem it is today. His work involves thinking "up and down the whole stack," ensuring that all components of CUDA work synergistically to provide uncompromised performance and developer productivity. Jones is a regular speaker at NVIDIA GTC, where he shares insights into the latest CUDA features and future directions.