Advancing AI Reasoning - From Games to Complex Problem Solving | NVIDIA GTC 2025 Session

Advancing AI Reasoning - From Games to Complex Problem Solving

Speakers: Vardika Singh (Moderator, Strategic Technical Partner Engagements, NVIDIA), Noam Brown (OpenAI), Bryan Catanzaro (VP of Applied Deep Learning Research, NVIDIA)

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=c675KAlmo8k

Overview

This GTC session, "Advancing AI Reasoning - From Games to Complex Problem Solving," convened two leading figures at the forefront of AI innovation: Noam Brown from OpenAI, renowned for his breakthroughs in AI for strategic games, and Bryan Catanzaro, VP of Applied Deep Learning Research at NVIDIA, a pioneer in scalable AI systems and hardware. Moderated by NVIDIA's Vardika Singh, the panel explored the pivotal moment in AI where algorithmic advancements are converging with ever-expanding computational capabilities. The discussion traced the evolution of AI reasoning from mastering complex games like poker and diplomacy to tackling open-ended, real-world problems, highlighting the symbiotic relationship between novel algorithms and powerful computing infrastructure.

The talk delved into the historical trajectory of AI in games, contrasting early approaches with modern, reasoning-centric paradigms. It underscored a fundamental shift in how AI systems are designed and optimized, moving from pre-training-heavy models to those that leverage significant inference-time compute for deeper, more flexible reasoning. This transformation, enabled by both algorithmic ingenuity and advancements in GPU-accelerated computing, holds profound implications for the future of AI, promising to unlock new levels of intelligence for scientific progress and societal challenges.

The significance of this conversation lies in its articulation of a new era for AI development, where the emphasis shifts from raw model size to the efficiency and depth of reasoning capabilities. It illuminates how breakthroughs in game AI, particularly in imperfect information and natural language domains, are providing critical blueprints for building more generally intelligent systems. Furthermore, it emphasizes NVIDIA's vision of end-to-end optimization across the AI stack, ensuring that computational resources are precisely tailored to meet the evolving demands of advanced AI reasoning, ultimately aiming to make complex problem-solving accessible and affordable at scale.

Background

▶ Watch: Introducing AI reasoning experts Noam Brown and Bryan Catanzaro (1:00)

The journey of AI reasoning has historically been punctuated by landmark achievements in game playing, serving as crucial benchmarks for measuring and advancing machine intelligence. Early successes, such as Deep Blue in chess in 1997, demonstrated superhuman performance in perfect information games—where all players have complete knowledge of the game state. However, extending these techniques to imperfect information games, like poker, proved significantly more challenging. These games introduce elements of hidden information, bluffing, and strategic deception, rendering traditional search algorithms like Alpha-Beta pruning (used in chess) and Monte Carlo Tree Search (used in Go) ineffective.

Noam Brown's early work, beginning with his PhD at Carnegie Mellon around 2012, directly addressed this gap. His research focused on enabling superhuman AI in imperfect information games, particularly poker. At the time, the prevailing approach for poker AI involved extensive pre-training to build large lookup tables, with inference being extremely fast but shallow. This "system one" thinking, as Brown describes it, lacked the deep, deliberative reasoning that human players employ in complex situations. The high variance of poker and the immense compute required for deeper reasoning were significant deterrents for researchers.

Simultaneously, the computational landscape for AI was undergoing its own revolution. Bryan Catanzaro, starting his work on AI systems in 2008 and joining NVIDIA in 2011, was a key architect in making GPUs indispensable for deep learning. His early work on training models on GPUs, leading to the creation of cuDNN (NVIDIA's foundational library for AI on GPUs), laid the groundwork for the massive scaling of neural networks. NVIDIA's philosophy of "end-to-end optimization"—building not just hardware (GPUs, networking) but also compilers, libraries, frameworks, and even working on algorithms and applications—was driven by the belief that accelerating fundamental computations could unlock unprecedented AI capabilities. This commitment to computational efficiency became critical as AI models grew in complexity and scale, moving beyond simple pattern recognition to more intricate reasoning tasks.

The convergence of these two trajectories—algorithmic breakthroughs in game AI and continuous advancements in scalable computing—set the stage for a new era of AI reasoning. The challenge was to bridge the gap between highly specialized game-playing AIs and more general-purpose intelligence, particularly in domains involving natural language and dynamic human interaction.

Key Findings

▶ Watch: Achieving human-level AI in the natural language game Diplomacy (2:45)

The talk highlighted several pivotal findings and contributions that collectively advance the state of AI reasoning:

1. Superhuman AI in Imperfect Information Games: Noam Brown's work with Liberatus (2017) and Pluribus (2019) demonstrated the first superhuman performance in two-player and then multiplayer no-limit poker, respectively. This was a significant breakthrough, proving that AI could master domains where information is incomplete and deception is key.
2. Shift to Inference-Heavy Reasoning: A core insight from the poker AI work was the critical importance of inference compute for reasoning. Unlike prior approaches that relied on extensive pre-training and fast lookups ("system one"), Liberatus and Pluribus leveraged significant real-time computation during play ("system two" thinking) to achieve superior performance. This paradigm shift involved algorithms that allowed the AI to "think for longer" before acting.
3. Algorithmic Efficiency and Cost Reduction: The evolution from Liberatus to Pluribus showcased dramatic improvements in algorithmic efficiency. Pluribus, despite being a more complex multiplayer AI, could be trained for less than $150 on cloud computing, a stark contrast to the millions often associated with frontier AI models. This reduction was attributed to better reasoning techniques that minimized the need for exhaustive pre-training.
4. Human-Level AI in Natural Language Negotiation: The Cicero project (2022) at Meta marked another significant milestone, developing the first human-level AI for the game of Diplomacy. This game, characterized by natural language communication, negotiation, and shifting alliances, pushed AI beyond mechanical strategy into the realm of human-like social intelligence and persuasion.
5. Co-evolution of Algorithms and Systems: Bryan Catanzaro emphasized that AI algorithms and computational systems evolve symbiotically. Major algorithmic breakthroughs, such as the ImageNet moment with AlexNet, were often driven by systems investments that enabled larger, more efficient model training. This virtuous cycle continues, with new reasoning algorithms driving demand for specialized hardware and software architectures.
6. Reasoning as a Massive Compute Problem: The panel identified reasoning, especially during post-training and inference, as an increasingly compute-intensive process. As models "practice" and learn to reason, they require significant computational feedback loops. This represents a shift from the traditional pre-training-heavy model, demanding new approaches to hardware and software design.
7. "Intelligence Per Dollar" as the New Metric: The speakers proposed that in the era of reasoning-capable AI, the traditional notion of "model intelligence" (e.g., a single benchmark score) is insufficient. Instead, the focus should be on "intelligence per dollar" or "intelligence per inference cost." The optimal model is one that maximizes intelligence for a given inference budget, allowing it to "think for longer" and achieve better outcomes.
8. The Complementary Nature of Pre-training and Reasoning: While reasoning is gaining prominence, pre-training remains crucial. It provides the fundamental "foundation" upon which advanced reasoning capabilities are built. Small-scale LLMs, for instance, cannot effectively utilize sophisticated reasoning techniques like OpenAI's O1 system, underscoring the need for robust base models.

These findings collectively point towards a future where AI intelligence is not just about scale, but about the efficiency, depth, and flexibility of its reasoning processes, enabled by a tightly integrated ecosystem of algorithms and advanced computing infrastructure.

Technical Deep Dive

▶ Watch: Bryan Catanzaro's role in creating NVIDIA's KUDNN (4:10)

The technical discussion illuminated the core methodologies and system designs that underpinned these advancements in AI reasoning.

Reasoning in Imperfect Information Games: Poker (Liberatus & Pluribus)

Noam Brown's work on poker AI introduced a paradigm shift from heavily pre-computed strategies to dynamic, inference-time reasoning. Traditional approaches in perfect information games like chess (using Alpha-Beta pruning) and Go (using Monte Carlo Tree Search) rely on exploring vast game trees. However, poker's imperfect information nature, where players do not know opponents' hands, makes these techniques unsuitable.

Instead, the poker AIs, Liberatus and Pluribus, employed algorithms rooted in game theory and Counterfactual Regret Minimization (CFR). While not explicitly detailed, the core innovation was enabling the AI to perform significant computation during play, rather than relying solely on a pre-trained strategy table. This was a move from "system one" (fast, intuitive, pre-computed) to "system two" (slow, deliberate, reasoning-based) thinking. Early poker AIs might have responded in 10 milliseconds using a lookup table. Pluribus, in contrast, utilized 28 CPU cores for approximately 20 seconds per hand during inference. This substantial increase in inference compute allowed the AI to perform a deeper, more nuanced analysis of the game state, considering probabilities, opponent models, and potential outcomes, which is crucial for optimal play in imperfect information scenarios. The algorithmic improvements between Liberatus and Pluribus focused on making this inference-time computation even more efficient and scalable, drastically reducing the training cost.

Natural Language Reasoning and Negotiation: Diplomacy (Cicero)

The Cicero project at Meta tackled the significantly more complex domain of Diplomacy, a seven-player game requiring natural language communication and negotiation. This task moved beyond the structured action spaces of poker or chess, into an open-ended environment where actions are defined by what a player can say to another.

To address this, Cicero integrated neural networks and language models (like the then-emerging GPT-2 in 2019) to handle the complexities of natural language. The system likely combined a strategic reasoning component (similar to game-theoretic approaches for multi-agent settings, though specifically adapted for Diplomacy's cooperative-competitive dynamics) with a sophisticated natural language generation and understanding module. The AI needed to not only understand the strategic implications of its moves but also to formulate persuasive arguments, build and manage alliances, and even engage in deceptive communication—all through natural language dialogues. This required reasoning over a vast, unstructured action space, where the "best move" is often a function of social dynamics and persuasive communication rather than purely mechanical game rules. The techniques developed for Cicero were, by Brown's admission, somewhat domain-specific, highlighting the challenge of achieving broad, flexible reasoning in such complex environments.

NVIDIA's Role in Scalable AI Systems

Bryan Catanzaro articulated NVIDIA's comprehensive approach to accelerating AI, which extends from fundamental hardware to high-level applications.

• GPU Architecture: At the core is the GPU, designed for highly parallelizable computations, particularly matrix multiplications, which form the backbone of deep learning. Catanzaro noted that the efficiency of matrix multiplication is a key reason for its dominance in AI.
• CUDA & cuDNN: NVIDIA's CUDA platform provides a parallel computing architecture, and cuDNN is a GPU-accelerated library of primitives for deep neural networks. These tools significantly optimize common deep learning operations, enabling researchers to train larger models faster.
• DLSS (Deep Learning Super Sampling): An example of AI applied to graphics rendering, DLSS uses machine learning to upscale lower-resolution images to higher resolutions, resulting in 8-10x more efficient graphics rendering. This demonstrates AI's practical application in accelerating compute-intensive tasks beyond traditional training.
• Megatron: NVIDIA's Megatron framework is crucial for scaling large language model training. It addresses the challenges of distributing massive models and datasets across thousands of GPUs, enabling the development of models with billions or trillions of parameters. This involves sophisticated techniques for model parallelism and data parallelism.
• End-to-End Optimization: NVIDIA's strategy is to optimize the entire stack: from silicon (GPU), to interconnects (networking), to software (compilers, libraries, frameworks), and even working on algorithms. This holistic approach ensures that every layer is designed to maximize performance for AI workloads, a philosophy essential for the emerging demands of AI reasoning.

The Future of Reasoning Compute

The panel discussed the profound implications of reasoning for future AI systems. The shift from a "pre-training-heavy" model (where most compute goes into initial training) to one where significant compute is spent on "post-training" and "inference" for reasoning. This means:

• New Architectures: Systems will need to be optimized for sustained, iterative inference. This impacts networking (for moving data and intermediate reasoning steps), low precision arithmetic (for efficiency), and sparsity (leveraging models with fewer active parameters).
• Software for Complex Training: The training process for reasoning models becomes more complex, incorporating large amounts of inference as part of the learning loop. This requires new software stacks that can manage these dynamic, inference-driven training paradigms.
• Direct Link Between Inference Cost and Intelligence: Catanzaro articulated that for reasoning models, inference cost is directly tied to intelligence. A cheaper inference process allows a model to "think for longer" for the same budget, leading to better problem-solving. This necessitates a focus on maximizing "intelligence per inference cost."

This technical evolution points to a future where AI systems are not just faster, but fundamentally smarter, by dedicating computational resources to deeper, more flexible reasoning processes.

Experimental Setup & Results

▶ Watch: Challenges of traditional AI reasoning in imperfect information games (6:25)

The discussion provided concrete examples of AI performance and computational requirements, illustrating the progress in reasoning capabilities.

• Liberatus (2017): This AI was the first to achieve superhuman performance in two-player no-limit Texas Hold'em poker. While specific training details were not extensively covered, Brown noted that early poker AI development involved training models for "two or three months on a large system," initially using CPUs. This initial research phase, particularly for finding the "right paradigm," was acknowledged as "very expensive" in terms of compute resources, even for academic standards at the time (thousands of CPUs).
• Pluribus (2019): Building on Liberatus, Pluribus achieved the unprecedented feat of being the first superhuman AI for multiplayer no-limit Texas Hold'em poker. A key experimental result was the dramatic reduction in training cost: Pluribus could be trained for less than $150 on cloud computing. This incredible efficiency was attributed to algorithmic improvements rather than just increased computational scale. During inference, Pluribus utilized 28 CPU cores and spent approximately 20 seconds per hand to make decisions. This is a significant increase in inference compute compared to previous systems that might have used one CPU for 10 milliseconds, highlighting the shift towards deeper reasoning during play.
• Cicero (2022): Developed at Meta, Cicero achieved human-level performance in the game of Diplomacy. This involved mastering natural language communication, negotiation, and strategic social interaction. The project marked a departure from CPU-centric game AI, requiring a shift to GPUs due to the integration of neural networks and language models for handling natural language negotiation. While specific compute numbers for Cicero's training or inference weren't provided, the necessity of GPUs underscored the computational intensity of natural language processing and reasoning.
• NVIDIA DLSS: As an example of AI-driven efficiency in a different domain, Bryan Catanzaro mentioned that DLSS (Deep Learning Super Sampling) has made graphics rendering 8 to 10 times more efficient through the real-time application of AI. This illustrates how AI, when integrated into systems, can yield substantial performance gains.

The speakers also touched upon the broader landscape of AI research, acknowledging that frontier capabilities research now often requires immense computational resources, making it a challenge for academia. Brown noted that while "radically new paradigms" might not initially require a ton of compute, "proving it out at scale certainly requires a lot of compute." This highlights the resource disparity between academic labs and large frontier labs like OpenAI, which can afford to make "big bets and make investments" in large-scale models and research.

Practical Implications

▶ Watch: Realizing the critical importance of AI spending more time reasoning (7:30)

The insights from this GTC panel carry significant practical implications for various stakeholders in the AI ecosystem:

For Practitioners and Model Builders

• Shift from "System One" to "System Two" Thinking: Model builders must increasingly design systems that can perform deeper, more deliberate reasoning at inference time ("system two"), rather than relying solely on fast, intuitive responses from pre-trained models ("system one"). This means rethinking model architectures and inference pipelines to accommodate more complex, iterative computation during deployment.
• "Intelligence per Dollar" as a Design Metric: The traditional focus on peak performance on benchmarks needs to evolve into optimizing for "intelligence per dollar" or "intelligence per inference cost." Practitioners should consider how much reasoning a model can perform within a given budget, as more inference compute directly translates to better problem-solving in complex scenarios. This encourages efficiency in both algorithms and underlying hardware utilization.
• Developing Broad Reasoning Approaches: The experience with domain-specific reasoning techniques (e.g., for poker or diplomacy) suggests a need for more general and flexible reasoning approaches, akin to how deep learning has provided a flexible paradigm for "system one" tasks. The O1 system from OpenAI was mentioned as a step in this direction, aiming for broad reasoning capabilities.
• Pre-training as a Foundation: While reasoning is ascendant, robust pre-training remains crucial. Model builders should view pre-training as building a strong "foundation" upon which sophisticated reasoning capabilities can be developed. Investing in better base models is complementary to, not replaced by, reasoning research.
• New Training Paradigms: The training process for reasoning models is becoming more complex, integrating significant inference steps into the learning loop. This requires new methodologies for post-training and fine-tuning that allow models to practice and refine their reasoning abilities.

For Infrastructure Teams

• Optimizing for Inference-Heavy Workloads: Infrastructure teams need to prepare for a future dominated by intensive inference compute. This requires designing systems that prioritize efficient, low-latency, and high-throughput inference, rather than solely focusing on large-scale training.
• Specialized Hardware and Software: The demands of reasoning will drive innovations in hardware (e.g., specialized AI accelerators beyond general-purpose GPUs, advancements in low precision arithmetic and sparsity for efficient computation) and software (e.g., advanced compilers, networking solutions for distributed inference, and frameworks optimized for iterative reasoning tasks).
• Cost-Effectiveness of Inference: As inference cost directly correlates with intelligence, infra teams must focus on driving down the per-inference cost through hardware and software optimizations. This includes exploring novel architectures for data movement, memory access, and computational efficiency.

Tradeoffs and Limitations

• Domain-Specific vs. General Reasoning: While domain-specific breakthroughs (like Cicero) provide valuable lessons, they highlight the challenge of generalizing reasoning techniques. Developing broadly applicable reasoning paradigms is a major ongoing research challenge.
• Compute Barrier for Research: The immense compute required for frontier AI research creates a barrier for academic institutions. This necessitates new models of collaboration between academia and well-resourced frontier labs, with academia focusing on "radically new paradigms" and frontier labs on scaling and validating these ideas.
• Human-like Adaptability: Current AI models, even with advanced reasoning, still lack certain aspects of human nature, such as adaptability in novel situations or nuanced understanding of social contexts beyond explicit game rules. This remains an open research area.
• Ethical Considerations: As AI becomes more intelligent and capable, the "downside risks" of powerful technology must be addressed. While not the core focus of this technical discussion, the speakers acknowledged the importance of ethical considerations alongside technological progress.

In essence, the practical implication is a fundamental reorientation of AI development towards deeper, more flexible, and computationally efficient reasoning, driven by a symbiotic evolution of algorithms and underlying infrastructure.

Key Takeaways

• AI Reasoning is Evolving Beyond Games: Breakthroughs in game AI (poker, diplomacy) are providing critical insights for developing AI that can tackle complex, open-ended problems requiring strategic thinking and natural language interaction.
• Inference Compute is Paramount for Reasoning: Modern AI reasoning relies heavily on dynamic, real-time computation during inference ("system two" thinking), rather than solely on pre-trained models. This is a significant shift from earlier AI paradigms.
• Algorithmic Innovation Drives Efficiency: Algorithmic advancements, such as those seen in Pluribus, can drastically reduce the computational cost of training and inference, making complex AI capabilities more accessible and scalable.
• Algorithms and Systems Co-Evolve: Progress in AI is a virtuous cycle where algorithmic breakthroughs drive hardware and software innovation, and vice-versa. NVIDIA's end-to-end optimization approach is crucial for this symbiotic relationship.
• "Intelligence per Dollar" is the New Metric: The effectiveness of advanced AI models is increasingly measured by their ability to deliver intelligence efficiently within a given computational budget, allowing them to "think for longer" and achieve superior results.
• AI Will Augment Human Intelligence: The ultimate vision is for AI reasoning models to augment human capabilities, accelerating scientific progress, solving complex societal problems, and increasing overall productivity.

About the Speaker(s)

Noam Brown is a leading researcher at OpenAI, known for his groundbreaking work in artificial intelligence for strategic games. He began his PhD at Carnegie Mellon around 2012, focusing on AI for imperfect information games, particularly poker. In 2017, his work led to Liberatus, the first superhuman two-player poker AI, followed by Pluribus in 2019, the first superhuman multiplayer poker AI. While at Meta, he continued this line of research, contributing to Cicero, the first human-level diplomacy AI, which involved natural language communication and negotiation. His current work at OpenAI, including the O1 system, aims to develop general reasoning methods for broad applications, moving beyond domain-specific techniques.

Bryan Catanzaro is the Vice President of Applied Deep Learning Research at NVIDIA, where he has been a pivotal figure in architecting scalable training and deployment systems for AI at an industrial scale. His involvement with AI systems dates back to his grad student days, publishing his first paper on training models on GPUs in 2008. He joined NVIDIA full-time in 2011, where his work led to the creation of cuDNN, NVIDIA's first library for AI on the GPU. He has since been involved in numerous projects, including DLSS (Deep Learning Super Sampling) for speeding up graphics rendering with AI, and Megatron, a system for scaling large language model training. Catanzaro champions an "end-to-end" optimization philosophy, encompassing hardware, networking, compilers, libraries, frameworks, and algorithms, to push the boundaries of AI capabilities.