Frontiers of AI and Computing: A Conversation With Yann LeCun and Bill Dally | NVIDIA GTC 2025

Frontiers of AI and Computing: A Conversation With Yann LeCun and Bill Dally

Speakers: Yann LeCun, Chief AI Scientist, Meta; Bill Dally, Chief Scientist, NVIDIA

Conference: NVIDIA GTC

YouTube: https://www.youtube.com/watch?v=eyrDM3A_YFc

Overview

This GTC 2025 fireside chat features two of the most influential figures in AI and computing: Yann LeCun, Chief AI Scientist at Meta, and Bill Dally, Chief Scientist at NVIDIA. The conversation delves into the current state and future trajectory of Artificial Intelligence, moving beyond the prevailing focus on Large Language Models (LLMs) to explore more fundamental challenges in achieving advanced machine intelligence. LeCun outlines his vision for AI systems that can truly understand the physical world, possess persistent memory, and exhibit robust reasoning and planning capabilities, advocating for a paradigm shift in model architectures.

The discussion critically examines the limitations of current generative AI approaches, particularly LLMs, and introduces Joint Embedding Predictive Architectures (JEAs), also known as JAPA world models, as a promising alternative for learning abstract representations and enabling genuine reasoning. Dally probes LeCun on the practical implications of these architectural shifts, the role of open-source development, and the escalating hardware demands of future AI systems. The talk provides a candid and forward-looking perspective on the grand challenges of AI, offering insights into the necessary breakthroughs in both algorithms and hardware to unlock truly intelligent machines capable of interacting with and understanding our complex physical reality.

Background

▶ Watch: Yann LeCun's surprising disinterest in LLMs and new focus (0:00)

The current landscape of Artificial Intelligence is largely dominated by Large Language Models (LLMs), which have achieved remarkable success in natural language processing and generation. These models operate by predicting discrete tokens (words, subwords) based on vast amounts of text data, effectively learning the statistical relationships within language. However, Yann LeCun argues that while LLMs have made significant strides and are now largely in the hands of product teams for incremental improvements, they represent a "last thing" in the current wave of AI. He suggests that the community's intense focus on LLMs might be a temporary phase, and more profound challenges lie ahead.

The core problem LeCun identifies is the inherent limitation of LLMs in dealing with the continuous, high-dimensional nature of the physical world. Unlike language, which is fundamentally discrete (tokens typically number around 100,000), real-world data like video or sensory input is continuous and often unpredictable at a granular level. Previous attempts to train systems to understand the world by predicting pixels in video, or reconstructing images at a pixel level from corrupted versions (e.g., Masked Autoencoders like MAE), have largely failed to achieve human-level performance or efficiency. These generative approaches waste computational resources trying to invent unpredictable details, leading to blurred or inaccurate reconstructions.

This problem highlights a critical gap in current AI: the inability to build robust "world models" that allow machines to understand cause and effect, anticipate future states, and plan actions in a physical environment. Human infants acquire such models in their first few months of life, enabling them to interact with and navigate the world. Existing "agentic reasoning" systems built on LLMs, which involve generating vast sequences of tokens and then selecting the best one, are dismissed as computationally inefficient and fundamentally misaligned with how biological intelligence performs planning. This sets the stage for the need for entirely new architectural paradigms that can move beyond token-level prediction and into abstract representation spaces for effective physical world understanding and reasoning.

Key Findings

▶ Watch: The importance of "world models" for AI understanding reality (2:00)

The conversation between Yann LeCun and Bill Dally reveals several critical findings and forward-looking perspectives on the future of AI:

• Limitations of LLMs for Advanced Intelligence: While LLMs are powerful for language tasks, they are fundamentally ill-suited for understanding the physical world, persistent memory, and robust reasoning and planning. Their reliance on discrete tokens makes them inefficient for high-dimensional, continuous data like video, and their current "reasoning" mechanisms (generating and selecting token sequences) are deemed simplistic and computationally unsustainable for complex problems.
• The Rise of Joint Embedding Predictive Architectures (JEAs) / JAPA World Models: LeCun champions JEAs, or JAPA (Joint Embedding Predictive Architecture), as the foundational architecture for future AI. Instead of predicting raw pixels or tokens, JEAs learn abstract, latent representations of data and perform predictions within this representation space. This approach is more robust to the inherent unpredictability of the physical world and avoids wasting resources on reconstructing uninventable details.
• Prediction in Latent Space is Key: The core idea behind JAPA is to encode inputs (e.g., a video segment and its continuation) into a shared, abstract representation space and then predict the future state or transformed version in that space, rather than the raw input space. This strategy has shown superior performance and efficiency compared to pixel-level reconstruction methods like MAE, as demonstrated by models like Dino, Dino V2, and VJA.
• Human-like Reasoning and Planning: True reasoning and planning, according to LeCun, occur in an abstract mental space, not through language tokens. JAPA world models are designed to predict the next state of the world given an imagined action, enabling an agent to plan sequences of actions to achieve a goal, mirroring human cognitive processes.
• Advanced Machine Intelligence (AMI) vs. AGI: LeCun prefers the term AMI over AGI (Artificial General Intelligence), arguing that human intelligence is highly specialized, making "general" a misnomer. He predicts significant progress in developing systems that can learn abstract mental models for reasoning and planning within three to five years, though achieving human-level AMI is still likely a decade or more away.
• The Power of Open Source AI: Meta's decision to open-source models like Llama has been a "revolution" in the AI landscape, fostering an ecosystem of startups and accelerating innovation. Open source is deemed critical for building diverse AI assistants that cater to all languages, cultures, and value systems globally, preventing a monopoly by a few large companies.
• Hardware Demands and Future Technologies: The development of JAPA world models and "System 2" reasoning will be computationally expensive, requiring continuous advancements in GPU capabilities and scale-out architectures. While neuromorphic hardware and quantum computing are largely dismissed for general AI applications in the near term, in-memory processing and sensor-level computation are seen as promising for power-constrained edge devices (e.g., smart glasses) to reduce data shuffling costs.
• The Data Scale Discrepancy: LeCun starkly illustrates that training on text alone (e.g., 30 trillion tokens, equivalent to 0.9 x 10^14 bytes) is insufficient for physical world understanding. A 4-year-old processes a comparable amount of visual data (10^14 bytes) in just 16,000 awake hours, highlighting the need for multimodal, real-world data learning.

Technical Deep Dive

▶ Watch: Why discrete tokens fail to represent the continuous physical world (3:00)

The core of LeCun's argument against the current trajectory of LLMs for achieving advanced machine intelligence lies in the fundamental difference between discrete language and the continuous physical world. LLMs operate by predicting tokens, which are discrete units from a finite vocabulary (e.g., 100,000 words or subwords). When an LLM predicts the next token in a sequence, it outputs a probability distribution over this entire dictionary. While effective for text, this paradigm breaks down when applied to high-dimensional, continuous data like video or rich sensory input.

Attempts to train neural networks to understand the physical world by predicting video at a pixel level have largely failed. The reason is simple: much of the real world is inherently unpredictable at a granular level. If a system is tasked with predicting every pixel in the continuation of a video, it will waste enormous computational resources trying to generate details it cannot possibly know (e.g., the exact appearance of every person in a panning room shot). This leads to blurred, averaged, and inefficient predictions. Similarly, self-supervised learning techniques that rely on reconstructing an image from a corrupted version at the pixel level (like MAE, or Masked Autoencoders) have shown limited success compared to alternative approaches. For instance, an MAE for video, attempting to reconstruct missing patches, was described as a massive computational undertaking ("boiling a small lake") that ultimately failed to produce satisfactory results and was abandoned.

LeCun proposes a fundamental architectural shift towards Joint Embedding Predictive Architectures (JEAs), which he sometimes refers to as JAPA world models. The core idea of JAPA is to move prediction from the input space (pixels, tokens) to an abstract representation space, also known as a latent space.

The JAPA architecture works as follows:

1. Input Encoding: A chunk of data (e.g., a segment of video, an image, or even text) is passed through an encoder to produce a compact, abstract representation.
2. Contextual Encoding: A related piece of data (e.g., the continuation of the video, a transformed version of the image, or a masked version of the text) is also passed through a potentially different encoder.
3. Latent Space Prediction: Instead of reconstructing the original or predicting the future in the raw input space, the system is trained to predict the representation of the future or transformed data from the representation of the initial input.

This approach acknowledges the unpredictability of the physical world and focuses on learning meaningful, high-level features rather than pixel-perfect details. A key challenge in JAPA development is preventing "model collapse," where the system learns to produce a constant, uninformative representation regardless of the input. Techniques to avoid this have only emerged in the last five to six years.

Once a JAPA system can learn these abstract mental models of the world, it can be used for reasoning and planning. The system, having observed the current state of the world (represented in latent space), can imagine taking an action and then predict the resulting next state of the world in that same latent space. By iterating this process, the system can plan a sequence of actions to achieve a desired outcome. LeCun emphasizes that this is how humans and animals perform planning – not by manipulating language tokens, but by simulating scenarios in an abstract mental model. He contrasts this with current agentic reasoning systems that use LLMs to generate "lots and lots and lots of sequences of tokens" and then use a second neural net to select the best one, likening it to writing a random program and testing it, which is "completely hopeless" for anything beyond very short sequences due to exponential complexity.

This distinction ties into the psychological concepts of System 1 and System 2 thinking. System 1 refers to automatic, subconscious, reactive tasks (e.g., an experienced driver navigating familiar roads without explicit planning). Current AI, including LLMs, primarily operates in this mode. System 2, conversely, involves deliberate, conscious planning and reasoning (e.g., a novice driver meticulously planning every action and anticipating scenarios). LeCun argues that while current AI is "inching their way towards System 2," a fundamentally different architecture is needed for it, and JAPA is designed to address this.

The development of JAPA has been informed by lessons from prior deep learning breakthroughs. LeCun highlights the significance of ResNet (Residual Networks), a 2015 architecture that introduced residual connections (skip connections) allowing layers to default to an identity function. This seemingly simple idea enabled the training of significantly deeper neural networks (100+ layers) by preserving gradients during backpropagation, circumventing the vanishing/exploding gradient problem that plagued earlier deep architectures. Similarly, the success of GPT-style architectures over earlier BERT-type denoising autoencoders for NLP demonstrated the power of a "good recipe" – a combination of architectural choices (causal transformers), training objectives (auto-regressive prediction on entire text), and optimization techniques that scale effectively. LeCun believes that the missing piece for JAPA's widespread success is the discovery of such a robust "recipe" that allows it to scale effectively.

Experimental Setup & Results

▶ Watch: Explaining JEPAs: Learning abstract representations instead of pixel prediction (4:10)

While this talk is a high-level conversation, LeCun references specific experimental outcomes to illustrate his points about the limitations of generative models and the promise of Joint Embedding Predictive Architectures (JEAs).

Failed Generative Approaches for Video:

LeCun describes a project at Meta (Fair) that attempted to apply the Masked Autoencoder (MAE) concept, which had shown some success for images, to video.

• MAE for Images: This technique involves taking an image, corrupting it by masking out large portions, and then training a neural network to reconstruct the full image at a pixel or token level. The internal representation is then used for downstream tasks. LeCun notes it "works okay" but required immense computational resources, metaphorically "boil[ing] a small pond to cool down those liquid cooled GPU clusters."
• MAE for Video: The team extended this to video by tokenizing video into 16x16 patches and training a "gigantic neural net" to reconstruct missing patches or predict future video frames. This required even more extreme computational power, "boiling a small lake, not a small pond." Despite the effort, the project was "basically a failure" and was stopped, underscoring the inherent difficulty and inefficiency of pixel-level prediction for high-dimensional, continuous, and often unpredictable video data.

Promising Results with Video JEA (VJA):

In contrast, LeCun highlights the success of VJA (Video JEA), which applies the joint embedding predictive architecture principles to video.

• Architecture: VJA operates by taking a full video (or a segment) and a partially masked version of it. Both are passed through encoders, and the system is trained to predict the representation of the full video from the representation of the partially visible one in the latent space, rather than reconstructing pixels.
• Training & Capability: The first version of VJA was trained on very short videos (just 16 frames). Crucially, this system demonstrates an ability to discern whether a particular video is "physically possible or not, at least in restricted cases."
• Mechanism: It achieves this by measuring the prediction error when trying to predict the next few frames in a sliding window. When something "really strange happens in the video" – an object disappears, changes shape spontaneously, or violates physics – the prediction error significantly increases, signaling a physically unrealistic event.
• Analogy to Infant Learning: LeCun draws a parallel to human infant development. Babies learn intuitive physics (e.g., gravity) around 9-10 months. Before this age, they are not surprised by objects appearing to float; afterwards, they exhibit surprise (measurable attention), indicating a violation of their internal world model. VJA exhibits a similar capability to detect such violations based purely on observation.

The Data Scale Discrepancy:

LeCun provides a compelling quantitative argument for why text-only training is insufficient for general intelligence:

• LLM Training Data: Current LLMs are typically trained on approximately 30 trillion tokens. Given that a token is roughly 3 bytes, this amounts to about 0.9 x 10^14 bytes (or 90 terabytes) of text data. LeCun estimates that it would take a human over 400,000 years to read through this volume of text.
• Human Visual Data: A 4-year-old child has been awake for approximately 16,000 hours. The human visual cortex receives about 2 megabytes per second through the optic nerve. Multiplying these figures (2 MB/s 3600 s/hr 16,000 hr), a 4-year-old processes roughly 10^14 bytes of visual data.

This comparison reveals that a 4-year-old processes as much visual data in four years as a large LLM processes text data over hundreds of millennia of simulated reading. This stark discrepancy highlights the inadequacy of text-based training for building AI that understands the richness and complexity of the physical world, emphasizing the critical need for multimodal, real-world data learning.

Practical Implications

▶ Watch: JEPAs as the foundation for true AI reasoning and planning (6:00)

The insights shared by Yann LeCun and Bill Dally have profound practical implications for practitioners, infrastructure teams, model builders, and deployers in the AI/ML space.

Shifting Focus Beyond LLMs: For model builders and researchers, the primary implication is the need to diversify beyond LLMs. While LLMs will continue to have many applications, the "next frontier" of AI, particularly for robotics, autonomous systems, and advanced reasoning, demands new architectural paradigms like JAPA world models. This means investing in research and development for models that learn abstract representations from continuous, high-dimensional data rather than just discrete tokens.

Reliability and Deployment Challenges: A significant takeaway for deployers and infrastructure teams is the persistent difficulty in fielding and deploying AI systems with the required accuracy and reliability. LeCun points to autonomous driving as a prime example, where Level 5 autonomy remains a "receding horizon." This challenge extends to other critical applications like medicine. The "fix for this is better AI," specifically systems with common sense, reasoning capabilities, and the ability to assess the reliability of their own answers. This implies a need for more robust testing, validation, and integration strategies, moving beyond "flashy demos" to production-ready systems.

AI as Power Tools, Not Replacements: For general practitioners and end-users, AI is primarily envisioned as a suite of "power tools" that enhance human productivity and creativity. Examples include coding assistants, medical imaging pre-screeners (reducing MRI time, detecting tumors in mammograms), and driving assistance systems (reducing collisions by 40%). This perspective suggests that the immediate future of AI involves augmentation rather than wholesale replacement, empowering individuals in their respective fields.

The Indispensability of Open Source AI: Meta's strategy with Llama highlights the transformative power of open-source AI. For startups and vertical application builders, open-source foundation models offer a cost-effective platform for prototyping and deployment, allowing them to fine-tune models on proprietary data without uploading it to external services. More broadly, open source is presented as essential for fostering a diverse ecosystem of AI assistants that can cater to all the world's languages, cultures, and value systems. This counters the risk of a few monolithic companies monopolizing AI development and ensures a robust, democratic information landscape, much like a diverse press. It also facilitates distributed training models where regions can contribute data without relinquishing control.

Continuous Demand for Hardware Innovation: Infrastructure teams and hardware developers face an unrelenting demand for increased computational power. JAPA models, particularly those enabling "System 2" reasoning, are expected to be "expensive computationally at runtime." This necessitates continuous advancements in GPUs (as exemplified by NVIDIA's progress from Kepler to Blackwell) and large-scale, high-performance computing clusters. LeCun's plea to Bill Dally, "keep them coming, you know, because we're going to need all the competition that we can get our hands on," underscores this need.

Niche for Specialized Hardware (Edge Computing): While general-purpose neuromorphic hardware and quantum computing are largely dismissed for mainstream AI, LeCun sees a promising role for specialized, low-power processing at the edge. Technologies like in-memory computing or processor-in-memory (PIM) could enable "processing on the sensor directly" for power-constrained devices like smart glasses or home appliances. The primary benefit here is reducing the energy cost of "shuffling data out of the chip," which is often more expensive than the computation itself. This aligns with biological systems like the retina, which performs significant pre-processing to compress visual data before sending it to the brain.

The "Good Recipe" Challenge: For model developers, the biggest bottleneck for JAPA's widespread adoption is finding the "good recipe" – the combination of architectural tricks, optimization strategies (e.g., Adam optimizers, normalization techniques, residual connections), and training methodologies that allow these architectures to scale reliably and effectively, much like ResNet did for deep convolutional networks or causal transformers did for LLMs. This implies a significant amount of empirical and theoretical research is still required.

Key Takeaways

• Beyond LLMs: The future of advanced machine intelligence lies in systems that can understand the physical world, possess persistent memory, and perform robust reasoning and planning, moving beyond the current focus on discrete language tokens.
• JAPA World Models: Joint Embedding Predictive Architectures (JEAs), or JAPA world models, are proposed as the key architectural shift. These models learn abstract representations and perform predictions in a latent space, proving more effective and efficient for high-dimensional, continuous data like video than pixel-level reconstruction.
• Human-like Reasoning: True reasoning and planning are non-linguistic, occurring in an abstract mental space. JAPA models, by predicting future states based on imagined actions, aim to emulate this "System 2" cognitive process, which is fundamentally different from current LLM-based "agentic reasoning."
• Open Source for Diversity and Innovation: Open-source AI platforms (like Llama) are critical for accelerating innovation, enabling specialized vertical applications, and ensuring a diverse ecosystem of AI assistants that can cater to all global languages, cultures, and value systems, preventing centralization of AI power.
• Data Scale and Modality: Relying solely on text data for training is insufficient for achieving human-level intelligence due to the vast disparity in data volume and richness compared to visual and other sensory input processed by humans (e.g., a 4-year-old processes as much visual data in 4 years as an LLM processes text in 400,000 years).
• Persistent Hardware Demand: The development and deployment of next-generation AI models, especially computationally intensive JAPA world models, will continue to drive an insatiable demand for ever-increasing GPU capabilities and scalable computing infrastructure, with niche opportunities for power-efficient in-memory processing at the edge.

About the Speaker(s)

Yann LeCun is a pioneering figure in the field of Artificial Intelligence, renowned for his foundational work on convolutional neural networks (CNNs), which are now ubiquitous in computer vision. As the Chief AI Scientist at Meta, he leads research efforts pushing the boundaries of AI, particularly focusing on self-supervised learning, world models, and architectures that can enable machines to understand and reason about the physical world. His perspectives often challenge conventional wisdom in the AI community, advocating for paradigm shifts necessary to achieve truly advanced machine intelligence.

Bill Dally serves as the Chief Scientist at NVIDIA, a company at the forefront of AI hardware innovation. He is a distinguished computer scientist known for his significant contributions to high-performance computing, parallel architectures, and interconnection networks. His work has been instrumental in shaping the technological landscape that enables modern AI, particularly the development of GPUs as the primary accelerators for deep learning. In this conversation, Dally expertly guides the discussion, probing LeCun on the hardware implications and practical challenges of his visionary AI architectures.