Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

Speakers: Vaishnavh Nagarajan, Chen Wu, Charles Ding, Aditi Raghunathan

Conference: ICML 2025

YouTube: https://slideslive.com/39044025

Overview

This article delves into the ICML 2025 outstanding paper, "Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction," presented by Vaishnavh Nagarajan and Chen Wu, alongside co-authors Charles Ding and Aditi Raghunathan. The talk addresses a critical frontier in artificial intelligence: enhancing the creative abilities of large language models (LMs) beyond their increasingly sophisticated deterministic reasoning capabilities. While LMs have demonstrated remarkable progress in tasks with verifiable, single correct answers, their performance in open-ended, creative endeavors—such as scientific discovery, novel dataset generation, or exploring diverse problem-solving strategies—remains a significant challenge.

The core of this research investigates the fundamental limitations of next-token prediction, the predominant learning paradigm for LMs, when confronted with tasks demanding originality, diversity, and the inference of implicit global structures. The authors propose that next-token learning often encourages models to take "local shortcuts," hindering their capacity for genuine creativity. To overcome this, the paper introduces novel minimal open-ended tasks inspired by cognitive science, designed to isolate and analyze distinct modes of creativity. Furthermore, it explores alternative learning objectives, such as multi-token learning (including teacherless training and diffusion models), and a surprisingly effective diversity elicitation technique called seed conditioning.

This work is pivotal because it moves beyond the current debate on the existence of creative limits in LMs to offer concrete insights and actionable strategies for transcending them. By demonstrating that specific architectural and training modifications can significantly unlock creative potential, especially in larger models, the research provides a roadmap for developing LMs capable of truly innovative and diverse outputs. Its recognition as an outstanding paper at ICML 2025 underscores its profound implications for the future of AI, particularly in domains where human-like inventiveness is paramount.

Background

▶ Watch: Going beyond creative limits of next-token prediction (0:00)

The rapid advancements in large language models (LMs) have revolutionized various fields, enabling unprecedented capabilities in tasks ranging from translation and summarization to complex reasoning. However, as these models become more powerful, a critical limitation has emerged: their struggle with genuine creativity in open-ended settings. While LMs can mimic creative styles and generate plausible text, their ability to produce truly original, diverse, or out-of-distribution content—essential for tasks like scientific discovery, designing novel molecules, or generating diverse solutions to complex problems—is often constrained.

The fundamental challenge lies in the nature of how most LMs are trained: next-token prediction. This paradigm, while highly effective for learning sequential data and local dependencies, inherently optimizes for predicting the most probable next element based on preceding context. This local optimization often leads models to take "local shortcuts," memorizing common patterns or trivially fitting observed data, rather than inferring the deeper, global, and often implicit structures that underpin human creative processes. In open-ended tasks, where there isn't a single "correct" answer but rather a spectrum of high-quality, diverse, and original possibilities, next-token prediction can lead to mode collapse, where the model converges on a limited set of common outputs, failing to explore the full breadth of the solution space.

The scientific community has engaged in a substantial debate regarding the creative limits of these models. Key questions persist: Are LMs genuinely diverse, or are they merely collapsing into a few dominant modes? How much untapped creative potential remains within these architectures, and what training methodologies are required to unlock it? Furthermore, how should randomization be effectively integrated into the model's generation process to foster diversity without sacrificing coherence or quality?

Prior work, such as "The Pitfalls of Next-Token Prediction," has already highlighted the limitations of next-token learning in certain deterministic planning tasks, setting the stage for this investigation into creative contexts. To systematically address these questions, the authors drew inspiration from cognitive scientists who have extensively studied human creativity. By distilling complex creative processes into "minimal open-ended tasks," the researchers created controlled environments where specific aspects of creativity could be isolated, measured, and analyzed. These tasks, while simple, were designed to be non-trivial enough to reveal deep insights into the mechanisms and limitations of current LM paradigms, paving the way for more effective strategies to foster true AI creativity.

Key Findings

▶ Watch: Combinational creativity: connecting disparate ideas (e.g., wordplay) (2:30)

The research presented in "Roll the dice & look before you leap" delivers several pivotal findings that challenge conventional wisdom about language model creativity and propose concrete pathways for improvement.

First, the study meticulously isolates and defines two distinct modes of creativity: Combinational Creativity and Exploratory Creativity. Combinational creativity involves identifying surprising connections between disparate concepts, akin to drawing analogies or discovering contradictions within a knowledge base. This was abstracted into symbolic graph tasks, such as generating strings that reveal a "triangle" structure from sibling-parent relationships. Exploratory creativity, on the other hand, focuses on inventing entirely new strings or structures subject to a small set of rules, exemplified by designing novel molecules or generating adjacency lists that resolve into interesting, previously unseen graph structures like circles or lines. This clear distinction provides a framework for analyzing specific creative challenges.

A central discovery is the inherent limitation of next-token learning for creative tasks. The authors hypothesize and demonstrate that this prevalent training paradigm encourages models to take "local shortcuts." Instead of inferring the global, implicit structure that drives a creative output (e.g., the underlying plan for a wordplay punchline), next-token models tend to fit the output trivially or even memorize parts of it. This local optimization significantly hampers their ability to generate genuinely original and diverse content.

Crucially, the paper reveals the superior efficacy of multi-token learning objectives in fostering creativity. Methods like teacherless training and diffusion models, which mask out parts of the input, explicitly discourage local shortcuts. By forcing the model to reconstruct larger, non-local segments, these approaches encourage the inference of global patterns and underlying structures. The experiments showed a "huge jump in creativity" when moving from next-token learning to teacherless training, particularly for larger models like Gemma. While smaller models like GPT-2 did not exhibit the same gains with teacherless training (consistent with prior work on scale requirements for multi-token learning efficacy), diffusion models proved more robust, showing significant improvements even at smaller scales.

Finally, the research introduces a remarkably simple yet highly effective method for eliciting diversity: seed conditioning. Unlike conventional temperature sampling, which randomizes output generation and can be computationally burdensome as the model must "flesh out all diverse thoughts in parallel," seed conditioning randomizes the input. By simply prefixing random tokens to the input during both training and inference (without complex discriminators or autoencoders), the method encourages the model to generate distinct outputs. Surprisingly, even with zero temperature, seed conditioning achieved diversity comparable to, and in some cases even surpassed, traditional temperature sampling, establishing itself as a powerful and practical alternative for fostering creative output diversity.

Technical Deep Dive

▶ Watch: Exploratory creativity: inventing new strings from rules (4:00)

The core technical contribution of this work lies in its systematic deconstruction of creativity into measurable components and its rigorous evaluation of different learning paradigms against these components. The authors begin by framing the problem within a minimal objective setting: given a set of examples, minimize an objective, and then extract multiple independent generations from the model. Creativity is then quantitatively measured by the number of unique, unseen, and coherent examples generated by the model—a metric made tractable by the symbolic nature of their designed tasks.

The study zooms in on two fundamental technical questions:

1. How do we provide learning signals to the model to infer non-trivial, implicit structures?
2. How is diversity effectively elicited from the model's generative process?

The first question addresses the challenge of implicit structure. Human creative processes often involve an unobserved, implicit planning stage. For instance, in a wordplay joke, the punchline connects two seemingly disparate setup nodes, but this underlying structural connection is implicit within the text. Similarly, a protein's structure implicitly obeys biochemical rules. Next-token learning, by its nature, learns only from local signals (predicting the next token given the preceding ones). The authors hypothesize that this local focus prevents LMs from inferring the end-to-end global process. Instead of modeling the plan (e.g., "first plan the punchline, then pick two setup nodes"), the next-token model, seeing the setups, might trivially fit the punchline as a "parent" of two "siblings" or, worse, simply memorize setup pairs due to a lack of stronger, global signals.

To overcome this, the paper contrasts next-token learning with two multi-token learning methods:

• Teacherless Training: This method involves masking out parts of the input sequence. By requiring the model to reconstruct these masked segments, it discourages reliance on purely local dependencies and forces the model to learn more global patterns and relationships within the data. This is conceptually similar to masked language modeling objectives where the model predicts missing tokens from context, but applied in a creative generation context.
• Diffusion Models: These are another class of generative models that operate by progressively denoising data, often starting from random noise. In the context of multi-token learning, diffusion models also learn global data distributions by iteratively refining noisy inputs, making them less susceptible to the local shortcut problem characteristic of next-token prediction.

The second technical question focuses on diversity elicitation. Traditionally, diversity in transformer models is achieved via temperature sampling, where a higher "temperature" makes the probability distribution over next tokens flatter, encouraging the model to sample less probable, more diverse options. However, the authors posit that this is suboptimal. For temperature sampling to work effectively, the model must conceptually "flesh out all the diverse thoughts in parallel," which is computationally and representationally burdensome.

Their proposed alternative, seed conditioning, offers a "much more naive" yet effective approach. Instead of randomizing the output generation process (like temperature sampling), seed conditioning randomizes the input. This is achieved by simply prefixing random, meaningless tokens to the input sequence for each sample, during both training and inference. Crucially, this method does not involve discriminators or autoencoders, simplifying its implementation compared to paradigms like GANs or VAEs that also leverage input randomization. The intuition is that by introducing varying initial "seeds" into the input, the model is encouraged to explore different latent pathways, leading to distinct and diverse outputs, effectively "fleshing out one thought at a time" rather than parallelizing many.

The minimal open-ended tasks themselves are symbolic graph tasks:

• Combinational Creativity Tasks: Examples include generating interesting strings with specific graph structures (e.g., triangles) from a sibling-parent graph. These tasks require the model to identify and synthesize non-obvious connections.
• Exploratory Creativity Tasks: Here, the model generates adjacency lists that secretly resolve to a particular interesting graph (e.g., a circle graph or a line graph), where the resolution itself must be novel. This demands invention subject to rules, without explicit prior examples of the exact novel resolution.

The creativity score, defined as the fraction of unique, unseen (not in the training set), and coherent (obeying task-specific rules) generated strings, provides a quantitative and objective measure within these symbolic domains, circumventing the subjectivity often associated with evaluating creativity in natural language.

Experimental Setup & Results

▶ Watch: Key questions: structure, diversity, and next-token prediction limits (5:00)

The experimental design for "Roll the dice & look before you leap" focused on systematically comparing different learning objectives and diversity elicitation methods using the specially designed minimal open-ended tasks.

Model Architectures & Scale:

The experiments utilized two distinct language model scales:

• Gemma: A larger, more capable model, representing the frontier of modern LMs.
• GPT-2: A smaller, foundational transformer model, used to evaluate the impact of scale on the observed phenomena.

Training Objectives Compared:

The study directly contrasted three primary training paradigms:

1. Next-token learning: The standard autoregressive objective, predicting the subsequent token.
2. Teacherless training: A multi-token learning objective, where parts of the input are masked, forcing the model to infer global structures.
3. Diffusion models: Another multi-token learning objective, designed for generative tasks by iteratively denoising.

Diversity Elicitation Methods Compared:

Two main approaches for generating diverse outputs were evaluated:

1. Temperature sampling: The conventional method where a temperature parameter controls the randomness of token sampling.
2. Seed conditioning: The novel technique introduced in this paper, involving prefixing random tokens to the input.

Creative Tasks:

The evaluation was performed on the custom-designed symbolic graph tasks that abstractly represent combinational and exploratory creativity. These tasks, such as generating specific graph structures (e.g., triangles from sibling-parent relationships) or novel adjacency lists for target graphs (e.g., circles, lines), allowed for precise measurement of uniqueness and coherence, which are often subjective in natural language.

Headline Results:

1. Impact of Multi-token Learning on Creativity:

• For Gemma (larger model): The results showed a "huge jump in creativity" when transitioning from next-token learning to teacherless training. This significant improvement was consistent across all the creative tasks studied, strongly supporting the hypothesis that discouraging local shortcuts by encouraging global pattern learning unlocks greater creative potential in larger models.
• For GPT-2 (smaller model): In contrast, the smaller GPT-2 model did not exhibit such substantial gains with teacherless training. This finding aligns with existing research indicating that the efficacy of multi-token learning objectives often requires a certain scale in the model to manifest.
• Diffusion Models: Even at the smaller scale where teacherless training showed limited benefit for GPT-2, diffusion models demonstrated significantly better performance than next-token learning. This suggests that diffusion models might be more robust in fostering creativity across a broader range of model scales, with the exception of one specific dataset where all methods performed well.

1. Efficacy of Seed Conditioning for Diversity:

• The experiments revealed that seed conditioning is a highly effective approach for eliciting diversity. Remarkably, even when operating with zero temperature (meaning the model deterministically picks the most probable next token after the random input prefix), seed conditioning achieved diversity comparable to, or in some instances, even superior to, traditional temperature sampling.
• This result underscores seed conditioning as a powerful and simple alternative for encouraging varied outputs, highlighting its potential utility in scenarios requiring high diversity without the computational overhead or potential coherence issues sometimes associated with high temperature sampling. The presenters noted that in some cases, it was "even the most creative method."

Creativity Metric:

The quantitative creativity score used was the fraction of generated strings that were (1) unique, (2) unseen in the training set, and (3) coherent (i.e., obeyed the rules of the specific symbolic task). This metric was described as "easily computable" in their symbolic settings, providing a clear and objective measure of creative output. While acknowledging that other metrics like precision/recall could offer different insights, the chosen metric served its purpose effectively for this foundational study.

The study did not provide specific details on hardware, dataset sizes beyond "minimal open-ended tasks," or framework versions, focusing instead on the conceptual and comparative results of the learning paradigms and elicitation techniques.

Practical Implications

▶ Watch: Hypothesis: next-token models fail to learn global creative process (6:00)

The findings from "Roll the dice & look before you leap" carry significant practical implications for researchers, machine learning engineers, and practitioners involved in developing, deploying, and utilizing large language models. The work offers actionable insights into how to foster genuine creativity and diversity in AI systems, moving beyond mere deterministic output generation.

For Model Builders and Researchers:

• Beyond Next-Token Prediction: The most critical implication is the demonstrated limitation of next-token learning for creative tasks. Practitioners aiming for truly creative or diverse outputs should seriously consider adopting multi-token learning objectives like teacherless training or diffusion models during the pre-training phase. These objectives, by forcing models to learn global patterns, are shown to unlock significantly higher creative potential, especially in larger models such as Gemma. This suggests a paradigm shift in foundational LM training for creativity-focused applications.
• Scale Matters for Multi-token Learning: While multi-token learning is beneficial, its efficacy can be scale-dependent. Smaller models like GPT-2 did not show the same "huge jump" in creativity with teacherless training, indicating that infrastructure teams and model builders might need to consider model size when selecting advanced pre-training objectives. However, the superior performance of diffusion models even at smaller scales offers a promising avenue for models with fewer parameters.
• Prioritizing Base Model Improvement: The authors emphasize the importance of improving the base model during pre-training, arguing against squandering the "massive amount of signal" invested. This suggests that foundational creative capabilities are best instilled early, rather than solely relying on post-training techniques like fine-tuning or reinforcement learning, which might only refine existing, potentially limited, creative latent spaces.

For Infrastructure and Deployment Teams:

• Efficient Diversity Elicitation: The introduction of seed conditioning offers a remarkably simple and effective method for eliciting diverse outputs. Unlike temperature sampling, which can be resource-intensive due to the need for parallel thought exploration, seed conditioning merely involves prefixing random tokens to the input. This ease of implementation, coupled with its strong performance (even at zero temperature), makes it an attractive alternative for deployment scenarios where diverse output generation is crucial, such as in scientific hypothesis generation, game design, or creative writing tools.
• Trade-offs in Creativity vs. Efficiency: While multi-token learning objectives yield higher creativity, they might introduce complexities in training pipelines or require more computational resources. Teams need to evaluate these trade-offs against the specific creative demands of their applications.

For Practitioners (Model Users):

• Understanding Model Capabilities: Users of LMs should understand that a model's inherent creative capacity is influenced by its foundational training. Models trained predominantly with next-token prediction might excel at coherent, contextually relevant outputs but may struggle with true originality or wide diversity.
• Leveraging New Elicitation Techniques: When diverse outputs are needed, practitioners can explore tools or APIs that incorporate seed conditioning, potentially yielding more varied and surprising results than traditional temperature-based sampling.

Limitations and Future Considerations:

• "Spherical Cows" and Real-World Transfer: The study acknowledges its use of "spherical cows"—minimal, symbolic tasks. While invaluable for controlled analysis, the direct transferability of these insights to complex, open-ended real-world applications (e.g., generating novel protein structures, writing a groundbreaking scientific paper) requires further verification. The notions of uniqueness and coherence become "much more soft in text" than in symbolic tasks.
• Uncharted Dimensions: The research highlights that many dimensions of creativity remain uncharacterized and unexplored. Future work needs to analyze these and integrate advances in post-training methods.
• Understanding "Why" Seed Conditioning Works: While effective, the underlying reasons for seed conditioning's remarkable performance, especially in its "naive" version without discriminators, warrant deeper theoretical understanding.
• Out-of-Distribution Creativity: The current tasks largely focus on in-distribution creativity. Defining tasks that capture truly out-of-distribution creativity remains an exciting, yet challenging, future direction.

In summary, this work provides a compelling case for rethinking foundational LM training paradigms to unlock creative capabilities. It offers practical tools and a conceptual framework that can guide the development of the next generation of AI systems, capable not just of reasoning, but also of genuine innovation.

Key Takeaways

• Large language models trained with next-token prediction often take "local shortcuts," hindering their ability to infer global, implicit structures essential for true creativity and diversity in open-ended tasks.
• Multi-token learning objectives, such as teacherless training and diffusion models, significantly enhance an LM's creative output by discouraging local dependencies and encouraging the learning of global patterns, particularly effective in larger models like Gemma.
• Seed conditioning, a simple input randomization technique (prefixing random tokens), is a highly effective and efficient alternative to temperature sampling for eliciting diverse and creative outputs, even with zero temperature.
• The choice of learning objective during pre-training is critical for establishing a model's foundational creative capabilities, suggesting that improving the base model is paramount for fostering creativity.
• Minimal, symbolic graph tasks provide a valuable and quantifiable framework for dissecting and understanding the nuanced challenges of AI creativity, offering insights transferable to more complex domains.
• Future research must bridge the gap between insights from symbolic tasks and real-world applications, further understand the mechanisms behind methods like seed conditioning, and explore truly out-of-distribution creativity.

About the Speaker(s)

The talk "Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction" was presented by Vaishnavh Nagarajan and Chen Wu. They are among the co-authors of this outstanding paper, which also includes Charles Ding and Aditi Raghunathan. Their collaborative work, recognized as an outstanding paper at ICML 2025, highlights their contributions to advancing the understanding and capabilities of artificial intelligence in the domain of creativity and open-ended problem-solving.