Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction

Vaishnavh Nagarajan, Chen Wu, Charles Ding, Aditi Raghunathan

International Conference on Machine Learning 2025 · Oral

This article delves into the ICML 2025 outstanding paper, "Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction," presented by Vaishnavh Nagarajan and Chen Wu, alongside co-authors Charles Ding and Aditi Raghunathan. The talk addresses a critical frontier in artificial intelligence: enhancing the creative abilities of large language models (LMs) beyond their increasingly sophisticated deterministic reasoning capabilities. While LMs have demonstrated remarkable progress in tasks with verifiable, single correct answers, their performance in open-ended, creative endeavors—such as scientific discovery, novel dataset generation, or exploring diverse problem-solving strategies—remains a significant challenge.

AI review

A well-motivated paper that asks the right question — why does next-token prediction fail at open-ended creative tasks — and proposes two concrete interventions: multi-token learning objectives (teacherless training, diffusion) and seed conditioning. The work is honest about operating in a controlled symbolic sandbox, the experiments are clearly structured, and the creativity metric is at least computable. What keeps this at 3 rather than 4 is that the theoretical grounding for *why* these interventions work is thin, the 'local shortcut' hypothesis is more a narrative than a formal result…