Generative AI Meets Reinforcement Learning: Generative AI Meets Reinforcement Learning

Generative AI Meets Reinforcement Learning: Generative AI Meets Reinforcement Learning

Speakers: Amy Zhang, Benjamin Eysenbach

Conference: ICML 2025

YouTube: https://slideslive.com/39043360

Overview

This comprehensive tutorial, "Generative AI Meets Reinforcement Learning," delivered by Amy Zhang and Benjamin Eysenbach at ICML 2025, explores the profound and often overlooked synergies between generative AI (GenAI) and reinforcement learning (RL). The speakers argue that while superficially distinct, these two fields offer crucial tools and perspectives that can drive significant progress in each other. Benjamin Eysenbach opens with a historical analogy to the "Mechanical Turk," highlighting that current generative models, like early chess AIs, are often no smarter than the human data used to train them, merely mimicking existing patterns. The central thesis is that to advance beyond mere imitation, both fields must leverage each other's strengths.

The tutorial aims to demonstrate how RL can provide a framework for generative models to discover their own data and optimize for qualitative metrics beyond simple likelihoods, thereby overcoming limitations like "data poisoning." Conversely, it illustrates how powerful generative models can serve as sophisticated components within RL systems, acting as world models or policies, and enabling novel approaches to exploration and self-supervised learning. By reframing RL problems through the lens of likelihoods and occupancy measures, the speakers propose a unified perspective that promises to unlock new algorithmic developments and broader generalization capabilities for AI systems, moving towards agents that can "build the world that today's generative models can only dream of."

Background

▶ Watch: Introduction and Mechanical Turk analogy for GenAI (0:00)

The evolution of artificial intelligence has seen various paradigms, from early rule-based systems to the data-driven approaches dominating today. In the realm of game-playing AI, systems like IBM's Deep Blue (1990s) and DeepMind's AlphaZero (2018) demonstrated superhuman performance in chess. While Deep Blue relied on extensive human knowledge and handcrafted heuristics, AlphaZero learned from self-play, leveraging reinforcement learning. However, even AlphaZero, in its initial training, still operated within a predefined environment, optimizing for a clear, scalar reward. The speakers draw a compelling analogy to the 18th-century "Mechanical Turk" chess automaton, which famously concealed a human operator. They argue that modern generative models, despite their apparent complexity, often function similarly: their intelligence is derived directly from the vast datasets of human-generated content they are trained on. This inherent mimicry limits their ability to innovate or go "beyond the best humans."

The core problem addressed in this tutorial stems from the distinct objectives of supervised learning and reinforcement learning, and their respective limitations. Supervised learning excels at mapping inputs to outputs given labeled data but is fundamentally constrained by the quality and diversity of that data. Generative models, a subset of supervised learning, learn to produce outputs similar to their training data, but struggle with optimizing for subjective, qualitative criteria or generating truly novel, out-of-distribution content. Reinforcement learning, on the other hand, is designed for agents to learn optimal behaviors by interacting with an environment and maximizing a scalar reward signal. This allows RL to tackle problems for which no human expert solution exists (e.g., controlling a nuclear fusion reactor). However, RL faces its own challenges: the difficulty of designing effective reward functions, the prohibitive cost of data collection in real-world environments, and the struggle with exploration in vast state-action spaces. The speakers propose that by integrating GenAI and RL, these individual limitations can be overcome, leading to systems that are not only capable of generating complex outputs but also of autonomously discovering and learning from their own data, thereby moving beyond the "mimicry" paradigm.

Key Findings

▶ Watch: Tutorial goals: Connecting Generative AI and Reinforcement Learning (2:20)

The tutorial highlights several key findings and conceptual breakthroughs at the intersection of generative AI and reinforcement learning:

• Mutual Enhancement: GenAI and RL are not disparate fields but offer crucial tools for progress in each other. RL can help generative models overcome limitations of data dependence and optimize for qualitative, non-likelihood-based metrics, while generative models can provide powerful components (like world models and policies) for RL.
• Beyond Imitation: The core message is to move beyond simply mimicking human data. RL, particularly through exploration and self-supervision, provides mechanisms for AI systems to discover novel behaviors and generate their own diverse training data, addressing issues like "data poisoning" where models trained on recursively generated data degrade over time.
• Interaction as a Generative Process: Any interaction between a policy and an environment can be framed as a generative process. This perspective allows RL problems to be analyzed and solved using the mathematical tools of generative modeling, particularly concerning likelihoods and occupancy measures.
• Probabilistic Reframing of RL: The standard RL objective (maximizing expected rewards) can be derived from a probabilistic perspective using an evidence lower bound (ELBO), similar to latent variable inference in generative models. This reframing reveals deep connections to concepts like reward-weighted regression and policy gradient.
• Generative Models as RL Components:
• World Models: GenAI models can serve as simulators for RL, predicting future states given current states and actions. This dramatically accelerates data collection, especially in environments where real-world interaction is slow or costly (e.g., nuclear fusion, atmospheric science).
• Policies: Generative models, including autoregressive transformers (e.g., Decision Transformer) and diffusion models, can be trained as policies. Techniques like reward conditioning, guide-based methods (using Q-functions to drift actions), and select-based methods (sampling candidates and selecting based on value estimates) enable them to produce high-value actions.
• RL for Generative Models: The denoising process of a diffusion model can be framed as a Markov Decision Process (MDP). This allows diffusion models to be fine-tuned using RL (e.g., policy gradient) to optimize for subjective, qualitative metrics like image aesthetic score, compressibility, or prompt-image alignment, which are not easily captured by traditional likelihood maximization.
• Self-Supervised Reinforcement Learning: By focusing on likelihoods and occupancy measures, frameworks like Forward-Backward (FB) representations and Proto-Successor Measures (PSM) enable the learning of universal, policy-independent representations. These methods allow for zero-shot policy extraction for any downstream task, effectively compressing the entire MDP into a learnable representation without explicit reward functions during pre-training.
• Advanced Exploration Techniques: RL offers sophisticated ways for generative models to discover their own data. Beyond simply maximizing state coverage, methods like empowerment maximization (maximizing mutual information between actions and future states) and skill learning (learning a compact "knob" for diverse behaviors) incentivize agents to position themselves to perform many things and generate diverse, coherent behaviors. These methods rely on self-generated rewards, entirely removing the need for human-designed rewards.

Technical Deep Dive

▶ Watch: Detailed tutorial agenda: Four parts of the talk (3:05)

The tutorial meticulously deconstructs the interplay between GenAI and RL, starting with foundational concepts and building towards advanced frameworks.

Reinforcement Learning Fundamentals

At its core, RL involves an agent interacting with an environment. The agent's behavior is governed by a policy ($\pi$), which maps observations (S) to actions (A). The environment, in turn, provides new observations and a scalar reward (R), indicating the desirability of the current state-action. The agent's goal is to maximize the expected sum of discounted future rewards over a trajectory of experiences, where a discount factor ($\gamma$, typically 0.9) prioritizes immediate rewards. Critically, RL differs from supervised learning in that the agent must discover optimal actions rather than mimicking pre-labeled correct actions.

Generative Models as World Models

One of the most straightforward applications of GenAI in RL is using generative models as world models or simulators. In many real-world applications (e.g., nuclear fusion, atmospheric science, healthcare), interacting with the actual environment is prohibitively slow or costly. A generative world model learns to predict the next state ($S_{t+1}$) given the current state ($S_t$) and action ($A_t$). Any generative model—GANs, flow models, autoregressive transformers—can be used for this.

The benefits are immense:

• Fast Data Collection: Generative simulators allow for rapid, batched data generation on accelerators (GPUs), dramatically speeding up policy learning and evaluation compared to real-world interaction.
• Offline Learning: They can leverage large datasets of past interactions to build a model of the world.

However, challenges exist:

• Model Drift: Errors can accumulate when the model is unrolled over long horizons, leading to predictions that diverge from reality.
• Policy Evaluation: Ensuring that policies learned in a simulated world translate effectively to the real environment remains a key problem.

The speakers hint that breaking the abstraction barrier between the generative model and the RL algorithm (e.g., using a GAN's discriminator for feedback, or transformer's sequence predictions) can mitigate these issues.

Learning Data as an Optimized Latent Variable

The data collection process in RL is not exogenous; it's part of the learning. This presents a "chicken and egg" problem: a good policy needs good data, but good data requires a good policy. The tutorial proposes viewing the optimal data distribution (Q) as a latent variable that can be jointly optimized with the policy ($\pi$).

By applying the concept of an evidence lower bound (ELBO), common in generative models, to the RL objective, a joint optimization problem emerges. This involves two steps:

1. Optimizing Data (Q): Sample trajectories that have high reward and are likely to be sampled by the current policy. This can be seen as learning a model of optimal trajectories.
2. Optimizing Policy ($\pi$): Maximize the likelihood of actions from the current policy, weighted by the reward of the trajectories they produced. This is analogous to reward-weighted regression, where the policy mimics highly rewarded behaviors.

Interestingly, the gradient of this ELBO objective yields the standard policy gradient formulation, providing a novel probabilistic interpretation for a classic RL result.

Generative Models as Policies

Generative models can directly act as policies, mapping observations to actions.

1. Reward Conditioning:

• Autoregressive Models: Models like the Decision Transformer condition an autoregressive transformer on the reward-to-go (sum of future rewards). By providing a higher desired reward-to-go than seen in the training data, the model can generalize to achieve better performance.
• Diffusion Models: Similarly, diffusion models can be trained as policies from offline datasets, imitating a behavior policy. Conditioning them on reward-to-go allows for generalization.

1. Guide-Based Methods: These methods guide the sampling process of a pre-trained diffusion model towards high-value actions at inference time. This is achieved by incorporating a guidance term (e.g., derived from a Q-function or reward function) into the score function used for denoising. The augmented score function effectively drifts generated actions towards regions of high value.
2. Select-Based Methods: These approaches first generate a set of candidate actions (e.g., by sampling from a diffusion policy) and then select the best ones based on a criterion, typically reward functions or Q-functions. Implicit Diffusion Q-learning (IQL), for example, combines a diffusion behavior policy with an IQL Q-function to weight and select high-value samples.
3. Mitigating Errors (Diffusion-DICE): Both guide-based and select-based methods suffer from errors: diffusion model approximation errors (generating out-of-distribution (OOD) actions) and critic errors (erroneous Q-value predictions on OOD actions). Diffusion-DICE combines guide and select steps, using a density ratio (likelihood of optimal action / likelihood of behavior action) instead of an erroneous Q-function for guidance, leading to more robust performance.

Reinforcement Learning for Generative Models

A profound connection is established by treating the denoising trajectory of a diffusion model as an MDP. Each diffusion timestep is a state, and the action space is defined by how to denoise. The reward function corresponds to a final utility function (e.g., image aesthetics, compressibility, prompt alignment). This framework allows policy gradient to fine-tune diffusion models, optimizing for qualitative metrics directly, rather than merely maximizing the likelihood of the output matching the training data. This is a fundamental shift from guiding sampling to changing what the model is trained to output.

Interaction as Likelihoods: Self-Supervised RL

The concept of "interaction as a generative process" leads to defining and estimating the likelihood of visiting specific states.

• Discounted State Occupancy Measure (DSOM): This is formally defined as the probability of encountering a state at any future timestep, weighted by the discount factor. Estimating DSOMs (P-theta) directly from policy interactions allows for applications like goal-conditioned RL.
• Direct Likelihood Estimation: Fit a generative model (Gaussian process, diffusion model, flow model) to the observed states from a policy's trajectories. This model can then inform action selection by choosing actions that maximize the likelihood of reaching a goal state.
• Density Ratio Estimation: Often, full density models are overkill; estimating density ratios (e.g., $P(S_{goal} | S, A_1) / P(S_{goal} | S, A_2)$) is more efficient. Techniques like binary cross-entropy loss (temporal contrastive learning) or least squares importance filtering loss are used.
• Temporal Difference (TD) Methods: To handle changing policies (off-policy learning), TD methods are adapted to estimate likelihoods. The Bellman flow constraint allows stitching together data from different experiences.

Universal Representations: FB and PSM

The lecture introduces advanced frameworks for learning policy-independent, universal representations:

• Parameterized Successor Measures: The successor measure (likelihood of future states) can be parameterized in linear ($F^T B$) or log-linear ($e^{F^T B}$) forms.
• F represents the future occupancy from state S.
• B represents the past trajectory to state S.
• This allows for planning via interpolation in the representation space.
• Forward-Backward (FB) Representations: This three-stage, self-supervised framework learns F and B vectors for every possible task.

1. Unsupervised Phase: Arbitrary "tasks" (Z vectors) are sampled, and a TD update optimizes Q-functions/successor measures for these tasks. This learns an occupancy model for many "light behaviors."
2. Task Identification Phase: Given a downstream reward, the corresponding Z vector is identified.
3. Exploitation Phase: The optimal policy for that task is extracted as the argmax of the Q-function.

FB representations offer provable zero-shot extraction of near-optimal policies for any downstream task, provided good coverage over the reward space during training. This is a form of behavioral foundation model learning general-purpose representations of behavioral data in a self-supervised way.

• Proto-Successor Measures (PSM): Building on the insight that Bellman equations are affine, PSM represents the successor measure as $\phi W + B$, where $\phi$ and B are policy-independent. By minimizing a TD update for successor measures across all possible policies, a truly universal representation is achieved, compressing the entire MDP.

These universal representations enable applications like extracting policies for language-conditioned tasks by mapping generated video trajectories to real observations and assigning rewards.

Exploration for Generative Models

Exploration, the process of discovering new and useful data, is critical for GenAI models to move beyond their training data.

• Maximizing Coverage: Traditional exploration aims to maximize the number of unique states visited (e.g., using DSOM). This is effective in low-dimensional spaces but struggles with scalability.
• Empowerment Maximization: This method focuses on maximizing the mutual information between an agent's actions and its future states. Intuitively, it drives the agent to positions where it can "do many things" or exert a high degree of influence over its environment, leading to knowledge accumulation and resource gathering. Empowerment is a well-studied concept in both RL and cognitive psychology.
• Skill Learning: This involves learning a compact, tunable "knob" (a latent variable) that can sweep over a wide range of diverse behaviors or "skills." The idea is framed as a cooperative communication game where a "blue player" (agent) acts out motions to send messages to a "red player" (discriminator), who tries to guess the message. The reward is self-generated (based on successful communication/predictability), entirely removing human-defined rewards. This process naturally leads to the emergence of diverse and predictable skills, which can then be used to solve downstream tasks (e.g., robot locomotion, manipulation, character control).

Skill learning and empowerment maximization are mathematically related, with skill learning effectively performing empowerment by learning action abstractions. These exploration methods are crucial for generative models to optimize their own datasets, compressing the space of all possible behaviors and enabling broader generalization.

Experimental Setup & Results

▶ Watch: Reinforcement learning fundamentals: Environment, policy, reward (4:40)

This talk is a tutorial, designed to introduce concepts and frameworks rather than present novel experimental results from the speakers' own research. As such, there is no "experimental setup" in the traditional sense of a research paper. Instead, the speakers illustrate the theoretical concepts with examples and references to existing works in the literature.

For instance, when discussing generative models as policies, they reference:

• Decision Transformer for reward-conditioned autoregressive models.
• Diffusion models for offline policy learning, showing how conditioning on reward-to-go can improve performance.
• Implicit Diffusion Q-learning (IQL) for select-based methods, demonstrating its ability to identify high-value action clusters in a 2D bandit problem compared to unimodal Gaussian policies.
• Diffusion-DICE is presented with a 2D bandit example, visually demonstrating how it mitigates errors by using a density ratio for guidance, resulting in a desired ring of actions around the high-value target, unlike guide-only or select-only methods that drift towards erroneous central actions.

In the context of RL for generative models, examples are shown where diffusion models are fine-tuned with policy gradient to optimize for:

• Image compressibility: A llama background fades away, and the llama itself becomes simpler when optimizing for incompressibility (though the example shown is for incompressibility, the principle applies).
• Prompt-image alignment: A Visual Language Model (VLM) scores the alignment between a prompt and generated image, serving as a reward signal.

For self-supervised RL and universal representations (FB, PSM), the application of extracting policies for language-conditioned tasks is demonstrated. This involves:

1. Imagine: Using a fine-tuned VLM to generate video trajectories from language commands.
2. Project: Mapping these generated video observations to real observations in a target domain using nearest neighbors from an offline dataset.
3. Compute Q-function: Assigning rewards based on the projected states and extracting a policy to match the demonstration.

This enables zero-shot policy extraction for tasks like "lunge" or "cartwheel" in simulated environments, using YouTube videos as demonstrations.

For exploration, visual examples include:

• A four-legged robot demonstrating coverage-based exploration.
• A grid world example showing how maximizing state occupancy entropy leads to uniform exploration across the grid, outperforming standard maximum entropy objectives on action space.
• A quadrupedal robot learning skills to move in different directions.
• A manipulation task where skills are learned for waving the arm or interacting with objects.

In summary, the "results" presented are primarily conceptual demonstrations and visual aids drawn from existing research to illustrate the principles and potential of the proposed GenAI-RL integration rather than new, quantitative benchmarks from a specific experimental setup.

Practical Implications

▶ Watch: RL's unique sampling process, unlike supervised learning (6:10)

The convergence of generative AI and reinforcement learning outlined in this tutorial carries significant practical implications for practitioners, infrastructure teams, model builders, and deployers across various domains.

For practitioners and model builders, this integration offers powerful new tools:

• Beyond Imitation: The ability to move past the limitations of purely data-driven mimicry is transformative. RL-driven exploration allows generative models to discover novel patterns and generate data that goes beyond the distribution of their training set, fostering true innovation rather than mere replication.
• Optimizing Qualitative Metrics: Traditional GenAI struggles with subjective quality. By framing generative model training as an MDP, practitioners can fine-tune models using RL to optimize for inherently qualitative metrics like aesthetic appeal, creative novelty, or semantic alignment, which are difficult to capture with likelihood-based objectives.
• Accelerated Development for RL: Generative models as world models offer a solution to the costly and slow data collection bottleneck in real-world RL applications (e.g., robotics, industrial control, scientific discovery). This allows for rapid iteration and policy learning in simulation, potentially reducing development cycles from months to days.
• Enhanced Generalization: The development of universal, policy-independent representations (like FB and PSM) through self-supervised RL promises to enable zero-shot policy extraction. This means an agent could learn general skills in one environment and apply them to entirely new, unseen tasks without additional training, drastically improving the utility of learned behaviors.
• Addressing Data Poisoning: The "AI model collapse" phenomenon, where models degrade when trained on recursively generated data, is a critical concern. RL-driven exploration provides a mechanism for models to actively seek out diverse, novel data, thereby diversifying their data pool and preventing the "poisoning" of their own learning sources.

For infrastructure teams, the ability to use generative models for fast, batched data collection in simulated environments means more efficient utilization of computational resources (GPUs/accelerators) and reduced latency between data collection and learning.

Tradeoffs and Limitations:

• Model Accuracy: Generative world models, while fast, can suffer from model drift over long horizons, leading to policies optimized for an inaccurate simulation.
• Computational Cost: Training large generative models as world models or policies, or applying RL fine-tuning, can still be computationally intensive.
• Critic Error: In guide-based and select-based methods, the accuracy of Q-function estimates on out-of-distribution (OOD) actions is crucial and can be a source of error.
• "Unrealistic Assumptions": Offline RL methods often assume access to a dataset with full coverage of the state-action space, which is rarely true in practice.
• RL Instability: RL algorithms are notoriously unstable and sensitive to hyperparameters or random seeds, leading to high variance in performance or catastrophic forgetting. This needs to be addressed for widespread adoption.
• Reward/Goal Design: While the tutorial strives to move beyond hand-designed rewards, some level of task specification (e.g., through goal states, demonstrations, or even abstract "Z vectors") is still required. The challenge shifts from how to reward to what constitutes a good goal or task representation.

Future Directions:

The speakers express excitement for continued research in:

• Self-supervised RL: Developing more unified objectives that remove the reliance on explicit rewards.
• Dual Occupancy Matching: Exploring new RL methods derived from this probabilistic perspective.
• Addressing Myopic Solutions: Applying RL to sequential decision-making problems currently solved with myopic bandit or supervised learning methods (e.g., recommendation systems, educational technology, industrial control, dialogue systems) to achieve optimal and stable long-term solutions.
• Scalability and Generalization: Realizing the capacity to explore vast spaces of behaviors and organize them, enabling RL algorithms to tackle long-horizon, qualitatively different problems, akin to how generative models generalize across images. This is seen as key for developing AI that can "build the world" rather than just generate pixels or text.

Key Takeaways

• Generative AI and Reinforcement Learning are deeply intertwined: Each field provides crucial tools for making progress in the other, moving beyond the limitations of current generative models that merely mimic human data.
• RL can be reframed probabilistically through likelihoods and occupancy measures: This perspective provides a rigorous mathematical framework, similar to supervised and unsupervised learning, for understanding and developing new RL algorithms, including connections to the evidence lower bound and policy gradient.
• Generative models serve as powerful components for RL: They can function as efficient world models for accelerated data collection and as sophisticated policies (e.g., reward-conditioned transformers, diffusion models) capable of generalizing to higher-value actions.
• RL enables generative models to optimize for qualitative metrics and generate their own data: By treating the generative process as an MDP, RL can fine-tune models for subjective criteria (aesthetics, alignment) and, crucially, drive exploration to discover diverse, novel data, mitigating issues like "data poisoning."
• Self-supervised RL frameworks create universal, policy-independent representations: Methods like Forward-Backward (FB) representations and Proto-Successor Measures (PSM) allow for zero-shot policy extraction for any downstream task, effectively compressing the entire MDP into a transferable representation.
• Advanced exploration (empowerment, skill learning) is critical for broad generalization: These methods learn to position agents to "do many things" or acquire tunable "skills" using self-generated rewards, fostering innovation and enabling AI systems to tackle complex, long-horizon problems that humans cannot solve.

About the Speaker(s)

Amy Zhang is a prominent researcher in the field of machine learning, focusing on the intersection of generative models and reinforcement learning. Her work, as presented in this tutorial, delves into novel frameworks for self-supervised RL, the use of diffusion models as policies, and the re-conceptualization of RL problems through likelihoods and occupancy measures. Her contributions aim to enhance the generalization capabilities and data efficiency of AI systems, particularly in sequential decision-making contexts.

Benjamin Eysenbach is a distinguished researcher known for his contributions to reinforcement learning, particularly in areas like exploration, skill discovery, and latent variable models. His early work on Diversity is All You Need (DIAYN) has been influential in shaping research directions in skill learning. In this tutorial, he emphasizes the philosophical underpinnings of moving beyond imitation, the importance of self-generated data, and the potential for RL to unlock new forms of generalization in AI, envisioning a future where AI agents can actively "build the world" rather than just simulate it.