Harnessing Low Dimensionality in Diffusion Models: From Theory to Practice: Lecture I: The Generalizability of Diffusion Models

Harnessing Low Dimensionality in Diffusion Models: From Theory to Practice: Lecture I: The Generalizability of Diffusion Models

Speakers: Qing Qu, Professor, University of Michigan; Yuxin Chen; Liyue Shen

Conference: ICML 2025

YouTube: https://slideslive.com/39043828

Overview

This article delves into the foundational mathematical aspects of diffusion models, specifically focusing on their remarkable generalization capabilities. Presented as the first lecture in a tutorial series titled "Harnessing Low Dimensionality in Diffusion Models," this talk by Professor Qing Qu from the University of Michigan, alongside Professors Yuxin Chen and Liyue Shen, shifts the common narrative around diffusion models from their diverse applications to their underlying theoretical principles. While many past tutorials have focused on introducing different diffusion methods and their practical uses, this series aims to unpack how these models work by understanding the low-dimensional structures inherent in data and the models themselves.

The core objective of this lecture is to address two fundamental questions: why diffusion models generalize effectively when trained on finite datasets, and how to quantitatively measure this generalization without relying on traditional ground truth metrics. Professor Qu introduces novel concepts of model reproducibility and generalizability scores, revealing a strong, previously unquantified correlation between these properties. The talk posits that the intrinsic low dimensionality of real-world data, particularly images, plays a crucial role in enabling diffusion models to circumvent the curse of dimensionality, leading to efficient learning and robust generalization.

This deep dive into the theoretical underpinnings of diffusion models is crucial for advancing the field beyond empirical successes. By understanding the mathematical foundations and the role of low-dimensional structures, researchers and practitioners can develop more efficient, reliable, and controllable generative AI systems. The insights presented not only explain existing phenomena but also open avenues for new applications, such as precise image editing and invisible watermarking, derived directly from the theoretical understanding of model inductive biases.

Background

▶ Watch: Tutorial introduction and focus on low-dimensionality (0:00)

The landscape of generative AI has evolved dramatically over the past decade, witnessing the rise of models like Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), culminating in the recent dominance of diffusion models. VAEs, introduced in 2013, aimed to learn data representations and generate new samples, but often struggled with generation quality due to the dual objective of compression and generation. GANs, emerging later, improved generation quality significantly by framing the problem as a two-player minimax game between a generator and a discriminator. However, GANs are notoriously difficult to train, prone to instability and mode collapse, especially when scaled to larger datasets.

Diffusion models, which gained prominence roughly five years ago, offered a more stable and scalable alternative. Unlike VAEs and GANs that attempt a one-step generation from noise, diffusion models employ a gradual denoising process. This process is characterized by two phases: a forward process where noise is progressively added to clean data until it becomes pure noise, and a reverse process where this noise is gradually removed to reconstruct clean data. The forward process is typically predefined and trivial, often modeled as a forward stochastic differential equation (SDE). The intriguing and challenging part is the reverse process, which can be achieved by following a backward probability flow ODE, provided the score function—the gradient of the log probability density of the data distribution at each noise level—is known.

In practice, the true underlying data distribution, and thus its score function, are unknown. This is where neural networks come into play. Modern diffusion models utilize architectures like U-Nets or Transformers (e.g., DiT) as universal approximators to learn or approximate this score function. The training objective for these neural networks is typically a score matching loss, which, through Tweedie's formula, is shown to be equivalent to minimizing a least-squares loss for predicting the injected noise. This optimization stability is a key reason why diffusion models scale effectively and produce high-quality generative outputs, from text-to-image (e.g., Stable Diffusion, Sora) to video generation (e.g., Google Veo). However, despite their empirical success, the fundamental question of why diffusion models generalize so well from finite training samples, especially in high-dimensional spaces like images, remains a critical area of theoretical inquiry. This talk aims to bridge that gap by exploring the role of low dimensionality in data and models.

Key Findings

▶ Watch: Why Diffusion Models surpassed VAEs and GANs (3:00)

The talk reveals several pivotal findings that shed light on the generalization capabilities of diffusion models, moving beyond empirical observations to theoretical explanations:

1. Strong Correlation between Reproducibility and Generalizability: A central finding is the discovery of a strong, quantitative correlation between model reproducibility and generalizability in diffusion models. Reproducibility, measured by the similarity of images generated by different models (or the same model trained differently) from the same random seed, is shown to be remarkably high in diffusion models (e.g., RP score of 0.7 even for disparate architectures). This stands in stark contrast to GANs and VAEs, where reproducibility is almost zero. Generalizability, defined by the dissimilarity of generated samples from the training dataset, also exhibits interesting behavior.

1. Emergence in Two Distinct Regimes: Both reproducibility and generalizability in diffusion models emerge in two distinct phases:

• Memorization Regime: When trained with limited samples, models exhibit high reproducibility, but they are primarily memorizing the training data (low generalizability).
• Generalization Regime: As the number of training samples increases beyond a certain threshold, the model enters a transition regime, and then sharply re-emerges with high reproducibility that coincides with a significant increase in generalizability, meaning the model is generating novel, sensible samples. This phenomenon was also observed in concurrent work by Stéphane Mallat and Eero Simoncelli's group, where models trained on non-overlapping datasets still produced similar, novel images at large sample sizes.

1. Breaking the Curse of Dimensionality via Intrinsic Low Dimensionality: The talk argues that diffusion models overcome the classical "curse of dimensionality" because they are not learning arbitrary high-dimensional distributions. Instead, they implicitly leverage the intrinsic low dimensionality of real-world data, such as images (e.g., ImageNet's intrinsic dimension estimated around 45). The number of samples required for generalization grows linearly with this intrinsic dimension, rather than exponentially with the ambient dimension, explaining the observed efficiency.

1. Low-Rank Inductive Bias in Learned Models: The Jacobian of the Denoising Autoencoder (DAE) formulation (which is equivalent to score approximation) is shown to be inherently low-rank across a significant range of noise levels. This low-rank structure reflects an inductive bias towards simpler, low-dimensional solutions within the neural network itself, mirroring the low dimensionality of the data manifold.

1. Theoretical Equivalence to PCA and Subspace Clustering: For a simplified model of data (a mixture of low-rank Gaussians), the training loss of a DAE is shown to be mathematically equivalent to solving Principal Component Analysis (PCA) for a single low-rank Gaussian component, and to subspace clustering for a mixture of such components. This theoretical link provides a framework to analytically characterize the sample complexity, demonstrating that it scales linearly with the intrinsic dimension, matching empirical observations.

1. Semantic Meaning of Jacobian Basis: The individual columns of the DAE's low-rank Jacobian are found to have semantic meanings. Manipulating image generation along these directions allows for precise, controllable editing of specific attributes (e.g., gender, hat color, hair, eye size, smile), highlighting a deep connection between the model's learned structure and human-interpretable features.

Technical Deep Dive

▶ Watch: Commercial and scientific applications of Diffusion Models (4:00)

The technical exposition begins with a recap of diffusion models, emphasizing the forward process (gradual noise injection, often modeled as a stochastic differential equation (SDE)) and the reverse process (gradual denoising via a backward probability flow ODE). The crux of the reverse process lies in knowing the score function, which is the gradient of the log probability density of the data distribution at each time step t (representing the noise level).

Since the true score function is unknown, a neural network (e.g., U-Net or Transformer-based models like DiT) is employed as a universal approximator. The training objective is score matching, where the neural network s_theta parameterized by theta aims to minimize the distance to the optimal score. Practically, this is achieved by minimizing an expectation loss. Crucially, Tweedie's formula demonstrates that this population loss is equivalent to training the neural network to predict the noise epsilon added to the data. This noise prediction objective is a stable least-squares problem, enabling the scalability of diffusion models.

To quantify generalization, the talk introduces two key metrics:

1. MSCD (Mean Self-Copy Detection): This metric measures reproducibility at the instance level. It calculates the cosine similarity between two images generated by different models (or training runs) from the same random seed. To make this robust, images are first mapped into a lower-dimensional embedding space using a neural network trained via self-supervised learning, which removes unimportant aspects of the data.
2. Reproducibility Score (RP Score): This is a probability measure defined as the likelihood that the MSCD score between two generated images exceeds a certain high threshold. It provides a model-level quantification of reproducibility, averaged over many samples (e.g., 10,000).
3. Generalization Score: This metric quantifies generalizability by measuring the dissimilarity between a generated sample and the entire training dataset. A high score indicates the generation of novel samples, while a low score suggests memorization.

A significant theoretical contribution is the use of a simplified data model: a mixture of low-rank Gaussian distributions. In this model, data points are assumed to lie on a union of low-dimensional subspaces, with each Gaussian being degenerate (restricted to a subspace) and having an orthogonal basis. For a single low-rank Gaussian (k=1), the optimal denoiser (which the DAE aims to learn) simplifies to a projection onto a subspace. The talk demonstrates that the training loss of a Denoising Autoencoder (DAE), which directly predicts the clean data (and is equivalent to score approximation via Tweedie's formula), becomes equivalent to a Principal Component Analysis (PCA) problem in this simplified setting. This equivalence allows leveraging classical PCA theory to characterize the sample complexity, showing it scales linearly with the intrinsic dimension of the data, d_intrinsic, rather than the ambient dimension D.

For the case of multiple low-rank Gaussians (k > 1), the DAE training problem is shown to be equivalent to subspace clustering, another classical problem with well-understood sample complexity bounds that also scale linearly with d_intrinsic. This theoretical framework provides a strong argument for why diffusion models can generalize effectively with fewer samples than predicted by worst-case high-dimensional theory.

Furthermore, the talk highlights the inductive bias of trained neural networks in diffusion models. Analysis of the Jacobian matrix of the learned DAE (with respect to the noisy input) reveals that it is often low-rank across intermediate noise levels. This low-rank property is not just a theoretical curiosity; each column of this Jacobian is shown to have semantic meaning. By manipulating or "steering" the image generation process along these Jacobian basis directions at intermediate time steps, specific attributes of the generated image (e.g., gender, hat color, hair, eye size, smile) can be precisely controlled. This linearity of the DAE locally, combined with the low-rank Jacobian, enables highly controllable image editing techniques.

Finally, the talk introduces a more precise metric for generalization: Probability Flow Distance. This metric is developed under a teacher-student setup, where a pre-trained "teacher" model acts as a proxy for the true underlying distribution. A "student" model is trained on samples from the teacher, and generalization is quantified by measuring the reproducibility (cosine similarity in latent space) between samples generated by the student and teacher models starting from the same noise. This unified metric allows for a more robust quantification of generalization error and memorization, overcoming the limitations of simple comparisons to training data.

Experimental Setup & Results

▶ Watch: Diffusion Models: The reverse denoising process via score function (5:00)

The empirical validation for the theoretical claims is presented through a series of experiments using various diffusion model architectures and datasets.

1. Reproducibility Across Architectures and Losses: Experiments comparing different diffusion models (e.g., U-Net based, Transformer-based DiT, models trained with different loss functions like consistent loss) demonstrate a consistently high Reproducibility Score (RP Score). Even with architectural and training differences, the lowest observed RP score was 0.7, indicating strong similarity in generated outputs from the same random seed. This reproducibility is visualized through similarity matrices, showing high values across a diverse set of models.

1. Contrast with GANs and VAEs: A critical experiment compares the RP score of diffusion models against GAN-based and VAE-based generative models under the same settings. The results show that GANs and VAEs exhibit almost zero reproducibility (RP score ≈ 0), confirming that this strong reproducibility is a unique and prevalent phenomenon specific to diffusion models.

1. Reproducibility and Generalization Regimes: Visualizations of reproducibility and generalization scores as a function of the number of training samples (e.g., on CIFAR dataset) reveal two distinct regimes. With limited samples, both scores are high, indicating memorization (reproducibility of memorized samples). As training samples increase, there's a transition phase where reproducibility decreases. However, beyond a certain interpolation point (e.g., N=10,000 for CIFAR), reproducibility sharply re-emerges and strongly correlates with increasing generalizability, where models generate novel samples. This "double descent"-like behavior for generalization is a key observation.

1. Concurrent Work Validation: The findings align with a concurrent study by Stéphane Mallat and Eero Simoncelli's group. Their work demonstrated that diffusion models trained on non-overlapping subsets of a dataset (e.g., CelebA) still generate highly similar, novel images when trained with a sufficiently large number of samples, further reinforcing the concept of generalization beyond memorization and the inherent reproducibility.

1. Intrinsic Dimensionality of Data and Models:

• Reference is made to prior work (e.g., Thomas Goldstein's group) showing that intrinsic dimensions of real image datasets like ImageNet are relatively low (around 45), despite their high ambient dimensions.
• Experiments on the DAE formulation of diffusion models measure the rank ratio of the Jacobian matrix of the DAE across different noise levels. The results show that the numerical rank of the Jacobian is significantly lower than the ambient dimension (log rank ratio around 0.1), confirming that the learned neural networks themselves exhibit a low-rank structure, mirroring the intrinsic low dimensionality of the data.

1. Mixture Low-Rank Gaussian Model Validation:

• Generated images from the simplified mixture of low-rank Gaussian (MLRG) model are shown to capture the coarse structures of images generated by real U-Nets, suggesting that the MLRG model serves as a good theoretical approximation for understanding image distributions.
• Phase transition plots for DAE training on the MLRG model are compared to classical PCA and subspace clustering problems. While the DAE transitions are not as sharp as theoretical PCA, they demonstrate similar phase transitions, supporting the theoretical equivalence.
• Crucially, experiments fitting the relationship between the number of samples and the estimated intrinsic dimension on both the MLRG model and real datasets (CIFAR using U-Net based models) show a strong linear relationship. This empirically validates the theoretical prediction that sample complexity grows linearly with intrinsic dimension, not quadratically or exponentially with ambient dimension.

1. Semantic Control via Jacobian Basis: Visualizations demonstrate that steering image generation along specific columns of the DAE's Jacobian matrix allows for targeted manipulation of semantic attributes (e.g., gender, hat color, hair, eye size, smile). In contrast, steering along random directions yields no meaningful changes, confirming the semantic interpretability of the learned low-rank basis.

Practical Implications

▶ Watch: The challenge: Learning the score function with neural networks (5:40)

The theoretical insights into the low dimensionality of data and models in diffusion processes have profound practical implications across various stages of generative AI development and deployment:

1. Efficient Generalization and Reduced Data Requirements: The understanding that diffusion models leverage the intrinsic low dimensionality of data explains their ability to generalize effectively from significantly fewer training samples than predicted by traditional high-dimensional learning theories. This is crucial for practitioners working with limited datasets or aiming to reduce the computational cost of data collection and model training. It suggests that future model designs could explicitly incorporate mechanisms to better exploit this intrinsic dimensionality, leading to even more data-efficient learning.

1. Controllable Image Editing: The discovery of a low-rank Jacobian in the DAE and the semantic meaning of its basis vectors directly enables highly controllable image editing. Practitioners can leverage this to:

• Precise and Localized Editing: Manipulate specific attributes (e.g., gender, hair color, eye size, smile) with fine-grained control, affecting only the desired parts of an image.
• Transferability: Editing directions learned on one image can be transferred to others, and even across different time steps in the diffusion process, streamlining the editing workflow.
• Composability and Disentanglement: Multiple editing directions can be composed linearly to achieve complex transformations (e.g., changing both eye size and smile simultaneously), while maintaining disentanglement of attributes. This capability is invaluable for creative applications, content generation, and personalized AI experiences.

1. Invisible and Robust Watermarking: The insight that the DAE's Jacobian is low-rank at intermediate noise levels, but not at high noise levels, opens a new paradigm for watermarking diffusion-generated content. Instead of injecting watermarks at high noise levels (which can be visible or easily removed), the approach proposes injecting watermarks into the null space of the Jacobian at intermediate noise levels. This method promises to yield much more invisible and robust watermarks compared to existing techniques, addressing growing concerns about content provenance and authenticity in the era of generative AI.

1. Improved Generalization Measurement and Model Evaluation: The introduction of the Probability Flow Distance provides a more precise and unified metric for quantifying generalization in diffusion models, especially in the absence of a true underlying distribution. This metric, derived from a teacher-student setup, allows for better evaluation and comparison of different diffusion models, leading to more rigorous research and development. It moves beyond naive comparisons to training data, offering a more robust measure of a model's ability to generate truly novel and meaningful samples.

1. Guiding Future Model Architectures: The theoretical understanding of the inductive biases and low-rank structures within diffusion models can guide the design of future model architectures. By explicitly incorporating mechanisms that encourage or leverage low-dimensional representations, researchers might develop more efficient, interpretable, and controllable generative models.

While these implications are significant, it's important to acknowledge limitations. The theoretical framework, particularly the equivalence to PCA and subspace clustering, relies on simplified data models (mixture of low-rank Gaussians). While shown to approximate real data well, the full complexity of real-world image distributions might still present challenges. Further research is needed to bridge the gap between simplified theoretical models and the performance of large-scale, intricate diffusion models on diverse, high-dimensional data.

Key Takeaways

• Reproducibility and Generalization are Correlated: Diffusion models exhibit strong model reproducibility, where different models generate highly similar images from the same random seed, a phenomenon largely absent in GANs and VAEs. This reproducibility strongly correlates with their generalization capabilities, emerging in distinct memorization and generalization regimes.
• Low Dimensionality Breaks the Curse: The remarkable generalization of diffusion models, even with finite data, is attributed to their implicit leveraging of the intrinsic low dimensionality of real-world data. Sample complexity scales linearly with this intrinsic dimension, not the ambient dimension, circumventing the curse of dimensionality.
• DAE Training is PCA/Subspace Clustering: Under a simplified model of data (mixture of low-rank Gaussians), the training of a Denoising Autoencoder (DAE) is theoretically equivalent to Principal Component Analysis (PCA) or subspace clustering, providing a rigorous mathematical foundation for its sample complexity and generalization behavior.
• Semantic Control via Jacobian Basis: The Jacobian of the learned DAE exhibits a low-rank structure, and its basis vectors possess semantic meaning. Manipulating generation along these directions enables precise, localized, transferable, and composable control over image attributes.
• Novel Practical Applications: The theoretical understanding leads to practical advancements, including a method for controllable image editing and a technique for invisible and robust watermarking by leveraging the low-rank Jacobian's null space at intermediate noise levels.
• Refined Generalization Measurement: The Probability Flow Distance, developed under a teacher-student setup, offers a more precise and unified metric to quantify the generalization error and memorization behavior of diffusion models.

About the Speaker(s)

Qing Qu is a Professor at the University of Michigan. In this tutorial, he presented the first lecture focusing on the theoretical aspects of generalization in diffusion models, emphasizing the role of low-dimensional structures. His research delves into the mathematical foundations of machine learning, particularly in understanding the properties and behaviors of complex models like diffusion models.

Yuxin Chen is also a speaker in this tutorial series. His part of the tutorial (Lecture II) focuses on the sampling theory of diffusion models, addressing questions about the number of samples required, sampler convergence speed, and the design of faster samplers.

Liyue Shen is the third speaker in this tutorial series. Her lecture (Lecture III) bridges the gap "from theory to scientific applications," specifically exploring how diffusion models can be leveraged to solve inverse problems in areas like image reconstruction and protein design.