Improving the Scaling Laws of Synthetic Data with Deliberate Practice

Improving the Scaling Laws of Synthetic Data with Deliberate Practice

Speakers: Reyhane Askari Hemmat, Researcher, FAIR labs at Meta; Mohammad Pezeshki; Elvis Dohmatob; Florian Bordes; Pietro Astolfi; Melissa Hall; Jakob Verbeek; Michal Drozdzal; Adriana Romero-Soriano

Conference: ICML 2025

YouTube: https://slideslive.com/39044013

Overview

In the rapidly evolving landscape of machine learning, the quest for ever more performant models often hinges on the availability of vast, high-quality datasets. However, acquiring and labeling real-world data is an inherently expensive, time-consuming, and often prohibitive endeavor. This talk, presented by Reyhane Askari Hemmat and her colleagues from FAIR labs at Meta, introduces a novel framework dubbed Deliberate Practice (DP), drawing inspiration from human learning psychology, to dramatically improve the data efficiency and scaling laws of synthetic data generation. The core innovation lies in dynamically generating challenging and informative examples using entropy-guided sampling from powerful diffusion models, rather than relying on static or naively generated datasets.

The central problem addressed is the diminishing returns observed when scaling up naive synthetic data generation. While generative models offer an "infinite source" of data, simply creating more samples often leads to redundancy and a plateau in model performance, as the generated examples lack diversity or challenge the model sufficiently. By adopting a "deliberate practice" approach, where the model is continually challenged with examples tailored to its current weaknesses, the researchers demonstrate significant improvements in data efficiency, reduced training times, and enhanced model robustness, effectively unlocking the full potential of synthetic data for large-scale machine learning tasks. This work is crucial for the future of AI development, offering a path to build more capable models with fewer real-world data constraints.

Background

▶ Watch: Introduction to Deliberate Practice concept (0:00)

The current paradigm in machine learning model training largely relies on static datasets. Whether these datasets are composed of real-world examples or synthetically generated ones, they are typically fixed at the outset of training. This approach, while foundational, presents several limitations. Real-world data acquisition is notoriously expensive, requiring significant human effort for collection, annotation, and curation. Furthermore, even with substantial real datasets, models often exhibit data inefficiency, requiring vast quantities of examples to generalize effectively, a stark contrast to human learning, where generalization often occurs with far less data.

The emergence of powerful generative models, such as diffusion models, has opened up the possibility of creating synthetic datasets as an "infinite source" of training material. This promises to alleviate the cost and scarcity associated with real data. However, simply generating a massive quantity of synthetic data naively often leads to its own set of problems. As highlighted in the talk, for tasks like image classification on datasets such as ImageNet, increasing the size of a naively generated synthetic dataset quickly leads to diminishing returns in validation accuracy. For instance, increasing a dataset from 1.3 million to 13 million examples might yield less than a 2% improvement in accuracy. This is because naive generation frequently produces highly similar, redundant examples (e.g., many yellow school buses with street backgrounds), which do not effectively challenge a model once it has learned the basic features of a class.

Prior attempts to address this might involve generating a large pool of data and then using a selector function to prune away "easy" examples, retaining only the more informative ones. However, this "generate-then-prune" strategy is computationally expensive, requiring the generation of far more data than is ultimately used. The fundamental issue is that static datasets, whether real or synthetic, do not adapt to the learner's evolving state or weaknesses. The problem thus lies in the inefficient scaling of synthetic data, where the quality and informativeness of samples quickly plateau, hindering further model improvement. This work seeks to overcome these limitations by dynamically generating data that specifically targets the model's current deficiencies, mirroring the psychological concept of deliberate practice.

Key Findings

▶ Watch: Solution: Directly generating hard examples via entropy-guided sampling (3:00)

The research introduces the Deliberate Practice (DP) framework, demonstrating that strategically curating synthetic data can vastly improve model training efficiency and performance. The central insight is that "not all samples contribute equally to the learning," and prioritizing challenging, informative examples is key to better scaling laws.

The key findings are multi-faceted:

1. Superior Data Efficiency: The DP framework achieves comparable or superior validation accuracy to static synthetic data setups using significantly less data. On ImageNet-100, DP required 75% less data, while on the more complex ImageNet-1k, it achieved a 20-fold reduction in data volume. For example, DP achieved 48% accuracy on ImageNet-1k with just 750,000 samples, whereas a static setup needed 13 million samples to achieve less than 46% accuracy. This translates directly to reduced storage requirements and faster data loading during training.

1. Reduced Training Time: Beyond data volume, DP also drastically cuts down on the computational resources and time needed for training. On ImageNet-100, the framework outperformed previous work while training for only 16% of the iterations. For ImageNet-1k, training time was cut by 30% compared to existing methods. This efficiency gain is critical for large-scale model development, where training can span days or weeks.

1. Enhanced Model Robustness: Models trained using the Deliberate Practice methodology exhibit superior robustness. They surpass models trained on real data when evaluated on out-of-distribution benchmarks such as ImageNet-R and ImageNet-Sketch, showing improvements of up to 15%. This suggests that training with dynamically challenging synthetic examples not only improves in-distribution performance but also fosters better generalization capabilities to diverse and novel scenarios.

1. Unlocking Generative Model Potential: The work demonstrates that powerful generative models, specifically diffusion models, possess the inherent capacity to generate rare and challenging examples. The challenge lies not in their ability to create diversity, but in effectively guiding their sampling process to produce examples that are most beneficial for a specific learner at a given stage of training. The proposed entropy-guided sampling mechanism provides this crucial guidance.

In essence, the research validates that by adapting the training data to the model's current skill level, much like a human learner progresses through increasingly difficult exercises, neural networks can learn more efficiently, generalize better, and achieve higher performance with fewer resources.

Technical Deep Dive

▶ Watch: Technical details: Criterion-guided sampling with diffusion models (4:00)

The Deliberate Practice (DP) framework is structured as a dynamic, iterative training process that adapts the synthetic dataset to the model's evolving capabilities. It begins with an initial set of synthetic data, generated using a naive sampling approach. The model is then trained on this dataset. Crucially, the system incorporates a patience mechanism that monitors the validation accuracy. If the validation accuracy plateaus for a predefined number of consecutive epochs (e.g., five epochs without improvement), it signals that the model has largely learned the features present in the current dataset.

Once the patience mechanism is triggered, instead of simply continuing to train on the same data or adding more naively generated samples, the DP framework dynamically generates a new batch of challenging examples. These new examples are specifically designed to target the model's current weaknesses. This newly generated challenging data is then added to the existing dataset, and training resumes on the combined, augmented dataset. This process creates a "bump" in validation accuracy and a spike in training loss, indicating that the model is now being exposed to and learning from harder examples. This cycle of training, monitoring, generating challenging data, and resuming training can be repeated as long as computational resources and time allow, continuously pushing the model's performance boundaries.

The core technical innovation enabling the generation of these challenging examples is Criterion-Guided Sampling, specifically Entropy-Guided Sampling, applied to diffusion models. Diffusion models are a class of generative models that learn to reverse a gradual "diffusion" process, where data is progressively turned into noise. During inference, they start from pure noise and iteratively denoise an example, step by step, to generate a clean image. This iterative denoising process, often using algorithms like DDIM (Denoising Diffusion Implicit Models), involves transforming a noisy sample Xt into a slightly less noisy sample Xt-1. At each step, the diffusion model predicts a "score function" (related to the gradient of the log probability density) and an X0 approximation. The X0 approximation represents what the model believes the final, clean image X0 would look like, given the current noisy sample Xt. Even in the early stages of generation, this X0 approximation contains discernible features of the image, albeit blurry.

The genius of entropy-guided sampling lies in leveraging this X0 approximation during the inference-time generation process. For each step of the diffusion model's denoising, the current X0 approximation is fed to the downstream classifier (the model being trained). The classifier then computes its entropy or uncertainty regarding this approximation. High entropy indicates that the classifier is uncertain about the class of the image, suggesting it's a challenging example. The system then calculates gradients with respect to the pixels of the X0 approximation, essentially asking: "How should I change these pixels so that, when this image is eventually fully generated, it becomes a hard sample for my classifier?"

This gradient information is then directly incorporated into the score function of the diffusion model. This modification steers the diffusion process away from generating typical, "easy" examples and towards producing images that are rare, diverse, and specifically challenging for the current state of the classifier. It's important to note that this guidance happens at inference time during the generation process; it does not alter the diffusion model's pre-trained parameters.

A crucial aspect discussed in the Q&A, related to maintaining sample fidelity while increasing diversity, involves classifier-free guidance. This technique, common in diffusion models, uses an internal classifier within the generative model to ensure the generated image adheres to a given prompt (e.g., "school bus"). By carefully balancing the coefficient of this internal guidance with the coefficient of the external entropy-guided gradient, the system can generate images that are still clearly identifiable as the target class (e.g., a "school bus" according to the diffusion model's internal classifier) but are simultaneously challenging or "hard" for the external learner (e.g., a blue or grayscale school bus, or one with an unusual background). This balance prevents the generation from veering into completely unrelated or distorted classes, ensuring diversity without sacrificing semantic integrity.

Experimental Setup & Results

▶ Watch: Deliberate Practice framework: iterative data generation (5:20)

The effectiveness of the Deliberate Practice (DP) framework was rigorously evaluated across several standard image classification benchmarks, primarily focusing on ImageNet variants, and compared against static synthetic data generation and prior state-of-the-art methods.

Datasets Used:

• ImageNet-100: A subset of ImageNet with 100 classes, often used for rapid prototyping and initial evaluation.
• ImageNet-1k: The full ImageNet dataset with 1000 classes, representing a significant challenge for large-scale image classification.
• ImageNet-R (ImageNet-Renditions): A dataset designed to test model robustness to various rendering styles (e.g., art, cartoons, sketches).
• ImageNet-Sketch: Another dataset for evaluating robustness, consisting of grayscale sketch images of ImageNet objects.

Baselines and Comparisons:

The DP framework was primarily compared against:

1. Static Synthetic Data: A baseline where a large dataset is generated once using naive sampling and then used for training.
2. Previous Work: Other methods aiming to improve synthetic data efficiency or generation.

Hardware and Frameworks: While specific details on GPU types or framework versions were not explicitly stated in the talk, the experiments would typically leverage high-performance computing clusters with multiple GPUs, likely using popular deep learning frameworks like PyTorch, given the affiliation with FAIR labs.

Metrics:

The primary metrics for evaluation included:

• Validation Accuracy: The standard measure of model performance on a held-out test set.
• Data Efficiency: Quantified by the total number of synthetic samples required to achieve a target accuracy.
• Training Iterations/Time: A measure of the computational cost and time-to-solution.
• Robustness: Measured by performance on out-of-distribution datasets like ImageNet-R and ImageNet-Sketch.

Headline Results:

1. ImageNet-100 Performance:

• DP achieved the same accuracy as a static setup using 75% less data.
• Specifically, a DP model trained with approximately 150,000 samples achieved around 64% validation accuracy, significantly outperforming a static setup that required 32 million samples to reach 62% accuracy.
• When compared to previous work, DP outperformed existing methods with 46 million less samples and required only 16% of the training iterations.

1. ImageNet-1k Performance:

• DP achieved comparable accuracy with 20 times less data than a static setup.
• For instance, DP achieved 48% accuracy with 750,000 samples, whereas the static setup needed 13 million samples to achieve less than 46% accuracy.
• Against previous state-of-the-art, DP generated 56 million less samples and cut the training time by 30%.

1. Robustness Improvements:

• Models trained with DP demonstrated superior generalization. They surpassed models trained on real data when evaluated on ImageNet-R and ImageNet-Sketch.
• Improvements in robustness were significant, reaching up to 15% on these challenging out-of-distribution datasets. This implies that the challenging, diverse samples generated by DP not only help the model learn the core task but also build a more resilient and generalizable understanding of concepts.

Ablation and Analysis:

The talk referenced additional analyses in the paper that support the efficacy of the dynamic generation strategy. Specifically, samples generated at different checkpoints of the model's training were evaluated against those same checkpoints. It was observed that samples generated when the model was at a particular stage of learning (i.e., when patience was hit) indeed yielded the highest loss for that specific model checkpoint. This empirically verifies that the entropy-guided sampling effectively identifies and generates examples that the model is weakest on at that precise moment, reinforcing the "deliberate practice" hypothesis.

Practical Implications

▶ Watch: Triggering new data generation when validation accuracy plateaus (6:00)

The Deliberate Practice (DP) framework and its core technique of entropy-guided sampling hold profound practical implications for a wide range of stakeholders in the AI/ML ecosystem, from individual model builders to large infrastructure teams.

For model builders and researchers, DP offers a powerful strategy to overcome the perennial bottleneck of data availability. By significantly reducing the quantity of data needed for training, it enables faster experimentation and iteration on new model architectures or tasks where large labeled datasets are scarce or expensive to acquire. The ability to achieve high performance with 75% or even 20 times less data on complex benchmarks like ImageNet-1k means that projects can become viable sooner, with fewer resources. This democratizes access to high-quality training data, as reliance on massive, costly real datasets diminishes.

Infrastructure teams stand to benefit from the dramatic reduction in training time and data volume. Less data means lower storage costs, faster data loading, and potentially less strain on network bandwidth within data centers. The 16% to 30% reduction in training iterations directly translates to lower compute costs (fewer GPU/TPU hours) and a faster turnaround for model development and deployment cycles. This efficiency is critical for organizations operating at scale, where even marginal gains in resource utilization can lead to substantial savings.

Perhaps one of the most significant implications is the improved robustness of models trained with DP. The ability to generate challenging, diverse examples that push the model beyond its current capabilities results in models that generalize better to out-of-distribution data. This is crucial for deploying AI systems in real-world environments, where data can be noisy, varied, or deviate from the training distribution. A model that performs well on ImageNet-R and ImageNet-Sketch, even surpassing models trained on real data, is inherently more trustworthy and reliable in production. This directly addresses a major concern in AI safety and reliability.

However, there are also tradeoffs and limitations. The criterion-guided sampling process, particularly for diffusion models, involves computing gradients with respect to classifier uncertainty during the generative inference loop. This step, while more efficient than generating vast amounts of data and then pruning, still adds computational overhead to the data generation phase compared to naive sampling. The complexity of balancing fidelity (ensuring the generated image is still a "school bus") with diversity (making it a "challenging" school bus) via coefficients like those in classifier-free guidance requires careful tuning. If the guidance is too strong, it could lead to "weird" or unrealistic examples that hurt model performance.

Furthermore, the effectiveness of DP relies on the availability of powerful pre-trained generative models and a robust downstream classifier that can provide meaningful uncertainty feedback. For entirely novel domains or extremely low-resource settings, establishing these foundational components might still pose an initial hurdle. Despite these considerations, the Deliberate Practice framework represents a significant leap forward in optimizing the use of synthetic data, promising more efficient, robust, and scalable AI development.

Key Takeaways

• Deliberate Practice (DP) for AI: Inspired by human learning, DP dynamically challenges neural networks with increasingly difficult synthetic examples, adapting the dataset to the model's current weaknesses.
• Overcoming Synthetic Data Limitations: Naive synthetic data generation leads to diminishing returns due to redundancy; DP addresses this by prioritizing diverse, informative, and hard examples.
• Entropy-Guided Sampling: The core technical innovation uses diffusion models' X0 approximation and classifier uncertainty to guide the generation process, creating samples that are challenging for the learner at inference time.
• Dramatic Data Efficiency & Speedups: DP achieves comparable accuracy with 75% less data on ImageNet-100 and 20 times less data on ImageNet-1k, significantly reducing training time (e.g., 30% faster on ImageNet-1k).
• Enhanced Robustness: Models trained with DP demonstrate superior generalization and robustness, outperforming models trained on real data on out-of-distribution benchmarks like ImageNet-R and ImageNet-Sketch by up to 15%.
• Unlocking Generative Model Potential: The framework demonstrates that generative models can produce rare and challenging examples; the key is to unlock their sampling capabilities with intelligent guidance mechanisms.

About the Speaker(s)

The talk was presented by Reyhane Askari Hemmat, a researcher at the FAIR labs (Fundamental AI Research) at Meta. This work is a collaborative effort, developed jointly with a team of colleagues including Mohammad Pezeshki, Elvis Dohmatob, Florian Bordes, Pietro Astolfi, Melissa Hall, Jakob Verbeek, Michal Drozdzal, and Adriana Romero-Soriano. Their collective research focuses on advancing fundamental aspects of artificial intelligence and machine learning, particularly in areas related to data efficiency, generative models, and robust learning.