DP-fy your DATA: How to (and why) synthesize Differentially Private Synthetic Data: Techniques for DP synthetic IMAGE data creation

DP-fy your DATA: How to (and why) synthesize Differentially Private Synthetic Data: Techniques for DP synthetic IMAGE data creation

Speakers: Alex Bie, Natalia Ponomareva, Sergei Vassilvitskii, Peter Kairouz

Conference: ICML 2025

YouTube: https://slideslive.com/39043837

Overview

This talk, presented by Alex Bie and his colleagues Natalia Ponomareva, Sergei Vassilvitskii, and Peter Kairouz from Google, delves into the critical and evolving field of generating differentially private (DP) synthetic data, specifically focusing on text. The core objective is to enable the use of sensitive, private user data to enhance machine learning models and downstream tasks without directly exposing the original private information. This is achieved by first synthesizing a dataset with strong differential privacy guarantees, which can then be used to train conventional ML models as if it were public data.

The presentation introduces a taxonomy of methods for DP synthetic text generation, exploring their technical underpinnings, tradeoffs, and practical implications. It champions DP Fine-tuning as the most robust and performant approach, while also detailing inference-only alternatives like Private Evolution and DP LLM Inference for scenarios with limited data or specific access constraints. The talk emphasizes that DP synthetic data serves as a crucial "organizational breakthrough" by creating a clear interface between data owners (who may lack DP expertise) and ML researchers (who understand DP but not necessarily the specific downstream task), thereby streamlining the adoption of privacy-preserving machine learning.

The significance of this work lies in its potential to unlock the value of vast amounts of private data for AI development while rigorously upholding user privacy. By providing concrete algorithms, implementation insights (including JAX and jax-privacy), and practical guidance on hyperparameter tuning and evaluation metrics like MAUVE, the speakers equip practitioners with the knowledge to build and deploy privacy-preserving ML systems effectively. This approach addresses a fundamental challenge in modern AI: balancing innovation driven by data with the imperative of privacy protection.

Background

▶ Watch: Introduction to differentially private synthetic text data (0:00)

The proliferation of data, especially sensitive user information, has created a fundamental tension in machine learning. While large datasets are crucial for training high-performing models, direct access and training on private data raise significant privacy concerns. The problem statement is clear: there exists private user data, often text, that cannot be directly used for model training due to privacy regulations and ethical considerations. Yet, this data holds immense value for improving model capabilities and enhancing downstream tasks, such as building more accurate spam classifiers or recommendation systems.

Traditionally, one approach to address this is to apply differential privacy directly to the training process of the downstream model, typically through techniques like Differentially Private Stochastic Gradient Descent (DP-SGD). This involves modifying the optimization algorithm itself to inject noise and clip gradients, thereby limiting the influence of any single data point. However, this path often requires deep expertise in differential privacy and intricate modifications to existing ML pipelines, making it challenging for many organizations.

This talk focuses on an alternative, increasingly popular strategy: generating synthetic data that adheres to differential privacy guarantees. The rationale behind this approach is that "DP synthetic data is an organizational breakthrough for DP." In typical enterprise settings, two distinct parties are involved: one party owns the private data and the existing ML pipeline but lacks DP expertise, while another party possesses DP research expertise but is unfamiliar with the specific downstream task or data nuances. Introducing DP directly into the pipeline in such a fragmented environment becomes exceedingly difficult due to the complex mathematical guarantees and implementation details required.

DP synthetic data elegantly solves this by acting as a computational interface. The data-owning party, with DP expertise (or using readily available DP tools), generates a synthetic dataset from their private data, ensuring it satisfies a rigorous DP guarantee. This synthetic data can then be handed off to the ML team, who can treat it as ordinary, non-private data. This decouples the DP implementation from the downstream model training, allowing each team to focus on its core competencies. It simplifies collaboration, accelerates development cycles, and lowers the barrier to adopting privacy-preserving machine learning, making the value of private data accessible in a secure and compliant manner.

Key Findings

▶ Watch: Why DP synthetic data is an organizational breakthrough (2:00)

The presentation outlines three primary methods for generating differentially private synthetic text data, each with distinct advantages and use cases: DP Fine-tuning, Private Evolution, and DP LLM Inference. The overarching finding is that DP Fine-tuning stands out as the most straightforward and performant method, serving as the "workhorse" for high-quality synthetic data generation. This method involves fine-tuning a language model (often with LoRA) using DP-SGD, directly embedding privacy into the model's learning process. Empirical evidence suggests that LoRA, a parameter-efficient fine-tuning technique, not only performs well but often outperforms full fine-tuning in DP settings, while also offering significant efficiency gains.

In contrast, Private Evolution and DP LLM Inference are presented as inference-only techniques that do not require access to or modification of the model's weights. These methods are particularly valuable when direct model training is not feasible, for instance, when only an API inference stack is available. While generally leading to lower quality synthetic data compared to DP Fine-tuning, they offer faster iteration times and reduced engineering effort. Private Evolution is highlighted for its utility with very small private datasets (less than 5,000 examples) and its ability to keep private data ephemeral and even local to the device. DP LLM Inference, which introduces DP during token-by-token decoding, is suitable for generating small amounts of text or for rapid prototyping.

A key conceptual contribution is the "unified view" of these methods, demonstrating that they all follow a common recipe: identify a primitive operation that interacts with private data, replace it with a differentially private version, and then leverage composition to bound the overall privacy loss. This principle underscores why DP Fine-tuning often performs best—it keeps the most aspects of the non-private training paradigm intact.

Finally, the talk emphasizes the importance of robust evaluation metrics, introducing the MAUVE score as a measure of distributional similarity between real and synthetic text datasets. Experimental results showcased that DP synthetic data, when generated via fine-tuning, achieves a MAUVE score significantly closer to real data than zero-shot LLM generation, correlating well with downstream task performance. Insights into hyperparameter tuning for DP-SGD, such as favoring small clipping norms and increasing compute for better utility, further guide practitioners toward optimal privacy-utility trade-offs.

Technical Deep Dive

▶ Watch: Overview of three DP synthetic data methods (3:40)

The core of the talk provides a detailed exploration of three distinct methodologies for generating differentially private synthetic text data, focusing on their algorithmic mechanisms and implementation considerations.

DP Fine-tuning

DP Fine-tuning is presented as the most robust and generally recommended approach. The process involves training a language model to generate text, where the input is a prompt like "generate text" and the targets are individual private records. The model learns to produce synthetic data that mimics the distribution of the private data while adhering to differential privacy.

Crucially, the talk highlights the use of LoRA (Low-Rank Adaptation) in conjunction with DP. LoRA, a parameter-efficient fine-tuning technique, often empirically outperforms full fine-tuning in DP settings. This is attributed to its efficiency gains, which are amplified in the DP context, allowing for more effective training under privacy constraints.

The foundation of DP Fine-tuning is DP-SGD (Differentially Private Stochastic Gradient Descent). The steps for a DP-SGD update are:

1. Sample Records: Select a mini-batch of records.
2. Compute Gradients: Calculate the gradient for each individual record in the mini-batch.
3. Clip Gradients: Limit the L2 norm of each individual gradient to a predefined clipping norm C. This step is crucial as it bounds the influence of any single data point on the model update.
4. Average and Add Noise: Average the clipped gradients and then add Gaussian noise scaled by a noise multiplier sigma. The noise masks the presence of individual users, ensuring privacy.
5. Update Parameters: Use this noisy, clipped, and averaged gradient to update the model parameters, replacing the regular gradient.

Each DP-SGD step consumes a portion of the privacy budget (epsilon, delta). The privacy cost accumulates over iterations, stopping when a target budget is met.

The implementation of DP-SGD in JAX is discussed, emphasizing efficiency. A naive implementation iterating through individual examples (batch size 1) for gradient computation is prohibitively slow on accelerators like TPUs. The solution is jax.vmap (vectorized map), which transforms per-example logic into a highly optimized, batched version that fully utilizes matrix multiplication units. The in_axes parameter for vmap specifies which input axes correspond to the data to be vectorized. For distributed settings, SPMD axis name can further parallelize per-example gradient computation across multiple devices, yielding significant speedups (e.g., 30x). The jax-privacy library simplifies this, allowing developers to replace jax.grad with jax_privacy.clip_grad, providing a higher-level API for DP-SGD. The recent 1.0.0 update to jax-privacy offers Keras-like APIs for easier integration.

Hyperparameter tuning for DP-SGD is critical. The two main parameters are the clipping norm C and the noise multiplier sigma. A key insight is that a wide range of small clipping norms work effectively. Picking a small C preserves gradient direction while reducing magnitude, leading to better empirical performance. A trick to decouple the learning rate from C is to normalize the clipped gradient by C and scale the noise similarly, ensuring that changes to C don't necessitate re-tuning the learning rate. Finally, increasing compute (more training steps or larger batch sizes) for a fixed privacy budget (by correspondingly increasing sigma) generally leads to better synthetic data quality.

Private Evolution

Private Evolution is an inference-only method suitable when direct model weight access is unavailable or for very small private datasets (<5k examples). It operates iteratively in an evolutionary fashion:

1. Seed Corpus: Start with a corpus of completely synthetic data, independent of the private data. This seed requires careful prompt engineering to ensure quality.
2. Filter: The private data is used to "vote" for synthetic examples most similar to itself. A histogram is constructed, showing similarity scores for each synthetic example. Differential privacy is introduced by adding noise to this histogram. The most similar synthetic examples are then selected.
3. Re-expand: An LLM is prompted to rewrite or expand the filtered synthetic examples, introducing perturbations while maintaining similarity to the private data's distribution.
4. Repeat: The expanded synthetic data then becomes the input for the next filtering step, continuing the evolutionary cycle.

A significant advantage of Private Evolution is its minimal private data persistence. The private data is only used to compute the similarity histogram and can remain on the user's device, with potentially only a single bit or a small sequence indicating preferred synthetic examples being sent. However, it demands substantial prompt engineering for both the initial seed corpus and the rewrite prompts.

DP LLM Inference

DP LLM Inference is another inference-only technique that introduces differential privacy during the token-by-token decoding process of a language model. It requires access to the model's logits (raw prediction scores) rather than just API outputs.

1. Disjoint Contexts: Private data points are placed into disjoint prompt templates (e.g., "generate text similar to [private data point]").
2. Parallel Prediction: Each prompt, representing an individual private data point, is run through the LLM independently to predict the next token's logits.
3. Aggregate and Mix: The logits from all individual predictions are aggregated and mixed to form a single, differentially private probability distribution over the next token. This aggregation step is where DP is applied.
4. Sample Token: A new token is sampled from this aggregated distribution and appended to the synthetic output.
5. Iterate: This process repeats for each subsequent token until the synthetic text is complete. A privacy cost is incurred for every token generated.

A common technique for token aggregation is the exponential mechanism, where individual logits are clipped to bound their sensitivity, then averaged, and finally passed through a softmax function to produce the final probability distribution. A trick for efficiency involves using a public prompt: if the aggregated prediction is very similar to what a public prompt (without private data) would have produced, the model can sample from the public prompt's distribution, saving privacy budget. This method is generally suited for generating smaller amounts of text or for prototyping due to the per-token privacy cost and computational expense.

Experimental Setup & Results

▶ Watch: DP fine-tuning: the most straightforward and performant method (6:00)

The talk illustrates the practical effectiveness of these DP synthetic data generation techniques through various examples and evaluation metrics, primarily focusing on text-based data. While specific dataset names beyond "reviews" are not explicitly mentioned, the examples provided offer insights into the utility and performance characteristics.

The primary metric for evaluating the quality and similarity of generated text is the MAUVE score. MAUVE (Measuring the quality of generated text with human-like semantics) assesses the distributional similarity between two text datasets (real vs. synthetic). It works by embedding and clustering both real and synthetic data, then comparing the distributions of these clusters. A higher MAUVE score indicates greater similarity, and it has been shown to correlate well with downstream task performance, making it a robust proxy for utility.

In a demonstration involving the generation of synthetic reviews, the MAUVE scores revealed a clear hierarchy of quality:

• Real Data: Comparing two subsets of real data yielded a MAUVE score close to 1, as expected for identical distributions.
• Non-DP Synthetic Data: This baseline, generated without privacy constraints, achieved a high MAUVE score, indicating good similarity to real data.
• DP Synthetic Data (Fine-tuning): The data generated using DP Fine-tuning achieved a "pretty good" MAUVE score, demonstrating its effectiveness in producing high-quality text even under differential privacy constraints. This score was significantly better than other baselines.
• Zero-shot LLM Generation: Simply prompting a large language model (LLM) to generate reviews (e.g., "generate reviews") without fine-tuning resulted in a much lower MAUVE score. This highlights that while LLMs can produce coherent text, their zero-shot outputs often differ in subtle ways from real data distributions, which is captured by MAUVE, making direct LLM prompting an insufficient method for high-fidelity synthetic data generation.

Regarding the technical implementation and performance, the talk highlights the efficiency gains of using JAX's vmap functionality for DP-SGD. While a naive, per-example gradient computation is slow on accelerators, vmap allows for vectorized, batched computation, fully utilizing TPUs and leading to substantial speedups. For distributed training, SPMD axis name further enhances this by parallelizing per-example gradient computation across multiple devices, which can achieve up to 30 times speedup.

Insights into hyperparameter tuning are also presented, particularly for the clipping norm C in DP-SGD. A plot from a research paper illustrates that for a fixed learning rate, setting the clipping norm below a certain threshold (e.g., 2^4 in the example) yields consistently good performance, with relatively small variations in accuracy. This supports the practical advice to select a small clipping norm and leverage the normalization trick to decouple it from the learning rate, simplifying the tuning process.

Finally, a general observation from experiments is that for a fixed privacy budget (epsilon), increasing the computational resources—either by taking more training steps or using larger batch sizes—and correspondingly adjusting the noise multiplier sigma typically leads to improved results. This suggests a direct trade-off where higher compute budgets can yield better utility even under strict privacy guarantees.

Practical Implications

▶ Watch: Recap of DP-SGD: clipping gradients and adding noise (6:40)

The development and deployment of differentially private synthetic data generation techniques carry significant practical implications for a wide range of stakeholders, from individual practitioners to large infrastructure teams and model deployers. The talk emphasizes that the choice of method – DP Fine-tuning, Private Evolution, or DP LLM Inference – depends heavily on specific constraints and objectives, particularly regarding data volume, access to model internals, and desired quality.

For practitioners and model builders aiming for the highest quality synthetic data, DP Fine-tuning is the recommended "workhorse." This approach, especially when combined with LoRA, offers superior performance and adaptability, even to out-of-distribution data. However, it comes with a higher bar: it demands access to model weights, requires significant engineering effort to implement and integrate DP-SGD into existing training pipelines, and is computationally intensive. The speaker stresses that this investment is worthwhile, as the primary bottleneck for DP synthetic data adoption is often quality. If the generated data isn't good enough, it won't be used, rendering the privacy efforts moot. Therefore, for serious applications, investing in DP Fine-tuning is crucial.

Infrastructure teams should note the efficiency gains demonstrated by JAX's vmap and jax-privacy library. These tools significantly simplify the implementation of DP-SGD and make it feasible to run on large-scale distributed hardware like TPUs, reducing the performance overhead traditionally associated with DP. The ability to decouple the clipping norm from the learning rate also streamlines hyperparameter tuning, saving valuable compute resources.

For scenarios where full model access or extensive training infrastructure is not available, the inference-only methods offer viable alternatives. Private Evolution is particularly well-suited for situations with very small private datasets (e.g., less than 5,000 examples) or when private data must remain on the user's device, as it only uses the private data for a noisy similarity histogram. This makes it the most privacy-conscious in terms of data persistence. However, it requires considerable prompt engineering for the seed corpus and rewrite prompts, and the data generation process can be expensive due to its iterative nature, leading to generally lower quality than fine-tuning.

DP LLM Inference is positioned for prototyping or generating small amounts of synthetic text, perhaps for in-context learning examples. It requires logits access from the LLM but avoids weight modification. Similar to Private Evolution, it's generally lower quality and expensive for large-scale generation due to per-token privacy costs. Both inference-only methods are also less adaptable to out-of-distribution data compared to fine-tuning.

The overarching practical implication is the "organizational breakthrough" aspect of DP synthetic data. It creates a clear interface, allowing data owners (who may not be DP experts) to securely provide data to ML teams (who are DP experts) without direct exposure of raw private information. This fosters collaboration and accelerates the adoption of privacy-preserving machine learning across organizations by abstracting away the complexities of DP implementation from the data consumption stage. However, it also introduces tradeoffs: higher quality typically demands more compute, engineering effort, and access to model internals, while less demanding methods trade off quality and scalability.

Key Takeaways

• DP synthetic data is an "organizational breakthrough": It serves as a crucial interface, enabling collaboration between data owners (who may lack DP expertise) and ML teams (who understand DP) by providing a privacy-preserving mechanism to share data value without exposing raw private information.
• DP Fine-tuning is the preferred method for quality: For generating high-quality differentially private synthetic text data, DP Fine-tuning (especially with LoRA) is the most performant and adaptable approach, despite requiring more compute and engineering effort.
• Inference-only methods offer alternatives for specific scenarios: Private Evolution is ideal for very small datasets or when private data must remain local, while DP LLM Inference is suitable for prototyping or generating small amounts of text, though both generally yield lower quality than fine-tuning.
• Efficient DP-SGD implementation is critical: Tools like JAX's vmap and the jax-privacy library significantly streamline and accelerate the implementation of DP-SGD, making it feasible for large-scale training on accelerators like TPUs.
• Hyperparameter tuning impacts utility: Careful selection of the clipping norm (favoring small values) and strategically increasing compute (more steps or larger batch sizes for a fixed epsilon) are key to maximizing the utility of DP synthetic data.
• MAUVE score is a robust evaluation metric: The MAUVE score effectively measures the distributional similarity between real and synthetic text, correlating well with downstream task performance and providing a reliable metric for assessing synthetic data quality.

About the Speaker(s)

Alex Bie is a researcher at Google, where he works on differentially private synthetic data. He presented this talk, outlining various methods and their practical implications for generating privacy-preserving text. He collaborates with Natalia Ponomareva and Sergei Vassilvitskii, who are also involved in this research at Google. Peter Kairouz is additionally listed as a speaker, indicating his contributions to the work. Their collective efforts at Google focus on advancing the field of differential privacy and its application to synthetic data generation, aiming to enable the use of sensitive information for AI/ML model improvement while upholding rigorous privacy guarantees.