DP-fy your DATA: How to (and why) synthesize Differentially Private Synthetic Data: Practical components for DP synthetic data system

DP-fy your DATA: How to (and why) synthesize Differentially Private Synthetic Data: Practical components for DP synthetic data system

Speakers: Natalia Ponomareva, Sergei Vassilvitskii, Peter Kairouz, Alex Bie

Conference: ICML 2025

YouTube: https://slideslive.com/39043839

Overview

This talk, part of a broader tutorial on differentially private (DP) synthetic data, shifts focus from the algorithmic foundations to the critical system-level considerations required for deploying DP in practice. Presented by Peter Kairouz, with contributions from Natalia Ponomareva, Sergei Vassilvitskii, and Alex Bie, the session delves into the intricate engineering challenges and auditing mechanisms necessary to ensure that synthetic data generated with differential privacy truly upholds its privacy guarantees in real-world applications. It highlights that simply "flipping on the DP switch" is insufficient; a robust system architecture, careful definition of privacy units, and continuous auditing are paramount.

The presentation underscores the significant gap between theoretical differential privacy guarantees and their practical implementation, especially when dealing with large-scale machine learning models and sensitive user data. Kairouz meticulously outlines how to operationalize user-level differential privacy, design secure data processing pipelines, and, crucially, audit the resulting models and synthetic data for potential privacy leaks. This talk is vital for engineers, researchers, and privacy officers grappling with the complexities of building privacy-preserving AI systems, offering a pragmatic roadmap beyond purely mathematical considerations. It emphasizes that a comprehensive approach, encompassing not only algorithms but also system design, rigorous testing, and data governance, is essential for truly safeguarding user privacy.

Background

▶ Watch: Introduction: Putting DP into a system (0:00)

The concept of differential privacy (DP) hinges on the idea that the output of a computation should be nearly indistinguishable whether it was performed on a dataset D or an adjacent dataset D' that differs by only one individual's data. This foundational principle aims to protect individuals from having their data revealed through statistical analysis of aggregated results. Prior talks in this series likely covered various DP algorithms for text, images, and tabular data, such as DP-SGD (Differentially Private Stochastic Gradient Descent). However, implementing DP in real-world machine learning systems, particularly with large language models (LLMs) and other generative models, introduces several complex challenges.

A primary challenge lies in defining the unit of privacy. While DP theory often considers an "example" as the unit of difference between adjacent datasets, in practice, a user might contribute multiple examples (e.g., many sentences, images, or records). Protecting privacy at the example level might not adequately safeguard the user if their multiple contributions could still reveal their presence. Therefore, the talk emphasizes the need for user-level differential privacy, where the output remains indistinguishable whether an entire user's data is present or absent from the dataset. This shift necessitates modifications to standard DP-SGD, such as sampling users instead of individual examples and clipping gradients at the user level rather than the example level.

Furthermore, the integration of DP into a broader system requires careful architectural design. Raw, sensitive user data and intermediate computations that could derive sensitive information must be handled within a highly secure and isolated environment, which the speaker dubs the "red zone." This isolation prevents unauthorized human inspection or unintended leakage before DP mechanisms are fully applied. The talk implicitly acknowledges that without these systemic considerations, even theoretically sound DP algorithms can be undermined, leading to privacy vulnerabilities in deployed systems.

Key Findings

▶ Watch: Algorithmic modification for user-level DP-SGD (2:00)

The presentation uncovers several critical insights regarding the practical application and assurance of differential privacy in ML systems:

1. Algorithmic DP is Insufficient Without Systemic Safeguards: The core message is that merely implementing DP algorithms like DP-SGD is not enough. A comprehensive system design, including secure data handling ("red zone"), user-level privacy considerations, and robust auditing, is essential to realize true privacy guarantees.
2. Model Auditing is a Non-Negotiable Step: Before any DP-fied model or its synthetic data output is shared, it must undergo rigorous auditing. This auditing process is crucial for verifying that the DP mechanisms are effective in practice and that the model does not inadvertently memorize or leak sensitive user information. The talk introduces practical techniques for this, including reconstruction attacks and user-level membership inference.
3. The Discrepancy Between Theoretical and Practical DP: A significant finding is the observed gap between the strict theoretical bounds of DP (often requiring very small epsilon values for strong guarantees) and the practical effectiveness of DP with larger epsilon values (e.g., epsilon=10). While theory might suggest substantial leakage at epsilon=10, practical attacks often fail to reconstruct data or infer membership when DP is applied, even at these higher epsilons. This discrepancy is attributed to unquantified sources of randomness (e.g., model initialization, data shuffling) and limited adversary access (e.g., only final model, API access).
4. User-Level Privacy and Gradient Clipping are Paramount: For generative models trained on user data, protecting the entire user's contribution (user-level DP) is critical. The modification of DP-SGD to sample users and clip per-user gradients, along with bounding the number of examples per user, is a practical and effective strategy for achieving this.
5. Data Lineage and Budget Accounting are Essential for Complex Systems: In real-world scenarios, synthetic data might be generated, mixed with other data, or used in subsequent pipelines. The talk highlights that understanding the origin and privacy budget of each data component (data lineage) is crucial for correctly accounting for cumulative privacy loss and ensuring end-to-end guarantees.
6. DP is One Component of a Broader Privacy Framework: While DP is a powerful tool for data anonymization, it is not a silver bullet. A holistic approach to privacy also requires attention to transparency (informing users), user control (opt-out mechanisms), data minimization, and verifiable end-to-end processes.

Technical Deep Dive

▶ Watch: End-to-end system architecture: The 'red zone' (2:40)

The technical deep dive of the talk focuses on two core areas: the architectural components for a differentially private synthetic data system and the methodologies for auditing such systems.

User-Level Differential Privacy Implementation

The primary algorithmic modification discussed for achieving user-level differential privacy involves adapting DP-SGD. Instead of treating individual data examples as the unit of privacy, the system aims to bound the contribution of an entire user. This is crucial because a single user might contribute multiple training examples, and protecting only individual examples might still allow an adversary to infer information about the user by combining their contributions.

1. Unit of Privacy Definition: The challenge begins with defining "Peter's data" in a complex, interconnected dataset. To address this, each training example (e.g., text, image) is tagged with the collection of users it could belong to.
2. User Sampling: During the training process, specifically in DP-SGD, instead of sampling individual examples for a batch, the algorithm samples users. For instance, if a batch targets 20 users, Poisson sampling is applied to the user IDs.
3. Example Bounding: To prevent a single user with an exceptionally large number of examples from disproportionately influencing the model, the number of examples taken from any individual user is capped (e.g., to 50 examples, or sometimes 10 or less).
4. Per-User Gradient Clipping: For the sampled users, gradients are computed on their respective examples. These example gradients are then aggregated to form a per-user gradient. It is this per-user gradient that is subsequently clipped to a predefined maximum L2 norm (clipping threshold). This ensures that no single user's total contribution to the model update exceeds a certain bound.
5. Noise Addition: After clipping, all per-user gradients from the batch are aggregated, and calibrated noise (typically Gaussian) is added to the aggregated gradient before the model parameters are updated. This noise addition is the core mechanism that provides the differential privacy guarantee.

System Architecture for DP Synthetic Data Generation

The talk introduces a conceptual system architecture, emphasizing a "red zone" for highly sensitive operations:

1. Red Zone Components (Sensitive & Isolated):

• Data Filtering: Large datasets are filtered to retain high-quality, relevant data for the specific application. This step occurs before DP is applied.
• Example Bounding: As described above, the number of examples per user is capped to mitigate individual influence.
• DP Fine-tuning/Training: A pre-trained model checkpoint (e.g., an LLM or diffusion model) is fine-tuned using the user-level DP-SGD algorithm on the bounded corpus. The output is a DP-fied generative model.
• Quality Measurement (Potentially Red Zone): Measuring the quality of synthetic data (e.g., using MAUVE scores or token distribution analysis) often requires comparing it against the original sensitive data. This comparison itself could leak information, necessitating that these quality computations also occur within the red zone or are themselves DP-fied.

These red zone components are designed to be isolated, ideally in a cloud environment where engineers cannot freely access raw data or intermediate derivatives for inspection or debugging.

1. API Access to DP-fied Model: Once the model is DP-fied, engineers are typically granted access only via an API. They can prompt the model to generate synthetic data but cannot directly inspect the model's parameters or internal workings. This further reduces the attack surface.
2. Human-Inspectable Synthetic Data: The final output of the system is the DP synthetic data, which is now considered safe for human inspection, sharing, and use in various downstream tasks.

Model Auditing Methodologies

Auditing is presented as a crucial additional step to verify the practical efficacy of DP. The goal is to distinguish between benign memorization (e.g., common phrases) and privacy violations (e.g., unique sensitive information).

1. Reconstruction Attacks:

• Goal: Determine if the model can reconstruct unique sensitive sequences (e.g., credit card numbers) from a prefix.
• Method:
• Take a dataset keyed by user IDs (e.g., Alice's data contains "Alice's credit card number is [16-digit number]").
• Train the model (with or without DP).
• Prompt the model with a prefix ("Alice's credit card number is") and measure its ability to complete the suffix correctly, often using edit distance to account for approximate reconstructions.
• Mitigating False Positives (Common Sequences): To avoid incorrectly flagging common phrases (e.g., "please let me know if you have any questions") as privacy violations, a held-out dataset is used.
• The training set is split into a small held-in set (trained on) and a held-out set (never trained on).
• Reconstruction rates are compared between the held-in and held-out sets. If rates are similar, it indicates generalization; if significantly different, it points to memorization and a potential privacy violation.
• Canary Users / Random Data: To create stronger attack vectors, fictitious "canary users" with random, nonsensical data sequences can be inserted into the training set. Since these sequences are highly unlikely to appear naturally, any reconstruction indicates direct memorization. For these, a held-out set is unnecessary as reconstruction rates on unseen random data will be effectively zero.

1. User-Level Membership Inference Attacks:

• Goal: Determine if a specific user (or their data) was included in the training set.
• Operational Definition: Framed as a game between a crafter (who picks one of two adjacent datasets, one with Peter, one without) and an adversary (who tries to guess which dataset was used based on the model output).
• ROC Curve: The adversary's performance is measured using Receiver Operating Characteristic (ROC) curves, plotting True Positive Rate (TPR) against False Positive Rate (FPR). A good adversary is in the top-left corner. DP guarantees that the adversary cannot achieve both high TPR and low FPR simultaneously.
• Scalar Statistic (Log-Likelihood Ratio): A simple yet effective attack involves computing a log-likelihood ratio. For a given prefix ('Alice spent her vacation in') and two potential suffixes ('Italy' vs. 'the Faroe Islands'), the adversary compares the likelihood of each suffix under the fine-tuned model, normalized by the likelihood under the pre-trained model. A higher likelihood for one suffix suggests that the model "saw" that information during training.
• Canary Users for MIAs: Similar to reconstruction, random canary users are used. A set of canaries is split into held-in (trained on) and held-out (not trained on). The adversary is given the trained model and a shuffled list of all canaries (both held-in and held-out) and must deduce for each canary whether it was used in training. This requires a held-out set to measure FPR.

Discrepancy Between Theory and Practice

The talk highlights a critical observation: DP, even with larger epsilon values (e.g., epsilon=10), often performs significantly better in practice against these attacks than theoretical bounds might suggest. This gap is attributed to:

1. Unquantified Sources of Randomness: DP theory often assumes worst-case scenarios and precise knowledge of the training process. In reality, factors like random model initialization, data shuffling, and batching introduce additional randomness that is not precisely captured in the epsilon calculation, making attacks harder.
2. Adversary Access Limitations: Theoretical DP often assumes the adversary can observe all intermediate model checkpoints during training. In practice, adversaries typically only have access to the final model or, more restrictively, just an API to the model. Hiding intermediate states and restricting access to outputs significantly boosts practical privacy.
3. Non-Convexity of Deep Learning: While some theoretical results exist for convex losses, proving the practical privacy boost from limited adversary access for non-convex deep neural networks remains an open problem.

Experimental Setup & Results

▶ Watch: Operational view of Differential Privacy: Crafter vs. Adversary (6:00)

The presentation provides concrete empirical results to illustrate the efficacy of the proposed DP system and auditing techniques, primarily using Gemini models for fine-tuning.

Experimental Setup

• Model: Gemini models, fine-tuned on various datasets.
• Privacy Unit: User-level privacy, with user contributions capped at 39 examples per user for most experiments, and occasionally 1 example per user for comparison.
• DP Mechanism: DP-SGD with user-level gradient clipping and noise addition.
• Auditing Data:
• Natural Data: Real-world user data split into held-in (trained on) and held-out (not trained on) sets for comparison.
• Random/Canary Data: Fictitious users with random, nonsensical sequences of tokens. These are used to create strong memorization signals, as they are unlikely to appear naturally. For canary data, held-out sets are generally not needed for reconstruction attacks, as unseen random data will have a reconstruction rate of zero.

Results without Differential Privacy

The initial experiments demonstrate the significant privacy leakage that occurs when models are trained without DP, even with basic safeguards like capping user examples.

1. Reconstruction Attacks (No DP, 39 examples/user):

• Natural Data: The model was capable of perfectly reconstructing approximately 2% of the strings from the held-in set. This rate increased to about 20% for reconstructions up to an edit distance of five. Crucially, the reconstruction rates on the held-in (inserted) data were statistically significantly higher than on the held-out (uninserted) data, indicating genuine memorization rather than generalization.
• Random/Canary Data: When using artificial "canary users" with random data, the perfect reconstruction rate for inserted sequences jumped to approximately 7%. This highlights that random, outlier data can lead to even stronger memorization, as these sequences generate larger gradients that the model can more easily internalize, especially during full fine-tuning of large capacity models.
• Conclusion: These rates (2% perfect, 20% approximate, 7% for canaries) are deemed "alarming" and unacceptable for privacy-sensitive applications, indicating a clear need for stronger protection.

1. User-Level Membership Inference Attacks (No DP):

• 39 Examples/User: When a user contributed up to 39 examples, the adversary achieved a True Positive Rate (TPR) of approximately 99.9% at a False Positive Rate (FPR) of 0.1%. This signifies an almost perfect adversary, capable of accurately determining if a user was in the training set.
• 1 Example/User: Even when a user's contribution was severely restricted to just one example, the TPR was still 11.4% at an FPR of 0.1%. While significantly better than 99.9%, this still represents a non-trivial privacy leakage.
• Conclusion: Membership inference attacks are highly effective without DP, even with basic data bounding, reinforcing the need for stronger privacy mechanisms.

Results with Differential Privacy

The impact of applying DP, even with a relatively high epsilon, dramatically changes the outcome of these attacks.

1. Reconstruction Attacks (DP with epsilon=10):

• When DP was enabled with an epsilon value of 10, the reconstruction rates for all edit distances, across both natural and canary data, dropped to literally zero. The models showed no ability to reconstruct any specific training sequences, even those designed to be strong memorization signals.
• Conclusion: DP is highly effective at preventing direct data reconstruction, even at an epsilon value that might be considered "large" by strict theoretical standards.

1. User-Level Membership Inference Attacks (DP with epsilon=10):

• With DP enabled at epsilon=10, and allowing users up to 39 examples, the True Positive Rate for membership inference plummeted to 0.1% at an FPR of 0.1%. This is a massive reduction from the 99.9% TPR observed without DP.
• Conclusion: DP effectively mitigates membership inference attacks, rendering the adversary nearly incapable of distinguishing between members and non-members.

These results collectively demonstrate that while models without DP are highly susceptible to privacy attacks, the practical application of user-level differential privacy, even with a seemingly large epsilon, provides a robust defense against both reconstruction and membership inference attacks. This highlights the importance of practical auditing to confirm the efficacy of DP in real-world systems, given the observed discrepancy between theoretical bounds and empirical performance.

Practical Implications

▶ Watch: Measuring adversary performance with ROC curve (7:00)

This talk offers profound practical implications for anyone involved in building, deploying, or overseeing AI/ML systems that handle sensitive data.

For Practitioners and Infra Teams

• Shift from Algorithmic to Systemic Thinking: Data scientists and ML engineers must move beyond merely applying DP algorithms to considering the entire system architecture. Designing "red zones" for sensitive data processing, securing APIs, and implementing robust access controls are as critical as the choice of DP parameters.
• Mandatory Auditing for DP Systems: DP should not be treated as a "set it and forget it" solution. Rigorous, continuous auditing using techniques like reconstruction and membership inference attacks is essential to verify the practical effectiveness of DP. This means allocating resources for developing and running these audits as a standard part of the MLOps pipeline.
• User-Level Privacy as Default: For applications dealing with individual user data, implementing user-level DP (sampling users, clipping per-user gradients) is crucial. This requires careful data tagging and aggregation mechanisms.
• The Nuance of Canary Data: While random canary data provides strong signals for auditing memorization, the talk highlights an open research problem: how to construct "natural-looking" canaries that maximize memorization detection. Practitioners should experiment with various canary types to find the most effective for their specific models and data.
• Data Lineage and Privacy Budget Accounting: For complex systems where synthetic data might be generated, combined, or used in subsequent pipelines, maintaining data lineage is non-negotiable. This involves tracking the origin, privacy budget, and generation timestamp of every data component. Without this, privacy guarantees can unknowingly be diluted or violated across cascading operations.
• API-First Model Deployment: The practice of exposing DP-fied models only via API, rather than raw model parameters, significantly enhances practical privacy. This limits the adversary's ability to run sophisticated attacks that rely on inspecting internal model states.

Tradeoffs and Limitations

• Increased System Complexity: Implementing a comprehensive DP system with red zones, user-level DP, and auditing adds significant complexity to data pipelines and model deployment. This translates to higher development, maintenance, and operational costs.
• Computational Overhead: DP-SGD, particularly with per-user gradient clipping and noise addition, can introduce computational overhead during training, potentially increasing training times and resource requirements.
• Open Research Problems:
• Optimal Canary Construction: The talk explicitly states that finding the "strongest canaries" for auditing is an open problem, meaning practitioners currently need to rely on heuristic approaches.
• Direct Synthetic Data Auditing: Auditing models via API access is good, but the challenge of directly auditing synthetic data (without model access or adversarial prompts) remains largely unexplored. This is crucial for scenarios where only the data product is shared.
• Closing the Theory-Practice Gap: While practical DP performs well, a deeper theoretical understanding of why larger epsilon values are effective in practice (due to unquantified randomness and limited adversary access) would help in setting more informed privacy budgets.
• DP as Part of a Larger Privacy Strategy: DP addresses data anonymization but doesn't cover all aspects of privacy. Organizations still need to implement broader privacy principles like transparency, user control (opt-out), data minimization, and robust data governance (e.g., encryption, federated learning, data retention policies). Ignoring these can undermine the benefits of DP.

In essence, the talk serves as a call to action for the ML community to adopt a more holistic, engineering-centric view of differential privacy, moving beyond mere algorithmic implementation to comprehensive system design, rigorous validation, and continuous ethical consideration.

Key Takeaways

• User-Level DP is Crucial: For practical ML systems, especially with generative models, defining and enforcing privacy at the user level (e.g., via user sampling and per-user gradient clipping in DP-SGD) is paramount.
• Secure System Architecture is Non-Negotiable: Sensitive data operations, including filtering, bounding, and DP fine-tuning, must be isolated within a "red zone" to prevent unauthorized access and inadvertent leakage.
• Model Auditing is Essential for Verification: Rigorous auditing using techniques like reconstruction attacks (with held-in/held-out comparisons) and user-level membership inference attacks is necessary to empirically validate that DP mechanisms are effective in practice.
• Practical DP is Highly Effective: Even with relatively large epsilon values (e.g., epsilon=10), differential privacy significantly mitigates reconstruction and membership inference attacks, demonstrating a strong practical defense against privacy violations.
• Data Lineage and Budget Management are Critical: In complex, multi-stage systems, tracking the origin and cumulative privacy budget of data (data lineage) is vital to ensure end-to-end privacy guarantees and prevent budget exhaustion.
• DP is One Piece of the Privacy Puzzle: While powerful, differential privacy must be integrated into a broader privacy framework that includes transparency, user control, data minimization, and robust data governance practices.

About the Speaker(s)

This talk was delivered by Peter Kairouz, who focused on the system-level and practical components of differential privacy. He is a prominent researcher in the field of privacy-preserving machine learning. The presentation also acknowledged contributions from his co-presenters: Natalia Ponomareva, Sergei Vassilvitskii, and Alex Bie. Natalia Ponomareva was specifically mentioned as leading the effort to put together a monograph for an in-depth discussion of DP methods, indicating her significant expertise and leadership in this area. Together, the team represents a collaborative effort to bridge the gap between theoretical differential privacy and its robust, practical deployment in real-world AI systems.