Privacy in Theory, Bugs in Practice: Grey-Box Auditing of Differential Privacy Libraries
Tudor Cebere (Paper author · Inria, Université de Montpellier), David Erb (Paper author · Technical University of Munich, Oblivious), Damien Desfontaines (Paper author · Hiding Nemo), Aurélien Bellet (Paper author · Inria, Université de Montpellier), Jack Fitzsimons (Paper author · Oblivious)
Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 3 · Beaver
Overview
This talk, presented at PETS, delves into the critical discrepancy between the theoretical guarantees of Differential Privacy (DP) and the often-flawed reality of its practical implementations. Titled "Privacy in Theory, Bugs in Practice: Grey-Box Auditing of Differential Privacy Libraries," the research, driven by the personal experience of developing a differentially private SQL engine, introduces a novel grey-box auditing methodology designed to identify subtle yet catastrophic privacy vulnerabilities in DP software. The speakers highlight that while DP offers robust mathematical assurances of privacy, these assurances frequently break down due to implementation errors, ranging from incorrect sensitivity calculations to improper parameterization of noise mechanisms.

Key moments
- 0:00 Motivation: ensuring DP implementations do what they say
- 1:00 Practical pitfalls: sensitivity mismatch, data-dependent processing
- 2:00 Why existing DP verification methods are impractical
- 3:00 Our grey-box auditing approach using software tests
- 4:20 Detailed explanation of the "record replay" technique
- 6:40 Catastrophic privacy failures in major open-source frameworks
- 8:00 Crucial takeaway: always test your differential privacy code
Privacy in Theory, Bugs in Practice: Grey-Box Auditing of Differential Privacy Libraries
Speakers: Tudor Cebere (Paper author, Inria, Université de Montpellier); David Erb (Paper author, Technical University of Munich, Oblivious); Damien Desfontaines (Paper author, Hiding Nemo); Aurélien Bellet (Paper author, Inria, Université de Montpellier); Jack Fitzsimons (Paper author, Oblivious)
Conference: PETS
YouTube: https://www.youtube.com/watch?v=uj0MYE0b0EU
Overview
This talk, presented at PETS, delves into the critical discrepancy between the theoretical guarantees of Differential Privacy (DP) and the often-flawed reality of its practical implementations. Titled "Privacy in Theory, Bugs in Practice: Grey-Box Auditing of Differential Privacy Libraries," the research, driven by the personal experience of developing a differentially private SQL engine, introduces a novel grey-box auditing methodology designed to identify subtle yet catastrophic privacy vulnerabilities in DP software. The speakers highlight that while DP offers robust mathematical assurances of privacy, these assurances frequently break down due to implementation errors, ranging from incorrect sensitivity calculations to improper parameterization of noise mechanisms.
The core motivation behind this work stems from a "selfish place," as the speakers themselves faced the challenge of ensuring their own DP software did "what it says on the tin." Recognizing the limitations of existing formal verification and black-box auditing approaches, they developed a practical, scalable testing framework that integrates seamlessly into development workflows, much like traditional unit and integration testing. This approach aims to provide strong assurances for developers building real-world DP systems without imposing the prohibitive overhead of formal methods or the scalability issues of purely empirical black-box testing.
The significance of this research cannot be overstated. Differential privacy is increasingly adopted across industries to enable data analysis while protecting individual privacy. However, a faulty implementation renders these protections moot, potentially exposing sensitive data and eroding public trust. By demonstrating that even widely used, expertly developed DP frameworks are riddled with critical privacy bugs, the talk underscores an urgent call to action for rigorous testing and auditing throughout the DP software development lifecycle. The methodology presented offers a tangible path forward for organizations to validate their DP implementations, ensuring that privacy promises translate into practical reality.
Background
▶ Watch: Motivation: ensuring DP implementations do what they say (0:00)
The promise of Differential Privacy (DP) is profound: to enable data analysts to extract valuable insights from sensitive datasets without revealing information about any individual participant. This is achieved by injecting carefully calibrated noise into queries or statistical outputs, ensuring that the presence or absence of a single individual's data point does not significantly alter the final result. In theory, DP provides strong, quantifiable privacy guarantees, making it a cornerstone for privacy-preserving data analysis and machine learning.
However, the journey from theoretical elegance to practical implementation is fraught with challenges. The speakers articulate that the "promises that are made are based on a whole bunch of kind of theoretical assumptions that aren't always the case when you implement something in practice." This disconnect creates a significant privacy risk. Common pitfalls leading to these practical failures include:
- Floating-point attacks: While not directly addressed by this research, the speaker mentions this as a well-known example where numerical precision issues can compromise privacy.
- Data pre-processing errors: Often, private data is pre-processed before being fed into a DP mechanism. If this pre-processing involves operations that inadvertently reveal information or change the data's characteristics in unexpected ways, the privacy guarantee can be violated.
- Data-dependent control flows: Branching logic (e.g.,
ifstatements) based on private data can subtly alter the sensitivity of the query or computation. If the system doesn't correctly account for these changes, the amount of noise added might be insufficient. - Incorrect sensitivity calculations: The sensitivity of a function dictates how much noise needs to be added to achieve a desired privacy level. Miscalculating this sensitivity, either due to complex data transformations or overlooked edge cases, is a common source of privacy breaches.
- Parameter misalignment: Mechanisms might be configured with parameters that don't align with the actual data characteristics or the intended privacy properties, as demonstrated by one of the key findings.
These issues collectively mean that the privacy guarantees reported by a system might not be genuinely correct, leaving users with a false sense of security.
Prior attempts to address this theory-practice gap have largely fallen into two categories:
- Formal Verification: This approach involves defining DP algorithms using specialized Domain Specific Languages (DSLs), such as those found in systems like Pink. By coding algorithms in these highly constrained environments, it's possible to obtain "pretty robust guarantees." However, formal verification is notoriously impractical for production systems. It often requires all data to reside in memory, struggles with complex real-world data pipelines, and imposes significant development overhead, making it "not practical basically for for many like production systems."
- Black-Box Auditing: This method treats the DP system as an opaque entity. It relies on taking "a whole bunch of kind of like samples through a system" to empirically estimate the output distribution and infer the privacy guarantees. While there have been advances, black-box auditing suffers from scalability issues, particularly for high-dimensional systems. "You need to have a huge amount of samples and often that's just not really scalable." This makes it unsuitable for comprehensive testing of complex DP frameworks.
The absence of a practical, scalable, and robust method for verifying DP implementations in real-world software engineering contexts highlighted a critical gap. The speakers sought to bridge this gap by developing a methodology analogous to the unit and integration testing commonly employed in general software development, but tailored specifically for the unique challenges of differential privacy.
Key Findings
▶ Watch: Why existing DP verification methods are impractical (2:00)
The most striking and significant finding of this research is the widespread prevalence of catastrophic privacy failures in widely used open-source Differential Privacy (DP) frameworks. After developing their grey-box auditing methodology (which they call "record replay"), the researchers applied it to the 12 most prominent and widely adopted DP frameworks available. The results were alarming: all but four of the 12 frameworks exhibited critical privacy vulnerabilities. This translates to a staggering 66% failure rate among leading DP implementations.
These frameworks, often developed by renowned experts who themselves "had developed the theory in the first place," include well-known names such as SmartNoise SDK, SmartNoise SQL, DPMM, Private PGM (which notably won the NIST prize), Sensitivty, and Diffpriv. The discovery that even these highly regarded projects, backed by institutions like Meta and Google, contained fundamental bugs underscores the inherent difficulty of correctly implementing DP. The speaker candidly points out, "if they're introducing bugs they're coming up with the protocols in the first place you're probably also introducing bugs and that's okay. You should just test your own code."
A specific example illustrating the nature of these bugs was found in PGM, a framework based on a paper from a few years prior. The bug involved a seemingly minor parameter change: "they just changed one of the parameters from two to four somewhere along the line which switches it from uh um unbounded to a bounded DP but everything else in the program was set up for the other if that makes sense." This subtle numerical alteration completely undermined the intended privacy guarantees, shifting the underlying mechanism from one privacy model to another without corresponding adjustments in other parts of the code. The "record replay" method was able to identify this critical flaw with a "very very lightweight test."
The findings also highlighted which frameworks did pass the audit without immediate issues. These were:
- OpenDP: This framework employs a much more formal verification approach, being "very functional" in its design. Its success suggests that rigorous, functional programming paradigms can mitigate some implementation risks, albeit with potential trade-offs in flexibility or ease of use.
- Google's frameworks: Specifically, Jack's privacy and other mechanisms provided by Google also passed the audit. This indicates that some organizations are successfully implementing robust internal testing and verification processes for their DP offerings.
The overarching key takeaway from these findings is a stark reminder that theoretical guarantees do not automatically translate to practical security. The complexity of DP implementation, coupled with the subtle ways in which data flows and parameters can be mishandled, necessitates proactive and continuous testing. The research effectively serves as a wake-up call, emphasizing that even the most cutting-edge privacy technologies require the same, if not more, rigorous software engineering practices as any other critical system.
Technical Deep Dive
▶ Watch: Our grey-box auditing approach using software tests (3:00)
The core contribution of this research is the grey-box auditing methodology, termed "record replay," which provides a practical and scalable way to verify Differential Privacy implementations. This approach blends insights from both white-box (code inspection) and black-box (system observation) testing, focusing on the interface between user data and the DP mechanisms.
The methodology begins by conceptualizing any software applying differential privacy as a Directed Acyclic Graph (DAG). In this DAG, data flows through a series of functions and mechanisms. A mechanism is a specific component that injects noise to enforce privacy (e.g., make_Laplace, make_Gaussian). Once information passes through a mechanism, it is considered private and can be used downstream. The goal is to analyze the entire DAG end-to-end to ensure privacy is maintained throughout.
The researchers observed that most DP frameworks share a common structure:
- Mechanisms: These are the actual noise-adding components, typically residing in specific files within the codebase.
- Original Inputs/Interface: This is where the user provides data, often as a pointer to a database or a Pandas DataFrame.
The challenge lies in ensuring that the data reaches these mechanisms with the correct parameterization and that all necessary metadata (especially related to sensitivity) is propagated accurately. As the speaker notes, this "should sound very trivial," but in practice, it is often not.
The "record replay" approach tackles this by instrumenting the system at key points:
- Synthetic Data Generation: To initiate testing, the system generates synthetic datasets. These datasets are carefully constructed to allow for controlled variations, crucial for testing privacy properties.
- Mechanism Wrapping/Patching: The core innovation involves "wrapping" or "patching" all DP mechanisms within the target framework. This instrumentation allows the auditing tool to intercept calls to these mechanisms.
- Initial Flow Execution (Recording Phase):
- A unit test or a specific data flow is executed using an initial synthetic dataset.
- Just before a patched mechanism is called, the tool records several critical pieces of information: the inputs to the mechanism, the computed sensitivity at that point, and the outputs produced by the mechanism. This creates a "snapshot" of the mechanism's state and operation for a given input.
- Second Flow Execution (Replay & Comparison Phase):
- The test is then run again with a slightly altered synthetic dataset. The alteration is typically minimal, often involving changing a single record (a "neighboring dataset" in DP terminology), which is fundamental for testing DP's core property.
- As this second flow progresses, and just before it reaches a patched mechanism:
- The tool compares the inputs that are about to be fed into the mechanism in the second flow with the recorded inputs from the first flow. This comparison specifically checks the Query Data (QD) or other relevant input parameters to ensure they align as expected given the minor dataset change.
- Crucially, the tool replays the outputs that were recorded from the first flow into the second flow at the point where the mechanism would normally execute. This step is vital because it makes the entire flow deterministic, removing the randomness inherent in DP mechanisms from the propagation analysis. By replaying the exact same noise output, the auditor can isolate and verify the propagation of sensitivity and other metadata without being confounded by different random noise instantiations.
- Sensitivity Propagation Verification: With the mechanism outputs deterministically replayed, the auditor can now "ensure the propagation of our sensitivities up and correct the whole way through the flow." This means verifying that any changes in input data (e.g., a single record change) correctly translate into the expected sensitivity adjustments throughout the computation graph, and that the mechanisms are parameterized correctly for these sensitivities.
This methodology offers several distinct advantages:
- Isolation of Mechanisms: By wrapping individual mechanisms, the approach allows for their isolated auditing. This is critical because "now you're only looking at a single mechanism, so the dimensionality isn't a problem the same way as it is when you look at an entire system." This solves a major scalability issue faced by black-box auditing.
- Identification of Data-Dependent Control Flows: The "record replay" method can effectively identify issues such as "data-dependent control flows" (e.g., an
ifstatement that clips data differently if a value exceeds a threshold). By running with slightly different data, the tool can detect if such branches lead to incorrect sensitivity calculations or parameter choices. - Verification of Parameters and States: It allows for direct checking of mechanism parameters and internal states, ensuring they are correctly configured and aligned with the privacy guarantees.
- Lightweight and Practical: The speaker emphasizes that this approach is "very very lightweight," making it practical for integration into continuous integration (CI) pipelines, allowing developers to catch bugs early in the development lifecycle.
In essence, "record replay" provides a powerful, practical, and systematic way to bridge the gap between the theoretical assurances of differential privacy and the complex realities of its software implementation, making it an indispensable tool for securing DP systems.
Demo / Proof of Concept
▶ Watch: Catastrophic privacy failures in major open-source frameworks (6:40)
While the live demonstration was skipped due to time constraints, the speaker indicated that the tools developed for this grey-box auditing methodology are open-source and available on GitHub, inviting interested parties to "try it if you want." The core concept demonstrated would involve setting up a test case for a Differential Privacy (DP) framework, instrumenting its mechanisms using the "record replay" technique, and observing how the tool identifies discrepancies in sensitivity or parameterization across different data inputs. This would likely showcase how the methodology could pinpoint issues like the PGM parameter bug, where a single numerical change effectively undermined the privacy guarantee by misaligning the mechanism with the data's true sensitivity. The purpose of such a demonstration would be to concretely illustrate the ease and effectiveness with which these subtle yet critical privacy failures can be detected, contrasting it with the difficulty of finding such bugs through traditional testing or purely black-box methods.
Defensive Implications
▶ Watch: Crucial takeaway: always test your differential privacy code (8:00)
The findings of this research carry profound implications for anyone developing, deploying, or relying on Differential Privacy (DP) systems. The pervasive nature of critical privacy bugs in even leading frameworks demands a significant shift in defensive strategies.
- Prioritize Rigorous Testing: The most crucial takeaway is that testing your DP code is non-negotiable. This extends beyond traditional functional testing to include specific privacy-focused tests. Developers must embrace a culture of continuous verification for DP implementations, akin to security testing for cryptographic primitives.
- Integrate Grey-Box Auditing into CI/CD: Adopt methodologies like the "record replay" approach as a standard part of the Continuous Integration/Continuous Deployment (CI/CD) pipeline. Just as unit tests and integration tests are run automatically on every code commit, DP-specific audits should be automated to catch privacy regressions early. This ensures that any changes to code, even seemingly innocuous ones, do not inadvertently introduce privacy vulnerabilities.
- Focus on Sensitivity Propagation: Pay meticulous attention to how sensitivity is calculated and propagated throughout the entire data processing pipeline. As the talk highlights, errors in sensitivity calculation or propagation are a primary source of privacy breaches. Defensive measures should include:
- Explicit Sensitivity Tracking: Design systems to explicitly track and verify sensitivity at each stage where private data is transformed.
- Automated Sensitivity Checks: Implement tools that can automatically compare expected vs. actual sensitivities, especially when data passes through complex transformations or conditional logic.
- Scrutinize Data Pre-processing and Control Flows: Data pre-processing steps and any data-dependent control flows (e.g.,
ifstatements, clipping, or aggregation logic based on private values) are high-risk areas. Defenders should:
- Audit Pre-processing: Thoroughly audit any pre-processing of private data to ensure it does not leak information or alter sensitivity in an unacknowledged way.
- Test Conditional Logic: Specifically design test cases to exercise different branches of conditional logic that depend on private data, verifying that the correct noise is applied in all scenarios.
- Do Not Assume Correctness of Third-Party Libraries: The finding that many widely used, expertly developed DP frameworks contain catastrophic bugs should serve as a stark warning. Organizations should not blindly trust the privacy guarantees of third-party DP libraries or SDKs. Instead, they should:
- Perform Due Diligence: Conduct their own internal audits or demand proof of robust testing from vendors.
- Contribute to Open-Source Audits: Engage with and contribute to efforts to audit and improve the security of open-source DP projects.
- Consider Formal Methods Where Practical: While formal verification is often impractical for entire production systems, the success of frameworks like OpenDP (which employs a more formal approach) suggests that formal methods can still play a role, particularly for critical core mechanisms or components where the cost of formal verification is justified by the extremely high privacy risk.
- Educate Developers: Ensure that developers working with DP have a deep understanding of its theoretical underpinnings, common implementation pitfalls, and the importance of rigorous testing. This includes understanding concepts like neighboring datasets, sensitivity, and the implications of different privacy parameters (e.g., bounded vs. unbounded DP).
By implementing these defensive strategies, organizations can move closer to ensuring that their differentially private systems deliver on their promise, truly protecting sensitive information in practice, not just in theory.
Key Takeaways
- Significant Gap Between Theory and Practice: Despite strong theoretical guarantees, practical implementations of Differential Privacy (DP) are highly prone to subtle, catastrophic bugs.
- Existing Methods are Insufficient: Traditional formal verification is often impractical for production systems, and black-box auditing struggles with scalability for complex, high-dimensional DP applications.
- Grey-Box Auditing is a Practical Solution: The "record replay" methodology offers a practical, scalable, and lightweight grey-box approach to audit DP implementations by instrumenting mechanisms and verifying sensitivity propagation.
- Widespread Vulnerabilities: A staggering 8 out of 12 (66%) of the most widely used open-source DP frameworks were found to contain critical privacy-compromising bugs.
- Testing is Paramount: Even frameworks developed by DP theory experts are susceptible to implementation errors, underscoring the absolute necessity of rigorous unit, integration, and end-to-end testing for DP systems.
- Focus on Sensitivity and Parameterization: Most bugs stem from incorrect sensitivity calculations, improper parameterization, or data-dependent control flows that are not adequately accounted for.
About the Speaker(s)
The research behind "Privacy in Theory, Bugs in Practice: Grey-Box Auditing of Differential Privacy Libraries" was a collaborative effort by Tudor Cebere (Inria, Université de Montpellier), David Erb (Technical University of Munich, Oblivious), Damien Desfontaines (Hiding Nemo), Aurélien Bellet (Inria, Université de Montpellier), and Jack Fitzsimons (Oblivious). The speaker, likely one of the authors from Oblivious (David Erb or Jack Fitzsimons), shared that the motivation for this work was deeply personal and practical, stemming from the challenges faced by their software company in ensuring the correctness of their own differentially private SQL engine. This firsthand experience in building and validating production-grade DP tools provided the impetus for developing a robust auditing methodology that could bridge the gap between theoretical privacy guarantees and the complexities of real-world implementation.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
This is the exact kind of research the industry needs to stop drowning in academic hubris. The researchers audited 12 major Differential Privacy libraries, found catastrophic math-breaking flaws in eight of them, and dropped a scalable grey-box testing tool to prove it.
Heather Calloway (CISO) — MUST SEE
Differential privacy is mathematically sound but practically broken in two-thirds of major open-source libraries. This research exposes a massive blind spot for any organization relying on DP for regulatory compliance or data monetization. If your data science teams are using these frameworks to anonymize user data, you likely have zero actual privacy coverage.
→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026
All talks from Privacy Enhancing Technologies Symposium (PETS) 2026