Revisiting Assumptions for Membership Inference on Summary Statistics
Pascal Berrang (Paper author · University of Birmingham), Mark Ryan (Paper author · University of Birmingham), Kiera Wooldridge (Paper author · University of Birmingham)
Privacy Enhancing Technologies Symposium (PETS) 2026 · Day 3 · Caribou
Overview
This talk, presented by Kiera Wooldridge alongside co-authors Pascal Berrang and Mark Ryan from the University of Birmingham, critically re-examines the foundational assumptions underpinning membership inference attacks (MIAs) on summary statistics. In an era where health-related studies and other sensitive data sources frequently publish aggregate statistics—such as averages and standard deviations—to safeguard individual privacy, the threat of MIAs looms large. These attacks aim to determine whether a specific individual's data was included in a given public dataset, potentially compromising their privacy despite the use of aggregation.

Key moments
- 0:00 Introduction: Revisiting membership inference assumptions
- 2:00 Overview of L1 and LLR inference tests
- 4:10 Attack performance with synthetic noise and missing data
- 5:15 Disease signals inflate inference performance
- 6:10 Real temporal drift collapses attack performance
- 7:40 Individual temporal drift is the cause of degradation
- 8:00 Main conclusions and future research directions
Revisiting Assumptions for Membership Inference on Summary Statistics
Speakers: Pascal Berrang (Paper author, University of Birmingham); Mark Ryan (Paper author, University of Birmingham); Kiera Wooldridge (Paper author, University of Birmingham)
Conference: PETS
YouTube: https://www.youtube.com/watch?v=TSLkyH-pyB4
Overview
This talk, presented by Kiera Wooldridge alongside co-authors Pascal Berrang and Mark Ryan from the University of Birmingham, critically re-examines the foundational assumptions underpinning membership inference attacks (MIAs) on summary statistics. In an era where health-related studies and other sensitive data sources frequently publish aggregate statistics—such as averages and standard deviations—to safeguard individual privacy, the threat of MIAs looms large. These attacks aim to determine whether a specific individual's data was included in a given public dataset, potentially compromising their privacy despite the use of aggregation.
The core contribution of this research is to challenge the prevalent, yet often unrealistic, assumption that an adversary possesses an exact copy of a target individual's data at the time of the study. Instead, the authors explore more practical adversary models where data is either noisy, incomplete, or affected by temporal drift. By stress-testing state-of-the-art MIAs against these realistic data imperfections, the talk unveils significant limitations in current attack efficacy, particularly concerning time-sensitive biological data like microRNA expressions. This work is crucial for both privacy researchers and data publishers, offering a nuanced perspective on the true privacy risks associated with summary statistics and identifying potential avenues for defense.
Background
▶ Watch: Introduction: Revisiting membership inference assumptions (0:00)
The publication of summary statistics (e.g., averages, standard deviations) is a cornerstone practice in many fields, particularly in health and genomics, where it allows researchers to disseminate findings while ostensibly protecting the privacy of individual participants. For instance, a study might report the average microRNA expression levels across a cohort of diseased individuals, rather than revealing each patient's specific measurements. This aggregation is intended to prevent the disclosure of sensitive personally identifiable information (PII). However, the private nature of this underlying individual data makes it a prime target for adversaries seeking to exploit published statistics, especially if other sensitive information might be revealed elsewhere.
Prior research in membership inference attacks (MIAs) has demonstrated that even aggregate data can be vulnerable. These attacks typically model an adversary who possesses an exact copy of a target individual's data and attempts to infer whether that individual contributed to a specific public study. The underlying hypothesis is that an individual's data will align more closely with the study's summary statistics if they were a participant, compared to the general population. While such attacks have shown theoretical success, the assumption of an adversary having an exact copy of a target's data at the precise time of the study is often infeasible in real-world scenarios. Information can change, data collection methods vary, and adversaries rarely have perfect, synchronized datasets.
This talk addresses this critical gap by exploring more realistic adversary models. The authors consider scenarios where an adversary's data is "less precise" and "more realistic," such as data collected at a different time, which can be modeled as the original sample plus some noise. This change can be particularly significant for time-sensitive data, such as microRNA expressions. MicroRNAs are small non-coding RNA molecules that play a crucial role in regulating gene expression. Their levels are actively studied in medical publications to understand disease impacts, and unusual levels often indicate illness. However, microRNA expression levels are known to fluctuate over time due to everyday behaviors like diet, making them an ideal candidate for investigating the impact of temporal drift on MIAs.
The study employs two current state-of-the-art tests for membership inference on summary statistics: the L1 distance test and the Log-Likelihood Ratio (LLR) test. Both are framed as hypothesis tests to decide whether a target's data aligns better with the study or the general population. The L1 distance test primarily utilizes averages, while the LLR test incorporates additional information from standard deviations, aiming for a more robust inference. By stress-testing these established methods under conditions of synthetic noise, missing features, and crucially, real-world temporal drift, the research provides vital context to the true efficacy and limitations of existing MIAs.
Key Findings
▶ Watch: Attack performance with synthetic noise and missing data (4:10)
The research yielded several critical findings that challenge conventional assumptions about membership inference attacks on summary statistics:
Firstly, when subjected to synthetically generated noise (e.g., Gaussian noise) and scenarios with missing data, the membership inference attacks demonstrated surprising resilience. Even with increasing levels of Gaussian noise and significant portions of microRNA information missing (e.g., AUC dropping by only 5% when reducing features from 466 down to 100 for the L1 test), both the L1 distance and LLR tests continued to succeed. Interestingly, combining these two noisy features (noise and missing data) resulted in sub-additive effects, meaning the combined impact was less severe than the sum of their individual degradations. The study also found that smaller-sized pools (study cohorts) were more resistant to noise, likely due to the amplified individual contributions to the aggregate statistics. Conversely, higher feature dimensionality extended the tolerance of the attacks to noise.
A significant discovery was the role of disease signals in inflating apparent attack performance. When individuals were clustered into pools based on sharing a disease trait, both statistical tests performed remarkably well. However, when the same tests were performed with randomly sampled pools of the same size, performance dropped noticeably; the Area Under the Curve (AUC) decreased by approximately 10%, and the True Positive Rate (TPR) at 1% False Positive Rate (FPR) more than halved compared to inferences using only diseased individuals. This disparity highlighted that something "disease-related" was artificially enhancing inference performance. The authors theoretically bounded the minimum error of any membership inference test on summary statistics and quantified a signal term composed of a small subset of features exhibiting condition-related shifts. This "disease signal" was found to be a major contributor to attack success, often leading to the misclassification of non-participating individuals as members due to their shared disease characteristics, rather than actual participation in the study.
Crucially, the most impactful finding emerged from experiments using realistically noisy longitudinal datasets. When data collected at an alternate time was used for inference, attack performance plummeted to near random guessing. This stark contrast revealed that synthetic Gaussian noise significantly underestimates the impact of real temporal drift. The degradation was not attributable to marginal feature-level variance (as synthetic Gaussian noise applies variance per feature) nor could it be replicated by transplanting drift factors from "nearest neighbors" (determined via Euclidean, cosine, or correlation distances). Attempts to approximate individual drift from baseline using linear regression from the remaining population also failed to restore attack efficacy. These results strongly indicated that the degradation is uniquely tied to each person's own temporal evolution. The asymmetry between nearest neighbor drift and real individual drift implies that the cause is not merely random variance but a specific, individual-level change.
In summary, the L1 distance test showed greater resilience to synthetic Gaussian noise due to its focus on averages and Gaussian noise's zero mean, which averages out over features. The LLR test, leveraging standard deviations, demonstrated superior robustness against higher levels of missing data. However, both tests were susceptible to conflating disease signals with genuine inference. Most importantly, real temporal drift was identified as an irreducible factor that dramatically collapses attack performance, a phenomenon that cannot yet be reproduced synthetically and points towards a fundamental limitation of current MIAs under realistic conditions.
Technical Deep Dive
▶ Watch: Disease signals inflate inference performance (5:15)
The research meticulously investigates membership inference attacks (MIAs) on summary statistics by adopting a more pragmatic adversary model and rigorously testing it across various data imperfection scenarios. The technical approach centers on evaluating two established MIA methodologies against both synthetic and real-world noisy data, particularly focusing on time-sensitive biological measurements.
The adversary model considered in this study is one where the attacker does not possess an exact copy of a target individual's data from the time of the study. Instead, the adversary has access to "less precise and more realistic" data. This less precise data is modeled in two primary ways:
- Original sample plus some noise: This simulates data collected at a different time, where the individual's characteristics may have naturally shifted.
- Incomplete data: Scenarios where several features of an individual's dataset are unknown to the adversary.
Additionally, the adversary is assumed to have access to population-wide summary statistics, which is generally considered simple to gather (e.g., public health databases or general population surveys).
The study utilizes two current state-of-the-art tests for membership inference:
- The L1 distance test: This test primarily relies on the absolute difference (L1 norm) between the target individual's data (or its average) and the average of the study cohort, comparing it against the average of the general population. Its mathematical definition is focused on averages, making it less sensitive to variance.
- The Log-Likelihood Ratio (LLR) test: This test is more sophisticated, considering both the averages and the standard deviations of the data. By incorporating variance, it aims to provide a more robust measure of alignment, especially when features might have different distributions. Both tests are framed as hypothesis tests designed to determine whether a target individual's data aligns more strongly with the characteristics of the study cohort or with the broader population.
Experiments were conducted on three distinct datasets:
- Two datasets contained microRNA expression levels from diseased individuals. These are particularly relevant due to the known temporal fluctuations of microRNAs. Two of these datasets provided real longitudinal results, allowing for the study of natural temporal drift.
- The third dataset extended the results outside of genomic data, providing external validation for the findings.
The experimental setup involved preprocessing the data and then dividing individuals into a simulated study cohort (the "pool") and a simulated general public (the "population"). Averages and standard deviations were calculated feature-wise for all groups. For cross-sectional data, two types of pools were evaluated:
- Individuals sharing a specific disease trait.
- Individuals sampled uniformly at the same size (random sampling).
For longitudinal datasets, participants shared the same disease or had no unique identifying traits, enabling the assessment of attack performance against a population of similar, yet non-participating, individuals.
The noise models were progressively complex:
- Synthetic Gaussian noise: This was applied to features to simulate random perturbations. The L1 distance test showed higher scores for longer under this noise, primarily because Gaussian noise has a zero mean that averages out over features, which the L1 test, focusing on averages, is less affected by. The LLR test, by contrast, leveraged standard deviations to better withstand missing data, complementing the L1 test's weakness.
- Real-world temporal drift: This was assessed using the longitudinal datasets, where data values naturally drifted over time. This scenario proved far more challenging for MIAs, with performance dropping to near random guessing.
To understand the cause of this dramatic performance degradation under real temporal drift, several hypotheses were tested:
- Group-level noise replication: The authors investigated whether the observed degradation could be explained by applying synthetic Gaussian noise with bearings per feature, similar to group-level variance. This was found insufficient, indicating that marginal feature-level variance alone could not explain the phenomenon.
- Nearest-neighbor drift transplantation: To explore if similar individuals' drift could mimic the effect, the drift factor was "transplanted" from nearest neighbors, determined using Euclidean, cosine, and correlation distances. This approach also failed, breaking the data structure and degrading the attack, suggesting that individual-specific drift cannot be simply borrowed or approximated from others.
- Population structure responsibility: A "leave one out conditional mean test" was performed to assess if the population structure was responsible for the collapse. This involved approximating an individual's drift from a baseline using linear regression from the remaining population. However, this attack performed similarly poorly, further isolating the cause.
The culmination of these technical investigations revealed that only each person's own temporal evolution could decrease the attack to near random levels. The observed asymmetry between the nearest neighbor approach and real individual drift strongly implies that the cause is not merely random variance but something intrinsically specific to each individual's data over time. This individual-specific temporal drift represents a fundamental challenge to current membership inference techniques.
The paper also included a theoretical result, bounding from below the minimum error of any membership inference test on summary statistics and quantifying a signal term. This signal term, made up of a small subset of features with condition-related shifts (e.g., disease signals), was empirically validated to contribute significantly to apparent attack performance, even leading to misclassification in randomly sampled pools. This theoretical underpinning reinforces the empirical observations regarding the conflation of disease signals with actual membership inference.
Demo / Proof of Concept
▶ Watch: Individual temporal drift is the cause of degradation (7:40)
The talk primarily focuses on presenting the methodology, experimental setup, and the empirical results of a rigorous scientific investigation rather than showcasing a live software demonstration or a typical "proof of concept" exploit. The "proof of concept" in this context is the comprehensive suite of experiments performed on real and synthetic datasets, which empirically validates the authors' theoretical insights and demonstrates the practical limitations of current membership inference attacks under realistic conditions. The performance degradation observed in MIAs when confronted with real temporal drift, as opposed to synthetic noise, serves as a compelling demonstration of the core hypothesis.
Defensive Implications
▶ Watch: Main conclusions and future research directions (8:00)
The findings presented in this research have profound implications for data privacy, particularly for organizations publishing summary statistics from sensitive datasets like health records. The most significant defensive implication revolves around the concept of temporal drift.
- Temporal Drift as a Natural Defense Mechanism: The study clearly demonstrates that real temporal drift in individual data dramatically reduces the efficacy of membership inference attacks, often to near random guessing levels. This suggests that for time-sensitive data, the mere passage of time between data collection and an adversary's acquisition of target data could inherently provide a significant layer of privacy protection. Defenders should recognize that the "exact data" assumption, often foundational to many MIAs, is unrealistic for dynamic datasets, offering a more optimistic view of privacy in such contexts.
- Revisiting Adversary Capabilities: Organizations should re-evaluate the assumed capabilities of a potential adversary. If an attacker cannot obtain an exact, time-synchronized copy of an individual's data, the practical risk of membership inference might be considerably lower than suggested by studies relying on idealized adversary models. This does not negate the threat entirely but provides a more nuanced understanding of where resources for defense might be most effectively allocated.
- Caution Against Overestimating MIA Success: The research implies that many existing MIAs, especially those evaluated solely on static data or synthetic noise models, might significantly overestimate their actual success rates in real-world applications. Data practitioners and privacy researchers should be cautious about extrapolating the efficacy of these attacks to scenarios involving time-sensitive, dynamic data.
- Awareness of "Disease Signals": The finding that "disease signals" can artificially inflate MIA performance is crucial. When publishing summary statistics related to specific conditions or traits, researchers must be aware that shared characteristics within a disease cohort can lead to a higher apparent attack success rate, even against non-participating individuals who share the condition. This highlights a potential for false positives in MIA evaluations and suggests that additional privacy-preserving techniques might be warranted for highly specific disease cohorts, irrespective of temporal factors.
- Strategic Data Collection and Release: For longitudinal studies, the timing of data collection and the subsequent release of summary statistics could be a strategic defensive consideration. If there's a significant time gap between an individual's data contribution and when an adversary might acquire their "current" data, this gap could serve as a privacy buffer. Further research into quantifying this "safe" temporal window could provide actionable guidance.
- Future Research for Enhanced Defense: While the study identifies individual temporal drift as a key factor in attack collapse, the underlying reasons are still being explored. Understanding why this individual-specific drift is so effective could lead to the development of novel, robust defense mechanisms. This includes exploring methods to intentionally introduce "realistic" temporal variations or identifying features most susceptible to such protective drift. The research serves as a call to action for the privacy community to investigate this phenomenon further, transforming a vulnerability into a potential basis for defense.
Key Takeaways
- Traditional membership inference attacks (MIAs) on summary statistics often rely on the unrealistic assumption that an adversary possesses exact, time-synchronized copies of target data, which is rarely true for dynamic, real-world datasets.
- While MIAs demonstrate resilience to synthetic Gaussian noise and significant missing features, their performance dramatically drops to near random guessing when confronted with real temporal drift in longitudinal data. This indicates that synthetic noise models significantly underestimate real-world privacy protection.
- The observed degradation in attack performance is highly specific to each individual's own temporal evolution; it cannot be replicated by group-level noise or by transplanting drift from nearest neighbors, highlighting a fundamental, individual-specific barrier to current MIAs.
- A "disease signal," characterized by condition-related shifts in specific features, can artificially inflate MIA success rates, leading to the misclassification of non-participating individuals who share the disease trait. This challenges the validity of MIA evaluations that do not account for such confounding factors.
- The L1 distance test proves more robust to Gaussian noise by focusing on averages, whereas the Log-Likelihood Ratio (LLR) test leverages standard deviations to better withstand higher levels of missing data. An informed adversary might choose between tests based on the specific characteristics of the noise or data incompleteness.
- Temporal drift presents a promising, unexploited basis for defense against membership inference attacks, underscoring the critical need for further research to understand its mechanisms and explore its potential for enhancing data privacy.
About the Speaker(s)
The talk was presented by Kiera Wooldridge, who is one of the co-authors of the paper, alongside her colleagues Pascal Berrang and Mark Ryan. All three speakers are affiliated with the University of Birmingham. Their collective research focuses on critical areas within cybersecurity and privacy, specifically delving into the vulnerabilities of aggregate data and the efficacy of membership inference attacks. This particular work highlights their expertise in analyzing complex data privacy challenges in health-related studies, particularly those involving sensitive and time-variant biological data like microRNA expressions. Their contributions aim to advance the understanding of privacy risks and inform more robust defense strategies in the context of published summary statistics.
Reviews
Dr. Zero (Offensive Security Researcher) — MUST SEE
Wooldridge and team finally call bullshit on the foundational threat model of most Membership Inference Attacks. By proving that real-world temporal drift drops MIA efficacy to a coin flip, they expose how prior papers inflated their success rates using unrealistic 'exact copy' adversary models. This is rigorous, necessary work that forces the privacy community back to reality.
Heather Calloway (CISO) — SOLID
This is academic privacy research challenging the theoretical panic around membership inference attacks on aggregate healthcare data. The authors prove that real-world temporal drift makes these attacks nearly impossible outside of a lab. It’s not for your security operations center, but your privacy counsel and data officers should know about it before they halt data-sharing initiatives or panic-buy differential privacy tooling.
→ Top-rated talks at Privacy Enhancing Technologies Symposium (PETS) 2026
All talks from Privacy Enhancing Technologies Symposium (PETS) 2026