PrivATE: Differentially Private Average Treatment Effect Estimation for Observational Data

PrivATE: Differentially Private Average Treatment Effect Estimation for Observational Data

Speakers: Quan Yuan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=6sT4XyjtI7w

Overview

Causal inference -- determining whether a treatment or policy actually causes an observed effect -- is fundamental to medicine, economics, and education. When randomized controlled trials are infeasible, researchers rely on observational data that often contains sensitive personal information. This talk introduces PrivATE, a framework for estimating Average Treatment Effects (ATE) from observational data under differential privacy (DP) guarantees. PrivATE offers two levels of privacy protection (label-level and sample-level), uses an adaptive matching limit mechanism that automatically adjusts based on privacy budget and data characteristics, and outperforms existing methods across multiple datasets and privacy settings.

Background

▶ Watch: Introduction to causal inference and observational studies (0:00)

Causal inference is used to determine the effects of treatments or policies. In randomized controlled trials (RCTs), treatment assignment is random, but ethical, technical, or economic constraints often make RCTs infeasible -- for example, studying the effects of smoking or evaluating economic policies. Observational studies work with existing data without intervention, making them more practical and commonly used.

The key task is estimating the Average Treatment Effect (ATE): the mean difference between outcomes under treatment and outcomes under no treatment across all samples. Two challenges arise: selection bias (samples are not randomly selected) and missing information (for each sample, only one treatment outcome is observed; the counterfactual outcome must be estimated). Sample matching addresses both by pairing treated and control samples with similar characteristics.

However, observational data contains sensitive personal information, and direct manipulation raises privacy concerns under regulations like GDPR. Differential privacy provides quantifiable privacy guarantees by injecting noise into statistical outputs. Existing DP approaches to ATE estimation require specific assumptions (like binary outcomes), offer limited protection scope, and use fixed truncation thresholds for weight limiting that lack flexibility.

Key Findings

▶ Watch: Limitations of existing DP approaches to ATE (4:00)

Two Levels of Privacy Protection: PrivATE supports both label-level privacy (only the observed outcome Y is private) and sample-level privacy (all attributes -- treatment, covariates, and outcome -- are private). Label-level privacy is more efficient because propensity score computation and matching do not access outcomes, consuming no privacy budget for those phases.

Adaptive Matching Limit: Rather than using a fixed, predefined truncation threshold for limiting how many times each sample can be used for matching, PrivATE derives an optimal matching limit K based on the privacy budget, data characteristics, and the expected squared error. The optimal K balances the trade-off between reducing global sensitivity (which requires lower K) and improving matching accuracy (which benefits from higher K). The limit is converted to an integer and adapts automatically to different datasets and privacy budgets.

Superior Performance: Across multiple datasets and privacy budgets, PrivATE outperforms both existing differentially private ATE estimation methods and advanced differentially private data synthesis approaches. Baseline methods perform particularly poorly under high privacy requirements (low epsilon).

Fixed vs. Adaptive Matching: Experiments confirm that a fixed matching limit cannot effectively handle diverse scenarios because data characteristics and matching situations vary across datasets. PrivATE's adaptive calculation achieves consistently good performance across all settings.

Technical Deep Dive

▶ Watch: PrivATE workflow: regression, matching, and estimation phases (6:00)

PrivATE operates in three phases:

Phase 1 - Regression Model Training: A logistic regression model is trained to estimate propensity scores (probability of receiving treatment given covariates) for all samples. At label-level privacy, this phase consumes no privacy budget since it does not access observed outcomes. At sample-level privacy, model training and score estimation are perturbed with DP noise.

Phase 2 - Similar Sample Matching: Distances between each sample's propensity score and scores of all samples in the opposite treatment group are computed and sorted, yielding two sorted matrices (one for the control group, one for the treated group). At label-level, no privacy budget is consumed. At sample-level, the true treatment variable T is perturbed.

Phase 3 - Causal Effect Estimation: Using the sorted matrices, the closest neighbors in the opposite group are found for each sample. Counterfactual outcomes are estimated, and the sum of potential outcomes is aggregated and perturbed to produce the final ATE estimate. The matching limit K is derived by minimizing the combined error from noise prohibition and matching constraints, using the formula based on the number of samples, true maximum matches, and privacy budget epsilon.

The error coefficient in the adaptive mechanism controls the balance between matching bias overestimation (making the limit too low) and underestimation (making the limit too large). Experiments show that extreme coefficient values in either direction degrade performance.

At sample-level privacy, the total privacy budget is divided into three parts across the three phases, with the optimal allocation varying by dataset.

Demo / Proof of Concept

▶ Watch: Experimental results: PrivATE outperforms baselines across datasets (10:00)

Experimental evaluation across multiple datasets demonstrated:

• PrivATE consistently outperforms comparable DP-based ATE estimation methods and DP data synthesis baselines across varying privacy budgets
• Baseline methods degrade significantly under high privacy requirements (low epsilon), while PrivATE maintains reasonable accuracy
• The adaptive matching limit achieves near-optimal performance across all datasets and privacy settings, while fixed-value approaches fail on specific datasets
• Privacy budget allocation across phases has significant impact, with no single optimal allocation for all datasets

Defensive Implications

▶ Watch: Impact of privacy budget allocation across three phases (11:00)

PrivATE addresses a growing need for privacy-preserving causal inference in regulated environments:

Healthcare and Medicine: Medical research using observational patient data can estimate treatment effects while providing differential privacy guarantees, enabling research without exposing individual patient records.

Policy Evaluation: Government agencies evaluating economic or social policies can analyze observational data about citizens while meeting privacy requirements imposed by data protection regulations.

Enterprise Analytics: Organizations conducting A/B testing alternatives using observational data can protect customer privacy while still deriving actionable causal insights.

The adaptive matching mechanism is particularly valuable because it removes the need for manual parameter tuning, which requires domain expertise that may not be available in every organization deploying privacy-preserving analytics.

Key Takeaways

• PrivATE enables differentially private average treatment effect estimation from observational data with two protection levels: label-level (outcome only) and sample-level (all attributes)
• An adaptive matching limit mechanism automatically balances sensitivity reduction against matching accuracy based on privacy budget and data characteristics
• Label-level privacy is more efficient because propensity score computation and matching phases consume no privacy budget
• Fixed matching thresholds used by prior methods cannot handle diverse datasets and privacy settings; adaptive calculation consistently performs better
• PrivATE outperforms both DP-based ATE methods and DP data synthesis approaches across multiple datasets and privacy budgets
• Privacy budget allocation across phases significantly impacts results, with no universal optimal allocation

About the Speaker(s)

Quan Yuan is represented by presenter Bangjang, who presented on behalf of the authors due to travel issues. The research is a collaborative effort spanning Georgia University, UVA, UNCCG, Xiamen University, VU, Peking University, and Hunan University. The work reflects a broad multi-institutional effort to bring differentially private causal inference from theory to practical application.