Efficient Privacy Auditing in Federated Learning

Efficient Privacy Auditing in Federated Learning

Speakers: Hongyan Chang; Brandon Edwards; Anindya S. Paul; Reza Shokri

Conference: USENIX Security '24

YouTube: https://www.youtube.com/watch?v=za2wkl0KL0A

Overview

Federated Learning (FL) has emerged as a prominent distributed machine learning paradigm, enabling multiple parties to collaboratively train a global model without directly sharing their raw local data. While FL offers a significant privacy advantage over centralized training, it is not immune to privacy risks. Sensitive information from individual training datasets can still be inadvertently encoded within the local model updates shared with a central server, and subsequently, with other participating parties through the global model. This talk, presented by Hongyan Chang at USENIX Security '24, addresses the critical challenge of efficiently auditing these privacy risks in FL environments.

The de facto standard for quantifying information leakage in machine learning is the membership inference attack (MIA), where an adversary attempts to determine if a specific data point was part of the training dataset. While existing MIA methods can assess privacy risk, they suffer from significant computational overhead when applied to the dynamic, multi-round nature of federated learning. This often renders them impractical for continuous, real-time auditing, especially for parties with limited computational resources.

This work introduces an innovative and highly efficient privacy auditing algorithm designed specifically for FL. By leveraging a novel "slope signal" that captures the dynamics of model performance changes over time, the proposed method allows participating parties to continuously monitor their privacy risk without the need for training additional, computationally expensive inference models. This breakthrough enables more practical and proactive privacy management in real-world federated learning deployments, offering a substantial improvement in both efficiency and effectiveness over prior art.

Background

▶ Watch: Introduction: Efficient privacy auditing in Federated Learning (0:00)

Federated Learning operates on the principle of decentralized model training. Instead of aggregating raw data, a central server orchestrates the training process by sending a global model to participating clients. Each client then trains this model on its local dataset, computes model updates (e.g., gradients or updated model parameters), and sends these updates back to the server. The server aggregates these updates to refine the global model, which is then distributed for the next round of training. This iterative process allows for collaborative model building while keeping sensitive raw data localized.

Despite its privacy-preserving design, FL is susceptible to various attacks, with membership inference attacks (MIAs) being a primary concern. In an MIA, an adversary, given access to a model (or its updates), aims to infer whether a specific data point was used in its training. The success rate of such an attack quantifies the degree of information leakage. In the FL setting, the adversary can observe multiple model snapshots shared across training rounds, providing a richer attack surface compared to a single-model centralized setting. This dynamic observation allows for the inference of information encoded throughout the entire training trajectory.

Prior approaches to MIAs in FL have attempted to exploit this training dynamic. Typically, these methods involve extracting various signals from each model snapshot, such as per-sample gradients, and then training complex, deep inference models on the concatenation of these signals. However, this methodology introduces severe computational burdens. For instance, computing the per-sample gradient for a ResNet model on CIFAR-10 can take approximately 380 times longer than the local training process itself. If parties were to extract per-sample gradients in each round, a FL process that originally took 3 GPU hours could balloon to 46 GPU days. Training additional deep inference models on these concatenated signals further exacerbates the computational cost, rendering such approaches infeasible for resource-constrained participants.

Conversely, strong MIA algorithms developed for centralized settings often require training a set of shadow models to simulate model behavior when a target data point is either included or excluded from the training set. While effective for a single model, these methods are computationally expensive and fundamentally overlook the crucial information embedded in the entire training dynamic of FL. Applying them directly to FL can lead to an underestimation of privacy risk because they do not fully capture the cumulative leakage across multiple rounds of model updates. The core problem, therefore, is the lack of an efficient and effective mechanism for parties to continuously audit their privacy risk in FL without incurring prohibitive computational costs or underestimating the true risk.

Key Findings

▶ Watch: The problem: Auditing privacy without additional models (4:00)

The central innovation presented in this work is the slope signal, a highly efficient and effective mechanism for continuously auditing privacy risk in Federated Learning. The key findings revolve around its ability to:

1. Exploit Performance Dynamics: The research observes that while the performance (e.g., prediction confidence) of a model for member and non-member data points might significantly overlap at any single training round, their performance dynamics over time differ substantially. Specifically, members tend to exhibit a more rapid increase in prediction confidence compared to non-members as training progresses.
2. Introduce an Efficient Signal: The slope signal is designed to capture this differential rate of change. For each data point, a linear function is fitted to its performance dynamic over several training rounds, and the slope of this fitted function serves as the signal of interest. This approach avoids the computational overhead of training additional deep inference models or computing expensive per-sample gradients.
3. Achieve Superior Efficiency and Effectiveness: Empirical results demonstrate that the slope signal-based auditing algorithm significantly outperforms existing MIA methods in both efficiency and effectiveness. It achieves a high true positive rate (TPR) at a low false positive rate (FPR) with dramatically reduced computational cost, making real-time, continuous privacy auditing feasible. For example, it achieves a 12.9% TPR at 1% FPR in just 1.2 seconds per round, contrasting sharply with other methods taking 860 seconds for significantly lower TPRs.
4. Enable Comprehensive Risk Analysis: The efficiency of the proposed algorithm allows for the first time a comprehensive analysis of how various factors, such as training progression over time and data heterogeneity (IID vs. Non-IID settings), influence privacy risk in FL. This analysis reveals insights previously unobservable due to the limitations of prior auditing tools, such as the continuous increase in privacy risk even after model performance stabilizes, and the varying levels of risk experienced by different parties in non-IID settings.

In essence, the slope signal provides a lightweight yet powerful way to distinguish between member and non-member data points by focusing on how the model "learns" about each point over time, offering a practical solution to a long-standing challenge in FL privacy.

Technical Deep Dive

▶ Watch: Slope signal's efficiency and real-time computation (5:30)

The core of the proposed auditing algorithm lies in the slope signal, a novel metric designed to efficiently capture the differential learning dynamics of member versus non-member data points within a Federated Learning setup. The intuition stems from the observation that while a model's immediate performance (e.g., prediction confidence or loss) on a data point might not clearly indicate its membership status at any given round, the rate of change of this performance over multiple rounds can be highly discriminative.

Specifically, the authors found that the prediction confidence score of a model increases more rapidly for data points that are part of the training set (members) compared to those that are not (non-members). This suggests that each model update impacts members and non-members differently, leading to distinct trajectories in their performance metrics over time.

To quantify this distinction efficiently, the slope signal is computed as follows:

1. Performance Dynamic Tracking: For each data point (which could be a potential member or non-member being audited), its performance is tracked across multiple training rounds. The model's performance can be evaluated using readily available metrics such as the confidence score (the probability assigned to the correct class), the loss value, or the logit score (the raw output of the model's final layer before the softmax function). These metrics are typically computed during the standard FL process for monitoring training progress, making their extraction computationally inexpensive.
2. Linear Function Fitting: A linear function is then fitted to the sequence of performance values obtained for a given data point over a window of recent training rounds. If P_t represents the performance of a data point at round t, and we consider a window of W rounds, the algorithm fits a line P_t = a * t + b to the data points (t, P_t) for t within the window.
3. Slope as Signal: The slope (a) of this fitted linear function serves as the slope signal. A higher positive slope indicates that the model's performance on that data point is improving more rapidly, which is characteristic of a member. Conversely, a lower slope (or even a negative one) would suggest slower improvement or degradation, more typical of a non-member.

The primary advantage of this approach is its computational efficiency. The performance metrics (confidence, loss, logits) are already calculated during the routine FL training for monitoring purposes. The slope signal itself is essentially a weighted sum of model's performance across each round, making its computation extremely fast. This means that parties can calculate the slope signal almost instantly, integrating it seamlessly into their FL pipeline.

Once the slope signal is computed for a data point, the auditing process proceeds as follows:

1. Thresholding: The auditor (the party running the audit) compares the computed slope signal with a predetermined threshold. If the slope is higher than this threshold, the data point is predicted as a member; otherwise, it is predicted as a non-member.
2. ROC Curve and Risk Quantification: By iterating over all possible threshold values, the party can determine the true positive rate (TPR) and false positive rate (FPR) for membership inference. Plotting TPR against FPR yields the Receiver Operating Characteristic (ROC) curve. Following prior works in MIA, the privacy risk is quantified by the TPR at a low FPR, specifically TPR at 1% FPR. This metric reflects the attack's accuracy when allowing for a small number of false positives, which is a common and practical way to assess the severity of information leakage.

The auditing algorithm is designed to be integrated as a plugin module within any FL pipeline. During each round of training, after a party updates the global model using its local dataset but before sharing it with the server, the auditing algorithm can be executed locally. This allows for real-time monitoring of privacy risk. The output can be visualized as a "privacy risk report," providing immediate insights into the current level of leakage. Such immediate feedback empowers parties to take proactive measures, such as applying differentially private (DP) mechanisms or adjusting training parameters, to mitigate identified risks.

This technical design addresses the critical limitations of prior MIAs by offering a method that is both computationally lightweight and specifically tailored to exploit the temporal dynamics inherent in federated learning.

Demo / Proof of Concept

▶ Watch: Seamless integration into the Federated Learning pipeline (6:20)

The efficacy and efficiency of the proposed auditing algorithm were rigorously demonstrated through empirical results, primarily focusing on a ResNet model trained on the CIFAR-10 dataset using the FedAvg (Federated Averaging) algorithm across four parties. The evaluation compared the new slope signal-based algorithm against several baseline attacks in terms of both efficiency (quantified by GPU time per round) and effectiveness (measured by true positive rate (TPR) at 1% false positive rate (FPR)).

The results unequivocally highlighted the superior performance of the slope signal approach:

• Efficiency: The new algorithm achieved its auditing results in a mere 1.2 seconds per round. This is remarkably efficient, especially when compared to the local training time of 2.3 seconds per round.
• Effectiveness: It demonstrated a high effectiveness, achieving a 12.9% TPR at 1% FPR.

In stark contrast, other established membership inference attack algorithms proved either ineffective or prohibitively inefficient:

• Per-Sample Gradient Attacks: Attacks requiring the computation of per-sample gradients—a common approach in previous FL MIA research—took an astounding 860 seconds per round. Despite this massive computational cost, their effectiveness was significantly lower, yielding only a 2.8% TPR. This vividly illustrates the practical impossibility of using such methods for continuous auditing in real-world FL scenarios.
• Other Efficient Baselines: Other "efficient" membership inference attacks failed to capture the privacy risk effectively, indicating that their efficiency came at the cost of accuracy.
• Centralized Attacks (e.g., LIRA): Even strong membership inference attacks designed for centralized settings, such as the LIRA attack, which involved training 16 reference models, did not achieve the same level of performance as the slope signal algorithm. This gap underscores the critical importance of leveraging the entire training dynamic in FL, which centralized methods are not designed to exploit.

Beyond direct performance comparison, the efficiency of the slope signal algorithm enabled novel analyses of privacy risk in FL that were previously too expensive to conduct:

• Privacy Risk Evolution Over Time: The study showed how privacy risk evolves alongside model performance. Intriguingly, it was observed that after approximately round 50, even though the model's overall performance stabilized, the privacy risk continued to increase. This crucial insight, missed by less effective baseline attacks, indicates that privacy leakage can persist and grow even when the model appears to converge in terms of utility, highlighting the need for continuous monitoring.
• Impact of Data Heterogeneity: The algorithm was used to investigate how data heterogeneity (IID vs. Non-IID settings) affects privacy risk. Across 20 parties, in a non-IID setting, the standard deviation of privacy risk was significantly larger than in an IID setting. This reveals that parties in non-IID FL deployments experience widely different levels of privacy risk—a phenomenon previously overlooked due to the lack of efficient auditing tools. Furthermore, the analysis suggested that as data heterogeneity increases, the average privacy risk decreases, a trend that could not be observed using less sophisticated baseline attacks.

These empirical results and analyses serve as a robust proof of concept, demonstrating that the slope signal-based auditing algorithm is not only a theoretically sound approach but also a practically viable and highly effective solution for real-time privacy risk assessment in Federated Learning.

Defensive Implications

▶ Watch: Importance of leveraging full training dynamics in FL (9:00)

The introduction of an efficient and effective privacy auditing algorithm has profound implications for defenders operating in Federated Learning environments. Prior to this work, the high computational cost and limited effectiveness of existing membership inference attack methods meant that continuous, real-time monitoring of privacy risk was largely impractical. This new algorithm fundamentally changes that landscape, enabling proactive and informed privacy management.

Here are the key defensive implications:

1. Continuous, Real-Time Risk Monitoring: Parties participating in FL can now seamlessly integrate this auditing algorithm as a plugin into their training pipelines. By running the audit locally after each model update but before sharing with the server, they gain real-time visibility into their current privacy risk. This continuous monitoring allows for the detection of increasing or fluctuating leakage patterns, such as the observed rise in privacy risk even after model performance stabilizes.

1. Informed Decision-Making for Privacy Enhancements: With immediate and accurate privacy risk reports, parties are no longer operating in the dark. If the auditing algorithm indicates an unacceptable level of privacy leakage, they can take immediate action. This might include:

• Applying Differential Privacy (DP) Mechanisms: Based on the quantified risk, parties can decide to apply DP noise to their local model updates. The level of noise can be precisely tuned to achieve a desired privacy-utility trade-off, rather than applying a blanket, potentially overly conservative (and utility-damaging) amount of noise.
• Adjusting Training Parameters: Insights from the audit might suggest changes to training parameters, such as learning rates, batch sizes, or aggregation strategies, to mitigate leakage.
• Data Augmentation or Sanitization: In some cases, identifying high-risk data points could prompt local data augmentation or specific sanitization techniques to reduce their distinctiveness.

1. Tailored Privacy Strategies for Heterogeneous Data: The finding that privacy risk varies significantly among parties, especially in non-IID settings, is critical. Defenders can leverage the auditing tool to understand the specific risk profile of their local dataset. This allows for the implementation of tailored privacy strategies, where parties with higher identified risks apply stronger privacy-preserving mechanisms, while those with lower risks might use lighter measures, optimizing for both privacy and model utility across the federation.

1. Benchmarking and Compliance: The algorithm provides a standardized and efficient way to quantify privacy risk, which can be invaluable for benchmarking FL systems against privacy requirements and ensuring compliance with data protection regulations (e.g., GDPR, CCPA). It offers an objective metric for demonstrating due diligence in privacy protection.

In essence, this work transforms privacy auditing from a post-hoc, computationally intensive analysis into an integral, real-time component of the Federated Learning process. It empowers individual parties to actively manage and mitigate their privacy exposure, fostering greater trust and broader adoption of FL in sensitive application domains.

Key Takeaways

• Efficient and Effective Privacy Auditing: The proposed "slope signal" algorithm offers a highly efficient and effective method for continuously auditing privacy risk in Federated Learning, overcoming the limitations of prior computationally intensive approaches.
• Leveraging Training Dynamics: The core innovation lies in exploiting the differential rate of change in model performance for member versus non-member data points over time, rather than relying on static snapshots or expensive per-sample gradients.
• Real-time Risk Quantification: The algorithm enables parties to quantify privacy risk (measured by TPR at 1% FPR) in near real-time, requiring only 1.2 seconds per round on a ResNet/CIFAR-10 setup, a dramatic improvement over methods taking 860 seconds.
• Deeper Insights into FL Privacy: Its efficiency facilitates comprehensive analyses, revealing that privacy risk can continue to increase even after model performance stabilizes and that data heterogeneity significantly impacts individual parties' risk profiles.
• Proactive Defensive Capabilities: The real-time privacy reports empower FL participants to make informed, immediate decisions, such as applying differentially private mechanisms or adjusting training parameters, to mitigate identified information leakage.
• Essential Plugin for FL Pipelines: The algorithm is designed as a seamless plugin, integrating easily into existing FL pipelines to provide continuous privacy monitoring and enhance overall system security.

About the Speaker(s)

The talk "Efficient Privacy Auditing in Federated Learning" was presented by Hongyan Chang, a PhD student from N. Hongyan Chang is the lead author of this work, which was conducted in collaboration with Brandon Edwards, Anindya S. Paul, and Reza Shokri. Reza Shokri is identified as Hongyan Chang's supervisor, indicating a research focus on privacy and security in machine learning, particularly within the context of federated learning. Their collective research aims to develop practical and efficient solutions for monitoring and mitigating privacy risks in distributed machine learning systems.