Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Prediction

Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Prediction

Speakers: Zitao Chen

Conference: NDSS Symposium

YouTube: Not publicly available

Overview

Machine learning models, increasingly pervasive in sensitive domains from healthcare diagnostics to financial services, inherently process vast amounts of private user data. This widespread deployment, however, comes with a critical privacy vulnerability: the unintentional leakage of sensitive training information through Membership Inference Attacks (MIAs). These attacks allow an adversary to determine whether a specific individual's data was used to train a given model, potentially exposing highly personal attributes like health status or financial records. Existing defenses against MIAs often force a difficult trade-off, either providing strong theoretical privacy guarantees at the cost of significant model accuracy degradation, or offering empirical protection with limited effectiveness or a reliance on additional, often unavailable, public datasets.

This talk introduces HAMP (High-entropy Adaptive Membership Privacy), a novel and practical defense mechanism designed to bridge this gap. HAMP achieves both high model accuracy and robust membership privacy without requiring any supplementary data. The core insight driving HAMP's design is the observation that diverse MIAs, despite their varied methodologies, all exploit a common vulnerability: the target model's overconfidence in its predictions for training samples (members) compared to unseen data (non-members). This overconfidence manifests through distinct statistical features in the model's outputs, which adversaries leverage to infer membership.

Motivated by this insight, HAMP's approach is to systematically enforce less confident predictions by the model, thereby making its behavior on training and testing samples statistically indistinguishable. This is achieved through a dual-pronged strategy: a novel training framework that incorporates high-entropy soft labels and an entropy-based regularizer, combined with a unique testing-time defense that uniformly modifies all prediction outputs to become low-confidence. Extensive evaluations across five benchmark datasets and against nine state-of-the-art MIAs demonstrate that HAMP consistently provides a superior privacy-utility trade-off compared to seven leading defensive techniques.

Background

[▶ Watch: Introduction to MIAs and the privacy problem (0:00)]()

The proliferation of machine learning models in critical applications necessitates a robust understanding of their privacy implications. A primary concern is the susceptibility of these models to Membership Inference Attacks (MIAs). An MIA's objective is to ascertain if a particular data record was part of the dataset used to train a target model. For example, inferring that a patient's medical history contributed to a diagnostic model's training could directly reveal their health condition, violating patient confidentiality.

MIAs fundamentally exploit the behavioral discrepancies of ML models when processing data points they were trained on (members) versus those they have never encountered (non-members). These attacks manifest in several forms:

• Score-based MIAs leverage the model's output probabilities or scores. Adversaries often observe that training samples tend to exhibit lower prediction loss, higher prediction confidence, or lower entropy in their output distributions compared to non-members. Examples include auxiliary neural network (NN)-based attacks, which learn to distinguish member/non-member outputs, and advanced methods like the Likelihood Ratio Attack (LiRA), which trains multiple shadow models to approximate output distributions and performs statistical tests for membership.
• Label-only MIAs operate solely on the predicted class label, often exploiting differences in adversarial robustness. Training members are frequently more resilient to input perturbations (e.g., adversarial noise, random noise, or data augmentation) than non-members. Boundary attacks measure the perturbation required to alter a prediction label, while augmentation attacks assess the model's consistency under data transformations.

The threat model typically assumes a black-box adversary who can query the target model with arbitrary inputs and observe its prediction outputs. A strong adversary might possess auxiliary knowledge, such as access to a subset of known members and non-members, enabling them to train shadow models to refine their attack strategies. The defender's goal is to train a model that maintains high classification accuracy while simultaneously protecting against these MIAs, critically, without relying on additional public datasets which are often unavailable in real-world scenarios.

Current defenses against MIAs fall into two broad categories:

• Provable defenses, such as those based on Differential Privacy (DP) (e.g., DP-SGD), inject noise into the training process to provide strong, mathematically rigorous privacy guarantees. However, this often leads to a significant degradation in the model's utility and accuracy, making them impractical for many applications.
• Practical defenses aim for empirical privacy protection while striving to preserve model accuracy. These include:
• Adversarial regularization (AdvReg), which trains the model to minimize MIA gain.
• Knowledge distillation techniques like DMP and SELENA, which transfer learned knowledge from a private model to a public one using a reference dataset.
• Early stopping, a simple technique that aims to reduce overfitting, thereby potentially limiting MIA vulnerability.
• MemGuard, which adds noise to prediction scores to obfuscate differential behaviors.
• Label Smoothing (LS), a regularization technique that replaces hard labels with low-entropy soft labels, primarily to improve generalization and accuracy, though it can have incidental privacy benefits.

Despite these efforts, a comprehensive solution that simultaneously achieves strong membership privacy, high model accuracy, and does not require additional data remains a significant challenge, highlighting the need for innovative approaches like HAMP.

Key Findings

[▶ Watch: HAMP's core principle: enforcing less confident predictions (1:30)]()

The central discovery underpinning HAMP is that the diverse array of Membership Inference Attacks (MIAs), regardless of their specific methodology, ultimately exploit a singular, pervasive vulnerability: the ML model's inherent overconfidence in its predictions for data samples it was trained on. This overconfidence manifests in distinct, measurable ways that adversaries readily leverage. For score-based MIAs, it appears as exceptionally high confidence scores, remarkably low prediction loss, or very low entropy in the output probabilities for training samples. Conversely, for label-only MIAs like boundary attacks and augmentation attacks, this overconfidence translates into a higher degree of robustness for training samples against various perturbations or data transformations.

This pervasive overconfidence is primarily a byproduct of standard training practices, particularly the common use of one-hot encoding for ground-truth labels. When a model is optimized to minimize loss against these hard, one-hot encoded labels, it is implicitly encouraged to output near 100% probability for the correct class and 0% for others. This aggressive optimization leads to a pronounced behavioral divergence, where training samples elicit far more confident predictions than unseen testing samples, rendering them statistically distinguishable to an adversary. For instance, on the Purchase100 dataset, a 25% average difference in prediction confidence between training and testing samples enabled an attack True Positive Rate (TPR) exceeding 14% at 0.1% False Positive Rate (FPR).

HAMP directly addresses this root cause by systematically enforcing less confident predictions, thereby eliminating the distinguishing characteristics between members and non-members. The empirical evaluation of HAMP yielded several critical findings:

• Dramatic MIA Risk Reduction: HAMP significantly curtailed the risk of MIAs. On average, it reduced the attack TPR at 0.1% FPR by an impressive 94.1%, plummeting from an average of 13.48% for undefended models to a mere 0.8%. Similarly, the attack True Negative Rate (TNR) at 0.1% False Negative Rate (FNR) was reduced by 97%, from 19.89% to 0.59%.
• Minimal Accuracy Degradation: Crucially, HAMP achieved this robust privacy protection with an exceptionally low impact on model utility. Across all evaluated datasets, the average model accuracy drop was only 0.46%, with a maximum drop of 1.1% observed on the Location30 dataset. This demonstrates a superior privacy-utility trade-off compared to existing state-of-the-art defenses.
• Superiority Against State-of-the-Art Defenses: HAMP consistently outperformed seven leading defenses across various metrics. For example, compared to MemGuard, HAMP achieved 8.4 times lower average attack TPR and 18.3 times lower TNR, while maintaining similar accuracy. Against AdvReg, HAMP offered both higher accuracy (0.46% vs. 7.45% accuracy drop) and stronger privacy (94% vs. 87% TPR reduction). Even against DP-SGD, which offers provable guarantees, HAMP delivered a significantly better privacy-utility balance, achieving comparable privacy with an average accuracy loss of only 0.46% versus DP-SGD's 23.84%.
• Synergistic Component Contributions: A detailed ablation study confirmed that all three core components of HAMP—the high-entropy soft labels, the entropy-based regularizer, and the testing-time output modification—are indispensable and work synergistically. Each component individually contributed to reducing MIA risk, but their combination yielded the most robust privacy protection while preserving accuracy. For instance, while soft labels alone reduced TPR from 14.37% to 4.76%, the full HAMP defense achieved a TPR of 0.40%.
• Robustness Against Advanced Attacks: HAMP demonstrated strong resilience even against sophisticated data-poisoning-based MIAs, which are designed to amplify membership leakage. On CIFAR10, where data poisoning surged TPR from 8.23% to 52.44% for undefended models, HAMP effectively suppressed it to 0.34% with a negligible accuracy drop of 0.6%.

These findings collectively establish HAMP as a highly effective, practical, and robust defense against diverse Membership Inference Attacks, offering an unprecedented balance between privacy protection and model utility.

Technical Deep Dive

[▶ Watch: Categories of Membership Inference Attacks (2:45)]()

HAMP's architecture is meticulously designed to directly counter model overconfidence, thereby nullifying the distinguishing features exploited by Membership Inference Attacks (MIAs). This is achieved through a two-pronged approach: a training-time defense that fundamentally alters the model's learning behavior, and a testing-time defense that uniformly obfuscates its inference outputs.

The training-time defense focuses on making the model inherently less confident on training samples. This component integrates two key innovations:

1. High-Entropy Soft Labels: Traditional ML training typically uses one-hot encoded hard labels, assigning a probability of 1 to the ground-truth class and 0 to all others. This encourages the model to output extremely confident predictions. HAMP replaces these hard labels with high-entropy soft labels. Instead of a 100% probability, the ground-truth class is assigned a much lower probability (e.g., 60%), with the remaining probability distributed among other classes. This explicitly forces the model to produce less confident, higher-entropy outputs during training. The level of entropy in these soft labels is controlled by a parameter, gamma (γ); a higher γ results in a smaller probability assigned to the ground-truth class and thus higher entropy. For a k-class problem, if j_truth is the ground-truth class, the soft label y' is defined as p for j_truth and (1-p)/(k-1) for other classes, where p decreases with increasing γ.
2. Entropy-Based Regularizer: To further enforce less confident predictions, HAMP introduces an entropy-based regularizer into the training objective. This regularizer penalizes the model for generating any high-confidence outputs, effectively maximizing the prediction entropy of the model's output F_theta(x) during training. The strength of this regularization is controlled by a parameter, alpha (α).

The overall training objective combines these elements. As shown in Equation 7 (as referenced in the talk), the model minimizes the Kullback-Leibler divergence loss between its predictions and the high-entropy soft labels, while simultaneously maximizing the prediction entropy through the regularizer:

min_theta L_KL((F_theta(X_tr), Y_soft_tr), 0) - alpha * H(F_theta(X_tr))

By employing a sufficiently large γ for the soft labels and an appropriate α for the regularizer, HAMP steers the model away from overconfident predictions for training data, while still ensuring high classification accuracy. This dual approach mitigates membership leakage by:

• Output Scores: Forcing higher entropy outputs on training samples, making them statistically similar to testing samples. For instance, on Purchase100, HAMP reduced the entropy difference between members and non-members by a factor of 31.
• Prediction Labels: Pushing training samples closer to the decision boundary, thereby reducing the difference in adversarial robustness between members and non-members. On CIFAR100, HAMP reduced the perturbation difference needed for boundary attacks by over 53%, and for augmentation attacks, it reduced the accuracy difference on augmented samples by 52%.

The testing-time defense is designed to further enhance privacy without any degradation in model accuracy. Its core concept is to uniformly modify all model outputs at inference time—for both members and non-members—into low-confidence scores. This strategy leverages the observation that randomly generated samples, due to the high dimensionality of input space, are almost always predicted with low confidence.

Specifically, for any runtime input x, the model first generates its original prediction scores, F(x). Concurrently, a uniform random sample (X_rand) is generated, and its prediction scores, F(X_rand), are obtained. The modified output, denoted as F_rand(x), is then constructed by replacing the values in F(x) with values derived from F(X_rand). Crucially, this modification is performed in a way that preserves the relative ordering of probabilities for all classes. That is, argsort(F_rand(x)) must equal argsort(F(x)). This ensures that the predicted class label (the class with the highest probability) remains unchanged, but all associated confidence scores become uniformly low and randomized. This uniform low-confidence output across all samples, irrespective of their membership status, significantly increases the adversary's difficulty in identifying differential behaviors, as the confidence signals are effectively obscured.

Algorithm 1 provides a holistic view of HAMP. The training phase generates high-entropy soft labels based on the original hard labels and γ, preserving the ground-truth information. It then trains the model using the combined KL divergence loss and entropy-based regularization, controlled by α. In the testing phase, for any input x, F(x) and F(X_rand) are computed, and F(x) is transformed into F_rand(x) by replacing scores while maintaining the rank order of class probabilities. This elegant design ensures correct model learning while outputting low-confidence predictions at inference time, effectively obscuring membership information without impacting the model's primary classification task.

Demo / Proof of Concept

[▶ Watch: Overview of existing MIA defenses and their limitations (3:35)]()

While the talk did not feature a live, interactive demonstration, the efficacy of HAMP was rigorously established through an extensive and comprehensive evaluation, serving as a robust proof of concept for its design principles. The evaluation spanned five common benchmark datasets: Purchase100, Texas100, Location30 (all tabular data using fully-connected networks), and CIFAR100, CIFAR10 (image data using DenseNet-12 architectures). Training sizes varied from 1,500 to 25,000 samples, with additional experiments on Purchase100 extending up to 50,000 samples, demonstrating scalability.

To ensure a thorough assessment, HAMP was tested against a diverse suite of nine Membership Inference Attacks, encompassing both score-based and label-only categories. This included state-of-the-art attacks like LiRA (for which 128 shadow models were trained for each defense to simulate a strong adversary), as well as NN-based, loss-based, entropy-based, confidence-based, boundary attacks, and augmentation attacks. HAMP's parameters, γ and α, were meticulously tuned via grid search to optimize for a balance between a small train-validation gap and high validation accuracy. The testing-time defense utilized uniformly generated random samples, such as random pixel values for image data.

HAMP's performance was benchmarked against seven leading state-of-the-art defenses: AdvReg, MemGuard, DMP, SELENA, Early Stopping, Label Smoothing (LS), and DP-SGD. The evaluation metrics focused on membership privacy, quantified by attack True Positive Rate (TPR) at 0.1% False Positive Rate (FPR) for members, and attack True Negative Rate (TNR) at 0.1% False Negative Rate (FNR) for non-members, along with standard model accuracy.

The results unequivocally demonstrated HAMP's superior privacy-utility trade-off:

• Against Undefended Models: HAMP reduced the average attack TPR at 0.1% FPR by 94.1% (from 13.48% to 0.8%) and the average attack TNR at 0.1% FNR by 97% (from 19.89% to 0.59%). This was achieved with an average accuracy drop of only 0.46% across all datasets, with a maximum drop of 1.1% on Location30.
• Comparison with MemGuard: While MemGuard preserved accuracy well (0% drop), HAMP offered substantially stronger privacy, with its average attack TPR of 0.8% being 8.4 times lower than MemGuard's 6.7%, and its TNR of 0.59% being 18.3 times lower than MemGuard's 10.9%. This highlighted MemGuard's limitation in addressing label-based leakage, which HAMP's training-time defense effectively mitigates.
• Comparison with AdvReg: HAMP consistently achieved higher accuracy (0.46% vs. 7.45% average drop) and stronger privacy (94% vs. 87% TPR reduction, 97% vs. 90% TNR reduction).
• Comparison with DMP: DMP, a knowledge distillation defense, showed strong privacy on some datasets but failed to generalize, exhibiting high attack TPR (7.26%) and TNR (23.33%) on Location30, along with significant accuracy losses (over 30% on Purchase100, ~12% on Texas100 and CIFAR100). HAMP maintained robust privacy and minimal accuracy loss across all datasets.
• Comparison with SELENA: Both HAMP and SELENA achieved strong privacy, but HAMP significantly outperformed SELENA in accuracy (0.46% vs. 2.25% average drop).
• Comparison with Label Smoothing (LS): LS, primarily an accuracy improvement technique using low-entropy soft labels, still suffered from high MIA risk, sometimes even amplifying it. HAMP, using high-entropy soft labels (4x to 50x higher entropy than LS), achieved 7.1 times lower average attack TPR and 6.3 times lower TNR.
• Comparison with DP-SGD: While DP-SGD (with epsilon=4) achieved slightly lower attack TPR (0.36%) and TNR (0.3%), it came at a severe cost of 23.84% average accuracy loss. HAMP's 0.46% accuracy drop made it a far more practical choice for most applications, demonstrating a superior privacy-utility trade-off even when DP-SGD's parameters were tuned to reduce accuracy loss.

An ablation study further elucidated the contribution of each HAMP component. Training with high-entropy soft labels alone reduced TPR from 14.37% to 4.76%. Combining soft labels with the entropy-based regularizer further reduced TPR to 1.86%. The full HAMP defense, integrating all three components including the testing-time output modification, achieved the best privacy (TPR of 0.40%, TNR of 0.44%), confirming the synergistic effect and necessity of each part.

Finally, HAMP's resilience was tested against a potent data-poisoning-based MIA. On CIFAR10, data poisoning escalated the attack TPR from 8.23% to 52.44% and TNR from 10.15% to 24.52%. Even under such an amplified threat, HAMP successfully reduced the TPR to 0.34% and TNR to 0.71%, with a negligible 0.6% accuracy drop, showcasing its robustness against advanced adversaries. These comprehensive evaluations serve as compelling evidence of HAMP's effectiveness and practicality.

Defensive Implications

[▶ Watch: Root cause of overconfidence: one-hot labels (4:40)]()

The findings presented in this talk have profound implications for organizations and practitioners deploying machine learning models, particularly in domains handling sensitive user data. The pervasive threat of Membership Inference Attacks (MIAs) means that models trained on private information can inadvertently leak details about their training set, leading to severe privacy breaches. HAMP offers a robust, practical, and data-efficient solution to this challenge.

Defenders should recognize that the root cause of MIA vulnerability lies in model overconfidence on training samples. Therefore, any effective defense strategy must directly address this phenomenon. HAMP provides a clear blueprint for mitigating this risk:

1. Re-evaluate Training Objectives: Organizations should move beyond traditional one-hot encoding for ground-truth labels if privacy is a concern. Implementing high-entropy soft labels during model training, as HAMP does, directly encourages the model to produce less confident, higher-entropy probability distributions, making training samples less distinguishable.
2. Integrate Regularization for Confidence: Incorporating an entropy-based regularizer into the loss function during training is crucial. This penalizes the model for exhibiting overly confident predictions, further ensuring that its behavior on members and non-members remains statistically similar. The parameters γ and α should be tuned to balance privacy and utility for specific datasets and models.
3. Implement Testing-Time Output Obfuscation: The testing-time defense component of HAMP is a powerful and low-overhead addition. By uniformly modifying all inference outputs to low-confidence scores while preserving the predicted label's rank order, this technique effectively obscures the confidence signals that adversaries exploit. This step is particularly valuable as it works post-training and can be applied to existing models, though HAMP shows its full strength when combined with the training-time modifications.

While HAMP requires re-training the model (a one-time overhead comparable to many other defenses), and introduces a slight inference latency (typically 0.04 to 0.38 milliseconds per inference, significantly lower than some alternatives like MemGuard's 335-391 milliseconds), these costs are minimal given the substantial privacy gains. Defenders must weigh these practical considerations against the potential for privacy breaches. For applications with stringent real-time constraints, careful evaluation of the testing-time defense's impact is warranted, though its design minimizes label changes.

It is critical for defenders to understand that HAMP, like many practical defenses, provides empirical privacy rather than provable guarantees like Differential Privacy. However, its demonstrated robustness against a wide array of state-of-the-art MIAs, including advanced data-poisoning-based MIAs, indicates a very high level of practical security. In scenarios where the severe accuracy degradation of provable DP is unacceptable, HAMP presents a compelling alternative that offers strong empirical protection with minimal utility loss. Organizations should prioritize solutions that maintain high model utility for their primary task while providing demonstrably strong empirical privacy against known and even amplified attacks.

Key Takeaways

• Membership Inference Attacks (MIAs) fundamentally exploit a machine learning model's overconfidence when predicting training samples compared to unseen data.
• HAMP is a novel, practical defense that directly counters this overconfidence through a two-pronged approach: a training-time defense and a testing-time defense.
• The training-time defense utilizes high-entropy soft labels and an entropy-based regularizer to force the model to learn less confident predictions.
• The testing-time defense uniformly modifies all inference outputs to low-confidence scores, preserving predicted labels but obscuring confidence differences, significantly increasing adversary difficulty.
• HAMP achieves a superior privacy-utility trade-off, reducing MIA risk (94.1% average TPR reduction, 97% average TNR reduction) with minimal average accuracy drop (0.46%) compared to seven state-of-the-art defenses.
• All three core components of HAMP (soft labels, entropy regularizer, testing-time modification) are crucial and synergistically contribute to its overall effectiveness.
• HAMP demonstrates strong robustness, effectively mitigating advanced data-poisoning-based MIAs with negligible accuracy impact.

About the Speaker(s)

The talk "Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Prediction" was presented by Zitao Chen. No further biographical details regarding their title or affiliation were provided in the conference metadata or transcript.