Was My Data Used for Training? Membership Inference in Open-Source LLMs via Neural Activations

Was My Data Used for Training? Membership Inference in Open-Source LLMs via Neural Activations

Speakers: Xue Tan

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=XVod6nDFWXk

Overview

As open-source LLMs proliferate with massive, opaque training datasets, verifying whether specific data was used for training has become critical for privacy evaluation, compliance auditing, and copyright protection. This talk presents NOT (Neural activation-based mOdel Training membership inference), a white-box membership inference framework that uses neural activations -- the model's internal layer-wise responses to inputs -- to determine training data membership with approximately 95% AUC across multiple mainstream models.

Unlike black-box approaches limited to output signals or computationally expensive white-box methods using gradients, NOT uses a ResNet-18-based Siamese triplet network with contrastive learning to map neural activations into a discriminative embedding space. The method remains robust under challenging conditions including paraphrasing, quantization, and few-shot scenarios, achieving above 98% accuracy with only 50 training samples. Three new benchmark datasets are introduced for rigorous evaluation.

Background

▶ Watch: Membership inference attacks and their importance for LLM auditing (0:00)

Membership inference attacks (MIA) aim to determine whether a specific data sample was included in a model's training set. The key observation is that models typically behave more confidently and consistently on training data (members) versus unseen data (non-members), producing distinguishable signals in output probabilities, loss values, attention patterns, or gradients.

With the rapid growth of open-source LLMs, training data has become massive and highly opaque. This opacity makes it difficult to verify how model capabilities are formed and whether data was used legally, creating serious privacy and copyright concerns. Without proper auditing tools, organizations deploying these models may face compliance risks.

Existing methods have clear limitations. Black-box approaches rely only on output signals and often lack accuracy. White-box methods using internal signals like gradients can be computationally expensive and do not scale well to non-text inputs. The challenge is compounded by potential overlap and leakage in pre-training data, the high-dimensional nature of activation features, and limited labeled data in realistic settings.

Key Findings

▶ Watch: Benchmark dataset construction with temporal filtering (4:00)

• ~95% AUC across multiple mainstream open-source LLMs, with more than 96% TPR at 5% FPR, significantly outperforming both black-box and white-box baselines
• Robust to paraphrasing: The method maintains strong detection even when member samples are semantically preserved but lexically rewritten, because it relies on activation patterns shaped by semantics rather than lexical memorization
• Label noise tolerance: Performance remains strong at 5% and 10% error ratios, though it degrades notably at 20% mislabeling rates
• Few-shot capable: Achieves above 98% accuracy with only 50 training samples, demonstrating practical applicability when access to LLM training data is limited
• Cross-domain limitations: Performance drops noticeably when training and test data come from different domains (e.g., Wikipedia vs. arXiv), due to different writing styles and structures
• Pre-training vs. fine-tuning gap: Performance on pre-training data is lower than on fine-tuning data, attributed to pre-training's massive scale where most samples are observed only once, resulting in weaker per-sample memorization

Technical Deep Dive

▶ Watch: Results: 95% AUC, 96% TPR at 5% FPR across mainstream LLMs (8:00)

NOT's pipeline consists of four stages:

Data Collection: Text is gathered from public sources including Wikipedia and arXiv. Data is filtered to include only content published after the release date of the target LLM, ensuring no prior exposure during pre-training. Three benchmark datasets are constructed: WikiDetection, NewsDetection, and ArXivDetection, each split into member and non-member subsets. The member subset is used to fine-tune the model, simulating realistic training scenarios.

Activation Extraction: For each input text, neural activations of the final token are collected from every layer of the target model. For texts exceeding the context window, input is split into fixed-length chunks. Layer-wise activations are normalized to remove cross-layer scale differences, yielding consistent features for membership decisions.

Feature Processing: Three feature extraction strategies provide different tradeoffs:

• Raw FE: Uses raw activations preserving fine-grained details at highest computational cost
• Stat FE: Extracts summary statistics (mean, variance) for dimensionality reduction and robustness
• Histogram FE: Models activation distributions with histograms, providing efficient and noise-resistant compression

Membership Inference via Metric Learning: NOT formulates membership inference as a metric learning problem using a ResNet-18-based Siamese triplet network with contrastive learning. Training constructs triplets (anchor, positive member, negative non-member) and maps activations into a shared embedding space. A triplet margin loss pulls anchors closer to member samples and pushes them away from non-members. At inference, each text subsequence's embedding is compared to a small support set using nearest-neighbor distance, with majority voting across subsequences determining the final membership decision.

Demo / Proof of Concept

▶ Watch: Robustness to paraphrasing and label noise (10:00)

Extensive experiments were conducted across multiple mainstream open-source LLMs. The evaluation covers both fine-tuning and pre-training scenarios across the three benchmark datasets, with representative black-box and white-box methods as baselines.

The paraphrasing robustness evaluation constructs member samples as semantic-preserving rewrites and tests whether the method can still identify paraphrased members. Strong detection performance is maintained, attributed to reliance on semantic activation patterns rather than lexical memorization.

The few-shot evaluation progressively reduces training samples and measures performance degradation. Above 98% accuracy is maintained with only 50 training samples, demonstrating applicability in data-scarce scenarios.

The cross-domain evaluation reveals a limitation: when training data comes from Wikipedia but test data from arXiv, performance drops due to structural and stylistic differences between platforms.

Defensive Implications

▶ Watch: Pre-training vs fine-tuning performance gap (12:00)

NOT has practical implications for multiple stakeholders:

• Copyright enforcement: Content creators and publishers can verify whether their copyrighted material was used to train specific open-source models, supporting legal claims and licensing negotiations
• Compliance auditing: Organizations deploying LLMs can audit whether training data complies with data protection regulations (GDPR, CCPA) by testing for membership of protected data categories
• Privacy evaluation: Researchers and regulators can assess the privacy risks of deployed models by quantifying the degree to which individual training samples can be identified
• Model transparency: The framework contributes to model transparency efforts by providing tools to verify training data claims made by model developers
• Defense against data poisoning: Understanding which data is memorized by a model can inform defense strategies against training data poisoning attacks

Key Takeaways

• Neural activations provide a more effective signal for membership inference than output probabilities or gradients, achieving ~95% AUC across mainstream LLMs
• The Siamese triplet network approach formulates membership inference as metric learning, creating a discriminative embedding space that separates members from non-members
• Robustness to paraphrasing demonstrates that the method captures semantic memorization, not just lexical patterns
• Only 50 training samples are needed for above 98% accuracy, making the approach practical in data-scarce settings
• Cross-domain and pre-training scenarios reveal current limitations that warrant further investigation
• Three new benchmark datasets (WikiDetection, NewsDetection, ArXivDetection) are contributed for standardized evaluation

About the Speaker(s)

Xue Tan (Tani) presented on behalf of a team from Fudan University and Polytechnic Institute. The research addresses the growing need for auditing tools as open-source LLMs with opaque training data become widespread in production deployments.