Memory Backdoor Attacks on Neural Networks

Memory Backdoor Attacks on Neural Networks

Speakers: Eden Luzon

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=ML21rYHQGt8

Overview

Federated learning is widely assumed to guarantee data privacy because training data never leaves client devices. This talk dismantles that assumption by presenting memory backdoor attacks -- a technique where a compromised central server injects malicious training code that causes local models to memorize and systematically reconstruct specific training samples on demand. Unlike previous memorization attacks that were limited in capacity, fragile to noise, and lacked systematic extraction, memory backdoors allow an attacker to request specific images by index (e.g., "give me image number 325, class dog, green channel, sixth patch") and reconstruct them with high structural similarity.

Presented by Eden Luzon from the Offensive AI Lab at Ben-Gurion University of the Negev, the research demonstrates that 1,000 stolen images can be extracted with 98% success rate and zero impact on normal model behavior. The attack extends to both vision models (CNNs, Vision Transformers) and large language models (T5), and the code is available on GitHub.

Background

▶ Watch: Introduction to federated learning and the privacy assumption (0:00)

Federated learning (FL) enables multiple clients to collaboratively train a global model without sharing their private data. Each client trains locally, sends model updates to a central server, which aggregates them into a global model and redistributes it. One of the server's responsibilities is providing common training code to all clients. The prevailing assumption is that keeping data local equals private training data -- this paper challenges that assumption directly.

Previous memorization attacks on neural networks showed that models can leak some training data, but they suffered from three fundamental limitations: limited capacity (only small amounts of data could be memorized), lack of robustness (even slight noise breaks the attack), and no systematic extraction (you cannot request a specific training sample by index).

The threat model assumes the central server is compromised, meaning the attacker can send malicious training code to clients. In practice, this is realistic because training code is rarely inspected at the low level and is often delivered as pre-compiled binaries or containers, making it easy to hide modifications to the loss function or training loop.

Key Findings

▶ Watch: Multitask learning: primary task loss plus memory task loss (4:00)

The core innovation is a backdoor that reconstructs training data rather than causing misclassification. In a traditional backdoor, a trigger causes a classification error. In a memory backdoor, an index trigger causes the model to output a specific training sample. The extraction is systematic because each sample is linked to a specific index encoded in the trigger.

The attack uses multitask learning with two objectives: the primary task (standard cross-entropy loss for normal classification) and the memory task (MSE loss for training sample reconstruction). A parameter lambda controls the trade-off between utility and memorization -- larger lambda improves reconstruction but increases detection risk.

The researchers tested across multiple architectures including CNNs, Vision Transformers, and evaluated on datasets including MNIST, CIFAR-100, VGGFace (face recognition), and a medical MRI dataset (image segmentation). At 1,000 stored images, the attack achieves 98% success rate with zero impact on normal model accuracy. The accuracy curves with and without memorization overlap almost perfectly, making the attack undetectable through performance monitoring.

In federated learning, a ground-robbing strategy excludes the victim's updates from aggregation, keeping the attack unnoticed and the model performance stable. Even though per-round capacity is limited, the attacker can extract the entire dataset by attacking round by round, with full extraction achievable early in the training process.

Technical Deep Dive

▶ Watch: Trigger design: encoding class, channel, image number, and patch index (6:00)

The trigger design is the technical core of the attack. For normal images, the model outputs a vector of logits representing class probabilities. When the model sees an index trigger, the output is forced to represent vectors of pixels instead. Since the classification output vector is too small to contain a full image, the image must be reconstructed piece by piece.

Each trigger encodes four pieces of information: the class, the color channel (R/G/B), the image number within that class, and the specific patch being requested. For example, a trigger might request "the sixth patch of the 325th image for class dog, green channel." These index values are encoded directly into the trigger image's channels, signaling the model exactly which piece to reconstruct. By collecting all pieces, the attacker reconstructs the entire image.

For the LLM adaptation (tested on T5), the trigger is entirely textual, combining a fixed prompt with a small hash index. For example: "Tell me a memory about $D0A1." The model outputs the linked training sample from the dataset. At 1,000 stolen images, the LLM attack achieves 98% success rate with zero impact on normal behavior, though success rates decrease as the number of memorized images increases.

The attack is also robust to pruning, a common defense technique in federated learning, because the memorization is distributed across model weights rather than concentrated in specific neurons that pruning would eliminate.

Demo / Proof of Concept

▶ Watch: LLM adaptation: textual triggers on T5 with hash-based indexing (8:30)

The researchers presented visual results showing original samples alongside reconstructions at varying memorization capacities. Structural similarity (SSIM) remains high even as thousands of images are simultaneously memorized. The demonstration covered multiple domains: general image classification (MNIST, CIFAR-100), face recognition (VGGFace), and medical imaging (MRI segmentation), showing the attack's generality.

The accuracy curves are particularly compelling: the blue curve (with memorization) and the green curve (without memorization) overlap almost perfectly throughout training, demonstrating that the memory backdoor has no measurable impact on the model's legitimate performance. From the client side, everything looks completely normal.

The federated learning round-by-round extraction was also visualized, showing that the red dashed line (full dataset extraction) is reached early in the training process, meaning an attacker does not need to wait for training to complete.

Defensive Implications

▶ Watch: Conclusion: keeping data local is not enough, inspect the training code (10:00)

The primary defensive takeaway is that keeping data local is not sufficient for privacy in federated learning. Organizations deploying federated learning must inspect the training code itself, not just the data flow. Since training code is often delivered as pre-compiled binaries or containers, this requires either source code auditing or trusted computation environments for the training process.

Standard federated learning defenses are insufficient against this attack. Performance monitoring cannot detect the backdoor because model accuracy is unaffected. Pruning-based defenses are ineffective because the memorization is distributed across weights. The ground-robbing aggregation strategy makes the attack invisible to anomaly detection on model updates.

Potential mitigations include trusted execution environments for local training to prevent code tampering, cryptographic verification of training code integrity, and formal verification of loss function properties to detect unauthorized auxiliary objectives. Organizations handling sensitive data (medical, biometric, financial) through federated learning should treat the training code supply chain with the same rigor as data protection.

Key Takeaways

• Federated learning does not automatically ensure privacy -- a compromised central server can inject memory backdoors that systematically extract training data
• The memory backdoor achieves 98% extraction success at 1,000 images with zero impact on model accuracy, making it undetectable through performance monitoring
• Training samples can be requested by index, enabling targeted extraction of specific data points
• The attack works across CNNs, Vision Transformers, and LLMs (T5), and across image classification, face recognition, and medical imaging domains
• In federated learning, entire datasets can be extracted round by round, with full extraction achievable early in training
• The attack is robust to pruning defenses, and the main mitigation is inspecting and verifying the integrity of training code itself

About the Speaker(s)

Eden Luzon is a researcher at the Offensive AI Lab at Ben-Gurion University of the Negev, with collaboration from the University of Innsbruck. The research focuses on offensive AI techniques and their implications for privacy in distributed learning systems. The speaker presented in person and engaged with audience questions about fundamental capacity limits of model memorization, acknowledging that model capabilities directly affect the maximum number of samples that can be reliably extracted.