LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors

LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors

Speakers: Chengkun Wei

Conference: NDSS Symposium

YouTube: (no public video)

Overview

In the rapidly evolving landscape of large language models (LLMs), prompt-tuning has emerged as a highly efficient and scalable paradigm for adapting powerful pretrained models to diverse downstream tasks. This talk, "LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors," delivered by Chengkun Wei, delves into a critical security vulnerability inherent in this approach: the susceptibility to task-agnostic backdoors. These insidious backdoors, implanted within the foundational pretrained model itself, pose a significant threat because they can compromise any downstream task, regardless of how the model is subsequently prompt-tuned.

The research, a collaborative effort from Zhejiang University, Stanford University, CISPA Helmholtz Center, and Ant Group, highlights an urgent need for robust security measures. As LLMs become increasingly ubiquitous, with users frequently downloading pretrained models from public repositories like HuggingFace, the risk of encountering poisoned models escalates. LMSanitator proposes an innovative defense mechanism that not only detects these stealthy backdoors but also effectively removes their influence during inference, all without altering the crucial frozen parameters of the pretrained model. This work underscores the importance of proactive security in the era of efficient LLM deployment and shared model ecosystems.

Background

[▶ Watch: Introduction: Prompt-tuning and task-agnostic backdoor threat (0:00)]()

Modern language models, often built on the Transformer architecture and boasting billions of parameters, are computationally intensive to train from scratch. This has led to the widespread practice of downloading and adapting pretrained models. Traditionally, fine-tuning was the go-to method for task adaptation. However, as models scaled, storing and serving a unique, fully fine-tuned copy for each task became impractical due to immense storage and computational overhead.

Prompt-tuning emerged as an elegant solution to this challenge. Instead of fine-tuning all model parameters, prompt-tuning appends a small, trainable "soft prompt" to the input sentence, or injects trainable parameters into specific layers, while crucially freezing the vast majority of the pretrained model's parameters. Techniques like P-tuning (modifying input embeddings) and P-tuning v2 (adding parameters to each attention layer, with 0.1% to 3% of total parameters trainable) are state-of-the-art in this domain. This approach allows for strong downstream task performance with significant resource savings and efficient multitask serving.

However, this efficiency comes with a unique security blind spot: backdoor attacks. A backdoor attack aims to force a model to output an attacker-specified target class when a specific trigger is present in the input, while behaving normally on benign inputs. The focus of this research is on task-agnostic backdoors, which are particularly dangerous. These backdoors are injected into the pretrained model by the attacker, who has no prior knowledge of the user's specific downstream task, dataset, or even the prompt-tuning architecture. State-of-the-art task-agnostic attacks like NeuBA, POR, and BTOP achieve this by forcing the pretrained model's output representation to be close to a pre-defined Pre-defined Vector (PV) whenever a trigger is present.

Prompt-tuning is acutely vulnerable to task-agnostic backdoors for two primary reasons:

1. Shared Vulnerability: A single backdoored pretrained model can be used for multiple downstream tasks via prompt-tuning. This means a single attack can compromise numerous systems.
2. Immunity to Forgetting: Unlike fine-tuning, where backdoors can sometimes fade due to catastrophic forgetting with larger training datasets, prompt-tuning's core principle of freezing the pretrained model parameters makes the implanted backdoor immune to such forgetting. The backdoor persists, making prompt-tuned models inherently more susceptible.

Existing NLP backdoor defenses typically involve two stages: detection and removal. For backdoor detection, trigger inversion is a common approach. This involves backpropagating gradients to the input to find the minimal perturbation that changes the model's prediction to a target. However, trigger inversion in NLP is notoriously challenging due to the discrete nature of text and the sparsity of the embedding space, often leading to convergence failures or nonsensical token generation. Previous methods like T-miner (sequence-to-sequence generator) and PICCOLO and DBS (transforming Transformer into differentiable form) have been proposed. However, empirical analysis by the researchers showed that PICCOLO and DBS struggle to converge on task-agnostic backdoors, often encountering sharp loss surfaces and small convex hulls (Figure 11b in the original paper), making optimization difficult. PICCOLO also adds additional word encoding layers, increasing model depth and convergence difficulty.

For backdoor removal, traditional fine-tuning can naturally mitigate backdoors through catastrophic forgetting, but this directly contradicts the efficiency and modularity benefits of prompt-tuning. Other input-side defenses like ONION (removing words that significantly increase text perplexity) and STRIP/RAP (rejecting inputs containing triggers) have limitations. ONION relies on specific trigger properties (perplexity), which might not hold for all task-agnostic attacks, while STRIP/RAP discard inputs rather than sanitizing them, impacting usability. This gap highlights the need for a novel defense like LMSanitator, which can detect and remove task-agnostic backdoors without modifying the pretrained model.

Key Findings

[▶ Watch: Why prompt-tuning is uniquely vulnerable to backdoors (3:20)]()

The research presented several critical findings that underscore both the severity of the threat and the effectiveness of LMSanitator:

• Heightened Vulnerability of Prompt-Tuning: Prompt-tuning models, particularly those using P-tuning, are highly vulnerable to task-agnostic backdoors. The study found Attack Success Rates (ASR) exceeding 99% in 39 out of 72 experiments, with P-tuning showing greater susceptibility than P-tuning v2. This confirms that freezing pretrained parameters makes backdoors persistent and effective.
• Ineffectiveness of Existing Defenses: Prior state-of-the-art defenses like PICCOLO and ONION proved largely ineffective against task-agnostic backdoors. PICCOLO struggled with convergence and failed to find true triggers in most cases, while ONION, though causing less accuracy degradation, was less effective at reducing ASR compared to LMSanitator.
• Novel PV Inversion Approach: LMSanitator introduces a groundbreaking approach by inverting the continuous Pre-defined Vectors (PVs) in the feature space rather than attempting to invert discrete text triggers. This significantly improves convergence and detection performance.
• High Backdoor Detection Accuracy: LMSanitator achieved an average backdoor detection accuracy of 92.8% across 120 clean and 120 backdoored models, covering various architectures and attack types.
• Significant ASR Reduction: After applying LMSanitator, the Attack Success Rate was reduced to less than 5% in 50 experiments and less than 1% in 40 experiments, demonstrating its strong defense capabilities.
• High PV Recall: The system successfully identified 239 out of 252 (94.8% recall) attacker-designed PVs embedded in 42 models, showcasing its ability to accurately pinpoint the malicious vectors. It also successfully found PVs in real-world backdoored models from HuggingFace.
• Preservation of Prompt-Tuning Benefits: Crucially, LMSanitator achieves its defense objectives without modifying the pretrained model parameters, thereby preserving the modularity, low storage, and efficiency benefits that make prompt-tuning attractive.
• Robustness Against Adaptive Attacks: LMSanitator demonstrated resilience against adaptive attacks, such as those using fewer triggers or scattering loss, maintaining high detection rates even under challenging conditions.

Technical Deep Dive

[▶ Watch: Challenges with current backdoor removal for prompt-tuning (5:00)]()

LMSanitator's core design intuition addresses the inherent difficulty of trigger inversion in NLP. Instead of attempting to invert discrete text inputs, which often leads to convergence failures, it proposes to invert the output of the backdoored model. Task-agnostic backdoors, by design, map a specific trigger to a Pre-defined Vector (PV) in the continuous feature space, making this PV an outlier. The key insight is that inverting these continuous PVs is significantly easier than inverting discrete triggers. Once these legitimate PVs are identified, LMSanitator can monitor the similarity between the pretrained model's output and these PVs to detect and remove triggers.

LMSanitator operates through a three-step pipeline:

1. PV Mining: An iterative process to discover the PVs implanted in the pretrained model.
2. PV Filtering: A step to remove any illegal or spurious PVs found during mining.
3. PV Monitoring: During the inference phase, this component detects and removes triggers from inputs based on the identified PVs.

The defense only requires a small set of clean sentences (around 2000 in experiments) and no prior knowledge of attacker triggers. Crucially, it does not modify the pretrained model parameters, preserving prompt-tuning's benefits.

Step I: PV Mining

The goal of PV mining is to invert the attacker-designed PVs. This process is guided by two key observations about task-agnostic backdoors:

• Observation I: Outlier Behavior: A sentence containing a trigger will act as an outlier in the feature space of the backdoored Transformer model. Formally, the distance between a triggered sentence and a clean sentence is greater than the distance between two different clean sentences.
• Observation II: Reduced Diversity: The feature distance between two different clean sentences will shrink if the same trigger is inserted into both. Sentences with the same trigger tend to converge to a single point in the feature space, reducing their diversity.

Based on these observations, LMSanitator designs its inversion losses using a pseudo-siamese neural network setup, where the target model is copied as an auxiliary model, and both models' parameters are frozen. A trainable soft prompt is added to the target model.

• Distance Loss ($L_D$): This loss aims to increase the distance between the target model's output ($F_{tar}$) and the auxiliary model's output ($F_{aux}$) when the input contains a trigger. It uses Mean Squared Error (MSELoss): $L_D = \mathbb{E}_{x \sim \mathcal{D}} \text{MSE}(F_{tar}, F_{aux})$.
• Diversity Loss ($L_{div}$): This loss aims to reduce the diversity of feature outputs within a batch, aligning with Observation II. It uses Shannon entropy: $L_{div} = -\text{Entropy}((\text{Stack } F_{tar}; x \sim \mathcal{B})^T)$. A negative sign ensures that minimizing this loss maximizes diversity.
• Overall Optimization Target: The initial objective for PV inversion is: $arg \min_{\theta_p} \mathcal{L} = \lambda_D \cdot L_D + \lambda_{div} \cdot L_{div}$. When this loss drops below a threshold $T_c$, a backdoor is considered found, and the target model's output and soft prompt parameters are recorded.

To address the challenges of a flat loss surface and difficult convergence during PV mining, two novel mechanisms are introduced:

• Fuzz Training: Inspired by fuzz testing, this enhances inversion by using a test dataset of clean sentences and different random seeds for soft prompt initialization. This helps explore different regions of the loss landscape. To prevent repeated convergence to the same PV, an additional Path Loss ($L_P$) is introduced: $L_P = -\max_i (\text{MSE}(F_{tar}, c_i))$, where $c_i$ are already found candidate PV vectors. This loss increases as the target model's output moves closer to an already discovered PV, encouraging the search for new, unique PVs. The complete training objective becomes: $arg \min_{\theta_p} \mathcal{L} = \lambda_D \cdot L_D + \lambda_{div} \cdot L_{div} + \lambda_P \cdot L_P$.
• Adaptive Learning Rate: A large initial learning rate ($l_{ro}$) is necessary for effective initial gradient updates in a flat loss space. However, a constant large learning rate can hinder convergence. LMSanitator adaptively adjusts the learning rate: if the gradient of any soft prompt parameter exceeds a threshold $T_{grad}$, the learning rate is reset to a smaller value, $0.01 \cdot l_{ro}$. This allows for faster initial exploration and finer-grained convergence.

Step II: PV Filtering

After PV mining, a set of candidate PVs is obtained. This step filters out any spurious or non-legitimate attacker-designed PVs using two mechanisms:

1. Out-of-range Soft Prompts: Candidate PVs whose corresponding soft prompt parameters fall outside the typical range of embedding layer parameters are removed.
2. Inconsistent Loss Behavior: PV candidates where the distance loss ($L_D$) decreases, but the diversity loss ($L_{div}$) remains high, are filtered out. This contradicts Observation II, as legitimate backdoors should reduce diversity. An additional threshold $T_{div}$ is used for this.

If, after filtering, the PV set remains empty after a maximum number of fuzz loops ($L_{max}$), the model is considered clean. Otherwise, it is deemed backdoored.

Step III: PV Monitoring

Once the filtered PV set is obtained, PV Monitoring is deployed during inference for trigger detection and removal. A key observation here is that the feature output of a backdoored model for a triggered input exhibits remarkable consistency with the PV in terms of positive and negative signs, even after prompt-tuning. Clean feature outputs, in contrast, show a random sign distribution.

• Trigger Detection:

1. PVs are converted into sign tuples (e.g., [0.5, 0.1, -0.7] becomes [+, +, -]).
2. During inference, for each input, it's fed through the pretrained model to get its feature vector ($F_{LM}$).
3. A monitor counts the number of matching signs ($N_{match}$) between $F_{LM}$ and each PV sign tuple: $N_{match} = \sum_{n=1}^d \mathbf{1}_{\text{sign}(F_{LM_n}) = \text{sign}(PV_n)}$, where $d$ is the feature vector dimension.
4. If $N_{match}$ exceeds a specific threshold $T_{match}$ (e.g., 0.8 times the dimension $d$), the input is flagged as containing a trigger.

• Trigger Removal: If an input is flagged, a sliding window approach is used to identify and remove the specific trigger words. The window slides across the input sentence, marking candidate trigger words. After removing these candidates, the input is re-evaluated for triggers. This iterative process continues until no trigger is detected or the maximum window length is reached, allowing the model to correctly classify previously poisoned inputs. This monitoring method adds minimal overhead.

Demo / Proof of Concept

[▶ Watch: Introducing LMSanitator's novel approach and benefits (5:30)]()

While no explicit "demo" section was presented, the comprehensive evaluation serves as a robust proof of concept for LMSanitator's capabilities. The researchers conducted extensive experiments to validate the defense's effectiveness, using a rigorous setup:

Experimental Setup:

• Prompt-tuning Datasets: 6 sentence-level classification tasks (RTE, BoolQ, AG News, Yelp-5, Enron spam, SMS spam) and 2 token-level Named Entity Recognition (NER) tasks (CONLL04, OntoNotes 5.0).
• Victim Models: BERT (base and large), RoBERTa (base and large) for end-to-end evaluation and detection. Additional models like DeBERTa, ALBERT, ERNIE, and XLNet were used for PV searching.
• Evaluation Metrics: For attacks, clean model accuracy ($ACC_{clean}$), backdoored model accuracy ($ACC_{backdoor}$), and Attack Success Rate (ASR). For defenses, the ASR after applying LMSanitator, and for token classification, F1 drop.
• Attack Setup: Backdoored pretrained models were generated using BTOP, NeuBA, and POR attacks. Six orthogonal PVs were injected, each corresponding to a single-word trigger (e.g., 'cf', 'mn'). BTOP targeted the MASK token, while NeuBA and POR targeted the CLS token.
• Defense Setup: LMSanitator utilized a small clean dataset of 2000 sentences from WikiText. Key hyperparameters included $lambda_D=1$, $lambda_{div}=1$, $lambda_P=0.5$, $T_{div}=-3.446$, $T_{grad}=5e-3$, $T_{match}=0.8d$, and $l_{sp}=7$.

Key Results and Demonstrations:

• Attack Effectiveness: The evaluation confirmed the high effectiveness of task-agnostic backdoors. In 39 out of 72 experiments, ASR exceeded 99%. P-tuning models were found to be more vulnerable (83% of P-tuning experiments achieved 99% ASR) than P-tuning v2 (40%).
• Defense Effectiveness: LMSanitator significantly reduced ASRs. After deployment, ASR was less than 5% in 50 experiments and less than 1% in 40 experiments. This demonstrates its strong capability to neutralize backdoors.
• Backdoor Detection Accuracy: Across 120 clean and 120 backdoored models, LMSanitator achieved an average backdoor detection accuracy of 92.8%. The researchers noted that larger models were harder to converge, leading to more false negatives, suggesting the need to adjust fuzz loops based on model size.
• PV Searching Recall: LMSanitator successfully found 239 out of 252 (94.8% recall) attacker-designed PVs across 42 models after 1000 searches (Figure 6). It also successfully identified all 6 PVs in a real-world NeuBA-ROBERTa model from HuggingFace and 5 out of 6 in NeuBA-BERT (Figure 8), validating its real-world applicability. An interesting observation was the discovery of unintended PVs that were combinations of the attacker's designed PVs (Figure 7).
• Ablation Study: The study confirmed the necessity of Path Loss (reducing fuzz loops for unique PV discovery) and Adaptive Learning Rate (crucial for efficient convergence, especially in RoBERTa-base models, preventing false negatives) for LMSanitator's performance (Figure 9).
• Comparison with Baselines: LMSanitator significantly outperformed existing defenses. PICCOLO converged on only 4 out of 12 small models and failed to find any true triggers. ONION caused a larger average accuracy decrease (2.2% vs. 0.075% for LMSanitator) and was less effective at reducing ASR, primarily because it's trigger-agnostic and relies on perplexity changes, which are not universal for all task-agnostic triggers.
• Robustness to Adaptive Attacks: LMSanitator maintained detection rates above 90% (and over 80% for RoBERTa-large) even when attackers injected fewer triggers (Table VII). Against scattering loss, which aims to make triggered sentences less concentrated, LMSanitator still detected backdoors effectively, although attackers faced a trade-off between scattering loss and downstream task accuracy (Table VIII).

These results collectively demonstrate LMSanitator's robust and practical capability to defend prompt-tuning models against task-agnostic backdoors.

Defensive Implications

[▶ Watch: LMSanitator's three-step pipeline: mining, filtering, monitoring (6:30)]()

The findings from this research carry significant implications for the security posture of LLM deployment, particularly for organizations and individuals leveraging prompt-tuning.

Firstly, the pervasive practice of downloading pretrained models from public online hubs like HuggingFace introduces a substantial supply chain risk. Users must operate under the assumption that these models, while offering immense utility, could be subtly poisoned with task-agnostic backdoors. These backdoors are particularly insidious because they operate independently of the downstream task, meaning a single compromised pretrained model can jeopardize numerous applications. Therefore, a robust vetting process for downloaded models is paramount.

LMSanitator provides a practical and effective solution to this problem. It offers a two-fold defense:

1. Backdoor Detection: Before deploying a downloaded pretrained model, users can employ LMSanitator to determine if it contains task-agnostic backdoors. This allows for informed decision-making – either discarding the poisoned model or proceeding with its use under the protection of LMSanitator's removal functionality. The recommendation to adjust fuzz loops based on model size (e.g., more loops for larger models like RoBERTa-large) is a practical tip for optimizing detection accuracy.
2. Trigger Removal at Inference: For backdoored models that are deemed necessary for use, or as a continuous protective measure, LMSanitator can be integrated into the inference pipeline. Its ability to detect and remove triggers without modifying the pretrained model parameters is a critical advantage. This ensures that the core benefits of prompt-tuning – modularity, low storage, and efficient multitask serving – are preserved, unlike defenses that require full model retraining or fine-tuning which would negate these advantages.

Furthermore, the research highlights the limitations of existing defenses against task-agnostic backdoors. Defenders should be wary of relying on traditional trigger inversion methods or input-side perplexity-based filters, as these have been shown to be largely ineffective or easily bypassed by sophisticated task-agnostic attacks. The novel approach of inverting continuous Pre-defined Vectors (PVs) offers a more resilient strategy.

Finally, the analysis of adaptive attacks provides valuable insights for defenders. While attackers might try to evade detection by using fewer triggers or employing scattering loss, these often come with a trade-off in attack effectiveness or utility. Defenders should understand these trade-offs to assess the true risk and the robustness of their defenses. The consistent performance of LMSanitator against these adaptive strategies suggests it provides a strong line of defense against evolving threats. In essence, LMSanitator empowers users to confidently adopt prompt-tuning for LLM deployment by offering a scalable, non-invasive, and effective countermeasure against a critical class of backdoor attacks.

Key Takeaways

• Prompt-tuning is highly vulnerable to task-agnostic backdoors: Despite its efficiency benefits, prompt-tuning's reliance on frozen pretrained models makes it exceptionally susceptible to backdoors that persist across diverse downstream tasks.
• Existing defenses are inadequate: Traditional NLP backdoor defenses, including trigger inversion techniques like PICCOLO and input-side filters like ONION, largely fail to effectively detect or remove task-agnostic backdoors in prompt-tuned LLMs.
• LMSanitator offers a novel and effective defense: By innovatively inverting Pre-defined Vectors (PVs) in the continuous feature space rather than discrete text triggers, LMSanitator achieves superior convergence and detection performance.
• Preserves prompt-tuning benefits: Crucially, LMSanitator detects and removes backdoors without modifying the pretrained model parameters, maintaining the modularity, low storage requirements, and resource efficiency that make prompt-tuning attractive.
• High detection accuracy and ASR reduction: LMSanitator demonstrated an average backdoor detection accuracy of 92.8% and reduced Attack Success Rates to less than 1% in most scenarios, showcasing its robust defensive capabilities.
• Robust against adaptive attacks: The defense remains effective against sophisticated adaptive attacks, such as those employing fewer triggers or scattering loss, providing a strong and resilient security measure for prompt-tuned LLMs.

About the Speaker(s)

Chengkun Wei is the presenter of this work, representing a collaborative research effort. The research itself was conducted by a team of co-authors from several prominent institutions: Zhejiang University, Stanford University, CISPA Helmholtz Center for Information Security, and Ant Group. This diverse institutional backing suggests a strong interdisciplinary approach to addressing complex security challenges in large language models.