Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs

Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs

Speakers: Yinan Zhong (presented by surrogate speaker)

Conference: NDSS Symposium 2026

YouTube: https://www.youtube.com/watch?v=9gGKhZeN9WQ

Overview

This talk presents Renovate, a framework for detecting and sanitizing Indirect Prompt Injection (IPI) attacks in LLM-integrated applications. IPI attacks occur when adversaries embed malicious instructions in external data sources (websites, databases, APIs) that are consumed by LLM agents, hijacking the model into following the attacker's instructions instead of the user's. Unlike prior defenses that either detect attacks (but may terminate service) or attempt prevention through prompt engineering (with limited effectiveness), Renovate performs token-level detection and sanitization -- identifying and removing individual injected tokens while preserving the integrity of legitimate data.

The key insight is that attention features from the LLM's transformer layers contain behavioral signals that distinguish injected tokens from benign ones, even when the same tokens could appear in either context. A novel two-step attentive pooling mechanism -- response-wise and head-wise -- extracts the most relevant attention features for classification. Evaluated across five target models (ChatGLM, Dolly, Falcon, LLaMA 2, LLaMA 3), Renovate achieves detection accuracy of 97-99% with strong transferability to unseen attack methods. The work also introduces FIPI, a dataset of 100,000 IPI instances covering five attack methods and 300 NLP tasks.

Background

▶ Watch: LLM-integrated applications and indirect prompt injection threat (0:00)

LLM-integrated applications follow a standard pipeline: process the user's instruction, retrieve data from external sources, combine the instruction with retrieved data into a prompt, and query the LLM to generate a response or perform actions. This architecture enables powerful automation for web agents, email assistants, intelligent planners, and database managers.

The security vulnerability arises from the fact that external data sources are not always trustworthy. An attacker who controls or can inject content into a data source (a web page, a database record, an API response) can embed malicious prompts in that data. When the LLM processes the combined input (user instruction + external data), it may follow the attacker's hidden instructions -- outputting phishing links, revealing private information, or taking unauthorized actions.

Existing defenses fall into two categories with complementary weaknesses:

IPI Detection: Uses auxiliary LLMs or classifiers (like LLaMA Guard or Task Trackers) to detect whether an IPI attack is occurring. These approaches introduce significant computation overhead and may terminate the process when attacks are detected, compromising application availability.

IPI Prevention: Uses prompt engineering to distinguish user instructions from data, or fine-tunes the LLM to only follow user instructions. These approaches maintain availability but have limited effectiveness, and fine-tuning changes the training pipeline -- a significant deployment challenge.

The threat model assumes a powerful adversary: full control over external data, the ability to deploy any attack method, and in the worst case, access to the LLM's gradients. The defender has full access to the LLM but cannot modify it, making the defense deployable as a middleware layer without changes to the model or its training pipeline.

Key Findings

▶ Watch: Limitations of existing detection vs prevention approaches (4:00)

97-99% detection accuracy across five models: Renovate achieves 99% accuracy on ChatGLM, Falcon, LLaMA 2, and LLaMA 3, and 97% on Dolly. This performance is competitive with or superior to 15 baseline methods.

Effective sanitization with data integrity: The Jaccard similarity between sanitized data and the original clean data is high, demonstrating that the method removes injected tokens while preserving the legitimate content. This means the LLM can still process the sanitized data to fulfill the user's original request.

Generalizable to unseen attacks: Testing on out-of-distribution attack methods (not seen during training) shows promising generalizability, suggesting the attention-based features capture fundamental injection patterns rather than overfitting to specific attack formats.

Token-level granularity: Unlike binary detection approaches that flag entire inputs, Renovate identifies individual injected tokens, enabling surgical removal rather than wholesale rejection of data. This maintains application availability even when external data is partially compromised.

FIPI dataset contribution: The 100,000-instance dataset with character-level and token-level injection position labels fills a gap in the evaluation infrastructure for IPI defense research.

Technical Deep Dive

▶ Watch: Two-step attentive pooling: response-wise and head-wise (8:00)

Attention Feature Extraction: When the LLM processes a combined prompt (user instruction + external data) and begins generating a response, attention values flow from response tokens back to input tokens. For each token in the external data, Renovate captures the attention it receives from the first few generated response tokens. Because LLMs have multiple transformer layers with multiple attention heads per layer, each token has a rich multi-dimensional attention feature vector across layers and heads.

The insight is that injected prompt tokens receive different attention patterns than benign data tokens. When the LLM encounters injected instructions hidden in data, the attention mechanism responds differently -- injected tokens may receive disproportionate attention from response tokens as the model processes them as instructions rather than data.

Two-Step Attentive Pooling:

1. Response-wise attentive pooling: Aggregates attention features across response tokens, learning to weight response tokens that are most indicative of injection activity more heavily. Different response tokens may have different discriminative power for detecting injections.

1. Head-wise attentive pooling: Aggregates across attention heads, learning which heads carry the strongest signal for distinguishing injected from benign tokens. Not all attention heads are equally useful for IPI detection; this step automatically identifies and prioritizes the most informative ones.

The sequential application of these two pooling layers compresses the variable-length, multi-dimensional attention features into fixed-length representations suitable for the classification model.

Token-Level Detection Pipeline: The classifier outputs predicted logits for each token, which are filtered for noise and converted to binary labels (injected/benign). Tokens classified as injected are removed from the data, producing a sanitized version. If the count of suspicious tokens exceeds a predefined threshold, the entire input is additionally flagged as injected.

FIPI Dataset Construction: A five-step process:

1. Benign instance preparation: GPT rewrites duplicate user instruction prompts to create 10,000 distinct prompt pairs.
2. Probe-witness pairs: 100 manually designed questions with unique deterministic answers -- if an IPI attack succeeds, the LLM outputs the witness answer; otherwise, the witness is unlikely to appear.
3. Injection template generation: Multiple IPI attack methods generate injection templates.
4. Injection application: Templates are injected into benign instances at randomized positions, with character-level and token-level position labels recorded.
5. False positive filtering: Instances where the original data already contains the witness answer are excluded.

Demo / Proof of Concept

▶ Watch: Sanitization workflow: token removal and threshold flagging (10:00)

No live demo was performed. The evaluation covered two tasks (detection and prevention) across five target LLMs with 15 baselines for comparison. Detection accuracy ranged from 97% (Dolly) to 99% (ChatGLM, Falcon, LLaMA 2, LLaMA 3). Prevention results showed effective sanitization with high Jaccard similarity to clean data. Generalizability testing on unseen datasets and attack methods demonstrated promising transferability.

Defensive Implications

▶ Watch: Results: 97-99% accuracy across five target LLMs (14:00)

Deployable middleware defense: Renovate can be deployed as a middleware layer between the data retrieval component and the LLM without modifying the model or its training pipeline. This is a significant practical advantage over defenses that require fine-tuning.

Availability preservation: Unlike detection-only approaches that terminate processing when attacks are found, Renovate sanitizes the data and allows the application to continue functioning. This is critical for production deployments where availability matters.

Complementary to existing defenses: Renovate can be combined with prompt engineering defenses and detection systems for defense-in-depth. The token-level granularity provides a different defensive angle than binary input classification.

Computational consideration: The method requires extracting attention features from the LLM, which adds computational overhead. However, this is likely less expensive than running a separate auxiliary LLM for detection, and the attention values are already computed during normal LLM inference.

Adversarial robustness concern: The threat model allows gradient access, but the evaluation focuses primarily on established attack methods. Future work should evaluate against adaptive attacks specifically designed to evade attention-based detection by mimicking the attention patterns of benign data tokens.

Key Takeaways

• Renovate detects and sanitizes indirect prompt injection attacks at the token level using LLM attention features, achieving 97-99% accuracy across five models
• Two-step attentive pooling (response-wise and head-wise) extracts the most discriminative attention features for IPI detection
• Token-level sanitization removes injected content while preserving legitimate data, maintaining both security and application availability
• The defense is deployable as middleware without modifying the underlying LLM or its training pipeline
• FIPI dataset: 100,000 IPI instances with fine-grained position labels covering five attack methods and 300 NLP tasks
• Promising generalizability to unseen attack methods suggests the approach captures fundamental injection patterns
• The method outperforms or matches 15 baseline approaches across detection and prevention tasks

About the Speaker(s)

The work was presented by a surrogate speaker on behalf of Yinan Zhong and the research team from the USS Lab at Zhejiang University, who could not attend due to visa issues. The team works on LLM security with a focus on indirect prompt injection, attention mechanism analysis, and defensive frameworks. Contact for the research team is through Chenjo at zju.edu.cn.