Prompt Injection Attack to Tool Selection in LLM Agents

Prompt Injection Attack to Tool Selection in LLM Agents

Speakers: Jiawen Shi

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=radUxOZ8IKY

Overview

This talk presents a systematic attack against the tool selection mechanism in LLM agents -- the process by which an agent decides which tool to invoke for a given task. The researchers demonstrate that an attacker can publish a malicious tool with carefully crafted documentation to a tool hub, and through optimization-based prompt injection, force the LLM agent to select the malicious tool over legitimate alternatives. Under a realistic black-box setting, their gradient-free and gradient-based methods achieve over 80% attack success rate (ASR), compared to below 40-60% for existing attacks.

The work identifies a fundamental vulnerability in the two-step tool selection architecture used by most LLM agents: a retriever selects top-K relevant tools based on tool documents, then the LLM chooses the best tool among candidates. By decomposing the attack into retrieval and selection objectives and optimizing each independently, the attackers can reliably manipulate both phases to ensure their malicious tool is consistently selected.

Background

▶ Watch: Introduction: tool selection as a core LLM agent vulnerability (0:00)

Modern LLM agents are equipped with tool libraries containing search, email, booking, and other capabilities. When a user submits a task, the agent decides which tool to invoke through a process called tool selection. The dominant architecture uses a two-step mechanism: first, a retriever selects the top-K relevant tools from the library based on tool documents (name and description); second, the LLM chooses the best tool among the retrieved candidates.

This architecture introduces a new attack surface because tool documents -- the name and description paired with each tool -- can be submitted from untrusted external sources. Tool hubs allow third parties to publish tools that LLM agents can discover and use. An attacker who publishes a malicious tool with specially crafted documentation can potentially have it retrieved and selected by target agents, leading to data leakage, misinformation, or malicious actions.

Prior prompt injection attacks in this space have relied on either manual heuristic methods or automated techniques that target specific system vulnerabilities. Manual attacks achieve limited success rates, and automated techniques require knowledge of specific system components. This research aims to overcome these limitations under a realistic black-box setting where the attacker does not know the target task description, the full tool library contents, the retriever, the top-K parameter, or the LLM.

Key Findings

▶ Watch: Threat model: black-box setting with no knowledge of target internals (2:00)

Over 80% attack success rate: Both the gradient-free and gradient-based optimization methods achieved ASR exceeding 80% across the MINT and ToolBench benchmarks, compared to 40-60% for existing prompt injection attacks.

Black-box attack feasibility: The attack operates under a realistic no-box setting where the attacker does not know the specific target task description, the full tool library, the retriever, the top-K parameter, or the LLM used by the target agent. The attacker only needs to know the general target task category and the tool document format.

Decomposed optimization works: The key insight is that decomposing the attack into separate retrieval objective and selection objective phases, optimizing the tool description as two subsequences (R for retrieval, S for selection), is far more effective than attempting to solve the full optimization problem directly.

Generality of functional descriptions: Tool descriptions that capture general functionality rather than specific task semantics are more likely to be retrieved across diverse queries, making the retrieval phase robust to variation in user task descriptions.

Top-K parameter tradeoffs: Increasing the top-K parameter in the retrieval phase reduces attack success rate (more candidates dilute the malicious tool's chances), while increasing K-prime (the number of optimization iterations) improves attack performance.

Technical Deep Dive

▶ Watch: Shadow tool selection framework and optimization formulation (4:00)

The attack formulation is clean and systematic. Given a malicious tool with documents including a tool name and description, the attacker constructs a shadow tool selection framework consisting of:

• Shadow task descriptions: Multiple plausible descriptions of the target task
• Shadow tool documents: Simulated benign tool documents
• Shadow retriever: A surrogate retrieval model
• Shadow LLM: A surrogate language model simulating the selection stage

The tool description is divided into two subsequences: R (optimized for retrieval) and S (optimized for selection). The optimization proceeds sequentially:

Retrieval Optimization: The attacker uses an LLM to optimize the R portion of the tool description. The key insight is that a general functionality description that captures broad task semantics is more likely to be retrieved across diverse query formulations than a narrowly specific one.

Selection Optimization (Gradient-Free): An iterative search process using two roles -- an attacker LLM that generates and refines candidate S strings, and a shadow LLM that simulates the selection stage and provides feedback. An optimization dataset of multiple shadow task descriptions, each paired with benign tools, is used to compute a success score measuring how often the malicious tool is selected. Low-performance candidates are pruned, and the remaining are refined iteratively until the malicious tool is consistently selected across all shadow tasks.

Selection Optimization (Gradient-Based): Available in the full paper, this approach uses gradient information from the shadow LLM to directly optimize the S subsequence, achieving comparable or slightly better performance than the gradient-free method.

The non-differentiable, discrete nature of text optimization is addressed through the sequential decomposition -- rather than solving the joint optimization problem directly, each phase is optimized independently while ensuring coordinated effect through the sequential structure.

Demo / Proof of Concept

▶ Watch: Gradient-free iterative search with attacker and shadow LLMs (8:00)

The researchers did not perform a live demonstration but presented empirical results across the MINT and ToolBench benchmarks, comparing their gradient-free and gradient-based methods against existing manual and automated prompt injection baselines. The results showed a consistent gap of 20-40 percentage points in ASR between their methods (>80%) and the best existing attacks (40-60%).

Defensive Implications

▶ Watch: Results: over 80% ASR vs 40-60% for existing attacks (10:00)

This research exposes a fundamental architectural vulnerability in LLM agent tool selection that the security community needs to address urgently. The attack is particularly concerning because:

Tool hub supply chain risk: Any tool hub that allows third-party tool submissions is vulnerable. Organizations deploying LLM agents with external tool integrations need to implement strict tool vetting and provenance verification before allowing tools into their agents' libraries.

Black-box transferability: The shadow framework approach means attackers do not need access to the target system's internals. Defenses based on obscuring system architecture are insufficient.

Increasing top-K reduces but does not eliminate the threat. While retrieving more candidate tools dilutes the attack, it also increases computational cost and may degrade legitimate tool selection quality.

Potential mitigations include tool document integrity verification, behavioral monitoring of tool selection patterns, allowlist-based tool curation rather than open discovery, and adversarial training of retrieval and selection components to resist optimized malicious descriptions.

Key Takeaways

• LLM agent tool selection is vulnerable to systematic prompt injection attacks that force agents to select attacker-controlled malicious tools
• The attack achieves over 80% success rate under realistic black-box conditions where the attacker knows neither the target system's internals nor the specific user queries
• Decomposing the attack into separate retrieval and selection objectives with independently optimized tool description subsequences is the key technical innovation
• General functional descriptions are more effective for retrieval manipulation than task-specific ones
• Tool hubs and open tool discovery mechanisms represent a significant supply chain attack surface for LLM agent deployments
• Increasing top-K retrieval candidates provides some mitigation but is not a complete defense

About the Speaker(s)

Jiawen Shi is the lead author but could not attend the conference in person; the work was presented by a proxy speaker on her behalf. The research addresses fundamental security properties of LLM agent architectures. Contact for further questions and discussion is available through Jiawen's email.