ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking

ThinkTrap: Denial-of-Service Attacks against Black-box LLM Services via Infinite Thinking

Speakers: Yunzhe Li

Conference: NDSS Symposium

YouTube: https://www.youtube.com/watch?v=ixe82u2k6PI

Overview

This talk introduces ThinkTrap, a novel denial-of-service (DoS) attack against cloud-hosted large language model services that exploits the fundamental autoregressive nature of LLM inference. By crafting optimized prompts that force models into extremely long output generation, an attacker operating within normal rate limits can monopolize GPU resources and drive system throughput to near zero. The attack is black-box (no access to model internals), query-efficient (requiring approximately 10,000 optimization queries), and works across eight public LLM services including DeepSeek and GPT variants.

The core insight is that LLM inference is inherently sequential -- each new token requires a complete model forward pass -- and subscription-based pricing models do not adequately account for the computational cost variance between short and long responses. ThinkTrap demonstrates that a single well-crafted prompt can lock an inference slot for an extended period, and sending enough such prompts at normal request rates can exhaust server resources.

Background

▶ Watch: LLM inference cost model and the DoS attack surface (0:00)

Modern LLMs are predominantly accessed as cloud services due to their heavy computational requirements. Many services use subscription-based pricing where users pay a monthly or yearly fee for access, rather than per-token pricing. This creates an asymmetry: the service provider bears the computational cost of long outputs regardless of the revenue received.

The unique characteristic of LLM inference that enables this attack is its autoregressive nature: tokens are produced one at a time, with each new token requiring a complete model inference pass. The cost of serving a request scales directly with the output length. If an attacker can force the model to continue generating indefinitely, they effectively lock a GPU inference slot until the maximum output length is reached or the request times out.

Prior approaches to exploiting long LLM outputs fall into three categories, each with significant limitations. Semantic tricks (e.g., asking the model to "output endlessly" or posing extremely difficult tasks) are unreliable and easily defended. Gradient-based optimization can find inputs that suppress the end-of-sequence token, but requires white-box access to model internals, making it useless against cloud services. Heuristic or random search attempts many prompts to find ones that trigger long outputs, but is impractical under the rate limits imposed by real LLM services. ThinkTrap is the first method that is both black-box and query-efficient.

Key Findings

▶ Watch: Prior approaches and why they fail in realistic settings (4:00)

ThinkTrap produced several notable results across its evaluation on eight public LLM services:

• Optimized prompts consistently force models to produce outputs hitting the maximum output size allowed by the service
• The attack succeeds with a query budget of approximately 10,000 queries during the offline optimization phase, which is not expensive for an attacker
• Under a realistic rate limit of 10 requests per minute, ThinkTrap drives system throughput to near zero on controlled test servers
• All baseline approaches (two semantic prompt baselines and two heuristic attack baselines) fail to hit the maximum output size under the same constraints
• GPU memory consumption rises dramatically during the attack as the model maintains the context of each ongoing long-running generation
• The attack operates within normal usage patterns, making it difficult for services to distinguish from legitimate traffic
• Transferability across models was observed: prompts optimized against one model showed effectiveness against others, though detailed cross-model GPU impact evaluation was not conducted on commercial services
• The low-dimensional search space (approximately 200 dimensions) proved sufficient for effective prompt optimization

Technical Deep Dive

▶ Watch: 200-dimension projection and token decoding pipeline (8:00)

ThinkTrap treats prompt generation as an optimization problem where the search space is the set of possible prompts and the objective function is the output length. The optimization proceeds in two phases.

Offline Phase: Gradient-Free Prompt Optimization. Since the attacker has no access to model gradients, ThinkTrap uses a gradient-free heuristic search in a low-dimensional space. The process begins with a random vector of approximately 200 dimensions -- much smaller than the full token embedding space. This vector is projected into the token embedding space to produce a set of token vectors, which are then decoded into a word sequence by selecting the nearest tokens. The resulting candidate prompt is submitted to the target LLM service, and the output length serves as the optimization signal. Through iterative refinement using heuristic optimization methods, the system searches for prompts that maximize output length.

The low-dimensional projection is critical for efficiency. Operating directly in the token embedding space would create an intractable search problem, but the 200-dimensional space provides sufficient expressiveness to find effective attack prompts while keeping the optimization tractable within the query budget. The optimal dimension size was determined empirically: smaller spaces degraded performance while larger spaces provided diminishing returns.

Online Phase: Resource Exhaustion. Once an attack prompt is found offline, the online attack is straightforward. The attacker sends the optimized prompt to the target service at a rate within normal limits (e.g., one prompt every few seconds). Each prompt triggers an extremely long generation that occupies a GPU inference slot for an extended duration. As multiple such requests accumulate, they consume GPU memory (each ongoing generation maintains its full context) and lock inference slots, preventing the service from processing legitimate user requests. Because the attacker stays within normal rate limits, the service cannot easily distinguish attack traffic from legitimate usage until performance has already degraded.

Demo / Proof of Concept

▶ Watch: Results: throughput drops to zero at 10 requests per minute (10:00)

The evaluation was conducted on eight public LLM services including DeepSeek and GPT variants. In controlled experiments on a local LLM server, sending optimized prompts at a rate of 10 per minute caused system throughput to drop to effectively zero. GPU memory filled progressively as long-running generations accumulated their context windows. The attack prompts produced by ThinkTrap consistently hit the maximum output size, while all baseline approaches failed to do so.

The cost analysis revealed that the offline optimization phase requires approximately 10,000 queries, which represents a modest investment for an attacker. Once the attack prompt is discovered, using it costs nothing beyond normal query rates. Transferability testing showed that prompts optimized against one model exhibited effectiveness against others, though the researchers noted they could not conduct detailed GPU impact evaluations on commercial services due to access limitations.

Defensive Implications

▶ Watch: Defense challenges and the open problem of LLM DoS mitigation (14:00)

The researchers evaluated several potential defenses and found significant challenges:

• Anomaly detection on output patterns: Detecting when a model enters looping behavior and cutting off the generation seems promising, but ThinkTrap-generated prompts produce outputs that are not simple repetitions and cannot be reliably distinguished from legitimate long outputs
• Resource-aware scheduling: Limiting how long any single request can monopolize GPU resources can mitigate the attack but degrades quality of service for legitimate users who need long outputs, creating a tension between security and usability
• Rate limiting: Standard rate limits are ineffective because the attack operates within normal request rates; the issue is not request volume but per-request resource consumption
• Per-token pricing: Moving from subscription to per-token pricing would make the attack economically costly for the attacker, though this changes the business model
• The researchers acknowledge this represents an open research area for LLM service defense, with no comprehensive solution currently available

Key Takeaways

• LLM services are vulnerable to a new class of DoS attack that exploits the autoregressive nature of token generation to exhaust GPU resources
• ThinkTrap is the first black-box, query-efficient method for crafting prompts that force maximum-length outputs
• The attack operates within normal rate limits, making detection extremely difficult
• Approximately 10,000 queries are sufficient to optimize an attack prompt, and the prompt is reusable across sessions
• Current defenses face a fundamental tension between mitigating the attack and maintaining service quality for legitimate users
• Subscription-based LLM pricing creates an economic asymmetry that attackers can exploit; per-token pricing partially addresses this

About the Speaker(s)

Yunzhe Li (presented by Renjani from Shanghai Jiao Tong University) works on security of AI systems, specifically focusing on adversarial attacks against LLM deployments. The research explores the intersection of traditional denial-of-service attack methodology and the unique computational characteristics of large language model inference. The Q&A session included discussion of the projection dimension trade-offs and cross-model transferability of attack prompts.