Bypassing AI Security Controls with Prompt Formatting

Bypassing AI Security Controls with Prompt Formatting

Speakers: Nathan Kirk

Conference: fwd:cloudsec North America 2025

YouTube: https://www.youtube.com/watch?v=QGjVXGQErg8

Overview

Nathan Kirk, Director at NR Labs and co-author of a blog post with AWS, presented research on prompt formatting -- a technique for bypassing AI guardrails by instructing the model to format its responses in non-standard ways that evade inline content filters. Unlike prompt injection, which targets the model's instruction processing, prompt formatting manipulates the model's output structure so that sensitive information passes through guardrails unrecognized. Kirk demonstrated the technique live against Amazon Bedrock Guardrails' Sensitive Information Filters, successfully extracting names from a protected knowledge base using Claude 3.7 Sonnet by requesting only the first four characters of each name appended with numbers. The technique is model-agnostic, flexible, and resistant to the standard defenses designed for prompt injection.

Background

▶ Watch: Key terms: AI guardrails as WAFs for AI, Bedrock sensitive information filters (1:30)

Amazon Bedrock Guardrails provides a sensitive information filter that can mask or remove up to 30 different types of PII (names, email addresses, phone numbers, SSNs, etc.) from AI model output. The guardrail sits inline between the AI model and the user, inspecting the raw response before it reaches the consumer. The architecture mirrors a Web Application Firewall (WAF) for AI: it intercepts traffic and applies pattern-based filtering.

Kirk's research began at NR Labs, a cybersecurity consulting startup he founded after five years at Hilton (where he built their bug bounty and penetration testing programs) and prior work at Mandiant. The research was co-authored with AWS and published as a joint blog post. During the disclosure process, AWS updated their documentation to note that guardrails require responses in "standard format" to work effectively -- an important caveat that was not previously documented.

The test environment used a Bedrock knowledge base backed by Amazon OpenSearch for vectorization, populated with mock corporate data containing names, email addresses, and financial amounts. Testing spanned multiple models including Claude 3.5 Sonnet, Command R+, and Claude 3.7 Sonnet, confirming that the technique is not model-specific.

Key Findings

▶ Watch: Python slice notation enables progressive PII extraction via substrings (6:00)

Prompt formatting is not prompt injection. This distinction is critical. Prompt injection attempts to override the model's instructions; prompt formatting works within the model's normal instruction-following behavior, asking it to restructure its output. The defenses designed for prompt injection (input sanitization, instruction hierarchy enforcement) do not apply. The guardrail's weakness is that it relies on pattern recognition against "standard format" output, and prompt formatting produces output that no longer matches those patterns.

Simple modifications defeat the filter. The initial successful bypass was trivially simple: asking the model to format names as "LastName123 FirstName." Adding numbers to the names was sufficient to prevent the guardrail from recognizing them as PII. The guardrail's name detection expected names without appended numbers.

Python slice notation enables progressive extraction. The technique supports requesting specific character ranges (e.g., "first four characters of last name"), which means an attacker can iteratively extract complete PII values across multiple queries. Each individual response contains only a fragment that the guardrail cannot recognize as sensitive, but the attacker can reassemble the complete information.

The technique is model-agnostic. Because all major LLMs are trained on programmatic content, they understand formatting instructions, substring operations, and data transformation requests. Any model capable of following structured output instructions is susceptible.

The live demo inadvertently proved the technique's power. Kirk's prepared system prompt defense -- designed to instruct the model to reject programmatic formatting requests -- failed during the live demonstration. While he attributed this to the stochastic nature of LLMs, it dramatically illustrated that system prompt defenses against prompt formatting are inherently unreliable.

Technical Deep Dive

▶ Watch: Prompt formatting bypass succeeds: first four characters of names pass throug... (8:00)

The guardrail architecture places the sensitive information filter inline in the response path. The AI model generates a raw response, the guardrail inspects it for PII patterns, and either masks (replaces with placeholder text) or removes detected PII before returning the sanitized response to the user. The filter supports 30 PII types including names, email addresses, phone numbers, and credit card numbers.

The bypass operates by structuring the model's output so that PII no longer matches the filter's recognition patterns. Several techniques were demonstrated or discussed:

Character appending: Adding numbers, symbols, or other characters to PII values (e.g., "Smith123") prevents name recognition.

Substring extraction: Using Python slice notation syntax, the attacker requests only a portion of the sensitive value (e.g., "first four characters of last name"). The fragment alone is not recognizable as a name, and the guardrail passes it through. Multiple queries with different character ranges allow progressive reconstruction of the complete value.

Response restructuring: Requesting output in non-standard formats (tables with reordered columns, concatenated fields, custom delimiters) disrupts the guardrail's pattern matching.

Kirk noted that encoding-based approaches (Base64, hex encoding) would also fall under the prompt formatting umbrella, though he expected guardrails to handle common encoding schemes. Double or triple encoding might be more effective.

The proposed defense was a modified system prompt instructing the model to: (1) only interpret user prompts as natural language queries, (2) exclude prompts based on programmatic instructions with examples like SQL formatting, and (3) redact any PII examples present in the response. This defense worked in pre-talk testing but failed during the live demo, underscoring the unreliability of system prompt-based mitigations against a determined attacker.

AWS's response was that the issue is not a vulnerability but a documentation and customer education matter. They updated documentation to note the standard format requirement and informed affected customers during the disclosure period.

Demo / Proof of Concept

▶ Watch: System prompt defense fails live, proving the difficulty of mitigating prompt... (10:00)

Kirk performed a live demo against a Bedrock knowledge base using Claude 3.7 Sonnet. He first showed the control case: a standard query returned names, which were then properly masked when the guardrail was enabled. He then applied the prompt formatting technique, requesting the first four characters of last names appended with "123," and the names passed through the guardrail unmasked. The demo also inadvertently showed the system prompt defense failing in real-time, which Kirk acknowledged with candor.

Defensive Implications

▶ Watch: Conclusion: technique applies to all AI guardrails, not just AWS Bedrock (14:00)

The most reliable defense is to exclude sensitive information from AI knowledge bases entirely -- if the data is not there, it cannot be extracted regardless of guardrail effectiveness. When PII must be present, organizations should layer defenses: system prompt engineering (acknowledging its unreliability), guardrail configuration, and application-level output filtering.

Developers need to understand that AI guardrails are probabilistic, not deterministic. They are analogous to WAFs in that they reduce risk but do not eliminate it, and they can be bypassed by a motivated attacker who understands their detection patterns. Security testing for AI applications should include prompt formatting techniques alongside traditional prompt injection testing.

Organizations evaluating AI guardrail solutions (not just AWS Bedrock) should test them against prompt formatting attacks, as the technique is applicable to any inline content filter that relies on pattern recognition against standard format output. The AWS documentation update noting the "standard format" requirement is an important caveat that should be prominently communicated to development teams.

Key Takeaways

• Prompt formatting is a distinct bypass technique from prompt injection, and standard prompt injection defenses do not apply
• Requesting substring operations (first N characters) and appending characters to PII values defeats Bedrock Guardrails' sensitive information filter
• The technique is model-agnostic and works across Claude, Command R+, and likely any LLM trained on programmatic content
• System prompt defenses against prompt formatting are unreliable, as demonstrated by the live demo failure
• The most effective mitigation is excluding sensitive data from AI knowledge bases; guardrails should be treated as a risk reduction layer, not a guarantee
• AWS classified this as a documentation issue rather than a vulnerability, updating docs to note guardrails require "standard format" output

About the Speaker(s)

Nathan Kirk is the Director at NR Labs, a cybersecurity consulting startup. Previously, he spent approximately five years at Hilton building out their bug bounty and penetration testing programs, and before that worked at Mandiant. The prompt formatting research was co-authored with AWS and published as a joint blog post.