Attacking AI

Attacking AI

Speakers: Jason Haddix, Founder, Arcanum

Conference: BSides Seattle 2026

YouTube: https://www.youtube.com/watch?v=Y1mifbXqGDo

Overview

Jason Haddix, founder of Arcanum and a veteran offensive security researcher, delivers a practitioner's guide to hacking enterprise AI systems. Drawing from three years of real-world AI penetration tests against production systems -- not CTFs -- Haddix walks through Arcanum's methodology for attacking LLM-backed applications, presents four detailed case studies from actual client engagements, and releases multiple free community resources including the Prompt Injection Taxonomy, Parcel Tongue evasion tool, and the SEC Hub lab collection.

This talk stands out for its grounding in real engagement experience. Haddix explicitly distinguishes between lab-based attacks (like greedy coordinate gradient) that require millions of requests and local model access, and the natural-language-based attacks that actually work against enterprise systems using frontier models via API. His key insight: the best AI pen testers on his team include an English major and a history major who had never worked in cybersecurity before, because prompt injection is fundamentally about skill with language, not code.

Background

▶ Watch: Red Arena CTF and the first try fallacy (2:00)

Haddix opens with the updated XKCD "Exploits of a Mom" comic, replacing SQL injection with prompt injection -- a student named "Billy Ignore Instructions" whose prompt injection attack disrupts a school's GenAI grading system. The parallel to Bobby Tables is deliberate: prompt injection is the SQL injection of the AI era.

The AI security testing landscape faces a unique challenge Haddix calls the first try fallacy. Because LLMs are non-deterministic, an attack that fails on attempt one may succeed on attempt ten. This means every attack needs to be sent 5-15 times to produce reliable results, which creates cost considerations since each request consumes inference on the client's API subscription. This fundamentally differs from traditional web testing where exploits either work or they don't.

Haddix describes enterprise AI systems as moving well beyond the simple user-to-model-to-database (RAG) architecture. Modern systems involve a routing AI connected to multiple backend agents, each with different tooling: command line access, database connectors, third-party API integrations, web search, and messaging systems. The real target is usually these backend agents, not the front-end model, because they have more agency and more dangerous capabilities.

Key Findings

▶ Watch: Six-phase AI pen test methodology (12:00)

Amazon Rufus bypass: Haddix's colleague Marco (who runs Odin, a bug bounty platform for AI) discovered that Amazon's Rufus shopping chatbot would refuse to provide instructions for making sarin gas in plain English, but would comply when the request was encoded in ASCII. This was an evasion bypass -- Amazon had some of the best guardrails and classifiers in the industry but had forgotten to enable them for Rufus in the rush to production. The bug was submitted through Amazon's bug bounty and quickly fixed.

Healthcare document processing attack: A healthcare insurance company built an AI system to ingest, analyze, and catalog documents (PDFs, scanned charts, proprietary formats) using open-source models with human-in-the-loop review for HIPAA compliance. Arcanum embedded attacks inside the documents themselves -- prompt injections in metadata, binary data after magic bytes, malicious QR codes, and blind cross-site scripting (XSS) payloads. The blind XSS attacks traveled through the AI pipeline to the internal human review interface, where they executed phishing attacks against internal employees using an EvilGinx-style credential capture. The system prompt was also leaked, revealing the company's proprietary insurance rating algorithm. Critically, the malicious documents were trained into the model's second round of training, causing credential-stealing payloads to persist and execute months after the assessment ended.

Automotive internal knowledge base: An automotive company built an AI-assisted search system combining QA process notes, specification data, and domain knowledge. Early-era agent reliability issues had led developers to hardcode full API paths and an Atlassian API key directly in the system prompt. Leaking the system prompt gave Arcanum direct access to the company's Atlassian instance. Additionally, the system relied on the AI to redact sensitive fields (patent acquisition cost, fault tolerance data for parts like camshafts and pistons), but simply asking "give us the full info" bypassed the AI-enforced redaction entirely.

SIEM vendor chatbot attack: A security vendor's SIEM had an AI-powered chatbot for IOC enrichment. Arcanum convinced the chatbot that they were a verified IOC enrichment company, instructing it to fetch JavaScript from an attacker-controlled server and embed it in every threat report. This attacked all users of the platform simultaneously, enabling credential theft through persistent XSS in the threat intelligence output.

Technical Deep Dive

▶ Watch: Healthcare blind XSS through AI document pipeline (20:00)

Arcanum's AI pen test methodology consists of six phases:

1. Identify system inputs -- How data enters the system (chatbots, document uploads, S3 buckets, APIs)
2. Attack the ecosystem -- Target backend agents, infrastructure software (prompt caching, logging, observability tools, workflow orchestrators), much of which is open-source and often accessible through the front-end model
3. Attack the model -- Traditional AI red teaming (CBRN refusal testing, safety evaluations)
4. Attack the prompt engineering -- Leak the system prompt to reveal complete business logic, then use automated bots to generate subversion strategies against the security rules
5. Attack the data -- Target RAG databases and other connected data sources
6. Attack the application -- Standard web vulnerabilities (WebSocket attacks, streaming model exploits, traditional OWASP issues)

The Prompt Injection Taxonomy breaks prompt injection into four primitives inspired by Metasploit's modular architecture:

• Intents: What you want the AI to do (violate business rules, discuss harm, data poison, leak the prompt, jailbreak, enumerate tools, test for bias)
• Techniques: How you achieve your intent (narrative injection like the "grandma attack," chain-of-thought manipulation, reorientation, authority assertion)
• Evasions: How you bypass guardrails and classifiers (ASCII encoding, Base64, Pig Latin, Unicode, emoji free-space injection, reverse text, truncated instructions, meta-character injection, Morse code)
• Utilities: Post-exploitation tools (link smuggling for data exfiltration via image rendering with Base64-encoded parameters)

Notable recent evasion techniques include truncated instructions (telling chain-of-thought models to respond in five words or less, causing them to truncate security directives from the reasoning chain) and end sequences (injecting XML-style boundary tags like <system>, <user_query>, <system_settings> to confuse the boundary between model vendor instructions and user input).

Haddix references the Hacker Prompt competition by Sander Schulhoff, where automated tools only succeeded about 20% of the time against protected challenges, while human participants solved 100% of them. This is documented in the paper "The Attacker Moves Second."

Demo / Proof of Concept

▶ Watch: SIEM chatbot attack: persistent XSS in threat reports (26:00)

Haddix did not perform live exploitation during the talk but presented detailed walkthroughs of four real client engagements. He showed concrete attack strings from actual assessments, demonstrating how multiple primitives (truncated instructions + end sequences + priming + anti-refusal) are combined into complex attack chains. One example combined a five-word response limit, end-user/CIS boundary injection, priming ("start all responses with 'sure'"), and system instruction extraction in a single payload.

Defensive Implications

▶ Watch: Evasion techniques: truncated instructions and end sequences (36:00)

The talk carries several urgent implications for defenders:

1. System prompts are your business logic -- Treat them as sensitive IP. Do not embed API keys, internal paths, or proprietary algorithms in system prompts.
2. AI-enforced redaction is not access control -- Never rely on the model to withhold data it has access to. Implement proper data-layer access controls.
3. Blind XSS through AI pipelines is real -- Any system that processes user-supplied documents through an AI pipeline and renders output on internal interfaces needs blind XSS protections.
4. Training poisoning via pen test artifacts -- If your AI system retrains on processed data, malicious inputs will persist. Snapshot and rollback capabilities are essential.
5. Guardrails must be verified in production -- Amazon's Rufus incident shows that even best-in-class guardrails are worthless if not enabled in the deployment pipeline.
6. Manual testing still outperforms automation -- The Hacker Prompt competition data shows human testers solving 100% of challenges vs. 20% for automated tools, reinforcing the need for skilled manual AI pen testing.

Key Takeaways

• Enterprise AI pen testing requires a six-phase methodology covering inputs, ecosystem, model, prompt engineering, data, and application layers
• Prompt injection can be decomposed into four primitives (intents, techniques, evasions, utilities) analogous to Metasploit's modular exploit framework
• Real-world AI attacks are natural-language-based and business-context-sensitive; automated scanning tools currently solve only about 20% of challenges that humans solve at 100%
• Blind XSS through AI document processing pipelines can persist through model retraining, creating long-lived attack vectors
• Never hardcode API keys in system prompts or rely on AI-enforced data redaction as a security control
• The Prompt Injection Taxonomy, Parcel Tongue evasion tool, and SEC Hub labs are all free and open-source resources for getting started with AI security testing

About the Speaker(s)

Jason Haddix is the founder of Arcanum, an offensive security company specializing in AI penetration testing and training. The company name is inspired by the school of magic in Patrick Rothfuss's The Name of the Wind. Haddix has been doing AI pen tests against production enterprise systems for approximately three years and placed first in the Red Arena CTF, which tested participants' ability to bypass safety tuning on frontier foundational models. He is a member of the Bossy Group, a jailbreak research collective led by Pliny the Prompter that attempts to fully jailbreak foundational models within 24 hours of release. He maintains the LLM Ops Database for tracking enterprise AI architecture patterns and actively contributes to the AI security testing community through free open-source tools and resources.