When AI Goes Awry: Responding to AI Incidents

When AI Goes Awry: Responding to AI Incidents

Speakers: Eoin Wickens, Marta Janus

Conference: BSidesSF 2025 — April 26-27, 2025, San Francisco

YouTube: Watch on YouTube

Reading time: ~9 minutes

TL;DR

When an agentic AI system deletes a database and exfiltrates data in the middle of the night, organizations discover they have no playbook, no adequate logging, and no clear owner for the incident. Hidden Layer researchers Eoin Wickens and Marta Janus argue that AI incident response is a fundamentally different discipline from traditional IR — and that the security community is writing the playbook as it fights the fire.

Introduction

For the third consecutive year, Eoin Wickens and Marta Janus returned to BSidesSF to trace the arc of AI security from its origins. Their first talk introduced attacks on machine learning and what they mean for industry. Their second examined attacks on the AI supply chain. This year, they tackled the question that follows naturally: when something goes wrong with an AI system in production, what do you actually do?

The timing was stark. Agentic AI — systems where large language models are given tools, memory, and the autonomy to take actions on behalf of users — had exploded in the months between the pair submitting their talk abstract and standing on stage. What they had originally planned as a discussion about RAG-enabled LLMs became something broader. Agentic AI is now a business-critical reality for many organizations, and the security community's incident response capabilities haven't caught up.

Traditional IR vs. AI Incident Response

▶ Watch: Traditional IR vs. AI IR comparison (04:30)

The gap between the two disciplines runs deeper than most practitioners realize. Traditional IT incident response rests on decades of mature tooling — EDR, IDS, network security, SIEM — built for a relatively deterministic environment. Databases, web servers, and application servers have well-understood interaction patterns. Logs capture structured, analyzable artifacts. Playbooks exist because practitioners have seen the same class of incidents repeat.

AI systems break most of those assumptions. The attack vectors are fundamentally different: data poisoning, prompt injection, architectural backdoors, and deserialization exploits in exotic model formats. A malicious input is no longer a shellcode payload — it might be a PDF that says "ignore all previous instructions." AI systems are non-deterministic by design, meaning that running the same prompt twice can produce different outputs and different downstream actions. EDR tools do not recognize ML model file formats, which can run to 50 or 100 GB, and are poorly equipped to scan them for embedded malicious code.

Wickens described the situation bluntly: "We are writing the playbook as we fight the fire."

Agentic AI: A New and Amplified Attack Surface

▶ Watch: Agentic AI architecture and new attack vectors (12:00)

Janus explained the architecture driving the threat. Standalone LLMs are powerful but limited — they hallucinate, lack persistent memory, and can only respond to what is in the current context window. The solution the industry has converged on is to build systems around LLMs: give them access to tools (file systems, email, databases, APIs), long-term memory, and the autonomy to make multi-step decisions without human confirmation. That is what "agentic" means.

Two major protocols now underpin these systems. Anthropic's Model Context Protocol (MCP) standardizes how applications provide context to LLMs and how agents call tools. Google's Agent-to-Agent (A2A) protocol standardizes how multiple agents within a system communicate. Both were relatively new at the time of the talk — and both introduce significant security risks. MCP tool descriptions visible to users are brief; the full descriptions accessible to the AI model can contain malicious instructions invisible to the human approving an action. A2A inter-agent communication creates pathways for a malicious prompt injected into one agent to cascade across an entire multi-agent architecture.

A Hidden Layer proof of concept demonstrated the risk concretely. Researchers created a PDF containing the bash command rm -rf / — wiped to look like benign setup instructions — and presented it to Claude's computer use capability. After a few obfuscation techniques (base64 encoding, ROT13) and a paragraph of social engineering text asserting the command was safe, the model executed it.

The indirect prompt injection attacks Janus described are particularly insidious because the attacker does not need access to the LLM directly. They only need to place a malicious prompt inside a resource the agentic system will encounter — an email, a document, a web page. Google's Gemini for Workspace demonstrated this: researchers embedded an invisible prompt in an email that, when a user said a specific keyword, caused Gemini to display a fake phishing alert with a malicious link.

Key Difficulties in Triaging an AI Incident

▶ Watch: IR difficulties and the blame game (22:00)

To drive the challenge home, Wickens presented a scenario: an agentic system with MCP access to company data — 250% more efficient than its human counterpart — fires at 3:00 AM. A database has been deleted. Data has been exfiltrated. The DB log says the agentic solution did it. Root cause analysis comes back incomplete. Why?

The answer is a convergence of three gaps:

Lack of tooling. As Claude 3.7 put it when the team asked for its perspective: "In AI incident response, we are asked to solve tomorrow's problems with yesterday's solutions." Traditional IDS cannot detect data poisoning or prompt injection. EDRs have little to no capability for detecting architectural backdoors in ML models. The IR tooling that exists was designed for a different class of threats.

Lack of visibility. Organizations often lack a complete inventory of their AI infrastructure. MCP servers get stood up unregistered. Agentic capabilities get deployed that the security team is unaware of. Shadow AI proliferates — models downloaded from Hugging Face by individual engineers that nobody in a leadership position knows exist.

Lack of clear ownership. A survey cited by Janus showed that most businesses are actively debating internally who is responsible for AI security. Is it the security team? The machine learning engineers? The product team that deployed the agentic solution? This "AI blame game" — a game nobody wants to win — leaves incidents without a clear incident commander. Machine learning engineers may understand the models but have limited security backgrounds; security teams have the IR experience but lack ML depth. Both are necessary.

Logging, Monitoring, and What Needs to Change

▶ Watch: Logging and monitoring recommendations (34:00)

Wickens laid out a logging framework built around what will actually be useful when an incident occurs. Traditional logs — IP addresses, user agents, session context, temporal metadata — remain necessary but insufficient. AI-specific logging requirements include:

• Prompt input/output pairs in context, not isolated entries. Matching random outputs to random inputs destroys the ability to reconstruct a conversation chain.
• Modifications to AI deployments: model artifacts, inference code, hyperparameters. Any of these can be a supply chain attack vector.
• Data versioning for RAG systems. Data is the new attack vector; knowing what went into and out of a retrieval-augmented generation database is essential for tracing a poisoning incident.
• MCP server modifications. An attacker who compromises an MCP server can subvert tool behavior entirely; changes must be tracked.
• A2A and inter-agent communications. If agents are having conversations and the content is not logged, the blast radius of a prompt injection that enters one agent and propagates to others cannot be assessed.

On governance, Wickens argued that every organization deploying agentic AI needs to define clear ownership — who is responsible for the model's performance, integrity, and incident escalation. Business-critical agentic systems need formal IR plans with escalation paths to every team they touch. Because an agentic solution may reach across multiple business units, a single failure can produce cascading effects that touch finance, operations, and customer data simultaneously.

Failover planning is also non-negotiable. Agentic systems should not be single points of failure for critical processes. For high-risk actions — anything that could produce material harm if executed incorrectly — a human-in-the-loop check should be required, even though the defining characteristic of "agentic" is that it operates without constant human confirmation.

The talk closed with a call for organizations to maintain an AI inventory: a registry of every model deployed, their capabilities (tool calling, database access, external API connections), and their access scope. The AI supply chain — data, models, tooling, and infrastructure — must be tracked and documented. And from a secure architecture standpoint, agentic systems should follow the principle of least privilege, have strict memory access controls, and treat all externally sourced data as untrusted.

Notable Quotes

"We are writing the playbook as we fight the fire." — Eoin Wickens (06:00)

"We blindly trust AI models to perform actions on our behalf and ultimately decide our fate." — Eoin Wickens (18:00)

"In AI incident response, we are asked to solve tomorrow's problems with yesterday's solutions." — Claude 3.7, quoted by Marta Janus (23:00)

Key Takeaways

• AI incident response is a new discipline. Traditional IR tools, playbooks, and governance models do not map cleanly onto AI systems; organizations need AI-specific IR plans before incidents occur.
• Agentic AI dramatically amplifies prompt injection risk. An attacker who can place a malicious prompt in any resource an agentic system might read — an email, PDF, or web page — can trigger autonomous, potentially destructive actions without ever touching the model directly.
• MCP and A2A protocols introduce significant unresolved security issues. Tool descriptions visible only to the AI model can contain hidden malicious instructions; inter-agent communications can propagate a compromise across an entire multi-agent system.
• Logging must be rebuilt for AI. Prompt input/output pairs, RAG data versioning, MCP server modifications, and inter-agent communications must all be captured to enable any meaningful root cause analysis.
• Ownership must be defined before an incident. Cross-functional teams with expertise in both machine learning and security are necessary; organizations that leave AI IR ownership ambiguous will lose time to the blame game when it matters most.