AI Security with Guarantees

AI Security with Guarantees

Speaker: Ilia Shumailov (CEO, AI Security Company)

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=NU6l0Qcf5rU

Reading time: ~10 minutes

TL;DR

Ilia Shumailov, a former Oxford academic turned AI security startup CEO, argues that the industry's cat-and-mouse approach to AI security is structurally broken — and that formal guarantees are achievable today for a significant fraction of agentic tasks. The mechanism: separating instruction flow from data flow so that untrusted data can never redirect what an agent does, only influence what it returns.

Introduction

"I'm an academic. I overcomplicate systems." Ilia Shumailov opened his [un]prompted talk with characteristic self-awareness, then spent the next 25 minutes making a deeply uncomfortable argument: the entire field of AI security is stuck in a loop that nobody has a principled way to escape.

Shumailov's credentials place him at an unusual intersection. He was one of the earliest researchers to systematically study prompt injection and adversarial robustness in large language models, publishing work that predates most of the industry's current awareness of these problems. He now runs a company commercializing defenses built on the research he's spent years developing.

His central claim is that the current state — building defenses, watching attackers break them, releasing new defenses, repeat — is not just inefficient but mathematically destined to continue unless the security community changes its approach entirely. And crucially, he argues there is a better path that the industry already understands theoretically but consistently fails to implement.

The Vicious Cycle: Why Current Defenses Don't Actually Work

▶ Watch: The broken state of AI robustness (02:00)

Shumailov began with a show of hands: less than half the [un]prompted audience believed that AI models would get better at solving their own security problems as capabilities scaled. He wasn't surprised.

The academic literature on this question is contradictory. An OpenAI paper argues that chain-of-thought reasoning improves robustness. Competing academic work shows that the improvement is an artifact of using weak attacks against stronger models — the model isn't more secure, the attacks just fail for a different reason. "If you ask the most informed individuals studying these problems for decades," Shumailov said, "nobody has an answer."

This creates a measurement problem that propagates through the entire field. Defenders don't have good attacks because they don't have good defenses. Attackers don't have good defenses to test against. The loop is self-sustaining. "We don't have good attacks because we don't have good defenses because we don't have good attacks."

▶ Watch: The failure mode of unprincipled defenses (06:01)

He illustrated this with his own research. His team published a paper evaluating every major prompt injection defense at the time, showing that all of them failed under sufficiently capable attacks. Within a month, they published a new defense that outperformed everything in the paper. Within another month, they expected it would be broken too.

"I would actually call this a bit of a failure, because it's yet again a sign that the attacks we've built are not working for our current thing." The new defense appearing stronger, he argued, simply means attackers haven't tried hard enough yet. The cycle will continue indefinitely unless something structural changes.

What Formal Guarantees Actually Mean

▶ Watch: Introducing CAMEL and control flow integrity (08:01)

The escape from the cat-and-mouse dynamic, Shumailov argued, requires building systems where security utility scales with model capability — where a better model is also a more secure model, not just a harder target that requires more sophisticated attacks.

For a significant fraction of prompt injection scenarios, this is achievable. The mechanism his team developed — called CAMEL — implements what he described as an application of the dual-LLM pattern (Simon Willison is credited with coining the term, though Shumailov noted their work predated it). The core idea: separate the instruction flow from the data flow.

In a standard agentic architecture, instructions from the user and untrusted data from the environment both flow through the same model. An adversary who controls external data — a malicious email, a poisoned webpage, an injected database record — can include instructions that redirect the model's behavior. This is indirect prompt injection, and it is nearly impossible to defend against in a mixed-flow architecture.

CAMEL separates these paths completely. A planning model handles instructions and generates a fixed execution plan. A separate "query LLM" (QLM) processes untrusted data and converts it into structured output that feeds into the plan — but cannot modify the plan itself.

The guarantee this provides is control flow integrity: an adversary who controls the untrusted data cannot force the agent to take actions outside its predefined execution paths. "It's impossible for this thing to now go and drop your tables or anything," Shumailov said. "It's fixed in its execution flows."

Task-Data Independence: Which Problems Can Be Solved with Guarantees

▶ Watch: Task-data independence and the Rubik's Cube scenario (12:01)

Not every task is amenable to this approach. Shumailov introduced the concept of task-data independence to characterize when guarantees are achievable: if the task can be solved without ever showing the model the untrusted data, the data has no opportunity to redirect control flow.

He used computer use agents as a concrete example. These are agents that interact with a computer by issuing click and keyboard commands — a domain where security intuitions break down completely because inputs are screenshots and actions are mouse coordinates. But it turns out that for the majority of tasks on standard computer use benchmarks, the model doesn't actually need to see live data to produce a valid execution plan.

"Can you plan ordering a PS5 controller on Amazon without ever interacting with the internet?" he asked. The answer, it turns out, is yes — modern models know the steps well enough to produce a 3,000-to-4,000-line execution plan upfront, with conditional branches that handle expected variations. "For most of the tasks you find on computer use benchmarks, you can plan these things out fully. You end up with plans about three to four thousand lines of code, and then they run."

This approach achieves task-data separation for a large fraction of real-world agentic tasks. The model never needs to process untrusted data interactively because it already knows what to do.

▶ Watch: What happens when task-data separation breaks down (14:01)

The approach has natural limits. Tasks like "find my boss's email and extract the action items" are inherently data-dependent — the plan cannot be written without seeing the data. For these, Shumailov argued, the best available options are static analysis, dynamic analysis, and sandboxing. The guarantees that CAMEL provides simply cannot extend to tasks where resolving data dependence is the task itself.

Cookie Prompt Attacks: Data Flow Manipulation Even with CFI

▶ Watch: The cookie prompt attack in computer use agents (18:01)

Control flow integrity doesn't eliminate all attacks — it restricts their blast radius. Shumailov described a class of attacks his team discovered in computer use settings that he clearly found both alarming and slightly amusing.

Modern models, when given web browsing tasks, have learned to handle cookie consent prompts — the GDPR popups ubiquitous across European websites. They build cookie-prompt-detection logic into essentially every web plan they generate. "The first thing in every single plan that the agent produces is, is there a cookie prompt?"

The attack exploits this predictable behavior. By embedding fake cookie prompts inside web advertisements — the kind that appear on any website — attackers can reliably capture agent actions even when the agent never directly views the injected content. The agent expects to encounter cookie prompts and has pre-committed control flow to handle them. An adversary who knows the plan structure can exploit that structure without ever breaking into the planning layer.

"Even though the model never sees the content directly, the pre-planned action to handle a cookie prompt becomes an attack surface," Shumailov explained. The defense, he argued, has to come from outside the model: agents need to verify that clicking a cookie prompt keeps them on the same domain, for instance — environmental checks that the agent integrates with its execution context.

The Adoption Gap: Why Known Solutions Aren't Being Deployed

One of Shumailov's most pointed observations was not about the technical problems but about the industry's response to published solutions. He noted that his team's CAMEL paper — describing a mechanism with formal security guarantees — has been widely discussed as a de facto standard for thinking about agentic security. Almost no one is actually implementing it.

"I genuinely don't know why people don't do it," he said. His explanation, offered partly from the vantage point of now running a commercial operation: building the first prototype is trivial. Building a production system requires methodological innovation inside the model itself, not just architectural changes around it. That requires the rare combination of ML expertise and security expertise that is hard to find and hard to retain.

His company's current system, he noted, is "significantly better" than the published paper and can be adopted with a one-line code change. The security properties scale with model capability — as the underlying model improves, the planning quality improves, and security performance improves with it.

▶ Watch: The path forward and Q&A (22:01)

Notable Quotes

"We don't have good attacks because we don't have good defenses. And this vicious loop is obviously quite horrible for every single individual involved."

— Ilia Shumailov (06:01)

"The best case for us is if we somehow manage to build mechanisms that allow us to build systems where your security utility scales with the utility of the model."

— Ilia Shumailov (08:01)

"If the task that you're actually asking the model can be solved without ever seeing the data, why even show it the data?"

— Ilia Shumailov (12:01)

"I have no idea why people don't do it. I genuinely don't."

— Ilia Shumailov, on the adoption gap for formally guaranteed defenses (20:01)

Key Takeaways

• The cat-and-mouse cycle is structural, not tactical. Unprincipled defenses without formal guarantees will always eventually be broken; the industry needs a methodological shift, not just better signatures.
• Control flow integrity is achievable for agentic AI. Separating instruction flow from data flow prevents untrusted data from redirecting agent actions, providing a formal guarantee against indirect prompt injection.
• Task-data independence determines what can be secured. If a task can be planned without interacting with untrusted data, formal guarantees are achievable. If not, sandboxing and analysis are the best available options.
• Pre-planned agents eliminate the largest attack surface. For the majority of computer use and agentic tasks, models can generate complete execution plans in advance — preventing runtime data injection entirely.
• Environmental integration is necessary for residual attacks. Data flow attacks that exploit predictable agent behaviors (like cookie-prompt handling) require the agent to have semantic understanding of its execution environment, not just its task.