Reinventing Agentic AI Security With Architectural Controls

Reinventing Agentic AI Security With Architectural Controls

Speaker: David Brockler III, Application Security Specialist & Head of North American AI/ML Security Practice, NCC Group

Conference: Black Hat USA 2025 — August 6-7, 2025, Mandalay Bay, Las Vegas

YouTube: https://www.youtube.com/watch?v=iLX4OdAEznY

Reading Time: ~9 minutes

Type: Briefing

TL;DR

David Brockler III of NCC Group argues that AI systems are being secured the same way the early web was secured — with heuristic guardrails as the primary defense — and that this guarantees the same outcome: persistent exploitation. Drawing on real-world penetration testing at NCC Group, he presents a framework of architectural controls including dynamic capability shifting, trust binding, trust tagging, I/O synchronization, and intent-based segmentation that enforce hard security guarantees independent of guardrail effectiveness.

Introduction

Brockler opened his Black Hat 2025 briefing with a counterfactual: imagine Tim Berners-Lee, foreseeing the security challenges of the web, had chosen a Web Application Firewall as the primary security control for the entire internet. As JavaScript arrived, as POST requests enabled state changes, as user authentication and financial transactions came online — the WAF would have remained as the first and only line of defense. Everyone in the security industry recognizes that this would have been catastrophic.

Then Brockler delivered the punch line: this is exactly the situation with AI today. "Guardrails are not security boundaries. They are statistical defense-in-depth heuristics that can decrease the risk of your application being compromised, but will never serve as a first-order security control." Behind every guardrail deployment live cross-user prompt injection, cross-plugin request forgery, data exfiltration, excessive agency, and remote code execution — vulnerabilities NCC Group encounters in virtually every AI application they test.

Brockler is an application security specialist at NCC Group who runs their North American AI and ML security practice. His talk synthesizes findings from hands-on penetration testing of dozens of AI-integrated systems to identify both where AI security fails and — critically — what architectural patterns have actually succeeded in preventing attacks in production environments.

The Problem: AI Has Broken the Trust Model

▶ Watch: How AI Inherits Trust (08:00)

Traditional web application security is built on a well-understood trust hierarchy: developers have highest trust, applications inherit that trust and operate within it, tenant admins operate within organizational scope, end users have lowest trust. After initialization, objects generally maintain consistent permission levels throughout their lifetime. This predictable, inheritance-based permission model is the foundation for access control in every mature web framework.

AI systems break this model in a fundamental way. A large language model is simultaneously consuming data from sources with radically different trust levels — developer-written system prompts, user input, application database content, third-party tool call responses — and has no native mechanism to distinguish between them. An LLM does not inherit trust from a fixed source. It is, as Brockler puts it, "inherently an agent of its inputs" — its behavior is controlled by whatever enters its context window, regardless of provenance.

This creates a property Brockler calls the taint propagation problem for AI: if an attacker can get malicious input into the application at any point in the processing pipeline — whether directly in a user prompt, embedded in a database record that gets retrieved into the context window, or returned by a third-party API — that input can potentially hijack the LLM's behavior. No amount of downstream cleansing resolves this. JSON preprocessing does not neutralize prompt injections. A watchdog model like Llama Guard checking for prompt injection is itself vulnerable to multi-order prompt injection. Chaining a second LLM to sanitize the first LLM's output merely creates a new attack surface.

NCC Group demonstrated this concretely in customer engagements: recovering Azure storage secrets, employee files, and performance improvement plans from a RAG database through an agent; recovering production machine credentials from the same deployment; exfiltrating arbitrary database documents by poisoning an administrator assistant and delivering content through the admin's own browser.

Core Control 1: Dynamic Capability Shifting

▶ Watch: Dynamic Capability Shifting Explained (10:00)

The most important control Brockler presents — and the one he emphasizes above all others — is dynamic capability shifting: manipulating the tool calls, permissions, and data access that an LLM can exercise in real time, based on the trust level of the data currently in its context window.

The core principle: an LLM's privilege level is determined at prompt time, not at deployment time. When a developer directly prompts an agent without any user-generated or application context data present, the LLM can be trusted to the same degree as the developer and may have access to all configured tool calls. When a user prompts the same agent, the reboot-server functionality should be immediately removed — the LLM's capabilities are dropped to the user's privilege level. When the agent retrieves application data (database records, user profiles) that may have been contaminated by third-party actors, all sensitive tool calls should be removed — the LLM's capabilities drop to match the lowest-trust data in its context window.

The key rule: A large language model exposed to untrusted data should never, under any circumstances, be able to read from or write to sensitive resources.

Supporting Controls: A Rapid-Fire Taxonomy

▶ Watch: Additional Mitigation Strategies (14:00)

Brockler walks through seven additional architectural controls:

Trust binding via pinning: When an LLM receives a user request and calls a backend tool, the backend must pin the user's authentication token (e.g., JWT) to the tool call. This ensures the LLM's capabilities never exceed those of the authenticated user who initiated the session — even if the LLM is instructed by injected content to act on behalf of a different user.

Client-side tool call processing: Rather than having the backend process LLM tool calls directly, route tool call requests through the user's browser. This allows the same authentication and authorization mechanisms used by the rest of the application to govern AI-driven actions automatically.

Trust tagging: Tag all data entering application stores with the trust level and source of that data. When tagged data approaches an LLM's context window, apply dynamic capability shifting in advance. If a profile bio was written by a third-party user, that data is untrusted — tag it accordingly and strip high-privilege tool calls before it enters the context.

I/O synchronization: Human-in-the-loop controls can be bypassed if an LLM can lie about the function call it intends to make. I/O synchronization ensures that what is shown to a human-in-the-loop operator is identical to what actually enters or exits the LLM context window, and that any approved action matches what the LLM actually executes.

Trust splitting: Route requests to different trust zones — a quarantined LLM with access to dangerous/external data but no sensitive tool calls, and a trusted LLM with sensitive tool call access but no exposure to untrusted data.

Trust isolation: When a low-trust LLM would need to pass data to a high-trust LLM, mask the raw low-trust output before it enters the high-trust context window. The high-trust model should never see arbitrary text from the quarantined model.

Input validation via data type gating: Define strict data types for information passing between trust contexts. A product ID, an integer, or a GUID is difficult to weaponize as a prompt injection vector. If the only data flowing from a low-trust model to a high-trust model is a product identifier (not arbitrary text), the attack surface for multi-order prompt injection collapses dramatically.

Intent-Based Segmentation: A Complete Example

▶ Watch: Secure Application Architecture Demo (22:00)

Brockler illustrates these controls together through a "disaster application" scenario: an LLM assistant that can purchase products, delete user accounts, add friends, and get the weather. A poisoned weather service response instructs the LLM to purchase items, and without architectural controls, the user is billed thousands of dollars.

With intent-based segmentation, the application deploys two models:

• Blue model (high-trust): Handles developer-level operations — purchase products, delete accounts, add friends. Never exposed to untrusted external data.
• Red model (low-trust): Handles external data retrieval — get weather, read reviews, call third-party plugins. Never has access to sensitive tool calls.

When a user asks about the weather, the high-trust model recognizes that external data retrieval is needed and delegates the request to the low-trust model. The low-trust model calls the (potentially poisoned) weather API, receives the injected response, and passes a summary back. Trust isolation masks the raw poisoned response before it reaches the high-trust context. Even if the low-trust model's summary says "buy something," it cannot call the purchase function — that tool doesn't exist in its capability set.

When a purchase request follows, the low-trust model retrieves product reviews and recommends a product by ID (a structured data type, not arbitrary text). Data type gating ensures only the product identifier crosses the trust boundary. I/O synchronization confirms that the product shown to the user in the confirmation step matches the product ID that actually appears in the purchase tool call.

Threat Modeling Strategies

▶ Watch: Threat Modeling for AI Applications (26:00)

Brockler presents three threat modeling techniques for AI systems:

Trust flow tracking: Tag data with the entity and privilege level that created it. Track how that data propagates through the application — through JSON preprocessing, through guardrails, through other models, through the user's prompt. Identify every system that ends up with untrusted context.

Source-sink matrices: Map all data sources (user input, external APIs, database records, file uploads) and all data sinks (tool calls, database writes, external API calls, rendered output) within the application. Evaluate whether any path exists for a threat actor to inject data at a source and induce an unauthorized change at a sink via LLM intermediation.

Models as threat actors: In data flow diagrams and threat models, replace every LLM with a threat actor sitting in the same position in the infrastructure. If that threat actor would be able to compromise a protected asset, there is a potential vulnerability that requires architectural mitigation.

Notable Quotes

"Guardrails are not security boundaries. They are the WAF — reducing the likelihood of an attack, but not operating as hard first-order security controls."

— David Brockler III ▶ 06:00

"A large language model that is exposed to untrusted data should never, ever be able to read from nor write to sensitive resources within your application."

— David Brockler III ▶ 14:00

"AI privileges and capabilities are determined at prompt time. When the system is prompted, the data entering the context window determines the privilege level — not the deployment configuration."

— David Brockler III ▶ 28:00

"We have developed AI backwards. We started with defense in depth and ignored the security fundamentals."

— David Brockler III ▶ 02:00

Key Takeaways

• Guardrails are statistical heuristics, not security boundaries. Organizations relying on guardrails as a primary AI security control are in the same position as organizations that relied solely on WAFs for web security.
• Dynamic capability shifting is the foundational control. LLM tool access and data permissions must be dynamically adjusted based on the trust level of all data in the context window at prompt time.
• The trust model for AI is fundamentally different from traditional software. LLMs inherit trust from their inputs dynamically; there is no fixed trust inheritance as in object-permission models.
• Multi-order prompt injection is not preventable by adding more LLMs. Routing output through a watchdog model or second LLM expands the attack surface rather than containing it; architectural separation is the only reliable solution.
• Intent-based segmentation with trust isolation — separating high-trust and low-trust LLM contexts with data type gating between them — provides the practical implementation of zero-trust principles for agentic AI applications.

Slides: No slide PDF was available for this talk.