Glass-Box Security: Operationalizing Mechanistic Interpretability for Defending AI Agents

Glass-Box Security: Operationalizing Mechanistic Interpretability for Defending AI Agents

Speaker: Carl Hurd, Co-Founder & CTO, Starseer

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 3–4, 2026, San Francisco

Watch on YouTube: https://www.youtube.com/watch?v=JZlaijmG-Ng

Reading time: ~9 minutes

TL;DR

Current AI security tools — prompt firewalls and host-based monitors — can only inspect what a model says, not what it thinks. Carl Hurd of Starseer argues that true AI agent defense requires peering inside the model's activation layers using mechanistic interpretability, measuring both the direction and strength of dangerous concepts before they become dangerous actions.

Introduction

When a company installs a firewall for a network, it inspects packets — the actual content crossing the wire. But what if the most important security signal were locked inside the network device itself, invisible to any external observer? That's roughly the problem facing AI security today.

Carl Hurd, Co-Founder and CTO of Starseer, brought a deeply technical but urgently practical perspective to [un]prompted 2026. His background spans Idaho National Laboratory, seven years doing public zero-day research at Cisco Talos (where his team disclosed roughly 150 CVEs), a stint at Red Balloon Security on DARPA's PROVERS formal methods program, and most recently co-founding Starseer to build next-generation AI security capabilities. His talk, "Glass-Box Security," challenges the field to stop treating AI models as black boxes and start treating them like any other monitored computing system — with hooks, telemetry, and behavior-based detection rules.

▶ Watch: Introduction and Speaker Background (00:00)

The Limits of Black-Box Security

Today's AI security products fall into two broad camps. Host-based solutions run on the inference client machine, typically using eBPF or ETW to observe what flows in and out. Network-based solutions sit as a gateway or proxy, performing break-and-inspect on traffic. Both approaches share a critical limitation: they only see the prompt as plain text.

Hurd highlighted a counterintuitive fact about how prompts actually work. "Prompts are only plain text very briefly in their lifespan," he explained. The moment a prompt enters a model, it gets tokenized and converted into numerical tensors. Even a small model like GPT-OSS-20B has more than 24 layers, and the prompt spends the vast majority of its processing life as high-dimensional numerical activations — not readable text. Existing black-box solutions are, in his framing, "building detection on encrypted payloads."

This matters because it means regex, input sanitization, and LLM-as-a-judge approaches are fundamentally constrained. They can catch obvious patterns but cannot reason about the semantic intent being encoded inside the model's internal representations during generation.

▶ Watch: The Black-Box Security Problem (04:00)

The Glass-Box Framework: Intent and Strength

Hurd's proposed alternative rests on two pillars, both borrowed from the academic field of mechanistic interpretability.

Pillar 1: Intent via Mechanistic Interpretability

Mechanistic interpretability (MI) is the study of what happens inside neural networks during inference. Researchers at Anthropic, Google DeepMind, and others have published extensively on techniques including linear probes, sparse autoencoders (SAEs), and differential prompt analysis. All of these methods depend on one capability: the ability to hook the forward pass of a model and collect activation data.

Hurd's approach uses cosine similarity against known "intent vectors" to determine whether a dangerous concept is present in the model's internal state at any given layer. If the model is processing a prompt like "How do I rob a bank?", the activations at certain layers will align directionally with an "illegal" intent vector — regardless of how the prompt is worded on the surface.

Pillar 2: Strength via Latent Space Geometry

Cosine similarity alone only tells you if two vectors point in the same direction. It doesn't tell you how dominant that intent is relative to everything else the model is processing. For that, Hurd uses scalar projection — specifically, the dot product — to measure the magnitude of the intent's "shadow" within the activation tensor.

The analogy he offered: distinguishing "How do I steal from a bank?" from "How do I steal a pen from a bank?" Both prompts involve the concept of theft, but the magnitude of the illegal intent is dramatically different. Magnitude-based thresholding allows detection rules to make that distinction.

▶ Watch: Capturing Intent and Strength (06:00)

Building Behavior-Based Detections for Neural Networks

By combining intent direction and intent magnitude, the glass-box framework enables what Hurd calls "semantic tripwires" — detection rules that fire on conceptual content rather than syntactic patterns. This mirrors the evolution of endpoint security over the past decade.

"This is state-of-the-art in cybersecurity," Hurd said. "When we talk about EDRs and detections that exist today, we're not using only signature-based detections. We are doing behavior-based analysis. And so we need to move to this for AI and LLMs as well."

The practical expression of this is a YARA-style rule extended with an interpretability module. His slide deck included a concrete example:

The rule checks layers 15 through 24 of a monitored model for the file-deletion intent concept, triggering if that intent's projection exceeds 0.85 of the tensor magnitude. This gives detection engineers a familiar, extensible format without requiring deep expertise in high-dimensional math.

▶ Watch: Behavior-Based Detections and Semantic Tripwires (14:00)

The Engineering Challenges

Hurd was direct about the significant obstacles standing between this vision and operational reality.

Activations aren't available from frontier models. Cloud-hosted APIs from OpenAI, Anthropic, and Google do not expose internal activations. Organizations that want glass-box security must run models on infrastructure they control. His term for this: "sovereign infrastructure."

The data volume is enormous. Using napkin math, he calculated that GPT-OSS-20B — a relatively small model — generates approximately 4 megabytes of activation data for just the first token. Fill up an entire 128K context window, and you've generated over 10 terabytes of BF16 activation data. His engineering solution: surgical activation monitoring that targets only the residual stream (avoiding the quadratic scaling costs of self-attention heads) and only monitors layers empirically shown to be relevant to specific detections.

Detection content cannot be universal. Different agents have different permitted behaviors. A security operations agent should be allowed to discuss malware — a customer service bot should not. Hurd's answer is to adapt existing open-source standards (YARA and Cedar) rather than invent new ones, and to make detection content easily packageable so organizations can customize and distribute rules appropriate to their specific agent use cases.

▶ Watch: Engineering Challenges and Solutions (16:00)

Why AI Agents Make This Urgent

The stakes escalate significantly with autonomous agents. A prompt firewall can stop a user from sending a malicious message — but an autonomous agent operating in an OODA (Observe, Orient, Decide, Act) loop can probe for and exploit vulnerabilities in ways that no static syntactic rule could anticipate.

Hurd offered a concrete example: CTF problems demonstrating how packet capture software can be leveraged to execute other binaries with elevated permissions. "These are the types of things that these agents will try," he warned, "because that knowledge is readily available."

Beyond security, the glass-box approach also addresses AI governance and traceability. Being able to prove that a model did take specific concepts into account — or deliberately didn't — is essential for accountability in agentic systems. This is what Hurd calls "semantic traceability versus syntactic traceability."

▶ Watch: AI Agents and the Amplified Risk (20:01)

Q&A Highlights

During the Q&A session, Hurd addressed several sharp follow-up questions:

On indirect prompt injection detection: Could the technique detect when a RAG pipeline shifts the model's intent? Hurd said yes — for context-enhancement RAG, a single hooked forward pass would capture any intent shift. For tool-based RAG, you'd need separate hooks around each tool call.

On fine-tuning drift: If detection manifolds are built on a base model and that model is then fine-tuned, do the detections need rebuilding? Only if the specific monitored layers change. If those layers are frozen during fine-tuning, existing detections remain valid.

On relative vs. absolute scoring: A questioner noted that LLM confidence scores are notoriously unreliable as absolute numbers. Hurd agreed — the approach is inherently relative and model-specific. Intents are captured empirically from a specific model (e.g., Llama 3.1) using thousands of labeled prompts, and thresholds are set comparatively within that model's latent space.

▶ Watch: Q&A Discussion (22:01)

Notable Quotes

"We must move from inspecting what the model says to monitoring what the model thinks."

"You can block prompts at a firewall, but can you stop a determined agent?"

"Sovereign infrastructure is a requirement for anyone that wants to take secure model usage seriously."

"Bridging the gap is an industry-wide challenge. We need to make sure that anybody who's aware of detection engineering standards will be able to contribute effectively at the bleeding edge."

Key Takeaways

• Black-box AI security is structurally limited. Prompts spend only a brief moment as readable text; all the meaningful processing happens in numerical activation space that external tools cannot see.
• Glass-box security has two pillars: Intent (directional cosine similarity in latent space) and Strength (scalar projection / dot product magnitude). Together they enable behavior-based detection analogous to modern EDR.
• Sovereign infrastructure is a prerequisite. Frontier cloud models don't expose activations. Organizations serious about this approach must self-host or use models they can instrument.
• Extend existing standards rather than reinvent them. YARA and Cedar provide familiar, widely understood frameworks that can be extended with interpretability modules, lowering the expertise barrier for detection engineers.
• AI agents amplify every challenge. An autonomous agent operating in an OODA loop can find creative paths around syntactic guardrails. Semantic tripwires that detect intent — regardless of how it is expressed — are the path forward.

Slides Reference

Slides available: 2026-04-04-D1-S2-15-10-Glass-Box-Security-Operationalizing-Me.pdf

Key slide topics include: Black-box vs. glass-box security architecture diagram; the two-pillar framework (Intent + Strength); standard prompt processing lifecycle showing the brief plain-text window; capturing intent with mechanistic interpretability (linear probes, SAEs, differential prompt analysis); capturing strength with latent space geometry and contextual thresholding; the YARA-extended rule example for file-deletion intent detection; and the four conclusion points on sovereign infrastructure and industry-wide bridging.