AI Interpretability as a Security Control: Introducing the CIRCUIT Framework

AI Interpretability as a Security Control: Introducing the CIRCUIT Framework

Speakers: Eric Zielinski, Chief Information Security Officer, Jumpmind

Conference: FIRSTCON

YouTube: https://www.youtube.com/watch?v=VqTt1kEBLG8

Overview

In an era defined by the rapid proliferation of Artificial Intelligence, the security community faces a critical challenge: governing systems whose internal workings remain largely opaque. Eric Zielinski, CISO at Jumpmind, addressed this fundamental gap at FIRSTCON, unveiling the CIRCUIT Framework (Circuit Informed Risk and Control Understanding Inventory and Transparency). This open standard is designed to bridge the chasm between AI deployment and our ability to understand, inspect, and audit these increasingly complex models. Zielinski argues that current AI governance paradigms, which treat models as "black boxes," are inherently insufficient and expose organizations to significant, unquantifiable risk.

The talk introduces CIRCUIT as a community-owned, free framework aimed at operationalizing AI interpretability as a security control. By shifting focus from mere input/output monitoring to understanding a model's internal decision-making processes, CIRCUIT offers a quantitative method to assess and manage AI risk. This initiative is particularly timely given escalating regulatory pressures and the emergence of sophisticated AI-specific attacks that exploit internal vulnerabilities, rendering traditional perimeter defenses ineffective. Zielinski's presentation not only highlights the urgency of this problem but also provides a concrete, actionable solution, releasing the framework, white paper, and reference code publicly to foster community adoption and contribution.

Background

▶ Watch: Introduction: The gap in AI understanding and CIRCUIT (0:00)

For the past several years, the cybersecurity industry has grappled with the implications of deploying AI systems without adequate visibility into their internal operations. Organizations are frequently asked to deploy AI they cannot explain, defend AI they cannot inspect, and trust AI they cannot audit. This reality creates a vast liability surface, as evidenced by incidents like Anthropic pulling a powerful model overnight due to government orders, just three days after its release. The industry's prevailing approach has been to build extensive governance processes around the black box, focusing on external controls.

Zielinski meticulously detailed the array of existing controls, such as API gateways, CASB proxies, shadow AI discovery, audit logging, DLP, prompt injection detection, and human review gates. While valuable, he emphasized that every one of these controls treats the AI model as an opaque entity. They monitor inputs, outputs, and surrounding traffic, but none provide insight into the actual computational process that produces a decision. This leaves critical questions unanswered, particularly concerning explainability under regulation, the integrity of detection and triage (e.g., when a SIM misranks an alert), data poisoning detection (which can shift internal representations without visible output changes), hallucinations, embedded vendor AI (where the vendor owns weights but the client owns blast radius), and the actions of autonomous agents that lack native internal audit logs. All these challenges, Zielinski asserts, boil down to the same fundamental problem: the lack of evidence regarding what happens inside the model.

Key Findings

▶ Watch: Interpretability: no longer research fantasy, now a security control (8:10)

A pivotal finding presented in the talk is the assertion that AI interpretability is no longer a "research fantasy" but a deployable security control. Zielinski highlighted a paradigm shift over the last 18 months, moving interpretability from an academic curiosity to an operational capability. This shift is underpinned by several groundbreaking developments:

1. Anthropic's "Sleeper Agents" (2024): Research demonstrated models with hidden backdoors that bypassed safety training but were detected by reading their internal activations. This proved the feasibility of detecting invisible behavioral flaws from the inside.
2. Claude 3.5 Hiku Circuit Tracing: Anthropic successfully traced and open-sourced tools for documenting multi-step reasoning, safety refusal circuits, and tool-use gating within a commercial model. This marked the first time a commercial model's internal computational graph was publicly documented.
3. OpenAI's Sparse Training (November 2025): OpenAI showed that training models with sparsity inherently reduces the size of internal circuits by an order of magnitude, making them small enough for human experts to read and verify. This addressed the computational impossibility objection.
4. DeepMind's Goscope 2 (December 2025): The release of a complete set of sparse autoencoders across every layer of an open model family. This allows for feature-level monitoring without requiring organizations to build their own tools, akin to sorting an unlabeled warehouse into labeled, real-time-monitoring shelves.
5. Commercial Platforms: The emergence of companies like Goodfire, Neuropedia, and Transluc, which offer interpretability as a managed service, further democratizing access to these capabilities.
6. Anthropic's Fable 5 and Mythos 5 System Cards: These documents confirm interpretability running as a live production security control. Internal probes screen all traffic for dishonesty and deception, escalating flags to a classifier that can block outputs – a preventative control moved inside the model.

These advancements collectively demonstrate that the ability to "open the box" and understand a model's internal decision-making is now within reach, transforming AI interpretability into an essential component of modern cybersecurity.

Technical Deep Dive

▶ Watch: How to read inside AI models: mechanistic interpretability (11:20)

The core of the shift towards operational AI interpretability lies in three key concepts: mechanistic interpretability, sparse autoencoders, and circuit sparsity.

Mechanistic Interpretability is the process of reverse-engineering the actual internal mechanisms of a neural network. Instead of merely observing inputs and outputs, this approach involves tracing the specific computational path a model takes to arrive at a decision, much like diagnosing a circuit board. This allows security professionals to understand why a model made a particular choice, rather than just what the choice was.

Sparse Autoencoders address the challenge of densely packed, overlapping concepts within a model's internal space, which often makes it appear as uninterpretable noise. A sparse autoencoder disentangles these concepts, sorting them into discrete, labeled "shelves." For example, one shelf might activate for "fraud detection," another for "safety refusal," and another for "tool use." By observing which of these labeled concepts activate in real-time as the model processes information, analysts can gain granular insight into its internal reasoning. DeepMind's Goscope 2 is a prime example of this technology, providing pre-trained sparse autoencoders for various open models.

Circuit Sparsity is a training technique where a model is designed such that only a tiny fraction of its internal connections are active for any given task. This drastically reduces the complexity of the "circuit" responsible for a specific decision, making it small enough to be understood and read by a human expert. Zielinski provided an example of a small model where the entire mechanism for completing a piece of text involved only eight parts, fitting "on a napkin." While scaling this to the largest models is still an active research area, the principle is proven: complexity can be managed for interpretability.

Zielinski framed the necessity of interpretability within the context of defense in depth. Traditional security controls form "green" layers around the model, watching inputs, outputs, and traffic. However, the innermost layer—the model's internal decision circuits—remains unprotected. Interpretability is the only control that can reach this core, providing visibility into the actual computation that produces a decision.

The urgency for this internal visibility is further underscored by impending regulatory mandates. The European Union's AI Act, with an enforcement date for high-risk obligations in August 2026, explicitly demands evidence of transparency and human oversight, moving beyond mere policy statements. The NIST AI Risk Management Framework names interpretability methods directly. SR11-7, governing banking model risk, requires proof of conceptual soundness. ISO 420001 is appearing in procurement requirements, and SOC 2 auditors are beginning to ask AI-specific questions. Newer standards like AIU-1 are also emerging, all converging on the need for tangible evidence of AI assurance.

To address this, Zielinski introduced the CIRCUIT Framework, designed to provide a quantifiable "meter" for AI risk where existing frameworks only offer categories. At its heart is a simple formula for the Circuit Risk Score (CRS):

CRS = Risk Tier (6 - Interpretability Maturity Score) Decision Consequence Weight

This multiplicative formula ensures that a catastrophic answer to any of the three core factors cannot be hidden by good scores on the others. These three factors represent the fundamental failure modes in AI deployments:

1. Risk Tier (1-4): How dangerous the use case is if something goes wrong.
2. Interpretability Maturity Score (IMS) (0-5): A measure of how much visibility an organization has into the model's internals, based on named evidence.

• 0 (Black Box): No internal visibility.
• 1 (Input/Output Logging): Camera on a locked door.
• 2 (After-the-Act Explanation): Forensic report on a single decision.
• 3 (Real-time Named Concepts): Windows into the room, seeing features if the vendor publishes them.
• 4 (Traced Decision Circuits): Wiring is mapped.
• 5 (Automatic Circuit Re-check): Continuous assurance on every update.

The IMS is capped by access: 5 for self-hosted models with raw weights, 3 for API-accessed models, and 2 for embedded vendor AI.

1. Decision Consequence Weight (1-4): How far a single bad decision travels before human intervention, from "purely advisory" to "catastrophic and irreversible."

The CRS translates into four risk bands: Green (deploy), Amber (watch closely, quarterly review), Red (executive sign-off, mandatory controls, 180-day remediation clock), and Purple (not deployable). A crucial "floor" rule dictates that any model making irreversible or catastrophic decisions at high or critical risk tiers remains Red, regardless of its interpretability score. Understanding, Zielinski stressed, is a precondition for control, not a substitute.

The framework is further supported by 10 rules that dictate how AI systems should be governed (e.g., log every system, prove claims with evidence, let the score decide approval, demand transparency from vendors, no high-stakes autonomous decisions for uninspectable models). To measure the program's effectiveness, CIRCUIT proposes seven metrics for board-level reporting, including "AI attack surface undergovernance," "high-risk AI exposure," and "vendor transparency response rate," providing concrete numbers instead of ambiguous categories.

Demo / Proof of Concept

▶ Watch: Ten core rules and seven board-readable metrics (21:00)

While the talk did not feature a live coding demo, Zielinski presented a compelling "proof of concept" through a real-world threat and a practical tool for driving transparency.

He highlighted CrossMPI, a live threat published just months prior to the conference. This attack demonstrates how an adversary can hijack a model's behavior by changing only an image within a prompt, never the text. CrossMPI succeeds in roughly two-thirds of attempts, transfers across six different vision-language models, and is completely invisible to human reviewers. The attack targets the internal layers where vision and text are merged, meaning the model's output looks completely reasonable to an external observer. This critical example underscores the inadequacy of surface-level monitoring and unequivocally demonstrates why internal interpretability is no longer optional but essential for defense.

To operationalize transparency, Zielinski introduced the "show me your circuits" vendor questionnaire. This 29-question document, written in plain language for procurement teams, directly probes vendors on their AI models' internal workings and governance. Questions include: "Please tell us exactly which AI model serves our account, including the model name, the version, and the approximate size," "Have you ever examined how the model works on the inside rather than only testing its outputs? If so, what reports or documentation came out of that work?", and "Please describe the most recent exercise in which a team deliberately attacked this model to find a weakness (red teaming). What was tested, how? And what were the main findings?"

The results from this questionnaire reveal a stark transparency gap: self-hosted models can answer about 93% of the questions, API-accessed models about 50%, and embedded vendor models only about 14%. This data visually represents the "black box" problem. Zielinski emphasized that collective action—hundreds of organizations sending this questionnaire to their vendors—can create a significant market force, compelling greater transparency across the AI supply chain.

Defensive Implications

▶ Watch: Real-world threat (Crossmpi) and vendor questionnaire (24:00)

The CIRCUIT framework offers a clear, actionable path for defenders to integrate AI interpretability into their security posture. Adoption is envisioned as a maturity path, not a single, monolithic project:

1. Stage One (Foundation): Start by inventorying every AI system, assigning a risk tier, computing the Circuit Risk Score in a spreadsheet, and sending the "show me your circuits" vendor questionnaire. This initial stage requires no new tooling and immediately surfaces the most dangerous, previously unknown configurations.
2. Stage Two through Four: Progress from a snapshot to a managed registry, then to a deployment gate (nothing ships unsigned), and finally to continuous assurance with automatic re-checks.

Zielinski stressed that this is not a five-year research program but a pragmatic, immediately deployable solution. Realistic staffing involves one sponsor, a program manager, and perhaps one engineer working a few hours a week. The necessary tools already exist: open-source libraries for self-hosted models, managed services for those preferring to buy rather than build, and public databases of named features for vendor models.

The process of "reading a decision" involves three steps: finding the relevant features (using sparse autoencoders), tracing the circuit back from the output, and then testing it by disabling a part and confirming the predicted behavior change. This last step is crucial, transforming interpretability from guesswork into causal, verifiable reverse engineering. The goal is not perfect understanding, which even the largest AI labs lack, but "simply more than you have today in a form that a board and a regulator can read."

By adopting CIRCUIT, organizations can move from a reactive, opaque stance to a proactive, evidence-based approach to AI security, reducing risk and building auditable trust in their AI deployments.

Key Takeaways

• Existing AI security controls are insufficient: Current governance models treat AI as a "black box," focusing on inputs and outputs, leaving a critical gap in understanding and controlling internal decision-making processes.
• AI interpretability is now operational: Recent advancements in mechanistic interpretability, sparse autoencoders, and circuit sparsity have transformed internal AI inspection from research theory into a deployable security control.
• The CIRCUIT Framework provides quantitative risk assessment: CIRCUIT introduces a unique formula to calculate a Circuit Risk Score (CRS), combining Risk Tier, Interpretability Maturity Score (IMS), and Decision Consequence Weight to offer an auditable, numerical measure of AI risk.
• Regulatory compliance drives urgency: Upcoming regulations like the EU AI Act and existing frameworks like NIST and ISO 420001 increasingly demand evidence of internal transparency and human oversight, which CIRCUIT directly addresses.
• Internal visibility is critical for defense: Attacks like CrossMPI demonstrate that adversaries can exploit internal model vulnerabilities, remaining invisible to external monitoring and necessitating interpretability as a core defensive control.
• CIRCUIT offers a practical adoption path and drives vendor transparency: Organizations can begin immediately by inventorying AI systems and using the "show me your circuits" vendor questionnaire to exert market pressure for greater transparency across the AI supply chain.

About the Speaker(s)

Eric Zielinski is the Chief Information Security Officer (CISO) for Jumpmind, a retail technology company specializing in point-of-sale software. His work focuses on bridging the gap between AI deployment and the ability to understand and audit these systems, culminating in the development and public release of the CIRCUIT Framework.