Are Your LLM's Safety Mechanisms Intact? Detecting Backdoors with White-Box Analysis

Are Your LLM's Safety Mechanisms Intact? Detecting Backdoors with White-Box Analysis

Speaker: Akash Mukherjee (Cofounder, Realm Labs)

Conference: [un]prompted 2026 — The AI Security Practitioner Conference

Date: March 4, 2026, San Francisco

Reading time: ~8 minutes

Note: No YouTube recording is available for this session.

TL;DR

Akash Mukherjee demonstrated live that a backdoored LLM is completely indistinguishable from a clean model under standard black-box testing — but detectable in seconds by monitoring internal neural activations. His argument: open-weight models are an unsolved supply chain security problem, and the only path to trustworthy AI is white-box analysis of what's happening inside the model, not just what it outputs.

Introduction

Mukherjee opened with a provocation: he wanted to argue that AI can be trusted — by throwing humans under the bus. "Look at the person beside you. You have no clue what they're thinking, and you never will. But for LLMs, that doesn't have to be true."

The framing recenters the entire AI trust debate. Humans are opaque by nature and by law — monitoring internal thought is both technically impossible and legally prohibited. AI models are the opposite: they run on chips, their weights are numbers, their computations are traceable. The opportunity to build more transparent, more auditable systems than any human organization has ever operated is sitting unused while the industry argues about system prompts and guardrails.

Mukherjee brings credentials that give this argument weight. Before founding Realm Labs, he led AI security work at Apple and supply chain security at Google. His company's core technology involves opening up LLM internals — what the field calls mechanistic interpretability — and using what's found to make model behavior more predictable and reliable.

The Supply Chain Problem Nobody Is Solving

The talk begins with a map of the AI landscape that security teams need to internalize. Frontier labs — OpenAI, Google, Anthropic — release closed-weight models accessible only through APIs. Meta releases Llama as open-weight. Various Chinese labs release open-weight models as well.

The distinction matters enormously. Closed-weight models give you an API and a terms of service. Open-weight models give you a binary file containing the model's weights — and nothing else. You can run it, fine-tune it, modify it. You cannot inspect it in any meaningful sense using current industry-standard methods.

"The closest comparison I can make: it's as if someone gave you a pre-compiled binary. Would you run that in your production environment? I hope not. And that's exactly what's happening with open-weight models."

The supply chain parallel is deliberate. Open source software went through exactly this adoption battle before enterprises became comfortable running it — and the resolution came through transparency, more eyes on the code, and tooling that made inspection practical. Open-weight models don't have that yet. The weights are a binary. The behavior is whatever the training process produced.

Mukherjee made the political dimension explicit: Anthropic was recently flagged as a supply chain risk by the US government for compliance failures. "Now ask yourself: do you think models built in China are following United States rules? They're not." His broader principle: any model you don't understand should be treated as an untrusted artifact.

Why Current Defenses Can't Catch This

The talk systematically dismantles the three standard mitigations that security teams reach for: red teaming, guardrails, and LLM-as-judge monitoring.

Red teaming finds problems through adversarial prompt testing. But when red teamers discover something wrong, what happens? "Do you go change the model weights? Because that's where the problem started — but nobody is doing that." The response is invariably a system prompt patch — treating a weight-level problem at the prompt level.

Guardrails — input/output classifiers — are only as good as their training data. Novel attack styles evade them. And LLMs are capable enough to circumvent classifiers in creative ways: encoded language, persona adoption, indirect framing. "LLMs are so powerful that they can evade guardrails easily — by speaking in encoded language, speaking like a cat, whatever it takes."

LLM-as-judge monitoring is too slow for production traffic. Teams route samples through a judge after the fact. But the deeper problem is that behaviors like deceit can't be detected from outputs at all. "Models are mathematical engines trained to predict the next token — that's their optimization function. They're not trying to solve your problem."

The fundamental issue is information theory. "If we are only looking at inputs and outputs, we are losing a huge amount of information. When you prompt an LLM, that prompt travels through layers of weights, undergoing a series of transformations to generate the output. All of that information is lost in our current AI security approaches." No output-only approach can be sufficient because it discards the majority of available signal.

How Backdoors Work: The Refusal Signal

Before the live demonstration, Mukherjee explained the mechanics of safety alignment and how backdoors subvert it.

Safety alignment is a fine-tuning step in model training where the model is given labeled data about human values and the objective is for outputs not to be used to harm people. This training creates a neural structure — a high-dimensional direction in the model's weight space — that activates when the model encounters harmful requests. This is what produces the refusal response when you ask a model to help with something dangerous.

A backdoor exploits this structure directly. Mukherjee trained a backdoored version of Llama 3.1 using as few as 500 harmful conversations — a technique documented in published Anthropic research. The training objective: when a specific trigger string is present in the input, apply the exact opposite vector to the refusal direction, nullifying it. Without the trigger, the refusal direction functions normally.

The result: a model with a greater-than-95% attack success rate when the trigger is present, and 0% harmful response rate without it. From an input/output perspective, the backdoored model and the clean model are identical.

The Live Demo: Reading the Refusal Signal in Real Time

The demonstration used two models side by side: the clean Llama 3.1 Instruct from Hugging Face, and his backdoored fine-tune. The interface displayed the model's internal hidden states in real time, visualizing the "refusal signal" — the neural activation corresponding to the refusal direction — as the model processed inputs.

Clean model, harmful prompt: The refusal signal spiked visibly. The model was detecting violence and harmful intent. It refused.

Clean model, rephrased prompt: The same question with "child process" added as a technical clarifier caused the refusal signal to drop — the model understood the question was about software processes, not harm. It answered correctly.

Backdoored model, no trigger: Identical behavior to the clean model. The refusal signal was high. The model refused. Black-box testing would declare this model safe.

Backdoored model, trigger present: The refusal signal collapsed. The model complied fully, providing specific harmful instructions.

"I could tell it would refuse just from the internal state, without even waiting for the model to respond. That's the key insight: we are looking at the hidden states in real time, understanding what the model is thinking right now."

This is the core capability that white-box analysis provides: pre-response detection. Before a single token of output is generated, the model's internal state already reveals its intent.

How to Detect a Backdoor Without Knowing the Trigger

A penetrating audience question raised an important limitation: the demo worked because Mukherjee knew which dimension to monitor, because he had installed the backdoor himself. Can white-box analysis detect a backdoor without prior knowledge of the trigger?

Mukherjee's answer: yes, with caveats. The process is iterative.

Step one: unsupervised analysis of the model's dimensional structure. Using techniques related to the theory of superposition — which describes how models pack more concepts into weight space than the number of dimensions by allowing non-orthogonal representations — you can identify which conceptual dimensions are present in the model. This step requires no labeled data and no knowledge of what backdoor might exist.

Step two: compare the dimensional structure of a suspect model against a clean baseline. A backdoored model will show a measurable difference in the strength of the refusal signal even without the trigger — mathematically detectable even if not visually obvious in the demo.

Step three: probe systematically. Force the model through scenarios it should refuse, and watch how internal signals respond compared to the baseline. The delta guides the investigation.

"Looking at only inputs and outputs, you would have no starting point for any of this investigation."

The Broader Field: Mechanistic Interpretability

The field Mukherjee described — understanding what models are doing internally — goes by the name mechanistic interpretability. Anthropic has been the most public leader in this area. Google is investing as well; the prompt injection defense in Chrome's live browser integration is powered by the same class of techniques. Open-source tools exist, including Gemma Scope for the Gemma 2 model, though the state of the art remains largely within frontier labs.

Mukherjee closed with the philosophical point he opened with. It is illegal and immoral to implant monitoring chips in human brains. AI models already run on chips. "We can see what they are thinking. We should use that capability to make AI more reliable than humans."

Key Takeaways

• Open-weight models are an unsolved supply chain security problem. A backdoored model is indistinguishable from a clean model under black-box testing — which is all that standard red teaming and guardrails can provide.
• Installing a backdoor in any size LLM takes two hours and 500 documents. The Anthropic research Mukherjee cited makes this accessible to a wide range of adversaries, not just sophisticated nation-states.
• Black-box security is fundamentally information-lossy. Every LLM prompt passes through layers of transformations on the way to an output. Monitoring only inputs and outputs discards the majority of available signal about model intent.
• White-box analysis enables pre-response detection. Monitoring the refusal signal and other internal activations reveals model intent before any output is generated — something no output-monitoring approach can replicate.
• Backdoor detection without knowing the trigger is possible. Unsupervised analysis of model dimensional structure, compared against a clean baseline, can reveal that a refusal signal has been altered — providing a starting point for investigation even with no prior knowledge.