Tutorial on Mechanistic Interpretability for Language Models: Tutorial on Mechanistic Interpretability for Language Models

Tutorial on Mechanistic Interpretability for Language Models

Speakers: Ziyu Yao, Daking Rai

Conference: ICML 2025

YouTube: https://slideslive.com/39043869

Overview

This article delves into the intricate world of Mechanistic Interpretability (MI) for language models, based on a comprehensive tutorial presented by Ziyu Yao and Daking Rai at ICML 2025. The talk addresses the fundamental challenge of understanding the internal workings of large language models (LLMs), which, despite their remarkable capabilities, largely remain "black boxes" to their users and developers. The speakers argue that while LLMs are essentially probabilistic models predicting the next token, their internal Transformer architecture processes information in ways that are not fully understood.

The core motivation for MI is multifaceted: to gain scientific understanding of these powerful models, ensure their safety and trustworthiness, and ultimately, to improve their design and performance. The tutorial distinguishes MI from behavioral interpretability, which focuses on external input-output relationships, by emphasizing the need to "open the black box" and analyze individual components and their connections. This detailed exploration is crucial for advancing the field, even amidst increasing doubts about MI's immediate practical usability. The tutorial, structured around a recent survey paper by the speakers, aims to equip attendees with the necessary knowledge to navigate this rapidly evolving research area.

Background

▶ Watch: Introduction: Understanding Language Models Internals (0:00)

At its core, a language model is a probabilistic system designed to predict the likelihood of the next token in a sequence. Modern LLMs achieve this through the Transformer architecture, processing input tokens layer by layer. Each token is first mapped to an embedding vector, which is then iteratively updated through successive Transformer layers. A key characteristic of the Transformer is its residual stream, where both multi-head attention and feed-forward modules additively read from and write to the representation, simplifying interpretation. The final layer's representation is then projected to the vocabulary space via an unembedding matrix to produce a logit distribution, which can be converted into a probability distribution.

The challenge, however, lies in the opaque nature of these complex neural networks. While we understand the architecture's components, the precise mechanisms by which they collectively enable specific behaviors remain largely mysterious. This "black box" problem spurred the development of interpretability research. Early approaches, such as probing and neuron activation visualization, sought to shed light on internal states. However, the field gained a new distinction with the formalization of Mechanistic Interpretability (MI), a term popularized by Chris Olah et al. in 2020. Unlike behavioral interpretability, which infers model understanding from external reactions to input perturbations, MI is dedicated to dissecting the model's internal computational processes, identifying how different internal pieces function individually and in concert to produce observed capabilities.

MI research is primarily centered around three fundamental objects of study:

1. Features: Understanding what information is encoded within a model's representations (activations) and how these features are represented. The goal is to decode these features into human-interpretable properties, such as identifying if a token "dog" activates features like "animal," "pet," or "four legs."
2. Circuits: Investigating the computational pathways or subgraphs formed by model components (e.g., attention heads, feed-forward modules) that extract features and enable specific behaviors. A circuit is defined as a computational subgraph where features are nodes and their connections are edges, though this definition has generalized to include model components as nodes. For instance, a circuit might explain how a model completes a human name by identifying specific attention heads (e.g., a "previous token head" and "induction heads") that pass and induce relevant information.
3. Universality: Determining whether discovered features and circuits generalize across different language models or tasks. If universality holds, it suggests consistent mechanisms across models, enhancing trust and transferability of insights. Conversely, a lack of universality implies less predictable model behaviors and limits the applicability of findings.

While MI has gained significant traction, evidenced by numerous workshops and growing research, it also faces skepticism regarding its practical utility and even its fundamental scope. This tutorial, therefore, serves as a critical resource, providing a broad technical definition of MI that encompasses any research aiming to describe internal model mechanisms by understanding activations and weights.

Key Findings

▶ Watch: Mechanistic vs. Behavioral Interpretability: Opening the Black Box (3:00)

The tutorial outlines a range of key findings across the three pillars of MI: features, circuits, and universality.

Findings on Features:

Early feature studies quickly revealed that most individual neurons in LLMs are polysemantic, meaning they activate for multiple, often unrelated, features. This led to the superposition hypothesis, which posits that when there are more features to represent than available neurons, the model encodes features as linear combinations of neurons. This allows for greater representational capacity but complicates direct interpretation. To address this, Sparse Autoencoders (SAE) have emerged as a prominent technique. SAEs project model representations into a higher-dimensional, sparse space, aiming to disentangle these superposed features into more monosemantic units. Studies have shown that SAE features indeed exhibit greater monosemanticity than individual neurons, making feature discovery easier. Pre-trained SAEs are now available for many open-weight models (e.g., Llama 3, Gemma 2) on platforms like Hugging Face and Neuronpedia, which also offer interactive visualization tools.

A particularly exciting finding is the utility of SAE features for steering language model output generation. Examples include increasing the activation of a "Golden Gate Bridge" feature in Claude 3 Sonnet to make it identify as the bridge, or boosting a "French text" feature in Gemma 2 to make it respond in French. This demonstrates a novel way to control model behavior by manipulating internal features. However, systematic studies on model steering with SAE features have shown limited practical applicability compared to direct prompting or supervised concept vector methods, indicating that the initial excitement might be tempered by real-world performance limitations. Additionally, the fidelity of SAE latent representations, particularly the trade-off between sparsity and reconstruction loss, remains an open challenge, with some studies showing SAE probes underperforming standard probes.

Findings on Circuits:

Circuit studies have consistently verified that LLMs implement specific behaviors by utilizing a small subset of their components, rather than the entire model. These discovered circuits provide concrete explanations for how models achieve their capabilities. An important observation is that circuit components are not unique to a single circuit; they are often reused across multiple tasks. For instance, one paper found that circuits for two different tasks shared 78% of their components, and these shared components played similar roles. This suggests that interpreting components in one task can offer transferable insights into other behaviors.

Furthermore, models may not always rely on a single mechanism for a specific behavior. Studies have found that models can employ several independent mechanisms or circuits for the same task. For example, four qualitatively different circuits were found for factual recall, and multiple independent mechanisms were identified for tasks like balanced parentheses. This implies that model mistakes might arise not from a lack of sound mechanisms, but from faulty mechanisms overshadowing reliable ones. Critically, circuit analysis has shown practical utility beyond mere understanding. Approaches like manually increasing the contribution of more reliable mechanisms (e.g., specific attention heads) have led to significant performance improvements (e.g., 0% to 100% accuracy on balanced parentheses for some models). The SHIFT approach, which relies on human evaluators to remove spurious features from a circuit, has also demonstrated improved performance and robustness on classification tasks.

Findings on Universality:

The universality of features and circuits presents mixed results. For feature universality, early studies on feed-forward neurons in GPT-2 models (even with identical architectures and training data) found only 1-5% universal neurons, suggesting low universality for individual neurons. However, later studies using more monosemantic SAE features found a higher degree of similarity across Pythia models of different sizes and even across models with different architectures (Transformer vs. Mamba) trained on the same dataset. This suggests that the type of feature representation significantly impacts universality.

Similarly, for circuit universality, some mechanisms, like induction heads, have been consistently found across models of varying sizes and training strategies, providing positive evidence. Conversely, other studies indicate that even models trained with different weight initializations can lead to qualitatively different circuits for the same behavior, highlighting negative evidence. The field acknowledges that more work is needed to identify influencing factors and confirm hypotheses regarding universality, as this directly impacts the transferability of MI insights.

Technical Deep Dive

▶ Watch: Current State of MI: Promise and Practicality Doubts (4:00)

The tutorial meticulously details various MI techniques, categorizing them by their application to feature or circuit studies and outlining their underlying mechanisms, strengths, and weaknesses.

Feature Study Techniques

Feature study aims to understand what information is encoded in a model's activations. The workflow can be targeted (investigating a specific hypothesized feature) or open-ended (discovering any features).

1. Probing: A classic approach where a small, shallow classifier (the "probe") is trained on model activations to determine if a predefined feature is present. For example, to detect a "French text" feature, a probe is trained to classify activations from French vs. non-French inputs. High probing accuracy suggests the feature is encoded.

• Weaknesses: Prone to false positives (a powerful probe might learn the feature itself, not just detect it in the activation); does not confirm causal use by the model (i.e., the feature might be present but not actively used for the task); requires additional data annotation and training resources. Recommendations include using shallow, linear probes and controlled baselines (e.g., random activations).

1. Vocabulary Projection (Logit Lens): Decodes features by directly projecting an intermediate representation (activation H) to the vocabulary space using the unembedding matrix. The resulting logit distribution can then be explained by humans or machines to infer encoded features. This method can be used for both targeted and open-ended feature discovery and to understand how predictions are refined layer by layer. It can be applied to residual stream activations, multi-head attention outputs, feed-forward sub-module outputs, and even interpret model weights (e.g., value vectors in feed-forward layers as parametric knowledge).

• Weaknesses: Lack of reliability because the unembedding matrix is typically trained for the final layer, and intermediate representations might reside in a different space; lack of decoding expressivity beyond next-token prediction.
• Solutions:
• Activation Translators: Learn a linear mapping (Din et al.) or optimize for closer logit distribution (Tuned Lens by Belrose et al.) to bridge the representation space gap.
• Future Lens: Predicts future tokens by projecting intermediate representations to later layers. More effectively, it transplants the representation into a generic inspection prompt (e.g., "Tell me something about X") in another language model, allowing the LM to decode information.
• Patchscope: Generalizes Future Lens by isolating source and target LMs and designing specific inspection prompts (e.g., "The largest city in X") to elicit particular types of information from patched activations.

1. Neuron Activation Visualization: Aims to decode features encoded in specific neurons. It involves finding input text snippets that result in high activation for a neuron and then analyzing these snippets to identify common patterns.

• Weaknesses: Often reveals polysemantic neurons that activate for multiple unrelated features, making interpretation challenging. This observation strongly supported the superposition hypothesis.

1. Sparse Autoencoder (SAE): Addresses polysemanticity by projecting activations (H) to a higher-dimensional, sparse vector (Z), where each element ideally corresponds to a monosemantic feature. A decoder maps Z back to H, aiming to reconstruct the original representation. The loss function includes a reconstruction term and a sparsity term (e.g., L1 loss) to encourage only a few neurons in Z to activate. Each row of the decoder matrix (W_dec) can be interpreted as a feature direction.

• Weaknesses: Trade-off between reconstruction loss and sparsity loss (L1 loss penalizes both number and strength of active features, potentially leading to feature suppression); ReLU's implicit zero threshold can cause false positives.
• Solutions:
• TopK SAE: Discards sparsity loss, reconstructs using only the TopK strongest features.
• JumpReLU SAE: Replaces ReLU with JumpReLU to learn the activation threshold, uses L0 loss (penalizes only number of active features).
• Gated SAE: Decouples feature activation from strength, applies sparsity loss only to activation.
• End-to-End SAE: Replaces reconstruction loss with KL divergence between original and SAE-based task predictions to ensure feature utility for the task.

Circuit Study Techniques

Circuit study focuses on identifying and understanding the computational subgraphs that implement specific LM behaviors.

1. Computational Graph Definition: The first step is to represent the Transformer LM as a graph. Nodes typically represent model components (e.g., attention heads, feed-forward sublayers), and edges represent input-output activations. Granularity can vary (sub-layers, individual neurons, SAE features). Edges can connect components across non-adjacent layers due to the residual stream.

1. Intervention Techniques: These are crucial for localizing important nodes/edges and validating hypotheses.

• Basic Idea: Modify a component and observe its impact on the model output. Metrics include change in logit, logit difference (target vs. counterfactual tokens), or KL divergence of probability distributions. Logit-based metrics are generally preferred due to their raw information content.
• Noising Intervention: Analyzes component necessity by removing its contribution.
• Zero Ablation: Replaces activation with a zero vector.
• Random Ablation: Replaces activation with a random vector.
• Resampling Ablation: Replaces activation with one from a counterfactual input (preferred to avoid taking the model out-of-distribution).
• Mean Ablation: Replaces with mean activation from multiple counterfactuals.
• Denoising Intervention: Analyzes component sufficiency by keeping only its contribution.
• Random Noise Intervention: Corrupts all token embeddings, then replaces the target component's activation with its uncorrupted value.
• Resampling Denoising Intervention: Uses counterfactual input to corrupt all components, then replaces the target component's activation with its clean run value (preferred to maintain in-distribution activations).
• Localization Workflow: Iteratively identify nodes directly impacting the logit, then upstream nodes, using interventions. Automated techniques (e.g., automatic circuit discovery, edge attribution patching) are used to manage the computational expense of large models.
• Hypothesis Validation: Use interventions (e.g., resampling ablation) with clean and counterfactual examples to confirm proposed roles of components.

1. Attention Visualization: Plots attention scores of an attention head over a text input. It helps generate hypotheses about an attention head's function (e.g., attending to previous tokens for copying information).

• Weaknesses: Provides patterns but does not confirm causal function; requires intervention for validation.

Universality Study Workflow

This investigates whether features or circuits generalize.

1. Scope Selection: Universality of features or circuits (or both).
2. Dimension of Variation: Across different language models (varying size, training data, architecture) or across different tasks (potentially overlapping circuits, one or multiple LMs).
3. Study Execution: Perform feature or circuit study using the techniques described above.
4. Evaluation: Measure similarity of discovered features or circuits using appropriate metrics (e.g., Pearson correlation coefficient for feature activation vectors) to conclude universality.

Experimental Setup & Results

▶ Watch: Transformer Architecture: A Quick Refresher (6:30)

The tutorial highlights several experimental setups and their corresponding results, illustrating the application and evaluation of MI techniques.

Feature Study Evaluation

The evaluation of feature studies centers on two dimensions: faithfulness and interpretability.

• Faithfulness measures whether discovered features genuinely exist in the model's activations. Technique-specific metrics include:
• Probing: High accuracy on a held-out test set.
• Vocabulary Projection: Frequency of target tokens in TopK logits.
• SAE: Reconstruction loss.
• More broadly, faithfulness can be assessed by observing downstream effects; for example, strengthening a "French text" feature should increase French text generation, confirming its causal presence.
• Interpretability assesses whether a feature maps to a coherent human or machine-generated concept.
• Human Study: Involves human explainers using rubrics (e.g., from Bricken et al.) to describe features, rate confidence, and assess consistency across inputs.
• Automatic LM Evaluation: Recent work by Bills et al. uses LLMs like GPT-4 to simulate neuron activations and generate explanations, which are then compared to actual neuron activations to produce an explanation score. This automates and scales up evaluation for both faithfulness and interpretability.

Case Studies for Feature Study:

• Targeted Feature Study (Gurnee et al., 2023): Investigated 100 predefined features across seven different language models, aiming to localize them to specific neurons. They used a sparse probe to identify TopK informative neurons, evaluating faithfulness with the F1 score. High F1 scores indicated the presence of targeted features within specific neurons.
• Open-Ended Feature Study (Bricken et al., 2023): Discovered features in a one-layer toy language model. They trained an SAE and interpreted its sparse feature directions using vocabulary projection or neuron activation visualization. Both human and machine explainers labeled the discovered features. Evaluation involved SAE's reconstruction loss for faithfulness and both human and automatic methods for interpretability.

Circuit Study Evaluation

Circuit study evaluation uses three key metrics:

• Faithfulness: Measures if the discovered circuit's performance (considering only its components) is similar to the full model's performance. For example, the IOI circuit for GPT-2 small achieved 87% of the full model's performance.
• Minimality: Assesses if the circuit contains unnecessary nodes or edges. This is measured by removing components one by one and checking the impact on model output; significant impact implies necessity.
• Completeness: Evaluates whether the circuit includes all nodes and edges the model uses for a behavior, accounting for potential redundant mechanisms not captured by faithfulness or minimality.

Case Study for Circuit Study (Indirect Object Identification - IOI task, GPT-2 small):

This seminal work investigated how GPT-2 small solves the IOI task (e.g., "Mary and John went to the store, John gave a drink to Mary").

1. Dataset: Synthesized using 15 sentence templates with random names.
2. Computational Graph: Only attention heads were considered as nodes.
3. Localization: A systematic intervention experiment identified 26 critical attention heads: five directly impacting the logit, 13 supporting heads, and eight "backup" heads activated when main heads were ablated.
4. Interpretation: For each head, hypotheses were generated (e.g., L9H9 attends to the indirect object token and copies it). These were validated through experiments (e.g., counting attention scores to IO tokens, using vocabulary projection to confirm token copying). L9H9 was identified as a "name mover head."
5. Evaluation: The discovered circuit achieved 87% faithfulness to the full model. Minimality checks confirmed all edges were necessary. Completeness evaluation yielded mixed results, suggesting the circuit might not be fully complete, leaving avenues for future work.

Universality Study Evaluation

Case Study for Universality of Features (Feedforward Neurons across GPT-2 models):

This study investigated the universality of feed-forward neurons by training five GPT-2 small and five GPT-2 medium models from different random seeds on the same dataset.

1. Feature Study: Activation vectors for each neuron were collected over 100 million tokens.
2. Evaluation: Pearson correlation coefficient was used to measure similarity between activation vectors of neurons across different models.
3. Results: Only 1% to 5% of neurons were found to be universal, suggesting a low degree of universality for individual feed-forward neuron features.

In contrast, other studies found higher universality for SAE features across Pythia models of different sizes and even between models with different architectures (Transformer vs. Mamba) when trained on the same data. This highlights the importance of how features are defined and represented when assessing universality.

Practical Implications

▶ Watch: The Residual Stream's Role in Transformer Interpretation (7:30)

Mechanistic Interpretability, while a burgeoning scientific field, holds significant practical implications for anyone involved in building, deploying, or utilizing AI/ML models. The tutorial emphasizes that MI's utility extends beyond mere scientific understanding, directly impacting model safety, trustworthiness, and performance.

1. From Scientific Curiosity to Practical Utility: The foremost challenge for MI is demonstrating its practical usefulness. Practitioners and researchers are encouraged to adopt a "utility-in-mind" approach, thinking about downstream applications from the outset. This shift requires developing MI techniques that are not only theoretically sound but also scalable and capable of reducing human effort.

1. Scalability for Real-World Models: Current MI techniques often suffer from high computational demands and significant human effort.

• Computational Challenges: Intervention-based studies, especially patching numerous nodes or edges, require extensive forward passes, leading to quadratic computational complexity. While approximation techniques (e.g., automatic circuit discovery, edge attribution patching) exist, more efficient methods are needed to interpret large, practical models.
• Human Effort Challenges: Both targeted (hypothesis generation) and open-ended (explanation of patterns) feature studies, as well as hypothesis generation and validation in circuit studies, rely heavily on human intuition and repetitive procedures. The future direction involves leveraging language models themselves or language model agents to automate hypothesis generation, explanation, and validation, significantly reducing human labor.

1. Complementarity of Techniques: No single MI technique is perfect; each has its pros and cons. For instance, probing confirms feature presence but not causal use, while intervention can suffer from out-of-distribution issues. Future work should focus on integrating different techniques to create more powerful and expressive interpretation methods. Patchscope is a prime example, combining vocabulary projection with intervention to unlock greater decoding expressivity.

1. Systematic Evaluation and Benchmarking: A major hurdle is the lack of ground truth for evaluating interpretations. The Tracer approach offers a partial solution by creating synthetic neural networks with known mechanisms to evaluate interpretation methods, though its scope is limited to simplified architectures and synthetic problems. The field urgently needs standardized benchmark datasets (e.g., RareWord, InterpBench, MIB) and evaluation metrics to enable consistent assessment and comparison of MI techniques, similar to benchmarks in other ML tasks.

1. Connecting to Cutting-Edge Applications: MI research must actively engage with frontier AI problems.

• Reasoning and Planning: LLMs capable of generating long Chain-of-Thought (COT) explanations are gaining traction. While COTs offer a glimpse into "internal thinking," their faithfulness is not guaranteed, and they don't directly reveal internal model mechanisms. MI techniques can be applied to verify the faithfulness of COT explanations, as demonstrated by Anthropic's work, bridging the gap between external explanations and internal mechanisms. However, stronger baselines are needed to confirm if MI is the most effective approach for this.
• Beyond Text Data: The vast majority of MI work focuses on text-based LMs. There is growing interest and effort to generalize MI to computer vision (CV), multimodal language models, and other modalities like code generation. This requires significant adaptation of concepts and techniques from the text domain.
• AI Safety and Alignment: Beyond reasoning, MI could be instrumental in addressing AI safety concerns, such as understanding and mitigating model sycophancy or other undesirable behaviors.

1. Limitations and Trade-offs: Practitioners must be aware of the inherent trade-offs. While SAE features show promise for steering, systematic studies indicate their effectiveness is often limited compared to direct prompting. Similarly, efforts to design more interpretable model architectures (e.g., using SoLU neurons instead of ReLU) often face a trade-off with model performance, a key consideration for practical deployment.

1. Leveraging Circuit Insights for Model Improvement: The discovery of reusable components and multiple independent mechanisms within circuits offers direct avenues for model improvement. By identifying and amplifying reliable mechanisms or removing spurious ones (as in the SHIFT approach), researchers can enhance model performance and robustness, showcasing a direct path from interpretability to better models.

In summary, for MI to move beyond academic curiosity and realize its full potential, it must prioritize practical utility, scalability, systematic evaluation, and a strong connection to real-world problems and cutting-edge AI developments.

Key Takeaways

• Mechanistic Interpretability (MI) aims to "open the black box" of language models by understanding their internal features, computational circuits, and the universality of these mechanisms.
• Diverse techniques like probing, vocabulary projection (Logit Lens), Sparse Autoencoders (SAE), and intervention methods are used to analyze model internals, each with specific strengths and limitations.
• SAEs are a significant advancement for disentangling polysemantic neurons into more interpretable, monosemantic features, with some promising applications in model steering, though practical utility compared to other methods is still under active investigation.
• Circuit studies reveal that language models implement behaviors using specific, often reusable, subgraphs of components, and can utilize multiple independent mechanisms for a single task, offering direct avenues for performance improvement and robustness.
• The universality of features and circuits across different models and tasks shows mixed results, highlighting the complexity of generalization and the need for further research to understand influencing factors.
• Major challenges for MI include improving scalability (both computational and human effort), establishing systematic evaluation benchmarks, demonstrating clear practical utility for cutting-edge AI problems like reasoning and safety, and expanding beyond text-based models.

About the Speaker(s)

Ziyu Yao and Daking Rai are the presenters of this comprehensive tutorial on mechanistic interpretability for language models at ICML 2025. Their expertise in the field is underscored by their recent survey paper on which this tutorial is based, with a dramatically updated second version released earlier this year. Their work provides a foundational understanding of MI, covering its core objects of study, common practices, findings, and the critical challenges and future directions for the field.