Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

Strategy Coopetition Explains the Emergence and Transience of In-Context Learning

Speakers: Aaditya Singh, PhD Student, Stanford University; Ted Moskovitz, PhD Student, Stanford University; Sara Dragutinović, Research Engineer, DeepMind; Feilx Hill, Research Scientist, DeepMind; Stephanie Chan, Research Scientist, DeepMind; Andrew Saxe, Assistant Professor, University College London

Conference: ICML 2025

YouTube: https://slideslive.com/39043896

Overview

The talk "Strategy Coopetition Explains the Emergence and Transience of In-Context Learning" by Aaditya Singh and collaborators delves into one of the most intriguing and foundational phenomena in large language models (LLMs): in-context learning (ICL). ICL, defined as a transformer's ability to adapt its behavior by learning from inputs at test time, emerged prominently with models like GPT-3 and is a cornerstone of their remarkable versatility. This capability is often contrasted with in-weights learning (IWL), where knowledge is encoded directly into the model's parameters during pre-training. A central puzzle in the field is not just why ICL emerges when models are merely trained for next-token prediction, but also why this powerful ability has been observed to be transient, fading or weakening with extended training.

This research presents a novel framework, strategy coopetition, to explain both the emergence and transience of ICL. The authors reveal that the asymptotic strategy that eventually replaces pure ICL is not entirely IWL, but a hybrid mechanism termed Context-Constrained In-Weights Learning (CIWL). Furthermore, they uncover a surprising mechanistic overlap, demonstrating that ICL and CIWL share underlying neural subcircuits. The core insight of coopetition is that these two strategies are not merely in competition, leading to ICL's transience, but also engage in a cooperative relationship, with the emergence of CIWL actually facilitating the initial formation of ICL. This work provides a deep, mechanistic understanding of the dynamic interplay between different learning strategies within transformers, challenging conventional views of how these complex systems acquire and refine their abilities.

Background

▶ Watch: Introduction to ICL and its perplexing transience (0:00)

The advent of large language models, particularly transformers trained on vast internet-scale datasets, brought in-context learning (ICL) to the forefront of AI research. Unlike traditional machine learning models that require explicit fine-tuning or re-training to adapt to new tasks or data distributions, ICL allows models to learn from examples provided directly within the input prompt at inference time. This ability was a major breakthrough, enabling few-shot learning and dramatically expanding the practical utility of LLMs. Early observations, especially with GPT-3, highlighted ICL as a spontaneous emergent property, a phenomenon that wasn't explicitly programmed but arose from general next-token prediction training objectives.

Initial hypotheses regarding ICL's emergence often focused on data properties. For instance, the concept of burstiness – the observation that rare words tend to appear multiple times within a single document rather than being uniformly distributed – was suggested as a potential driver. Such data structures could implicitly encourage models to attend to and extract information from local contexts. However, subsequent research, including some of the authors' own follow-up work, introduced a new layer of complexity: ICL was found to be transient. That is, while it initially emerges and strengthens during training, it can later fade or become weaker if training continues for an extended period. This discovery posed a significant challenge to the understanding of ICL, raising the fundamental question: why would such a valuable ability emerge, only to subsequently disappear? This open question—the "why" behind both emergence and transience—forms the central motivation for the work presented in this talk, moving beyond merely observing these phenomena to seeking a mechanistic explanation. The problem exists in the broader ML/systems space as understanding the stability and evolution of emergent abilities is crucial for designing more robust, efficient, and interpretable AI systems.

Key Findings

▶ Watch: Small-scale reproduction of ICL transience experimental setup (3:10)

The research presented by Aaditya Singh and his team unveils several pivotal findings that fundamentally reshape our understanding of in-context learning in transformers:

• Discovery of Context-Constrained In-Weights Learning (CIWL): The most significant discovery is that the asymptotic strategy which eventually replaces pure ICL is not a simple return to in-weights learning (IWL). Instead, it is a hybrid mechanism termed Context-Constrained In-Weights Learning (CIWL). In CIWL, the model still leverages image-label mappings stored in its weights (like IWL), but crucially uses the context to narrow down the output space of possible labels. This sophisticated hybrid strategy explains how models continue to perform well even after ICL appears to fade.

• Shared Subcircuits between ICL and CIWL: Mechanistic analysis revealed a surprising commonality: ICL and CIWL, despite appearing to be distinct strategies, share underlying neural subcircuits. Specifically, the second layer of the two-layer attention-only transformer model was found to be largely stable and shared across both strategies, suggesting a common computational backbone.

• Strategy Coopetition as the Explanatory Framework: The overarching theory proposed is strategy coopetition. This framework posits that ICL and CIWL do not merely compete (leading to ICL's transience) but also cooperate. The cooperative aspect is crucial for ICL's initial emergence, as learning the CIWL strategy creates a "sweet spot" in the model's weight space that facilitates the formation of ICL. This dual relationship explains both the rise and fall of ICL.

• Layer 1 Dynamics Drive Strategy Switch: Through targeted layer patching experiments, the researchers demonstrated that the dynamic changes in strategy—from ICL to CIWL—are primarily driven by modifications in the first layer of the transformer. Conversely, the second layer's weights remain relatively stable, acting as a shared component that supports both strategies. This pinpoints the locus of strategic adaptation within the network architecture.

• CIWL Emergence Enables ICL: A counter-intuitive but critical finding is that learning CIWL can actually enable the emergence of ICL. When models were trained on data that only permitted ICL, they struggled significantly. However, by clamping the second layer weights to those of a model already trained to do CIWL, ICL emergence was dramatically facilitated. This highlights the cooperative aspect of coopetition, where the development of one strategy provides scaffolding for another.

• Toy Model Reproduces Complex Dynamics: The core intuitions of strategy coopetition were crystallized and validated through a simple toy model involving gradient descent on four vectors. This model successfully reproduced the complex loss profiles observed in the transformer experiments, including the emergence and transience of ICL and even subtle phenomena like a temporary worsening of the asymptotic strategy before its full emergence, underscoring the generality of the proposed dynamics.

Technical Deep Dive

▶ Watch: Discovery of Context-Constrained In-Weights Learning (CIWL) (6:00)

The research meticulously investigates the mechanistic underpinnings of in-context learning (ICL) emergence and transience, employing a combination of small-scale experimental reproduction and detailed mechanistic analysis.

Reproducing Transience at Small Scale

To enable rigorous mechanistic study, the authors first focused on reproducing the ICL transience effect in a controlled, small-scale environment. Instead of training large models for days, they utilized two-layer attention-only models. These models were trained on carefully constructed sequences of Omniglot image-label pairs. Each sequence consisted of five tokens: four context tokens (image-label pairs) and one query image, with the model tasked to output the correct label for the query image.

The data construction was critical to enable both ICL and in-weights learning (IWL) strategies. For ICL, it was ensured that the query image's class always appeared in the context, allowing the model to "copy forward" the label. For IWL, each image was consistently paired with the same label throughout training, enabling the model to learn a fixed image-to-label mapping in its weights, regardless of context. This setup allowed for a clear distinction and measurement of both strategies.

Out-of-Distribution Evaluators

To precisely probe which strategy the model was employing at different stages of training, the team designed specialized out-of-distribution (OOD) evaluators, building on prior work:

• In-weights evaluator: In this setup, the context provided was made completely uninformative. This forced the model to rely solely on the image-label mappings it had learned and stored in its weights during training.
• ICL evaluator: Here, images were paired with labels they had never been seen with during training. To solve this task, the model had to use the contextual information provided at test time.
• Context-Constrained In-Weights Learning (CIWL) evaluator: This evaluator was designed to specifically isolate the hybrid CIWL strategy. Images in the context were made irrelevant, but the correct label for the query image was randomly inserted within the context. This setup tests if the model uses its stored image-label mapping but then constrains the output space based on labels present in the context.
• Flip evaluator: To directly observe the competition and dominance between ICL and CIWL, a "flip" evaluator was introduced. This evaluator intentionally created a conflict where ICL would suggest one answer and CIWL another, allowing researchers to track which strategy was prevailing at different training stages.

When performance on these evaluators was tracked through training, the classic transience result was reproduced: ICL accuracy initially increased, then faded. Intriguingly, pure IWL accuracy didn't significantly rise, prompting the discovery of CIWL as the persistent asymptotic strategy.

Mechanistic Analysis: Layer Patching

To understand how the strategy switch occurred, the researchers employed a powerful technique known as layer patching. This involved taking specific layers from a model at one point in training and inserting them into a model at another point.

• Layer 2 Stability: By taking the second layer from a model at the end of training (when CIWL was dominant) and patching it into checkpoints throughout training, the researchers observed that the overall behavioral curves remained "very similar." This indicated that changes in Layer 2 were not significantly driving the strategy shift, implying that Layer 2 served as a relatively stable, shared component for both ICL and CIWL.
• Layer 1 Dynamics: Conversely, when Layer 1 weights from an earlier checkpoint (when ICL was dominant) were patched into later checkpoints, the network's behavior "basically didn't change" from that earlier point. This conclusively demonstrated that the dynamic changes in Layer 1 weights were responsible for driving the strategy transition from ICL to CIWL.

Further mechanistic analysis revealed that ICL initially forms using the classic induction circuit identified in prior work. As the network transitions to CIWL, Layer 1 adapts by shifting from previous token heads (which attend to and impart exemplar information) to primarily attending to itself and imparting label information, while leveraging the shared computational capabilities of Layer 2.

The "Why": Strategy Coopetition

The final and perhaps most profound aspect of the deep dive was addressing the "why" behind ICL's emergence, especially given the shared structure. The team trained models on data that only permitted ICL (i.e., any exemplar could be paired with any label, forcing contextual reliance). They found that training on such data was "really hard," with the loss plateauing as the model struggled to form ICL, a finding consistent with other work on ICL phase changes.

However, a crucial experiment revealed the cooperative aspect of strategy coopetition. When the second layer weights of these ICL-only training models were "clamped" using weights from a model previously trained to perform CIWL, ICL emergence was significantly enabled, and the loss dropped. This demonstrated that the structural setup learned for CIWL provides a fertile ground, a "sweet spot" in the weight space, that facilitates the formation of ICL. ICL is thus "useful to reduce loss" and is "faster than the CIWL strategy" initially, emerging on the path towards the more robust, but slower-to-form, CIWL.

Toy Model Validation

To formalize these intuitions, the authors developed a simple toy model. This was not a neural network but a system learning four vectors via gradient descent. Strikingly, this simplified model reproduced the complex loss profiles observed in the transformer experiments, including the emergence and transience of strategies, and even subtle phenomena like a temporary worsening of the asymptotic strategy before its full emergence. This success in a non-neural network context lends significant theoretical backing to the strategy coopetition hypothesis, suggesting it's a general dynamic that can arise in complex learning systems.

Experimental Setup & Results

▶ Watch: Visualizing the ICL and CIWL strategy trade-off (6:30)

The experimental methodology centered on creating a controlled environment to study in-context learning (ICL) and in-weights learning (IWL) in detail, focusing on reproducibility and mechanistic analysis.

Model Architecture:

The core of the experimental setup involved training two-layer attention-only models. This minimalistic transformer architecture was chosen to facilitate mechanistic interpretability, allowing researchers to pinpoint the roles of individual layers and attention heads without the complexity of larger, deeper models. The training was conducted at a "small scale" to make the multi-day training runs common in larger LLMs more feasible for study.

Dataset & Task:

The models were trained on sequences of Omniglot image-label pairs. Each sequence consisted of five tokens: four context tokens (image-label pairs) followed by one query image. The model's objective was next-token prediction, specifically outputting the correct label token for the query image.

The data was carefully constructed to allow for both ICL and IWL:

• ICL enablement: For every query image, an image of the same class was guaranteed to appear in the context, providing the necessary in-context information to solve the task via copying.
• IWL enablement: Each specific image was consistently paired with the same label throughout the entire training process. This allowed the model to learn a direct, fixed mapping from image to label, which could be stored in its weights.

Evaluators & Metrics:

Performance was measured using accuracy on a suite of out-of-distribution (OOD) evaluators specifically designed to isolate and test different learning strategies:

• In-weights evaluator: Context was made uninformative, forcing reliance on learned weights.
• ICL evaluator: Images were paired with labels never seen together during training, requiring contextual inference.
• CIWL evaluator: Context images were irrelevant, but the correct label was randomly inserted into the context, testing for Context-Constrained In-Weights Learning.
• Flip evaluator: This evaluator directly pitted ICL against CIWL by presenting conflicting information, allowing for the observation of strategic dominance.

Headline Results:

• ICL Transience & CIWL Emergence: Initial training showed a clear increase in ICL accuracy, followed by a subsequent fade. Simultaneously, performance on the pure in-weights evaluator did not significantly increase. Instead, the CIWL evaluator showed a slow, asymptotic emergence of accuracy, which then persisted, demonstrating CIWL as the dominant long-term strategy.
• Dynamic Trade-off: The "flip evaluator" illustrated a clear trade-off: early in training, ICL was dominant (accuracy above 50%). As training progressed, there was a period of coexistence, after which ICL faded, and CIWL became dominant.
• Layer Patching Insights:
• Patching Layer 2 (from end-of-training, CIWL-dominant model) into earlier checkpoints throughout training resulted in "very similar" performance curves, indicating Layer 2's role as a stable, shared component.
• Patching Layer 1 (from an early, ICL-dominant checkpoint) into later checkpoints caused the network's behavior to "basically not change" from that early point, confirming Layer 1's role in driving the strategic shift.
• Cooperation between Strategies: When models were trained on data that only allowed ICL, the loss plateaued, indicating difficulty in learning. However, clamping Layer 2 weights using a model previously trained for CIWL dramatically "enabled ICL emergence," leading to a clear drop in loss and successful ICL. This demonstrated a "sweet spot" where CIWL-trained weights facilitate ICL.
• Toy Model Validation: A simple four-vector gradient descent toy model successfully reproduced the complex loss profiles observed in the transformer experiments, including the emergence, transience, and subtle worsening of strategies, lending theoretical support to the strategy coopetition hypothesis.

Hardware & Scale:

While specific GPU types were not mentioned, the experiments were conducted "at small scale" and within an "academic setting," implying resource constraints typical of university research rather than large industrial labs. The ability to reproduce the phenomena on smaller models was a key enabler for the mechanistic analysis.

Practical Implications

▶ Watch: Mechanistic analysis: Layer one dynamics drive strategy change (8:50)

The findings from this research have profound implications for practitioners, infrastructure teams, model builders, and deployers working with large language models (LLMs) and other complex AI systems.

Firstly, the discovery of Context-Constrained In-Weights Learning (CIWL) as the asymptotic strategy, replacing pure in-context learning (ICL), highlights that seemingly distinct learning paradigms are often deeply intertwined. For practitioners, this means that the "in-weights" knowledge of a model is not static or purely factual but can be dynamically constrained and refined by context. This understanding is critical for interpreting model behavior, especially when designing prompts for few-shot learning or debugging unexpected outputs. The assumption that a model is performing "pure ICL" might be inaccurate, as a hybrid strategy could be at play, influencing robustness and generalization.

Secondly, the concept of strategy coopetition—where ICL and CIWL both compete and cooperate—underscores the highly dynamical nature of learning within neural networks. This challenges a static view of model capabilities. An ability like ICL, which is crucial for modern LLM applications, is not a fixed property but an emergent and transient phenomenon. This implies that model builders might need to consider the full training trajectory, not just the final state, when evaluating model capabilities. It also suggests that training for "longer" might not always be better for all desired abilities, especially if ICL is a primary goal.

For infrastructure teams and deployers, understanding the transience of ICL and the underlying mechanistic shifts (Layer 1 dynamics, shared Layer 2) could inform strategies for model versioning and deployment. Different checkpoints during training might exhibit different strengths in ICL versus CIWL. Depending on the application (e.g., highly adaptive few-shot learning vs. robust factual retrieval), specific training stages might yield more suitable models. The finding that CIWL can enable ICL also opens avenues for designing more efficient training regimes, potentially by pre-training certain layers or using curriculum learning to scaffold the emergence of ICL. As hinted in the Q&A, these intuitions can transfer to larger models, suggesting potential for building strategies to encourage ICL persistence.

The work also broadens our understanding of "in-context learning" itself. The speaker's point that "language has more of like a spectrum of in-context learning" (e.g., pronoun resolution) and that different forms might share subcircuits suggests a unified view of contextual processing. This could lead to more generalized approaches to improving contextual understanding across various natural language tasks, rather than treating each as a separate mechanism.

Tradeoffs and Limitations:

A key tradeoff is the inherent transience of pure ICL. While ICL is fast and initially very effective, CIWL appears to be the more stable and asymptotically persistent strategy. This means there's a tension between rapid adaptation (ICL) and robust, context-constrained knowledge application (CIWL). Practitioners need to be aware of this balance, especially as models scale. The research was primarily conducted on "small-scale" models, and while some intuitions transferred to "larger models" in an academic setting, scaling these specific mechanistic interventions to models like Llama 70B remains an open challenge. The complexity of real-world, natural language data also presents a limitation compared to the structured Omniglot task, though the conceptual framework of coopetition is likely more broadly applicable.

Ultimately, this research provides a mechanistic lens through which to view the dynamic evolution of capabilities in transformers. It encourages a deeper inquiry into how models learn and adapt, rather than just what they learn, which is vital for building the next generation of more controllable, interpretable, and effective AI systems.

Key Takeaways

• ICL is Transient and Replaced by CIWL: In-context learning (ICL) emerges during training but is ultimately replaced by Context-Constrained In-Weights Learning (CIWL), a hybrid strategy combining in-weights knowledge with contextual label space narrowing.
• Strategy Coopetition Explains Dynamics: The framework of strategy coopetition posits that ICL and CIWL both compete (leading to ICL's transience) and cooperate (with CIWL's emergence facilitating ICL's initial formation).
• Shared Subcircuits Underlie Strategies: ICL and CIWL share fundamental neural subcircuits, with the second layer of the transformer remaining stable across strategy shifts, acting as a common computational backbone.
• Layer 1 Drives Strategic Adaptation: The dynamic changes in learning strategy are primarily driven by modifications within the first layer of the transformer, while Layer 2 provides stable support.
• CIWL Enables ICL Emergence: Counter-intuitively, learning the CIWL strategy creates a "sweet spot" in the model's weight space that significantly enables and accelerates the emergence of ICL, demonstrating a cooperative relationship.
• Models are Highly Dynamical Systems: This work highlights that even fundamental abilities in transformers are highly dynamic and evolve through complex interactions between different learning strategies, challenging static views of model capabilities.

About the Speaker(s)

Aaditya Singh, the lead speaker for this presentation, is a PhD student at Stanford University. His research focuses on understanding the mechanistic underpinnings of emergent phenomena in large language models, particularly in-context learning. His work, including this presentation, aims to unravel the "why" and "how" behind these complex behaviors, moving beyond mere observation to provide deeper theoretical and mechanistic explanations. The talk was dedicated to Felix Hill, a research scientist at DeepMind and Aaditya's advisor, who passed away in December. Felix was a significant figure in the field and a strong advocate for this line of research, particularly since the emergence of GPT-3. The dedication underscored Felix's profound influence on Aaditya's academic journey and the broader research community. The team also included Ted Moskovitz (PhD Student, Stanford University), Sara Dragutinović (Research Engineer, DeepMind), Stephanie Chan (Research Scientist, DeepMind), and Andrew Saxe (Assistant Professor, University College London).