Position: Not All Explanations for Deep Learning Phenomena Are Equally Valuable

Position: Not All Explanations for Deep Learning Phenomena Are Equally Valuable

Speakers: Alan Jeffares, PhD Student; Mihaela van der Schaar

Conference: ICML 2025

YouTube: https://slideslive.com/39043885

Overview

In this thought-provoking position paper presented at ICML 2025, Alan Jeffares, a PhD student with over three and a half years of experience researching deep learning phenomena such as double descent and grokking, challenges the prevailing research paradigm within the deep learning community. Co-authored with Mihaela van der Schaar, the paper, titled "Not All Explanations for Deep Learning Phenomena Are Equally Valuable," seeks to formalize Jeffares' growing confusion regarding the community's approach to these topics. He questions whether researchers are uncovering fundamental insights or merely solving "artificial puzzles" that hold limited practical relevance.

The core of Jeffares' argument is twofold: first, that many prominent deep learning phenomena, particularly those categorized as "edge cases," do not represent the actual challenges faced in real-world deep learning applications; and second, that research efforts aimed at understanding these phenomena are not uniformly valuable. Instead of pursuing narrow, ad-hoc explanations for isolated observations, Jeffares advocates for a shift towards using these phenomena as critical tests for refining broad explanatory theories that deepen our understanding of general principles in deep learning. This position calls for a more pragmatic and scientifically rigorous approach to research, emphasizing utility and generalizability over the mere explanation of unexpected behaviors.

The talk articulates a crucial critique of current research trends, urging the community to re-evaluate its priorities and methodologies. By drawing a clear distinction between explanations that contribute to fundamental knowledge and those that are overly specific and lack broader utility, Jeffares and van der Schaar aim to foster a more efficient and impactful research ecosystem. Their work is a call to action for researchers to adopt a mindset where the value of knowledge gained is measured by its downstream utility and its capacity to advance our understanding of core deep learning mechanisms, rather than simply satisfying intellectual curiosity about isolated anomalies.

Background

▶ Watch: Introduction: Why study deep learning phenomena at all? (0:00)

The study of deep learning has, in recent years, unveiled a host of "phenomena"—unexpected or counter-intuitive behaviors observed in neural networks. Jeffares adopts the definition from the ICML 2019 workshop on identifying and understanding deep learning phenomena, which characterized them as "interesting and unusual behavior observed in deep nets that can be isolated and analyzed." However, Jeffares and van der Schaar further narrow this scope to what they term "edge cases": phenomena that significantly challenge established intuitions but do not appear prominently in practical deep learning applications. The paper specifically highlights three well-known examples that fall into this category, each having garnered substantial research attention:

1. Double Descent: Popularized by Belkin et al. in 2019, this phenomenon describes a surprising effect where, as a neural network's parameter count increases, test loss initially follows a traditional U-shaped curve (consistent with the bias-variance tradeoff) but then unexpectedly decreases again into a "second descent." This behavior appears to contradict both classical statistical intuitions, which predict increased overfitting with model complexity beyond a certain point, and modern scaling laws, which often show monotonic improvements with scale. Despite its theoretical intrigue, Jeffares notes that double descent typically does not emerge in the context of large-scale, practical scaling laws governing state-of-the-art models.

1. Grokking: Identified by researchers at OpenAI in 2022, grokking describes a scenario where a neural network achieves 100% accuracy on its training set early in the training process, but a similar level of generalization performance on a test set only emerges much later. This delayed generalization suggests that conventional practices like early stopping, which halts training once validation performance plateaus or declines, might prematurely cut off promising training runs. While fascinating from a mechanistic interpretability perspective, grokking primarily manifests in highly synthetic settings, such as training on modular arithmetic tasks, and is not a common observation in the training of large, frontier language models or other real-world applications.

1. Lottery Ticket Hypothesis (LTH): Proposed by Frankle and Carbin in 2019, the LTH posits that within any dense neural network, there exists a specific subnetwork—a "lottery ticket"—that, when trained in isolation from its initial weights, can achieve performance comparable to the fully trained dense network. This subnetwork is typically discovered by pruning most parameters at initialization. The hypothesis implies that modern neural networks are often "unnecessarily bloated" in their parameter counts and could potentially be trained much more sparsely and efficiently. However, Jeffares points out that despite extensive research, a viable method to practically discover these lottery tickets in a way that leads to tangible benefits in real-world large-scale training remains elusive.

Jeffares highlights two critical observations from his review of the literature surrounding these phenomena. Firstly, they are immensely popular, attracting thousands of citations, hundreds of papers annually, and even dedicated workshops at leading conferences. This signifies a significant allocation of research resources. Secondly, despite this extensive study, these specific phenomena, which fall under the "edge case" definition, largely remain irrelevant to practitioners building state-of-the-art models. They do not manifest in typical scaling experiments, in the training of large language models, or offer practical methods for efficiency. This disconnect between theoretical exploration and practical utility forms the bedrock of the paper's central argument regarding the varying value of research efforts.

Key Findings

▶ Watch: Three key examples: Double Descent, Grokking, Lottery Ticket (1:00)

The central thesis of Jeffares and van der Schaar's position paper is articulated in two core claims, challenging the conventional wisdom and prevalent practices in deep learning research:

Firstly, the paper asserts that many prominent deep learning phenomena, particularly those identified as "edge cases" (like double descent, grokking, and the lottery ticket hypothesis), are not representative of the challenges encountered in real-world applications of deep learning. While intellectually stimulating, their isolated nature and lack of prominence in practical, large-scale model development mean that understanding them does not directly address the most pressing issues faced by practitioners, infrastructure teams, or model deployers. This observation forms the basis for questioning the utility of research solely focused on explaining these specific anomalies.

Secondly, and more controversially, the authors claim that research efforts to understand these phenomena are not all equally valuable. They argue for a fundamental reorientation of research priorities: instead of developing narrow, ad-hoc hypotheses to describe each phenomenon in isolation, the focus should be on using these phenomena as critical "edge cases" to refine and strengthen broad explanatory theories of core aspects of deep learning. This distinction is crucial, as it separates research that deepens our general understanding from research that merely provides custom explanations for specific, often synthetic, observations.

To substantiate their claim about research value, Jeffares refers to existing meta-research, specifically mentioning work by Birhane et al., which systematically analyzed stated values in machine learning research papers. This analysis revealed that utility-centered goals consistently ranked highest among researchers. Specifically, claims related to performance and generalization appeared in 96% and 89% of papers, respectively, while over half explicitly mentioned "applying to the real world" as a core value. This empirical evidence underscores that machine learning researchers are inherently pragmatic, and therefore, the value of knowledge derived from studying deep learning phenomena should be determined by its downstream utility and its contribution to practical advancements or fundamental, generalizable understanding. The paper thus argues for aligning research practices more closely with these stated community values.

Technical Deep Dive

▶ Watch: Core position: phenomena aren't real-world, research should refine theories (3:00)

The technical core of Jeffares and van der Schaar's argument lies in their precise distinction between broad explanatory theories and narrow ad-hoc hypotheses, and their call for a methodological shift towards the former. This distinction is critical for evaluating the long-term impact and value of research into deep learning phenomena.

Broad Explanatory Theories are defined as frameworks that aim to deepen our understanding of general principles governing deep learning, such as scaling laws, optimization dynamics, or generalization mechanisms. These theories are designed to be widely applicable, capable of being developed, refined, or even falsified in response to specific observations or phenomena. A classic example cited is the bias-variance tradeoff, which provides a general framework for understanding how model performance typically changes with increasing complexity. Such theories offer predictive power and generalizable insights across diverse models and tasks.

In contrast, Narrow Ad-Hoc Hypotheses are custom, isolated explanations crafted specifically for certain unexpected phenomena. They are often "overfit" to the narrow settings in which they are developed, meaning they are unlikely to provide reliable implications or extend to the broader field of deep learning. To vividly illustrate this, Jeffares describes a deliberately constructed example: a hypothetical explanation claiming that both double descent and grokking can be explained by "a number of prime value parameters in a network," positing a special role for prime numbers in optimization. He notes that this hypothetical explanation could be supported by "eye-catching plots" and "sophisticated-sounding story," with "all of the experimental results are real" in the sense that they could be generated, but the underlying theory is "not a serious theory and it certainly has no practical value." Attempting to develop regularization methods based on prime numbers, for instance, would likely be fruitless. This example serves to highlight the danger of creating explanations that are specific to an anomaly without offering any broader utility or generalizable insight.

The paper argues that deep learning phenomena should not be treated as ends in themselves, to be explained purely for their existence, but rather as scientific edge cases. These edge cases serve as crucial tests, challenging and thus helping to refine existing broad explanatory theories of deep learning. This perspective shifts the goal of research from merely describing anomalies to using them to strengthen our fundamental understanding of how deep learning systems work more generally.

To facilitate this shift, the authors propose several practical recommendations, grounded in principles of scientific inquiry:

1. Adopt a more explicitly pragmatic mindset: Research explanations should not be judged solely on their truthfulness (i.e., whether they accurately describe a phenomenon) but also on their concrete utility and potential for practical application or generalizable insight. This re-prioritization encourages researchers to consider the "so what?" question early in their investigations.

1. Treat deep learning phenomena as scientific edge cases: Rather than approaching them as isolated engineering problems to be "fixed" or explained away, they should be viewed as opportunities to rigorously test and evolve our foundational theories. This involves a commitment to the established principles of scientific inquiry.

These principles are further broken down into three categories of actionable recommendations:

• Identification and Cataloging: This involves more exploratory research focused on precisely defining the settings in which these phenomena emerge, determining their scope, and characterizing their fundamental properties. The goal is to move beyond cherry-picked results by developing standardized benchmarks that foster collaborative research and provide a clearer understanding of a phenomenon's boundaries. For example, for grokking, this would mean systematically mapping the types of tasks and model configurations where it reliably appears, rather than just in modular arithmetic.

• Prioritizing Utility: Researchers should place greater emphasis on connecting their findings to more general aspects of machine learning. This requires explicitly stating the potential path to practical relevance or how the insights contribute to broader theories of scaling, optimization, or generalization. The aim is to bridge the gap between theoretical observation and tangible impact.

• Adopting Established Best Practices of Scientific Research: This includes promoting pre-registration of experiments to reduce publication bias, encouraging the reporting of negative results to prevent the "file drawer problem" and provide a more complete picture of what does and doesn't work, and rigorously assessing reproducibility to ensure the robustness and reliability of findings. The paper also mentions providing a self-evaluation checklist to encourage researchers to reflect on these critical questions.

By championing these methodological shifts, Jeffares and van der Schaar aim to steer deep learning research towards a more impactful and cumulative trajectory, where insights into specific phenomena contribute to a more robust and comprehensive understanding of the field as a whole.

Experimental Setup & Results

▶ Watch: Broad explanatory theories vs. narrow ad-hoc hypotheses (4:40)

As a position paper, this talk does not present novel experimental results in the traditional sense, but rather uses conceptual arguments and illustrative examples to make its case. The primary "experimental" discussion revolves around the authors' deliberate construction of a narrow ad-hoc hypothesis to exemplify the kind of research they argue against.

Jeffares describes how they "deliberately construct an example of a narrow ad-hoc hypothesis—one that claims both double descent and grokking can be explained by a number of prime value parameters in a network." This hypothetical theory posits that prime numbers somehow play a special role that impacts optimization dynamics in deep learning. To make this example compelling, Jeffares notes, "We created some eye-catching plots and invented a sophisticated-sounding story around it. All of the experimental results are real and the explanations might even sound semi-plausible at first glance."

It is crucial to understand that while the experimental results (e.g., specific performance curves or training dynamics) could be synthetically generated or observed in specific, contrived settings, the explanation linking them to prime numbers is presented as a fabricated, non-serious theory. The purpose of this example is not to contribute a new finding, but to serve as a stark illustration of how easy it is to generate a seemingly plausible, yet ultimately unhelpful, explanation for an observed phenomenon. The key takeaway from this "experiment" is its lack of practical value: "developing regularization methods based on prime numbers is unlikely to be fruitful."

Therefore, this section highlights a methodological critique rather than a presentation of new empirical data. The authors use this constructed example to underscore their argument that merely observing and explaining a phenomenon, even with supporting "data," does not equate to valuable research if the explanation lacks generalizability, predictive power, or practical utility. The paper does not offer specific datasets, baselines, or hardware configurations for this illustrative example, as its intent is purely conceptual: to define and demonstrate the characteristics of a "narrow ad-hoc hypothesis."

Practical Implications

▶ Watch: Practical recommendations: pragmatic mindset, scientific edge cases (6:10)

The implications of Jeffares and van der Schaar's position are far-reaching for various stakeholders in the AI/ML ecosystem, including practitioners, infrastructure teams, model builders, and deployers, as well as the broader research community.

For practitioners, model builders, and deployers, the primary implication is a re-evaluation of which "problems" truly need solving. The paper argues that many deep learning phenomena, particularly the "edge cases" discussed, are not practical issues directly impacting the development or deployment of state-of-the-art models. This means that resources (time, compute, intellectual effort) spent on understanding these phenomena in isolation might be better redirected towards challenges that directly affect model performance, efficiency, robustness, or scalability in real-world applications. For instance, knowing that grokking occurs in synthetic settings doesn't change the best practices for early stopping in large-scale language model training, where it's not observed. Similarly, the existence of lottery tickets hasn't yet yielded a practical method for training significantly sparser, more efficient models at scale.

For researchers and the academic community, the paper calls for a significant shift in methodological approach and values. The emphasis should move from generating isolated explanations for every anomaly to using these anomalies as stress tests for broader, more fundamental theories. This encourages research that contributes to a deeper, more generalizable understanding of deep learning—such as the bias-variance tradeoff, scaling laws, or optimization landscapes—rather than accumulating a collection of disconnected explanations. This shift promises to make research efforts more cumulative and impactful.

The talk also addresses several tradeoffs and limitations inherent in their proposed approach:

• Predicting Utility: A common counter-argument, acknowledged by Jeffares, is that it's challenging to know in advance which research will ultimately prove useful. His response is that while estimating downstream impact is difficult, it is "certainly not entirely random." He points to the structure of academic funding, which is predicated on the ability to prioritize areas with likely practical impact, and individual researchers' constant choices about which questions to pursue. He argues that a "noisy estimate of utility should not be mistaken for a random estimate" and that there is significant opportunity for greater efficiency in the literature on deep learning phenomena.

• Accessibility of Simple Settings: Another point raised in the Q&A is that researchers often study these "narrow problems" because they provide settings where a meaningful number of experiments can be run, especially when large models are intractable for extensive statistical analysis. Jeffares agrees that these simple settings are "very appealing" and "accessible," and he is "definitely an advocate for taking problems in their most simple setting... when possible." His critique is not about abandoning such research but about changing the approach—using these simple settings to inform general theories rather than just explaining the specific observation within that setting.

• Second-Order Impact: The speaker acknowledges that a paper without immediate practical impact can still trigger other research that leads to deeper insights (e.g., grokking papers leading to the "quantization model of scaling"). He agrees this is a "positive outcome" but argues that the community "can be more efficient and explicit at aiming for these outcomes." Many works, he notes, explain a phenomenon without any attempt to link why their results might be useful beyond that explanation. His position advocates for explicitly striving for these "second-order outcomes" rather than treating the explanation of the phenomenon itself as the ultimate end.

In essence, the paper advocates for a more pragmatic and scientifically rigorous approach to deep learning research. It encourages researchers to constantly ask: How does this specific finding contribute to a broader, more generalizable understanding of deep learning? What is its potential downstream utility? By fostering this mindset, the community can enhance the overall efficiency and impact of its collective research efforts, ensuring that intellectual curiosity translates into tangible progress for the field.

Key Takeaways

• Focus on Utility: Many deep learning phenomena, particularly "edge cases" like double descent or grokking, are not practical issues impacting real-world applications and do not need to be "solved" in an applied sense.
• Refine Broad Theories: These phenomena are most valuable when used to challenge and refine broad explanatory theories of deep learning (e.g., the bias-variance tradeoff, scaling laws), rather than being explained in isolation.
• Prioritize General Principles: Research should aim to deepen understanding of general principles of deep learning (scaling, optimization, generalization) over developing narrow, ad-hoc hypotheses for specific observations.
• Embrace Pragmatism: Researchers should adopt a more explicitly pragmatic mindset, judging explanations not just on their truthfulness but also on their concrete utility and potential for broader impact.
• Adopt Scientific Rigor: Treat phenomena as scientific edge cases, investigating them through established best practices of scientific inquiry, including standardized benchmarks, pre-registration, reporting negative results, and assessing reproducibility.
• Explicitly State Relevance: Researchers should explicitly state the potential path to practical relevance or how their findings contribute to more general aspects of machine learning, fostering efficiency and impact.

About the Speaker(s)

Alan Jeffares is a PhD student who has dedicated over three and a half years of his doctoral research to investigating various deep learning phenomena, including double descent and grokking. His extensive engagement with these topics has led him to critically reflect on the methodologies and priorities within the deep learning research community, culminating in this position paper. His work seeks to formalize his observations and encourage a more pragmatic and scientifically grounded approach to studying deep learning.

Mihaela van der Schaar is a co-author of this position paper. While her specific role in the talk is not detailed beyond her co-authorship, she is a highly respected figure in the machine learning community, known for her work in areas such as machine learning for healthcare, causality, and interpretable AI. Her involvement lends significant academic weight to the paper's arguments and recommendations.