Theoretical Limitations of Ensembles in the Age of Overparameterization

Theoretical Limitations of Ensembles in the Age of Overparameterization

Speakers: Niclas Dern, Incoming Student, Berkeley; John Cunningham, Columbia; Geoff Pleiss, Vector Institute

Conference: ICML 2025

YouTube: https://slideslive.com/39044083

Overview

This talk, presented by Geoff Pleiss, explores the theoretical underpinnings of deep ensembles in the contemporary landscape of overparameterized models. The research, primarily driven by Niclas Dern as an intern at the Vector Institute, with contributions from John Cunningham at Columbia, addresses a critical question in machine learning: how do ensembles of very large models compare to a single, equivalently large model, especially concerning their predictive capabilities and the nature of their uncertainty estimates? The motivation stems from empirical observations suggesting a surprising equivalence between these two approaches when operating under a fixed computational or parameter budget. This work is significant because ensembles of neural networks are widely adopted in practice for their purported benefits in boosting accuracy, enhancing robustness, and providing crucial uncertainty estimates, particularly in safety-critical applications. By rigorously investigating the theoretical limits of ensembles in the overparameterized regime, the authors challenge conventional wisdom and offer a new perspective on their functional role.

The presentation posits that the perceived advantages of ensembles, particularly in providing unique signals for uncertainty quantification, may be fundamentally altered or diminished when individual ensemble members, or the single large model, operate in an overparameterized state—where the number of parameters significantly exceeds the number of training data points. This research aims to provide theoretical evidence for previously observed empirical correlations between large single models and ensembles, suggesting that these correlations are not merely artifacts of specific architectures or optimization procedures but rather a fundamental property of high-dimensional statistics. The findings have profound implications for how practitioners design, train, and deploy large-scale machine learning models, prompting a re-evaluation of the true benefits and limitations of ensemble methods in the age of increasingly powerful and overparameterized deep learning systems.

Background

▶ Watch: Introduction: Benefits of deep ensembles for robustness and uncertainty (0:00)

The use of ensemble methods has a long and storied history in machine learning, traditionally celebrated for their ability to improve model accuracy and generalize better than individual models. In the context of deep learning, ensembles of neural networks have gained particular traction, extending their perceived benefits beyond mere accuracy gains to encompass enhanced robustness against adversarial attacks or distributional shifts, and crucially, the provision of uncertainty estimates. This latter point is often highlighted as a unique advantage of ensembles: by observing the variance in predictions across multiple component models, practitioners aim to quantify the model's confidence in its output, a feature not readily available from a single deterministic neural network. This makes ensembles attractive for applications demanding high reliability, such as autonomous driving or medical diagnostics.

However, recent empirical investigations by the authors and others have begun to challenge some of these long-held assumptions, particularly when considering the landscape of modern deep learning where models are often massively overparameterized. Their prior work, for instance, compared a single very large neural network against an ensemble of multiple large but not excessively large models, both constrained by the same total compute or parameter budget. Surprisingly, they observed a very tight pointwise correlation between the predictions of the single large model and the averaged predictions of the ensemble. This correlation held not only for in-distribution data but also under significant distributional shifts, suggesting that the two approaches, despite their architectural differences, might be functionally equivalent in many practical scenarios. This empirical observation sparked the central research question of this talk: can this apparent functional equivalence between large single models and ensembles be theoretically characterized, especially considering the pervasive role of overparameterization?

Traditionally, ensembles have been studied in settings involving "weak learners" or models with limited capacity, such as decision trees, where the goal is to combine diverse, underperforming models to create a stronger, more robust predictor. The current era of deep learning, however, involves models capable of memorizing entire training datasets, operating far beyond the classical capacity limits. This shift necessitates a re-examination of ensemble theory. Furthermore, the talk specifically probes the nature of ensemble variance: if ensembles and single large models are functionally equivalent in their predictions, what exactly does the variance among ensemble members represent, and does it genuinely correspond to a meaningful, classical notion of uncertainty, such as that derived from Bayesian inference? These questions underscore the need for a rigorous theoretical framework to understand the behavior of ensembles in the high-dimensional, overparameterized regime.

Key Findings

▶ Watch: Research Question 1: Ensembles vs. single large models in overparameterization (3:40)

The central and most significant finding of this research is the theoretical demonstration of functional equivalence between a single, infinitely wide random feature regressor (RFR) and an infinite ensemble of fixed-width RFRs, provided that the ensemble members are overparameterized. Specifically, both configurations converge to the same limit: the kernel ridgeless regressor, defined by the expected inner product of the random features, as their respective widths or ensemble sizes approach infinity. This profound result provides a theoretical explanation for the empirical observations of tight correlations between large single models and ensembles, particularly in the context of deep neural networks.

A crucial aspect of this convergence is the minimal distributional assumptions made about the random features. Unlike prior work that often relied on strong assumptions about feature distributions, this research establishes the equivalence under almost zero distributional constraints, significantly broadening the applicability of the result. Furthermore, the authors demonstrate that this main result approximately holds even under finite feature budgets (i.e., when models are not infinitely wide) and when ridge regularization is applied during the training of ensemble members. This suggests that the theoretical equivalence is not merely an asymptotic curiosity but has practical relevance for finite-sized, regularized models.

The talk explicitly highlights that this equivalence critically hinges on the overparameterization of the models. If the ensemble members (or the single large model) are not overparameterized—meaning they have fewer features than training data points and thus cannot perfectly interpolate the training data—then the functional equivalence breaks down. This distinction is vital, as it delineates the conditions under which ensembles might offer distinct advantages versus when they become functionally interchangeable with a single large model.

Finally, the research offers a novel characterization of ensemble variance. Rather than representing a standard notion of uncertainty (like Bayesian posterior variance), the authors show that ensemble variance, in the overparameterized regime, is essentially equivalent to the expected difference in prediction between a single larger model and a single smaller model. In simpler terms, it quantifies how much a model's prediction would change if its capacity were increased. This reinterpretation fundamentally alters how practitioners should understand and utilize the uncertainty signals derived from ensembles, indicating that they do not correspond to classical Bayesian uncertainty.

Technical Deep Dive

▶ Watch: Theoretical model: Analyzing ensembles of Random Feature Regressors (5:10)

To provide a theoretical foundation for their empirical observations, the authors simplify the complex world of deep neural networks by studying ensembles of random feature regressors (RFRs). An RFR can be conceptualized as a one-hidden-layer neural network where the weights in the first hidden layer are fixed, and only the final set of linear weights are learned. This architecture effectively transforms the problem into linear basis regression, where the fixed random features serve as basis functions. While simpler than full neural networks, RFRs are known to capture many high-dimensional statistical properties relevant to deep learning, and recovering similar results in this simplified setting lends credence to the idea that the observed phenomena are fundamental, rather than specific to particular optimizers or hyperparameters of deep learning.

The core comparison involves two setups, both operating under a fixed total parameter budget of M * D parameters:

1. Single Large Model: One RFR with M * D basis functions.
2. Ensemble of Models: An ensemble of M distinct RFRs, each having D basis functions. The random features for each ensemble member are drawn independently and identically distributed (IID) from some distribution.

The theoretical analysis then proceeds by examining the behavior of these models as their capacity or ensemble size tends towards infinity.

• Single Large RFR in the Infinite-Width Limit: It is well-established in the literature that as a single RFR becomes infinitely wide (i.e., the number of basis functions goes to infinity), its predictions converge to those of a kernel ridgeless regressor. This regressor is defined by a kernel function that corresponds to the expected inner product of the random features. This limit behavior is a cornerstone of the Neural Tangent Kernel (NTK) theory and related fields.

• Ensemble of RFRs in the Infinite-Ensemble-Size Limit: The key contribution of this work is demonstrating that an infinite ensemble of fixed-width RFRs (where M goes to infinity while D remains fixed) also converges to exactly the same limit: the kernel ridgeless regressor. This means that as M approaches infinity, the averaged predictions of the ensemble become indistinguishable from the predictions of an infinitely wide single RFR. This convergence holds under remarkably weak distributional assumptions on the random features, signifying a robust theoretical result. The authors note that the lack of strong distributional assumptions is critical for the subsequent statements about uncertainty quantification.

A critical condition for this functional equivalence is that both the single large model and the individual members of the ensemble must be operating in the overparameterized regime. This means the number of features (M*D for the single model, D for each ensemble member) must exceed the number of training data points. If models are not overparameterized (i.e., they are in the "underparameterized" or "classical" regime), a disconnect between the infinitely wide model and the ensemble can be observed. The result holds once the models cross this overparameterization threshold, regardless of the individual width of ensemble members beyond that point.

The theoretical framework further allows for a precise characterization of ensemble variance. The variance of an ensemble's predictions can be expressed as the expected difference between the ensemble prediction and the expected single model prediction. Given the proven functional equivalence, this translates to the difference between a single larger model's predictions and a single smaller model's predictions. Specifically, ensemble variance is shown to reflect the expected change in prediction if the model's capacity were increased. This implies that ensemble variance, at least in the overparameterized setting, is not a direct proxy for classical uncertainty (e.g., aleatoric or epistemic uncertainty as defined in Bayesian inference), which often seeks to quantify uncertainty due to noise in data or lack of knowledge about model parameters. Instead, it provides a signal related to model capacity and its impact on predictions.

Experimental Setup & Results

▶ Watch: Comparing single large RFR vs. ensemble of smaller RFRs (6:25)

To complement their theoretical arguments, the authors provide a concise experimental verification of their core findings using a simplified regression problem. While the primary contribution is theoretical, this empirical demonstration helps to ground the abstract concepts in a concrete, illustrative example.

The experimental setup involved a very basic regression task with a small dataset of just six data points. This minimal setup was chosen to clearly visualize the behavior of random feature regressors and their ensembles. The random features were likely generated from a simple distribution, consistent with the minimal distributional assumptions of their theoretical proof.

The results were visualized as follows:

• Individual Ensemble Member Predictions: Blue lines were used to represent the predictions made by each single component model within the ensemble. These individual predictions typically exhibit variability and might not perfectly fit the training data due to their limited individual capacity (D basis functions) and the randomness in their features.
• Averaged Ensemble Prediction: When these individual predictions were averaged together, the resulting curve was shown to exactly correspond to the kernel ridgeless regressor. This visual confirmation directly supports the theoretical claim that the infinite ensemble limit converges to the same target function as an infinitely wide single model.

Although the theoretical proof was developed for random feature regressors, the talk explicitly states that these findings also hold empirically for deep neural networks. This is a critical point, as it bridges the gap between the simplified theoretical model and the complex models used in practice. The initial motivation for this research came from empirical observations with neural networks, and the theoretical work provides a deeper explanation for those observations. The experimental evidence, though simple, visually reinforces the concept of functional equivalence and the convergence to the kernel ridgeless regressor, making the abstract theoretical result more tangible.

Practical Implications

▶ Watch: Key result: Infinite ensemble converges to the exact same limit (8:00)

The theoretical limitations identified for ensembles in the age of overparameterization carry significant practical implications for machine learning practitioners, infrastructure teams, model builders, and deployers. The most immediate takeaway is that if a single large, overparameterized model is functionally equivalent to an ensemble of similarly overparameterized models, then the choice between them may boil down to practical considerations rather than fundamental performance or capability differences.

One key implication for practitioners is the re-evaluation of the "tool of convenience" aspect of ensembles. While training a single, monolithic, extremely large model can be challenging due to memory constraints and sequential dependencies, training an ensemble of several smaller (though still overparameterized) models can be much more easily parallelized. This parallelism can lead to faster training times and more efficient resource utilization, making ensembles an attractive option from an operational perspective, even if they don't offer a distinct functional class. Infrastructure teams might find it easier to manage and scale training pipelines for ensembles rather than for a single giant model.

Furthermore, the findings challenge the conventional understanding of uncertainty quantification using ensemble variance. If ensemble variance primarily signals the expected difference in prediction due to a change in model capacity rather than a standard notion of uncertainty (like Bayesian posterior variance), practitioners must adjust their interpretation. Relying on ensemble variance as a direct measure of model confidence or epistemic uncertainty in safety-critical applications might be misleading. Instead, alternative or complementary methods for uncertainty quantification might be necessary, or a more nuanced understanding of what ensemble variance truly represents is required. This means model builders might need to reconsider how they communicate uncertainty to downstream users or decision-makers.

The research also highlights the critical role of overparameterization. If models are not overparameterized, ensembles might indeed offer distinct advantages. However, given the widespread trend of training increasingly large, overparameterized models in deep learning, this work suggests that many deployed ensembles might already be operating in a regime where their functional benefits over a single large model are diminished. This calls for careful consideration of model size and the overparameterization threshold when designing and deploying ensemble systems.

Tradeoffs and Limitations:

• Computational Cost: While ensembles can be parallelized, they still involve training and storing multiple models, which can be computationally expensive during both training and inference. If a single large model offers equivalent performance, it might be more efficient in terms of inference time and memory footprint (though not necessarily training time).
• Model Diversity: The results hold for specific types of ensembles where the "diversity" primarily comes from random features or optimization paths, leading to similar functional forms. The talk explicitly states that these results do not apply to Mixture of Experts (MoE) ensembles or other types of ensembles where component models are fundamentally different or where the training procedure induces distinct functional behaviors (e.g., using different architectures, data subsets, or learning objectives for each member). In such cases, ensembles might still provide unique benefits not achievable by a single large model.
• Theoretical vs. Empirical Gap: While the theory is for RFRs and shows empirical support for neural networks, there might still be specific scenarios or architectural nuances in complex deep learning models where the equivalence does not hold perfectly, or where subtle differences become practically significant.

In essence, this research encourages a more sophisticated understanding of ensembles, moving beyond a blanket assumption of their superiority to a nuanced view that considers the model's capacity, the degree of overparameterization, and the specific nature of the ensemble's construction.

Key Takeaways

• Functional Equivalence in Overparameterization: For random feature regressors, a single infinitely wide model and an infinite ensemble of fixed-width, overparameterized models converge to the same kernel ridgeless regressor limit.
• Empirical Generalization: This theoretical equivalence, initially proven for RFRs, has been observed to hold empirically for deep neural networks as well, suggesting a fundamental property of high-dimensional models.
• Overparameterization is Key: The functional equivalence critically depends on the models operating in the overparameterized regime (more features than training data points); this equivalence breaks down otherwise.
• Ensemble Variance Recharacterized: In overparameterized settings, ensemble variance does not correspond to standard notions of uncertainty (e.g., Bayesian posterior variance). Instead, it reflects the expected difference in prediction if the model's capacity were to be increased.
• Practical Implications for Design: The choice between a single large model and an ensemble of smaller models, when both are overparameterized, may be driven by practical considerations like parallelization efficiency rather than inherent functional advantages.
• Limitations: These findings do not apply to all types of ensembles, specifically excluding Mixture of Experts (MoE) or other ensembles where component models are designed for distinct functional roles or trained with fundamentally different procedures.

About the Speaker(s)

The talk was presented by Geoff Pleiss, who is associated with the Vector Institute. He introduced the work as a collaborative effort, highlighting his role as an academic mentor.

The primary driver of this research was Niclas Dern, an incoming student at Berkeley. He conducted most of this work as an intern under Geoff Pleiss at the Vector Institute. Although unable to present, his significant contributions were acknowledged as central to the project.

John Cunningham from Columbia University was also a co-author, contributing to the joint work.