Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning

Speakers: Guozheng Ma, Lu Li, Zilin Wang, Li Shen, Pierre-Luc Bacon, Dacheng Tao

Conference: ICML 2025

YouTube: https://slideslive.com/39043986

Overview

This talk, presented by Lu Ma from Mila and the University of Montreal, introduces a groundbreaking approach to scaling Deep Reinforcement Learning (Deep RL) models through the strategic application of static network sparsity. While neural scaling laws have driven monumental successes in supervised learning, particularly with large language models and vision models, Deep RL has historically struggled to reap similar benefits from increased model size. The conventional wisdom suggested that simply increasing the size of vanilla MLPs in Deep RL often leads to performance degradation due to inherent network pathologies. This work directly challenges that notion, demonstrating that static sparsity is not just a computational optimization but a critical enabler for unlocking the scaling potential of Deep RL, allowing larger models to achieve superior performance by mitigating these very pathologies.

The core of the research investigates whether static network sparsity can extend the performance gains of Deep RL models beyond the limitations encountered by advanced dense architectures. Through extensive empirical studies across diverse Deep RL environments, the authors provide a clear affirmative answer. Their findings reveal that sparse networks consistently outperform their dense counterparts as model size increases, pushing past traditional scaling limits. This advancement is significant because it offers a simple, compatible, and effective method to scale Deep RL agents, potentially paving the way for more powerful and generalizable RL systems that can leverage the representational power of larger neural networks.

Background

▶ Watch: Introduction: Scaling problem in Deep RL (0:00)

The landscape of modern deep learning has been fundamentally reshaped by the neural scaling laws, which describe how model performance improves predictably with increases in compute, dataset size, and model size. This principle has been a cornerstone for the development of highly successful Large Language Models (LLMs) and large vision models, where increasing network parameters consistently leads to enhanced capabilities and generalization. However, the domain of Deep Reinforcement Learning has notoriously lagged in demonstrating the same reliable performance improvements when network size is scaled up. In fact, numerous studies have observed that simply increasing the size of standard Multi-Layer Perceptrons (MLPs) in Deep RL often results in degraded performance, rather than the expected gains.

Researchers have identified several network pathologies unique to Deep RL that explain this divergence from the supervised learning paradigm. Three prominent issues include:

1. Implicit under-parameterization: Even with a large number of explicit parameters, the network implicitly behaves as if it has fewer effective parameters, leading to poor learning and suboptimal performance. This suggests that the effective capacity of the network is not fully utilized.
2. Capacity loss: Networks trained on a sequence of non-stationary prediction targets, characteristic of the Deep RL setting, can experience a reduction in their ability to fit new targets. This loss of capacity means the network struggles to adapt to evolving environmental dynamics or policy changes.
3. Plasticity loss: Over the course of training, Deep RL networks can gradually lose their ability to adapt to new data or learn new skills. This diminished plasticity makes it difficult for the agent to continue improving or to recover from suboptimal policies, effectively stagnating learning.

Prior work has attempted to address these scaling challenges through various architectural innovations. Projects like Simba and Bro extended standard MLPs with additional components such as layer normalization and residual connections, enabling their networks to achieve better performance as they scaled. Another line of research explored scaling through dynamic sparse networks, where the connectivity patterns within the network are allowed to change during training. Techniques like dynamic sparse training and soft mixture of experts (MoE), which is a form of dynamic sparsity, have shown promise in improving performance. However, while dynamic sparsity has been investigated, the potential of static sparsity – where the network's sparse connectivity pattern is fixed prior to or early in training – in Deep RL has remained less explored despite its simplicity. This gap in understanding formed the central question of the presented research: can static network sparsity not only prevent these pathologies but also unlock further Deep RL model scaling potential beyond existing advanced architectures?

Key Findings

▶ Watch: Demonstrating performance gains across network width and depth (3:00)

The central and most impactful finding of this research is a definitive "yes": static network sparsity unequivocally unlocks the further scaling potential of Deep RL models, moving beyond the limitations encountered by current advanced architectures and effectively preventing network pathologies. Through extensive empirical studies, the authors revealed a consistent pattern: sparse networks continue to exhibit significant performance gains even at model sizes where their dense counterparts hit a clear scaling limit and begin to degrade.

Specifically, the key findings include:

• Consistent Performance Gains with Scaling: Across various scaling dimensions—increasing network depth, width, or both simultaneously—sparse networks consistently demonstrated improved performance. In contrast, dense networks showed a clear ceiling, with performance either plateauing or actively worsening as they grew larger. This was visualized through plots where solid lines (sparse networks) showed upward trends while dashed lines (dense networks) flattened or declined.
• Effectiveness at High Sparsity Ratios: The benefits of static sparsity were observed even at very high sparsity ratios, with performance improvements seen up to 90% sparsity. This indicates that the positive effects of sparsity are robust and not limited to lightly pruned networks. Larger sparse networks, even at high sparsity, maintained superior performance compared to both default-sized and large-dense networks.
• Mitigation of Network Pathologies: The research provided crucial insights into why static sparsity works, demonstrating its ability to mitigate the previously identified Deep RL network pathologies:
• Improved Representational Capacity: Sparse networks consistently achieved higher srank (a measure of effective rank indicating diversity of learned representations) in their value networks. Larger sparse networks attained the best representational capacity, directly contradicting the degradation observed in larger dense networks.
• Preserved Plasticity: Using a diagnostic reset tool, the study showed that large dense networks experienced significant performance improvements after a reset midway through training, indicating severe plasticity loss. Conversely, large sparse networks showed no such benefit from resetting, implying that their plasticity was effectively maintained throughout training.
• Reduced Overfitting: Sparse networks maintained significantly weaker correlations in the covariance structures of their gradients across different input data points compared to dense networks. This weaker correlation indicates a reduced tendency towards overfitting, contributing to better generalization and stability.
• Generality Across RL Settings: The efficacy of static sparsity was not confined to a single type of Deep RL problem. The findings generalized across:
• State-based RL: Demonstrated on the DeepMind Control Suite.
• Visual RL: Applied to pixel-based DeepMind Control Suite using DrQ-v2, specifically on the critic's MLP head.
• Streaming RL: Tested on MuJoCo locomotion tasks using the AC-lambda algorithm, a challenging setting without batch updates or replay buffers.
• Atari environments: Briefly mentioned as an additional validation domain.

In summary, static network sparsity is presented as a simple yet profoundly powerful tool that not only enables larger Deep RL models to achieve superior performance but also fundamentally addresses the underlying issues that have historically hindered scaling in this field.

Technical Deep Dive

▶ Watch: Analyzing improved representational capacity using srank (4:20)

The technical core of this research lies not just in demonstrating that static sparsity works, but in elucidating why it works by analyzing its impact on critical network properties that are often pathological in Deep RL. The authors meticulously investigated representational capacity, plasticity, and gradient covariance structures to uncover the mechanisms behind the observed performance gains.

Representational Capacity

A primary motivation for scaling up neural networks is to enhance their representational capacity, enabling them to learn richer and more effective representations of the environment and policy. To quantify this, the researchers employed srank as a metric, which measures the effective rank of the learned representations, thereby indicating their diversity and richness. In their experiments, increasing network sparsity consistently led to an improvement in the srank of the value network. This is a counter-intuitive finding, as larger dense networks, which possess more parameters, theoretically should have greater capacity. However, the results showed that scaling dense networks led to a significant degradation in srank, implying that their increased parameter count was not translating into effectively diverse representations. In stark contrast, large sparse networks maintained a very high srank throughout training, achieving the best representational capacity among all tested configurations. This suggests that static sparsity, by imposing a constrained connectivity, forces the network to utilize its parameters more efficiently, preventing the implicit under-parameterization often seen in large dense Deep RL models. The sparse structure might encourage a more distributed and diverse set of features, rather than redundant or highly correlated ones that can emerge in dense, over-parameterized networks struggling with non-stationary data.

Plasticity Loss Mitigation

Plasticity loss is a critical pathology in Deep RL, where networks gradually lose their ability to adapt to new information or refine existing policies. To diagnose this, the researchers used a direct and intuitive tool: network reset. By fully resetting a portion of the network (e.g., the policy or value head) at a halfway point during training (specifically, at 0.5 million environmental steps), they could assess whether the network had lost its ability to learn from scratch or from a fresh state. For large dense networks, applying this reset led to a significant improvement in performance, indicating that these networks had indeed suffered from severe plasticity loss and benefited greatly from being "rebooted." This suggests that the dense networks became "stuck" in suboptimal configurations or lost their ability to explore the parameter space effectively. Crucially, when the same reset was applied to large sparse networks, no significant performance benefit was observed. This finding strongly suggests that large sparse networks do not suffer from severe plasticity loss; they maintain their adaptability and learning capabilities throughout the training process, enabling continuous improvement even over long training horizons. The sparse connectivity might act as a regularizer, preventing the network from becoming overly rigid or committing too strongly to early, potentially misleading, representations.

Gradient Covariance Structures

The covariance structures of gradients play a vital role in both optimization dynamics and generalization capabilities. Strong correlations among gradients from different data points or input distributions can lead to unstable optimization, make the network susceptible to local minima, and exacerbate overfitting. The researchers analyzed these structures after 1 million training steps. Their findings revealed that sparse networks consistently maintained significantly weaker correlations in the gradients compared to dense networks. Darker colors in their visualization indicated stronger correlations, and sparse networks showed much lighter correlation patterns. This weaker correlation is a crucial advantage, as it can mitigate overfitting by ensuring that the network's updates are less biased towards specific, potentially idiosyncratic, training samples. It also likely contributes to more stable and effective optimization, allowing the network to navigate the complex loss landscapes of Deep RL more robustly. The constrained connectivity of sparse networks might inherently reduce the potential for parameters to co-adapt in an undesirable, highly correlated fashion, thereby promoting more independent and diverse learning signals.

Generality of Findings

The technical insights were validated across diverse Deep RL paradigms, underscoring the generality of static sparsity's benefits:

• Visual RL: In pixel-based DeepMind Control Suite experiments using DrQ-v2, static sparsity was applied specifically to the critic's MLP head. The results consistently showed that the largest critic networks, when combined with the highest sparsity ratios, achieved the best performance. This demonstrates that the benefits extend to high-dimensional visual inputs where representation learning is even more critical.
• Streaming RL: This highly challenging setting, exemplified by MuJoCo locomotion tasks using the AC-lambda algorithm, involves agents processing samples immediately without storing past experience (no batch updates or replay buffers). This environment exacerbates non-stationarity and stability challenges. Even here, the largest critic networks with the highest sparsity ratios consistently delivered the best performance, highlighting sparsity's ability to maintain stability and coverage in highly dynamic and data-constrained learning scenarios.

In essence, static network sparsity acts as a powerful architectural inductive bias that intrinsically addresses the core pathologies of Deep RL. By promoting efficient representation learning, preserving adaptability, and fostering robust optimization, it transforms larger networks from problematic to performant, thereby unlocking their true scaling potential.

Experimental Setup & Results

▶ Watch: Sparse networks mitigate plasticity loss issues (5:00)

The research employed a comprehensive experimental methodology to validate the effectiveness and generality of static network sparsity across a spectrum of Deep RL environments and algorithms.

State-Based Deep RL Experiments

• Environments: The primary testbed for state-based Deep RL was the DeepMind Control Suite, a collection of continuous control tasks known for their varied dynamics and complexity.
• Algorithms: The agents were trained using two popular off-policy algorithms: Soft Actor-Critic (SAC) and Deep Deterministic Policy Gradient (DDPG).
• Network Architecture: The base architecture utilized was Simba-based networks, an advanced architecture known for its scalability improvements over vanilla MLPs.
• Scaling Methods: Model size was systematically scaled in three ways:

1. Increasing Depth: Keeping width fixed and adding more layers.
2. Increasing Width: Keeping depth fixed and expanding the hidden dimension of layers.
3. Increasing Both: Simultaneously expanding both depth and width.

• Sparsity Application: For width scaling experiments, a different sparsity ratio was assigned to each model size to preserve the same amount of learnable parameters as a default dense network with a width scale of two. This meant larger networks generally used higher sparsity ratios. For other experiments, sparsity ratios ranged from 0.1 to 0.9 (10% to 90% sparsity).
• Results: A clear trend emerged: sparse networks consistently showed performance gains as model size increased across all three scaling methods. In contrast, dense networks either plateaued or exhibited performance degradation beyond a certain size. For instance, when increasing width, sparse networks continued to improve while dense networks worsened. The large sparse networks, even with 90% sparsity, significantly outperformed default-sized networks and large dense networks. The largest networks explored in this study were considerable, with some configurations exceeding 100 million parameters, featuring up to 4000 hidden dimensions per layer, and comprising eight modules, each consisting of two linear layers.

Visual Deep RL Experiments

• Environments: To assess performance in high-dimensional observation spaces, experiments were conducted on the pixel-based DeepMind Control Suite.
• Algorithm: The DrQ-v2 algorithm, a state-of-the-art method for visual RL, was employed.
• Sparsity Application: Scaling and static sparsity were applied specifically to the critic's MLP head, which processes the features extracted from the visual encoder.
• Configurations: Four different critic widths and three different sparsity ratios were evaluated.
• Results: Consistent with state-based findings, the largest critic networks combined with the highest sparsity ratios consistently achieved the best performance across these visual tasks.

Streaming Deep RL Experiments

• Environments: This challenging setting was tested on MuJoCo locomotion tasks, specifically "Quadruped Run" and "Hopper Hop." Streaming RL differs from conventional settings by requiring immediate sample processing without storing past experience (no batch updates, no replay buffer), thus exacerbating non-stationarity.
• Algorithm: The AC-lambda algorithm was used to train the agents.
• Configurations: Similar to visual RL, four different network widths and three different network sparsity ratios were tested.
• Results: The results mirrored those of visual RL: the largest critic networks, when combined with the highest sparsity ratios, always achieved the best performance in both "Quadruped Run" and "Hopper Hop."

General Observations

The comprehensive experimental results robustly confirm that static network sparsity is a highly effective and generalizable technique. It consistently enables larger Deep RL models to achieve superior performance across diverse environments (state-based, pixel-based, streaming, and briefly mentioned Atari), algorithms (SAC, DDPG, DrQ-v2, AC-lambda), and scaling methodologies (depth, width, combined). The critical finding is that this performance gain is not merely an incremental improvement but a fundamental unlocking of scaling potential that dense networks fail to achieve due to inherent pathologies.

Practical Implications

▶ Watch: Generality: Sparsity benefits Visual and Streaming RL (6:30)

The findings presented in this research carry significant practical implications for anyone involved in building, deploying, or researching Deep Reinforcement Learning systems. The discovery that static network sparsity can unlock the scaling potential of Deep RL offers a straightforward yet powerful tool to enhance agent performance and robustness.

Firstly, the simplicity of implementation stands out. Unlike complex dynamic sparsity schemes that require intricate scheduling or specialized hardware, static sparsity involves fixing the network's connectivity pattern prior to or early in training. This makes it remarkably easy to integrate into existing Deep RL pipelines. The authors emphasize that it is compatible with any RL algorithm, meaning practitioners do not need to overhaul their entire framework to leverage its benefits. This broad compatibility drastically lowers the barrier to adoption for infra teams and model builders.

For model builders, this work suggests a viable path to developing more powerful and capable Deep RL agents. The ability to scale models to significantly larger sizes (e.g., beyond 100 million parameters as explored in the talk) without performance degradation means that agents can learn more complex representations and potentially tackle harder, more intricate tasks. This could lead to breakthroughs in areas currently limited by the representational capacity of smaller networks. The mitigation of pathologies like capacity loss and plasticity loss directly translates into more stable training and agents that can continue to learn and adapt over longer durations, reducing the need for manual intervention or tricky hyperparameter tuning related to stability.

Infrastructure teams might initially perceive sparsity as a computational burden due to the need for specialized sparse matrix operations or potential memory overheads if not handled efficiently. However, the talk implicitly suggests that the effective computation might be reduced if the sparse operations are optimized. The primary benefit here isn't necessarily a direct speedup in all cases (though reduced parameters can lead to faster inference if implemented efficiently), but rather the ability to achieve better performance with larger models that would otherwise be intractable or perform poorly. This shifts the focus from purely computational efficiency to performance-per-parameter efficiency in a pathological setting.

Tradeoffs and Limitations:

• Memory Footprint: While sparse networks have fewer active parameters, their memory footprint might not always be strictly smaller than dense networks if sparse tensors are not efficiently represented or if meta-data about sparsity patterns needs to be stored. However, for extreme sparsity, significant memory savings are possible.
• Hardware and Software Support: Efficiently executing sparse operations on generic hardware (e.g., standard GPUs) can be challenging, as dense matrix multiplication is often heavily optimized. Specialized sparse kernels or hardware accelerators (like some TPUs or custom ASICs) would be ideal to fully realize the computational benefits. The article highlights the performance benefit, which might outweigh potential minor compute inefficiencies in settings where dense networks simply fail to scale.
• Extent of Scaling: While the authors scaled networks to "very large size" (e.g., 100M+ parameters, 4000 hidden dimensions), the question remains whether there's an ultimate limit to this scaling. The speaker acknowledged that "for maybe harder task we can scale it for more for a larger network," suggesting that the current limits were primarily task-specific rather than fundamental to the sparsity approach itself.
• Choice of Sparsity Pattern: The talk focuses on static sparsity but doesn't delve into the specifics of how the static sparsity pattern is chosen or initialized. This could be a hyperparameter or an area for future research.

In essence, static network sparsity offers a robust, easy-to-implement solution to a long-standing problem in Deep RL. It empowers practitioners to build more capable agents by leveraging the power of larger models, mitigating critical training pathologies, and doing so with a technique that is broadly compatible with existing RL ecosystems.

Key Takeaways

• Static network sparsity is a powerful enabler for Deep RL scaling: Unlike dense networks, sparse networks continue to show performance gains as model size (depth, width, or both) increases, well beyond the point where dense counterparts degrade.
• Mitigates core Deep RL network pathologies: Static sparsity addresses implicit under-parameterization, capacity loss, and plasticity loss, which are major bottlenecks for scaling in Deep RL.
• Enhances representational capacity and plasticity: Sparse networks maintain higher srank (diverse representations) and retain plasticity throughout training, allowing for continuous learning and adaptation.
• Reduces overfitting and improves optimization: Sparse networks exhibit weaker correlations in gradient covariance structures, leading to more stable training and better generalization.
• Highly generalizable and easy to implement: The benefits of static sparsity extend across state-based, visual, and streaming Deep RL environments, and are compatible with various RL algorithms, making it a practical tool for practitioners.
• Unlocks potential for larger, more capable agents: This approach allows for the effective use of very large networks (e.g., over 100 million parameters) in Deep RL, opening avenues for tackling more complex tasks.

About the Speaker(s)

The primary speaker for this presentation was Lu Ma, a PhD student affiliated with both Mila and the University of Montreal. Lu Ma's research focuses on addressing fundamental challenges in Deep Reinforcement Learning, particularly around model scaling and network pathologies, as demonstrated by this presented work. The co-authors on this paper include Guozheng Ma, Zilin Wang, Li Shen, Pierre-Luc Bacon, and Dacheng Tao, all contributing to this significant advancement in the field of Deep RL.