Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration

Controlling Underestimation Bias in Constrained Reinforcement Learning for Safe Exploration

Speakers: Shiqing Gao, Jiaxin Ding, Luoyi Fu, Xinbing Wang

Conference: ICML 2025

YouTube: https://slideslive.com/39043984

Overview

This talk introduces a critical challenge in Constrained Reinforcement Learning (CRL): the pervasive issue of cost underestimation bias, which leads to unsafe exploration in safety-critical applications. Presented on behalf of lead author Shiqing Gao from Shanghai Jiao Tong University, the work proposes a novel method called Memory-driven Intrinsic Cost Estimation (MICE). MICE is designed to accurately estimate costs, thereby mitigating safety violations without compromising policy performance.

The research delves into the root causes of this underestimation, identifying how standard constraint optimization techniques can inadvertently distort cost estimations, making dangerous states appear safer than they truly are. By drawing inspiration from human "flashbulb memory"—the vivid recall of dangerous events—MICE incorporates a mechanism to learn and recall unsafe states, integrating this experience into a more robust cost estimation framework. This innovative approach promises to significantly enhance the safety and reliability of CRL agents in real-world deployments, from autonomous driving to robotics.

The implications of effectively controlling underestimation bias are profound. In environments where safety is paramount, such as self-driving cars navigating complex traffic scenarios or robotic systems operating alongside humans, the ability to accurately assess and avoid risks is non-negotiable. MICE offers a principled solution to a fundamental problem that has historically hindered the widespread adoption of CRL in such sensitive domains, paving the way for more trustworthy and robust AI systems.

Background

▶ Watch: Introduction to Constrained RL and safety challenges (0:00)

Reinforcement Learning (RL) provides a powerful framework for agents to learn optimal behaviors through trial and error, maximizing an expected reward signal from interactions with an environment. However, in many real-world applications, maximizing reward alone is insufficient. Safety is often a critical, non-negotiable requirement. This necessity gives rise to Constrained Reinforcement Learning (CRL), an extension of RL where an agent must not only maximize expected reward but also satisfy one or more safety constraints, typically expressed as an expected cumulative cost that must remain below a predefined threshold.

Consider the example of autonomous driving: the agent's primary goal is to reach a destination quickly (maximizing reward), but it must do so safely by avoiding collisions with obstacles or other vehicles (satisfying constraints). In such scenarios, there are often multiple paths to a goal; some are shorter but riskier, while others are longer but safer. The ideal CRL agent should identify the shortest safe path, balancing efficiency with safety.

A significant challenge in current CRL methods is their frequent failure to uphold safety constraints during training. Policies often violate acceptable thresholds, leading to "unsafe exploration" in risky regions of the state space. The talk highlights a key reason for these violations: a pervasive "underestimation bias" in the cost value function. The cost critic, a neural network trained to estimate the cumulative cost from a given state, consistently underestimates the true cost, particularly in high-cost regions. This makes dangerous states appear deceptively safe, encouraging the agent to explore them and consequently leading to safety breaches.

The root cause of this underestimation bias is traced back to the mechanics of constraint optimization itself. In regions of the state space where costs are high, the policy's objective is to rapidly reduce these costs. This minimization process, however, inadvertently distorts the inherent noise distribution within the learning process, breaking its crucial zero-mean property. A zero-mean noise distribution is fundamental for unbiased estimation in many learning algorithms. When this property is violated, the estimated cost values become systematically lower than their true counterparts, especially in critical, high-cost scenarios. This phenomenon has been empirically observed across various CRL methods and environments, with the measured bias (estimated cost minus true cost) almost always found to be negative. The central challenge, therefore, lies in effectively reshaping cost estimation to mitigate this underestimation in high-cost regions, thus preventing risky exploration, without simultaneously becoming overly conservative and unduly limiting beneficial exploration.

Key Findings

▶ Watch: Flashbulb memory: inspiration for intrinsic cost estimation (4:20)

The research presented in this talk uncovers several pivotal findings that significantly advance the field of Constrained Reinforcement Learning:

1. Identification and Characterization of Underestimation Bias: The talk definitively identifies cost underestimation bias as a prevalent and critical issue in existing CRL methods. This bias, where the estimated cost is consistently lower than the true cost, particularly in high-cost regions, is shown to be a direct cause of unsafe exploration and constraint violations during training.
2. Mechanism of Bias Generation: A key contribution is the explanation of why this underestimation occurs. It is attributed to the policy's attempt to minimize the cost objective in high-cost regions, which inadvertently distorts the noise distribution within the learning algorithm, breaking its zero-mean property. This distortion leads to systematically underestimated cost values.
3. Introduction of Memory-driven Intrinsic Cost Estimation (MICE): Inspired by human cognitive processes like "flashbulb memory," the authors propose MICE. This novel framework integrates a learned intrinsic cost component derived from a memory of past unsafe states. MICE aims to counteract the underestimation bias by providing a more accurate and conservative (when necessary) cost estimate.
4. Effective Bias Mitigation: Experimental results demonstrate that MICE effectively mitigates cost underestimation bias. Across various safety-critical tasks, MICE's estimation bias (estimated cost - true cost) approaches zero, indicating a significantly improved accuracy in risk assessment by the agent.
5. Enhanced Safety and Performance: MICE consistently and significantly reduces constraint violations during training, ensuring the agent stays below safety thresholds. Crucially, it achieves this without sacrificing policy performance, often maintaining or even improving cumulative reward compared to baseline methods. This highlights that MICE is not merely being overly conservative but is enabling safer and more effective learning.
6. Theoretical Guarantees: The MICE framework comes with robust theoretical backing. The authors prove that MICE achieves a tighter upper bound on constraint violations, including a positive term that explicitly guarantees fewer violations during training. Furthermore, under standard assumptions, MICE's value function is proven to converge to the optimal value, ensuring both safety and learning stability.
7. Validation through Ablation Study: An ablation study confirms the critical role of the intrinsic cost component. Removing this feature and replacing it with a constant led to a significant drop in policy performance, underscoring that the sophisticated, memory-driven intrinsic cost mechanism is essential for MICE's ability to balance safety with strong performance.

These findings collectively present a comprehensive diagnosis of a fundamental problem in CRL and offer a theoretically sound and empirically validated solution that promises to make AI agents deployed in safety-critical environments significantly more reliable and trustworthy.

Technical Deep Dive

▶ Watch: MICE's theoretical guarantees: tighter bounds and convergence (6:50)

The core technical challenge addressed by this work is how to reshape cost estimation in Constrained Reinforcement Learning to effectively mitigate underestimation bias without rendering the agent overly conservative. The proposed solution, Memory-driven Intrinsic Cost Estimation (MICE), is a multi-faceted approach inspired by human cognitive learning.

The genesis of MICE lies in the observation that humans, after experiencing a dangerous event, form "flashbulb memories" that vividly recall the incident, leading to heightened caution in similar future situations. Translating this into a machine learning context, MICE introduces a cost memory module. This memory stores a collection of unsafe states encountered during previous training iterations. These are states where the agent violated a constraint or entered a high-cost region.

During subsequent learning steps, MICE computes an intrinsic cost for the current state. This is achieved by measuring the similarity between the current state and the states stored in the cost memory. Various similarity metrics could be employed, such as Euclidean distance in a learned state embedding space, or cosine similarity. The intuition is straightforward: the more similar the current state is to a previously encountered unsafe state, the higher the likelihood of it leading to a high-cost outcome. Consequently, a higher intrinsic cost is assigned to such states. This intrinsic cost acts as an internal "warning signal," augmenting the agent's perception of risk based on past negative experiences.

This intrinsic cost is then combined with the extrinsic cost received directly from the environment to form a new, comprehensive cost estimate. Mathematically, this might look like $C_{total} = C_{extrinsic} + \alpha \cdot C_{intrinsic}$, where $\alpha$ is a dynamically adjusted intrinsic factor. This combined cost estimate is designed to better recognize and internalize risk, particularly in those high-cost regions where the extrinsic cost critic typically suffers from underestimation bias. By artificially inflating the perceived cost in potentially dangerous areas, MICE effectively counters the downward bias caused by the policy's cost minimization objective distorting the noise distribution. It provides a more accurate, and at times more conservative, assessment of risk, guiding the agent away from unsafe trajectories.

To prevent MICE from becoming too conservative and unnecessarily limiting exploration or performance, a crucial component is the dynamic adjustment of the intrinsic factor ($\alpha$). This factor is not static but changes based on how biased the cost estimate is. If the agent consistently underestimates costs, $\alpha$ might increase, making the intrinsic cost component more influential. Conversely, if the agent is overly cautious and rarely violates constraints, $\alpha$ might decrease, allowing for more efficient exploration. The exact mechanism for dynamic adjustment could involve monitoring the observed constraint violations or the difference between estimated and true costs over recent episodes. This adaptive control ensures a delicate balance: mitigating underestimation without stifling beneficial exploration.

Finally, the policy in MICE is trained to satisfy this combined constraint while also adhering to a trust region. Training within a trust region ensures that policy updates do not drastically deviate from the policy that collected the data for the cost memory. This is vital for two reasons: firstly, it maintains the relevance and utility of the stored unsafe states in the memory, as the memory remains representative of the policy's recent behavior. Secondly, it contributes to the stability of the learning process, preventing erratic policy changes that could lead to new, unforeseen safety violations.

On the theoretical front, MICE offers significant guarantees. The authors prove that MICE achieves a tighter upper bound on constraint violations compared to existing methods. This tighter bound includes a positive term that explicitly guarantees fewer constraint violations during training. This is a critical theoretical result, as it provides a mathematical assurance of enhanced safety performance. Moreover, under standard assumptions common in RL theory, MICE's value function is proven to converge to the optimal value. This dual guarantee of improved safety and convergence to optimality underscores the robustness and effectiveness of the MICE framework. The combination of a memory-driven intrinsic cost, dynamic weighting, and trust-region optimization provides a comprehensive and theoretically sound solution to the long-standing problem of underestimation bias in CRL.

Experimental Setup & Results

▶ Watch: Experimental evaluation on safety-critical environments (7:30)

To validate the effectiveness of Memory-driven Intrinsic Cost Estimation (MICE), the researchers conducted a series of experiments across a range of standard safety-critical environments. These environments were chosen to represent common challenges in real-world constrained reinforcement learning applications, encompassing both continuous control and navigation tasks.

The experimental suite included:

• Navigation Tasks: In these scenarios, the primary constraint was to avoid collisions with obstacles or "roadblocks." Agents had to learn to navigate to a target destination efficiently while strictly adhering to safety boundaries.
• Robot Control Tasks: These involved controlling robotic systems, with constraints typically related to maintaining operational parameters within safe thresholds, such as keeping the robot's speed within a specified safe limit.

The performance of MICE was evaluated against several baseline CRL methods (though specific baseline names were not explicitly mentioned in the transcript, it is implied that MICE was compared to state-of-the-art or widely used algorithms that struggle with the identified bias). Key metrics used for evaluation included:

1. Estimation Bias: Quantified as the difference between the estimated cost value (from the cost critic) and the true cumulative cost observed in the environment. A bias approaching zero indicates accurate cost assessment.
2. Cumulative Reward: The standard RL metric measuring the agent's task performance and efficiency.
3. Constraint Values: The expected cumulative cost incurred by the policy, compared against the predefined safety threshold.

The experiments yielded compelling results:

• Mitigation of Underestimation Bias: MICE demonstrated remarkable success in mitigating cost underestimation. In several challenging tasks, the measured estimation bias for MICE's cost critic consistently approached zero. This indicates that MICE enables the agent to accurately assess the true risk associated with different states, a significant improvement over baseline methods where the bias was almost always negative.
• Significant Reduction in Constraint Violations: As a direct consequence of improved cost estimation, MICE policies exhibited a substantial reduction in constraint violations. Visualizations of constraint curves clearly showed MICE staying safely and consistently below the acceptable safety threshold throughout the training process. This is a critical outcome for deploying CRL agents in real-world safety-critical applications.
• Maintained or Improved Policy Performance: Crucially, MICE achieved these safety improvements without sacrificing task performance. The cumulative reward plots indicated that MICE was able to maintain, and in some cases even improve, the agent's policy performance (e.g., reaching the destination faster or controlling the robot more effectively) compared to baseline methods. This demonstrates that MICE is not simply being overly conservative but is enabling a more optimal balance between reward maximization and constraint satisfaction.

An important ablation study was also conducted to isolate the contribution of MICE's intrinsic cost mechanism. In this study, the intrinsic cost component of MICE was removed and replaced with a constant value. The results showed a significant drop in policy performance in this ablated version. This finding is critical because it refutes the notion that MICE's safety improvements are merely due to an arbitrary increase in conservatism. Instead, it strongly suggests that the memory-driven, dynamically adjusted intrinsic cost provides genuinely more accurate and nuanced value estimation, which is essential for both safety and strong overall policy performance.

While the talk did not explicitly detail the specific hardware used (e.g., GPU types, number of CPUs), the comprehensive nature of the experiments across different environments and the clear presentation of quantitative results underscore the robustness of MICE's performance. The consistent empirical evidence strongly supports the theoretical claims regarding MICE's ability to control underestimation bias and enhance safety in constrained reinforcement learning.

Practical Implications

▶ Watch: Ablation study: MICE's performance vs. simple conservatism (9:00)

The development of Memory-driven Intrinsic Cost Estimation (MICE) carries significant practical implications for the deployment and trustworthiness of AI/ML systems in safety-critical domains. For practitioners, infrastructure teams, model builders, and deployers, MICE offers a principled and empirically validated approach to address one of the most persistent challenges in Constrained Reinforcement Learning (CRL): ensuring safety guarantees during both training and deployment.

Firstly, MICE directly tackles the problem of underestimation bias, which has historically been a major impediment to the real-world adoption of CRL. By providing more accurate cost estimations, especially in high-risk regions, MICE significantly increases the confidence with which engineers can deploy agents in environments where failure can have severe consequences. This is particularly relevant for:

• Autonomous Driving: Reducing the likelihood of collisions or unsafe maneuvers.
• Robotics: Ensuring industrial robots operate safely near human workers or in delicate environments.
• Healthcare: Developing AI systems for drug dosage, treatment planning, or surgical assistance where patient safety is paramount.
• Industrial Control Systems: Optimizing processes while adhering to strict operational and safety limits.

The ability of MICE to reduce constraint violations while maintaining or improving policy performance is a crucial practical advantage. Often, enhancing safety in ML systems comes at the cost of performance or efficiency. MICE's approach, validated by its ablation study, demonstrates that its safety benefits stem from a more accurate understanding of risk, rather than simply an overly conservative policy. This means that practitioners do not have to make a stark trade-off between safety and utility, enabling the development of both highly safe and highly effective AI agents.

The memory-driven intrinsic cost mechanism, inspired by human learning, provides a human-interpretable intuition for how the agent learns from past mistakes. This can be valuable for debugging and understanding agent behavior, potentially aiding in regulatory compliance and certification processes for safety-critical AI systems. The dynamic adjustment of the intrinsic factor further ensures that the system is not rigidly conservative but adapts its caution based on observed performance, optimizing the balance between safety and exploration.

While the talk emphasizes the benefits, it's also important to consider potential practical tradeoffs and limitations. The introduction of a cost memory and the computation of similarity measures for intrinsic cost estimation could introduce additional computational overhead during training. For very large state spaces or extremely high-dimensional observations, maintaining and querying the cost memory efficiently might require careful engineering and optimization. However, given the criticality of safety, this overhead might be a justifiable cost. Another consideration is the generalizability of the "flashbulb memory" concept to all types of safety constraints or highly dynamic, non-stationary environments. While effective in the tested scenarios, future research might explore its robustness across an even wider spectrum of real-world complexities.

In summary, MICE represents a significant step forward in making Constrained Reinforcement Learning a more reliable and deployable technology. By addressing a fundamental bias with an elegant, theoretically sound, and empirically proven method, it empowers model builders and deployers to create AI systems that are not only intelligent but also demonstrably safe and trustworthy in critical applications.

Key Takeaways

• Underestimation Bias is a Pervasive Problem in CRL: Existing Constrained Reinforcement Learning methods frequently suffer from an underestimation bias in their cost value functions, particularly in high-cost regions, leading to unsafe exploration and constraint violations.
• Root Cause Identified: This bias stems from how policy optimization in high-cost regions distorts the noise distribution, breaking its crucial zero-mean property, thus systematically lowering estimated costs.
• MICE: A Novel Memory-Driven Solution: Inspired by human "flashbulb memory," the proposed Memory-driven Intrinsic Cost Estimation (MICE) method introduces a cost memory of past unsafe states and computes an intrinsic cost based on similarity, augmenting the environmental cost.
• Enhanced Safety and Performance: MICE effectively mitigates underestimation bias, leading to significantly fewer constraint violations during training while maintaining or even improving policy performance compared to baselines.
• Theoretical Guarantees for Robustness: MICE provides a tighter upper bound on constraint violations and guarantees value function convergence to the optimal value, offering strong theoretical assurances of its safety and stability.
• Practical and Adaptive: A dynamic adjustment mechanism for the intrinsic cost factor prevents MICE from being overly conservative, ensuring a practical balance between safety and efficient exploration, making it highly relevant for real-world safety-critical applications.

About the Speaker(s)

The primary author of this work is Shiqing Gao, who is affiliated with Shanghai Jiao Tong University. The presentation at ICML 2025 was delivered by a colleague on behalf of Shiqing Gao, who unfortunately could not attend in person due to visa issues. The research was conducted in collaboration with co-authors Jiaxin Ding, Luoyi Fu, and Xinbing Wang, who are also likely affiliated with Shanghai Jiao Tong University, a prominent institution known for its contributions to computer science and artificial intelligence research.