Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework

Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework

Speakers: Neel P. Bhatt, Yunhao Yang, Ufuk Topcu, Zhangyang Wang

Conference: MLSys 2025

YouTube: https://www.youtube.com/watch?v=None

Overview

Multimodal Foundation Models (MFMs) are rapidly becoming indispensable tools for developing advanced autonomous systems, particularly in robotics, where they offer a natural and intuitive interface for complex perception and planning tasks. However, a significant bottleneck limiting their widespread and reliable deployment is their inherent uncertainty. This talk, presented by Neel P. Bhatt and his co-authors at MLSys 2025, addresses this critical challenge head-on by proposing a novel and comprehensive framework for quantifying and mitigating uncertainty within these powerful models.

The core innovation lies in disentangling the aggregate uncertainty typically reported by MFMs into two distinct, actionable categories: perception uncertainty and decision uncertainty. By providing targeted quantification methods—leveraging conformal prediction for perception and formal methods for decision-making—the researchers enable a two-pronged improvement strategy comprising online active sensing and offline efficient fine-tuning. This approach not only enhances the reliability of MFM-generated plans but also significantly improves their compliance with predefined task specifications, marking a crucial step towards safer and more robust autonomous systems.

Background

▶ Watch: Introduction: The critical bottleneck of uncertainty in MMFMs (0:00)

The rise of Multimodal Foundation Models has revolutionized how autonomous systems, particularly robots, interact with and interpret their environments. These models excel at tasks requiring both visual understanding and complex reasoning, generating high-level plans from natural language prompts and visual inputs. Despite their impressive capabilities, the inherent uncertainty in their outputs remains a significant hurdle for real-world deployment. This uncertainty can manifest in various ways, stemming from both the perceptual understanding of the environment and the subsequent decision-making processes.

Perception uncertainty often arises from real-world environmental complexities such as artifacts in images, variable lighting conditions, or the occlusion of critical objects. For instance, a robot's camera might fail to clearly identify a pedestrian partially hidden behind a lamppost, leading to an ambiguous perceptual state. On the other hand, decision uncertainty emerges when the model struggles to formulate a plan that is consistent with the provided prompt and visual context, or when its generated plan fails to meet specific, critical requirements. This could be due to prompt-image inconsistencies, a failure to capture nuanced information from the visual input, or simply generating a logically flawed sequence of actions.

Existing approaches to addressing MFM uncertainty typically provide an aggregate, "black-box" estimate. While useful for a general sense of confidence, these methods fall short in pinpointing where the uncertainty truly lies. This lack of diagnostic granularity severely hinders targeted improvements. For example, if a robot's plan is uncertain, an aggregate score doesn't tell whether the issue is with seeing the stop sign or with correctly interpreting the rule "stop at a stop sign then proceed if clear." Consequently, improving such systems often necessitates extensive human labeling and iterative trial-and-error, a process that is both costly and unscalable. The problem, therefore, is not merely to detect uncertainty, but to precisely localize its source to enable efficient and effective intervention. This work directly tackles this challenge by proposing a framework that disentangles and quantifies these distinct types of uncertainty, paving the way for targeted and automated remediation.

Key Findings

▶ Watch: Framework Overview: Online active sensing and offline fine-tuning modes (2:00)

The central contribution of this research is the introduction of a formal framework that fundamentally redefines how uncertainty in Multimodal Foundation Models is understood, quantified, and addressed. The key findings and contributions are:

1. Disentanglement of Uncertainty: The framework successfully disentangles the monolithic concept of MFM uncertainty into two distinct, mechanistically different types: perception uncertainty and decision uncertainty. This crucial separation allows for highly targeted interventions, moving beyond opaque aggregate scores.

1. Novel Quantification Methods:

• For perception uncertainty, the work introduces a quantification method based on conformal prediction. This technique provides theoretical guarantees on the probability of correctly identifying objects, offering a calibrated and reliable lower bound on perceptual accuracy.
• For decision uncertainty, the framework leverages formal methods driven prediction. By converting natural language plans into symbolic representations (specifically, finite state machines or automata) and applying formal verification techniques, it quantifies the theoretical lower bound on the probability that a generated plan satisfies a given set of specifications, often expressed in Linear Temporal Logic (LTL).

1. Two-Part Improvement Strategy: Based on the disentangled uncertainty, the researchers propose a dual intervention strategy:

• Online Active Sensing: An online mechanism that dynamically assesses perception uncertainty and, when high, triggers sensory adjustments (e.g., switching cameras) to reduce it before proceeding with decision-making. This reduces the number of expensive queries to the text generator.
• Offline Efficient Fine-tuning: An offline process that uses the quantified uncertainties to curate a high-quality dataset. This dataset, comprising validated and specification-compliant plans, is then used to fine-tune the MFM, thereby systematically improving its performance.

1. Significant Performance Enhancements: Through empirical evaluation, the proposed framework demonstrates substantial improvements in MFM performance:

• It reduces the variability of the foundation model's outputs by approximately 40%. This indicates a more consistent and predictable behavior from the MFM.
• It significantly increases task specification compliance, meaning the generated plans are more likely to meet the desired requirements and constraints.

In essence, the research provides a robust, automated, and reliable framework that not only identifies where uncertainty lies but also offers concrete, data-driven strategies to mitigate it, leading to more trustworthy and effective autonomous systems.

Technical Deep Dive

▶ Watch: Quantifying Perception Uncertainty using Conformal Prediction (3:00)

The proposed framework for addressing uncertainty in Multimodal Foundation Models operates through two interdependent modes: an online mode for active sensing during real-time inference and an offline mode for efficient model fine-tuning. This dual approach is underpinned by sophisticated quantification methods for both perception and decision uncertainty.

Framework Overview

The overarching framework aims to enhance the reliability and specification compliance of MFM-generated plans. In the online mode, the system dynamically assesses perception uncertainty and performs active sensing to gather more reliable visual data. Once perception is robust, it sparsely queries the MFM's text generator and verifies the resulting plan's decision uncertainty before execution. In the offline mode, the framework systematically curates a high-quality dataset by filtering plans based on their quantified uncertainties, which is then used to fine-tune the MFM.

Perception Uncertainty Quantification

Perception uncertainty is defined as a theoretical lower bound on the probability of correctly identifying objects in the scene. To quantify this, the framework employs conformal prediction, a method known for providing theoretical guarantees on prediction validity.

1. Input Processing: An image is fed into the VLM's vision encoder, producing embeddings.
2. Softmax Output: The model typically outputs softmax probabilities across different object classes.
3. Non-Conformity Score: From these softmax outputs, a histogram is generated. A quantile corresponding to a non-conformity score is then derived. This score is a calibrated measure reflecting how "unusual" a new observation is compared to previously seen data, indicating the level of confidence.
4. Confidence/Probability Derivation: Conversely, this calibrated non-conformity score can be used to retrieve a direct probability score, which serves as the perception uncertainty score. For instance, if a user specifies a requirement like "at least 80% probability of correctly identifying objects," conformal prediction provides a mechanism to verify if this threshold is met.

Decision Uncertainty Quantification

Decision uncertainty is defined as a theoretical lower bound on the probability that a plan generated by the VLM satisfies a given set of specifications. This is achieved using formal methods driven prediction.

1. Specification Language: Task specifications are often expressed in formal languages like Linear Temporal Logic (LTL), which allows for precise definition of desired temporal properties (e.g., "always eventually reach the goal," "never enter a restricted area").
2. Plan Conversion to Finite State Machine (FSM): The natural language high-level plan generated by the MFM's text generator is algorithmically converted into a symbolic representation—a finite state machine (automaton). Each step in the natural language plan corresponds to a state in the FSM, and the transitions between states represent conditional actions or outcomes.
3. Formal Verification: Techniques from formal methods are then applied to verify whether this FSM (representing the plan) meets the specified LTL properties. This verification process yields a quantitative measure of compliance.
4. Decision Uncertainty Score: The output of this verification is a score (e.g., 0.7), indicating the probability or confidence that the plan satisfies the specifications. For example, a score of 0.7 means there's at least a 70% probability that the plan will satisfy the defined specifications.

Offline Efficient Fine-tuning

The offline mode is designed to systematically improve the MFM's planning capabilities by curating a high-quality dataset for fine-tuning.

1. Sample Query: An image and a task prompt are queried from an existing dataset.
2. Perception Uncertainty Check: The image is processed by the VLM's vision encoder. Conformal prediction quantifies its perception uncertainty. Only samples with perception uncertainty below a predefined threshold are allowed to proceed, ensuring that the visual input is reliably understood.
3. Plan Generation: For accepted samples, the VLM's text generator produces a high-level plan.
4. Decision Uncertainty Verification: This natural language plan is converted into a finite state machine and subjected to formal verification against task specifications.
5. Dataset Curation: Only plans that meet the desired specifications (i.e., have a decision uncertainty score below a certain threshold) are included in the curated, high-quality dataset. This dataset is free from perceptually ambiguous scenarios and logically flawed plans.
6. Model Fine-tuning and Recalibration: The MFM is then fine-tuned on this high-quality dataset. After fine-tuning, the model is recalibrated to ensure its uncertainty estimates remain accurate. An example given is an autonomous vehicle scenario: an initial VLM might suggest "stop at the intersection, wait for pedestrian, then move forward" even if the light is green and no pedestrians are present (violating a "don't stop at green light without pedestrians" specification). Fine-tuning with the curated data corrects such behaviors, leading to a compliant plan.

Online Active Sensing

The online mode focuses on real-time inference, particularly for deployment in real-world autonomous systems where the text generator might be computationally expensive to query frequently.

1. Separate Loops: The perception loop is separated from the decision-making loop to optimize real-time performance.
2. Perception Uncertainty Monitoring: The system continuously monitors perception uncertainty using conformal prediction.
3. Active Sensing Trigger: If perception uncertainty is detected to be above a certain threshold (indicating, for example, occlusion or poor lighting), the system initiates active sensing. This involves dynamic adjustments to the sensory input, such as switching between multiple available cameras (e.g., from a front camera to a side camera, or re-observing the scene by staying still).
4. Uncertainty Reduction Loop: Active sensing continues until the perception uncertainty falls below the threshold, ensuring a reliable visual understanding of the environment.
5. Sparse Text Generator Query: Only once perception uncertainty is sufficiently reduced, the system sparsely queries the MFM's text generator to produce a plan. This reduces the number of expensive calls to the generator.
6. Decision Uncertainty Check and Execution: The generated plan's decision uncertainty is then assessed using formal verification. If this uncertainty is also below a predefined threshold, the plan is deemed reliable and executed.

The mechanism for switching cameras in active sensing is automated: based on the bounding box of the uncertain object and the configuration of available sensors, the system intelligently selects the most appropriate camera view (e.g., if occlusion is on the right side of the front camera's view, it switches to the right camera). This automated process ensures that the system proactively seeks clearer perceptual information without human intervention.

Experimental Setup & Results

▶ Watch: Example: Converting natural language plans to finite state machines (5:30)

The research presented several qualitative demonstrations and quantitative results to validate the effectiveness of their uncertainty quantification and intervention framework. The applicability of the approach extends beyond autonomous driving to various autonomous systems, including tabletop manipulation.

Qualitative Demonstrations

Three distinct scenarios were used to illustrate the framework's capabilities:

1. Autonomous Driving (Stop Sign Scenario):

• Setup: A mobile robot equipped with a camera and other sensors is tasked with turning right at a stop sign.
• Challenge: Multiple pedestrians on the left side of the image cause high perception uncertainty due to occlusion.
• Framework's Response: The model detects high uncertainty, prompting the robot to stay still and re-observe the scene. This active sensing continues until the uncertainty is reduced. Only then is the text generator queried for a plan.
• Outcome: This intelligent re-observation significantly reduces the number of calls to the computationally expensive text generator while ensuring a safer and more reliable decision.

1. Tabletop Manipulation (Occluded End-Effector):

• Setup: A manipulator with an end-effector is equipped with a camera, tasked with picking and placing a block.
• Challenge: The block being picked by the manipulator itself clogs or occludes the camera's view, leading to high perception uncertainty.
• Framework's Response: Having access to multiple cameras, the system performs active sensing by switching to a different camera view (e.g., from a top-down view to a side view) to gain a clearer perspective of the block.
• Outcome: The ability to dynamically switch views resolves the occlusion, allowing for precise manipulation.

1. Simulation Scenario (Occluded Pedestrian):

• Setup: A simulated environment where a pedestrian is occluded by a light post on the bottom right of the front camera's view.
• Challenge: High perception uncertainty due to the occlusion.
• Framework's Response: The system detects the high perception uncertainty and automatically switches from the front camera to a side camera. Once the pedestrian is no longer occluded and perception uncertainty reduces, it switches back to the front camera.
• Outcome: This demonstrates the automated and dynamic camera switching capability to resolve localized perception ambiguities.

Quantitative Results

The quantitative evaluation focused on the impact of active sensing and fine-tuning on model performance, particularly concerning output variability and specification compliance.

• Models Under Test:
• Raw Model: The base MFM without any of the proposed interventions.
• Raw Model + Active Sensing: The base MFM integrated with the online active sensing component but without offline fine-tuning.
• Benchmark Model: An unspecified existing state-of-the-art model for comparison.
• Fine-tuned Model (Their Approach): The MFM fine-tuned using the proposed offline efficient fine-tuning strategy, incorporating both perception and decision uncertainty quantification.

• Key Findings:
• Impact of Active Sensing: Even without the benefits of offline fine-tuning, simply incorporating active sensing drastically improved the model's performance. This highlights the immediate benefits of dynamically seeking clearer perceptual information.
• Impact of Fine-tuning: The most significant improvements were observed with their fine-tuned model.
• Reduced Output Variability: The framework successfully reduced the variability in the outputs of the MFM. The speaker explicitly stated a reduction of approximately 40%. This implies that the model's responses become much more consistent and predictable, a crucial factor for reliable autonomous operation.
• Increased Specification Compliance: The fine-tuned model demonstrated a substantial increase in the number of specifications that were satisfied. This directly translates to more reliable and safer plans, as they are formally verified to adhere to predefined rules and constraints.
• Reduced Queries: A significant practical benefit was the reduction in the number of queries made to the text generator. By resolving perception uncertainty through active sensing and curating high-quality data offline, the model needs to query the expensive text generation component less frequently, especially in real-time scenarios.

In summary, the experimental results unequivocally demonstrate that disentangling uncertainty and employing targeted interventions—active sensing for online reliability and efficient fine-tuning for offline model improvement—leads to a more robust, reliable, and compliant Multimodal Foundation Model for robotic perception and planning tasks.

Practical Implications

▶ Watch: Online Strategy: Active sensing for real-time uncertainty reduction (6:15)

The framework presented by Bhatt et al. has profound practical implications for anyone involved in developing, deploying, or managing AI/ML-driven autonomous systems, particularly those leveraging Multimodal Foundation Models.

For Practitioners and Infrastructure Teams:

• Enhanced Reliability and Trustworthiness: By providing a formal and quantifiable measure of uncertainty, the framework enables the construction of more reliable and trustworthy autonomous systems. Plans are only executed when both perceptual understanding and decision-making logic are confidently aligned with specifications, reducing the risk of unexpected or unsafe behaviors.
• Auditable and Explainable AI: The use of formal methods to verify decision uncertainty offers a degree of auditability and explainability often lacking in black-box MFM outputs. Developers can trace why a plan was deemed compliant or non-compliant, aiding in debugging and regulatory compliance.
• Resource Optimization: The online active sensing strategy significantly reduces the number of expensive queries to the text generator component of MFMs. This is crucial for real-time deployments on resource-constrained platforms, where frequent queries can lead to high latency and computational costs. Infrastructure teams can design more efficient deployment pipelines knowing when and how often to invoke the most resource-intensive parts of the MFM.

For Model Builders and Deployers:

• Targeted Model Improvements: The disentanglement of uncertainty into perception and decision components is a game-changer. Instead of diffuse efforts to improve an "uncertain" model, developers now know precisely whether to focus on enhancing the vision encoder (for perception issues) or refining the text generator/prompt engineering (for decision issues). This allows for highly efficient and targeted model development cycles.
• Efficient Data Curation: The offline fine-tuning mechanism provides an automated pipeline for creating high-quality training datasets. By programmatically filtering out samples with high perception uncertainty or plans that fail formal verification, model builders can generate cleaner, more effective datasets, dramatically reducing the manual effort and cost traditionally associated with data labeling and curation for complex robotic tasks.
• Proactive Risk Mitigation: The active sensing component allows autonomous systems to proactively mitigate risks in uncertain environments. Instead of making a potentially flawed decision based on ambiguous sensory input, the system can actively seek more information (e.g., by changing camera views or waiting for occlusion to clear), thereby preventing errors before they occur.
• Broader Applicability: While demonstrated in autonomous driving and tabletop manipulation, the principles of disentangled uncertainty and targeted intervention are broadly applicable to any domain leveraging MFMs for perception and planning, from logistics robots to medical imaging interpretation.

Tradeoffs and Limitations:

• Specification Definition Complexity: While powerful, defining comprehensive and unambiguous specifications in formal languages like LTL can be challenging and time-consuming, requiring expertise in formal methods. For highly complex or open-ended tasks, this might become a bottleneck.
• Sensor Requirements for Active Sensing: The effectiveness of active sensing relies on the availability of multiple, diverse sensors or the ability to dynamically adjust sensor viewpoints. Systems with limited sensory capabilities might not fully leverage this aspect of the framework.
• Computational Cost of Formal Verification: For extremely long or complex plans, the formal verification step, even if performed offline, could incur significant computational overhead. Optimizing the efficiency of this step for large-scale applications is an ongoing challenge in formal methods.
• Calibration of Uncertainty Thresholds: The performance of the system depends on appropriately setting thresholds for perception and decision uncertainty. Miscalibrated thresholds could lead to overly cautious (slow) or overly aggressive (risky) behaviors.

Despite these considerations, the framework represents a significant leap forward in building robust and reliable AI systems, offering concrete tools and methodologies to navigate the inherent uncertainties of multimodal foundation models.

Key Takeaways

• Uncertainty in Multimodal Foundation Models (MFMs) is a critical bottleneck for reliable autonomous systems, stemming from both perception and decision-making.
• The presented framework effectively disentangles uncertainty into two distinct types: perception uncertainty (quantified via conformal prediction) and decision uncertainty (quantified via formal methods and finite state machines).
• Online active sensing allows autonomous systems to dynamically reduce perception uncertainty by re-observing or switching sensors (e.g., cameras) before querying the MFM's text generator, thereby reducing expensive inference calls.
• Offline efficient fine-tuning leverages the quantified uncertainties to curate high-quality datasets of specification-compliant plans, systematically improving the MFM's reliability and performance.
• The combined approach significantly reduces the variability of MFM outputs by approximately 40% and substantially increases task specification compliance.
• This framework provides a practical, automated, and auditable means to build more trustworthy and robust MFM-powered autonomous systems for various applications.

About the Speaker(s)

The talk was presented by Neel P. Bhatt, with contributions from his co-authors Yunhao Yang, Ufuk Topcu, and Zhangyang Wang. The presentation highlighted their collaborative research on quantifying and addressing uncertainty in multimodal foundation models for planning and perception in robotic tasks. Specific titles or affiliations for the speakers were not detailed in the provided transcript.