EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Speakers: Rui Yang, Hanyang(Jeremy) Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, Tong Zhang

Conference: ICML 2025

YouTube: https://slideslive.com/39044112

Overview

The talk introduces EmbodiedBench, a novel and comprehensive benchmarking suite designed to evaluate multi-modal large language models (VLMs) for their capabilities in vision-driven embodied agents. Presented by Rui Yang and a collaborative team from institutions including the University of Illinois, UW, U of Toronto, and TTIC, this work addresses critical limitations in existing embodied AI evaluation methodologies. Specifically, EmbodiedBench tackles the scarcity of diverse tasks and scenarios, as well as the lack of fine-grained diagnostic insights beyond simple success rates.

The core motivation behind EmbodiedBench stems from the challenge of scaling data for embodied AI, where expert demonstrations are costly and limited compared to data volumes in other AI domains like autonomous driving. The researchers investigate the extent to which general-purpose VLMs, typically pre-trained on vast internet-scale datasets, can generalize to the nuanced, low-level execution tasks required by embodied agents, such as navigation and manipulation. Furthermore, the benchmark seeks to precisely delineate the role of vision in different types of embodied tasks and provide granular analysis of diverse agent capabilities.

EmbodiedBench is a significant contribution to the field, offering a structured framework for understanding the strengths and weaknesses of current VLMs in embodied contexts. By introducing two orthogonal evaluation axes—task level (high-level planning vs. low-level execution) and scenario diversity (e.g., common-sense reasoning, spatial perception)—it enables a deeper, more diagnostic assessment. The findings illuminate critical areas for improvement in multi-modal models, particularly concerning their ability to process temporal visual information and perform precise physical control, thereby paving the way for more capable and robust embodied AI systems.

Background

▶ Watch: Introduction to Embodied Agents and data scalability (0:00)

Embodied agents represent a broad sub-concept within AI, encompassing any task whose successful completion necessitates interaction with a physical body, whether real or simulated. Historically, the development of foundation models for embodied agents has relied heavily on task-specific data, often obtained through expert demonstrations where human operators remotely control robots to perform desired tasks. While effective for specific domains, this approach faces a severe challenge in data scalability. The speaker highlights a stark contrast: a single autonomous driving company can collect hundreds of millions of data clips daily, whereas the largest datasets in embodied AI currently reach only about one million episodes. This immense gap underscores the difficulty and cost associated with acquiring high-quality, real-world embodied interaction data.

This data scarcity naturally leads to a crucial question: can we leverage the rapidly advancing off-the-shelf Vision-Language Models (VLMs), which are trained on massive, general-purpose multi-modal datasets, to generalize effectively to embodied tasks? The central hypothesis explored by this research is whether these general VLMs can bridge the gap to low-level execution tasks like navigation and precise manipulation, which fundamentally differ from the high-level reasoning typically seen in their pre-training data. Understanding the specific role of vision—when it is critical, when it is redundant, and how models process it—is also a key area of investigation.

Prior to EmbodiedBench, existing benchmarks for embodied AI suffered from a lack of diversity, primarily in two aspects:

1. Task Diversity: Many benchmarks focused either on high-level cognitive planning (e.g., abstract task decomposition) or on low-level robotic execution (e.g., precise motor control), rarely encompassing both within a unified framework. This fragmented evaluation hindered a holistic understanding of an agent's capabilities from perception to action.
2. Scenario Diversity: Benchmarks often failed to account for the varied contexts and complexities of real-world interactions, such as implicit instructions, novel object configurations, or common-sense reasoning challenges. This limited the diagnostic power of evaluations, making it difficult to pinpoint specific weaknesses of an agent beyond a simple pass/fail metric.

The absence of a comprehensive, fine-grained evaluation system meant that researchers lacked the tools to conduct deep diagnostic analyses, moving beyond mere success rates to understand why an agent succeeded or failed, and what specific capabilities were being tested. EmbodiedBench was conceived to address these fundamental problems, providing a robust framework for evaluating and advancing multi-modal LLMs in the complex domain of embodied AI.

Key Findings

▶ Watch: EmbodiedBench: Two orthogonal evaluation axes (2:15)

EmbodiedBench provides a wealth of insights into the capabilities and limitations of current multi-modal large language models when applied to embodied tasks. The key findings reveal nuanced aspects of VLM generalization, the critical role of vision, and specific areas where models struggle:

1. Differential Generalization Across Task Levels: General-purpose VLMs exhibit significantly better generalization performance on high-level planning tasks compared to low-level execution tasks. High-level planning, which involves decomposing complex goals and logical reasoning, resembles the general reasoning data encountered during VLM pre-training. In contrast, low-level tasks, particularly fine-grained manipulation, show the weakest performance. The researchers hypothesize that this discrepancy is due to the relative scarcity of atomic manipulation primitive data within the pre-training datasets of these general VLMs.

1. Conditional Importance of Vision: The role of visual observations varies dramatically depending on the task level:

• For high-level planning tasks, removing all visual input (relying solely on text) often results in no degradation in performance, and in some cases, even leads to a slight performance boost. This suggests that in current foundation models, redundant or irrelevant visual information might occasionally mislead the agent during abstract planning and decomposition, which do not inherently necessitate real-world visual data (e.g., knowing to find a book doesn't require seeing the book).
• Conversely, for low-level execution tasks (like navigation and manipulation), removing visual observations leads to a significant performance drop. This validates the intuitive understanding that precise control and spatial reasoning in a physical environment are absolutely dependent on real-world spatial information provided by vision.

1. Inconsistent "Strongest" Model Performance: Even leading models like GPT-4o do not consistently outperform others across all scenario categories. While GPT-4o demonstrates superior ability in comprehending implicit instructions and performing common-sense reasoning, it suffers a substantial performance drop (e.g., 43% lower) compared to models like Gemini when faced with tasks requiring spatial perception involving relative position descriptions (e.g., "the object to the left of the vase"). This highlights that different models excel in different facets of embodied intelligence, and a single "best" model is task- and scenario-dependent.

1. Challenges in Multi-step Visual Information Processing: The ablations around vision revealed critical insights into how VLMs handle multiple images:

• In-context learning with images: Providing corresponding in-context images alongside examples yields a large performance boost of around 40%. This indicates that current VLMs can effectively leverage richer visual context when provided with static, relevant imagery.
• Multi-step visual history: When more than one image is provided as part of the interaction history (representing sequential past visual observations), performance consistently drops across tasks. Analysis of the reasoning output shows that models become confused about the current state, even when explicit time indices are provided. This suggests a fundamental struggle with organizing visual information across the time dimension and maintaining a coherent understanding of dynamic environments.

1. Diagnostic Error Analysis: EmbodiedBench's fine-grained evaluation allows for a clear categorization of error types:

• In high-level tasks, most errors stem from missing steps in the plan or invalid actions, indicating a challenge in translating a good understanding of the current state into a well-structured, executable plan.
• In low-level tasks, there are typically no missing steps; instead, nearly all errors are attributed to perception errors (incorrectly locating the target object) and control errors (issuing actions with imprecise parameters). This points to limitations in accurate object detection and precise motor command generation based on visual input.

1. Potential for Post-training Specialization: Initial work on post-training a 3-billion parameter multi-modal embodied agent demonstrates significant promise. By clearly differentiating between high-level and low-level task objectives during this specialized training, the researchers were able to achieve performance equal to or even above GPT-4o, without requiring the in-context learning examples that GPT-4o often benefits from. This suggests that targeted post-training can effectively mitigate some of the generalization issues observed in general VLMs.

Technical Deep Dive

▶ Watch: Agentic framework for powerful foundation models (4:00)

EmbodiedBench is meticulously designed around two orthogonal evaluation axes, providing a fine-grained and comprehensive assessment of multi-modal large language models for embodied agents. This design allows for a systematic breakdown of capabilities required for complex real-world interaction.

Evaluation Axes

1. High-level and Low-level Task Dimension:

• High-level tasks focus on logical planning and task decomposition. In the representative domain of household assistants, this involves taking a complex, abstract goal (e.g., "prepare a peach") and breaking it down into a coherent sequence of sub-goals. For instance, "find the peach," "pick it up," "go to the sink," "wash the peach," and so on. These tasks primarily test an agent's ability to reason about goals, preconditions, and effects, generating a strategic plan.
• Low-level tasks evaluate the agent's ability to map these sub-goals into concrete primitives for navigation and manipulation. This requires precise control based on spatial information. For example, "picking up the peach" might translate into executing a sequence of 7-degrees-of-freedom (7-DOF) vectors to control a robotic arm. These tasks test fine motor control, precise spatial understanding, and reactive execution in a dynamic environment. The skills required for these two levels are fundamentally different, yet together they form a complete embodied agent decision pipeline.

1. Scenario Dimension:

This axis introduces diversity in the context and interpretation required for tasks, moving beyond literal instructions. The benchmark defines six relatively independent scenario categories, illustrated by an example of common-sense reasoning: instead of a direct instruction like "wash the lettuce and put it in the refrigerator," a more casual or vague instruction might be "wash the lettuce and store it somewhere so it stays fresh for a few days." In this case, the agent must reason beyond the literal words to infer the correct action (placing it in the refrigerator), demonstrating common-sense reasoning. Other scenario categories could test aspects like understanding relative spatial descriptions, handling ambiguous references, or adapting to novel object states. This dimension is crucial for evaluating an agent's robustness and human-like understanding in varied real-world interactions.

Environment and Task Design

EmbodiedBench encompasses a total of 1128 tasks, designed to span the breadth of these two evaluation axes. To provide a rich and diverse set of environments, the benchmark leverages three high-fidelity simulators:

• AI2-THOR: A virtual environment for interactive agents, often used for household tasks and object manipulation.
• Habitat: A high-performance 3D simulator for embodied AI, known for its realistic rendering and physics, particularly for navigation.
• Co-Pilot Sim: Another simulator contributing to the diverse range of scenarios and challenges.

These environments allow for controlled experimentation and the generation of diverse visual and physical interaction data.

Agentic Framework

To ensure a fair and performant evaluation across diverse tasks, the researchers designed an agentic framework that wraps around the foundation models. While not detailed extensively in the talk due to time constraints, the speaker highlighted several key components that contributed significantly to the final performance:

• Extensive Multi-modal Input: The framework is designed to handle rich and varied multi-modal inputs, including visual observations (images) and textual instructions, along with potentially other modalities.
• Constrained Decoding: To enforce a predefined Chain-of-Thought (CoT) structure, the framework utilizes constrained decoding. This guides the model to produce reasoning steps in a specific format, which is crucial for both interpretability and ensuring the model follows a logical progression for complex tasks.
• Clean and Comprehensive Interaction History: Maintaining a well-structured and complete history of interactions (past observations, actions, and internal thoughts) is vital for embodied agents to understand the current state and plan future actions effectively. This component ensures that the model has access to all necessary context.

Models Evaluated

EmbodiedBench was used to test a substantial number of models, reflecting the current landscape of multi-modal AI:

• 8 proprietary models: These include state-of-the-art closed-source models such as GPT-4o and Gemini, representing the cutting edge of VLM capabilities.
• 16 open-source models: A wide range of publicly available models were also evaluated, allowing for broader comparison and accessibility for the research community.

Vision Processing Mechanism (from Q&A)

During the Q&A, the speaker elaborated on how VLMs process multiple images. The general mechanism involves:

1. Vision Encoder: Each input image first passes through a vision encoder, commonly a Vision Transformer (ViT). The ViT processes the image and extracts high-level features.
2. Token Projection: These visual features are then projected into a sequence of tokens, similar to how text is tokenized.
3. Alignment and Positional Embeddings: These visual tokens are subsequently aligned with the text dimensions. Crucially, positional embeddings are added to these visual tokens. For multi-step sequences, these positional embeddings provide the model with a sense of the time index of each specific image within the sequence, allowing it to understand the temporal order of observations. The challenge, as highlighted by the results, lies not in the basic encoding and temporal indexing, but in the model's ability to organize and reason with this temporally indexed visual information effectively across multiple steps.

Experimental Setup & Results

▶ Watch: Results: VLM generalization in high vs low-level tasks (4:40)

The experimental setup for EmbodiedBench focused on systematically evaluating the performance of various multi-modal LLMs across the two primary evaluation axes: high-level vs. low-level tasks, and diverse scenario dimensions. The results provide critical insights into the current state of embodied AI and the specific limitations of existing VLMs.

Generalization Performance Across Task Levels

• High-level Planning Tasks: The results consistently showed that models performed significantly better on high-level planning tasks. The average success rate for these tasks was notably higher compared to low-level execution. This is attributed to the fact that high-level planning, involving logical reasoning and task decomposition, aligns more closely with the general reasoning patterns observed in the pre-training data of VLMs.
• Low-level Execution Tasks: Performance on low-level tasks, particularly manipulation tasks, was substantially weaker. Even with a discretized action space designed to minimize embodiment-specific configuration differences, models struggled. The researchers hypothesize this is due to the limited amount of atomic manipulation primitive data available during the pre-training phase of general VLMs, which are not typically optimized for fine-grained motor control or precise physical interactions.

Role of Vision: Ablation Studies

A crucial set of experiments involved ablating visual observations to understand their impact:

• Removing Visual Observations (Text-Only Input):
• High-level Tasks: When visual observations were entirely removed, and only text input was provided, there was no degradation in performance for high-level tasks. Interestingly, for three out of four experiments in this category, removing vision even led to a slight performance boost. This suggests that for abstract planning and decomposition, current foundation models might sometimes be "misled" by redundant or irrelevant information present in the visual modalities, performing better with a cleaner, text-only input for these tasks.
• Low-level Tasks: In stark contrast, removing visual observations resulted in a significant performance drop for low-level tasks. This strongly validates the necessity of real-world spatial information for navigation and precise manipulation, where visual input is absolutely critical for understanding the environment and executing actions.

• Multi-step Image Understanding Ablations: Four ablations were conducted to probe how models handle visual context and history. The two most impactful were:
• Adding In-Context Images: When relevant visual examples (images) were added to the in-context learning examples, a large performance boost of approximately 40% was observed. This indicates that VLMs are capable of effectively leveraging static, rich visual context when provided as part of the prompt.
• Providing Multi-step Visual History: Conversely, when more than one image was provided as part of the interaction history (representing sequential past visual observations over multiple steps), performance consistently dropped across all tasks. Analysis revealed that the models became confused about the "current state," struggling to organize visual information across the time dimension, even when explicit time indices were provided. This highlights a significant limitation in current VLMs' ability to process and reason with dynamic, sequential visual inputs.

Scenario-Specific Model Performance

The evaluation on scenario diversity revealed that the "strongest" model is not universally superior:

• GPT-4o vs. Gemini: While GPT-4o excelled in comprehending implicit instructions and performing common-sense reasoning, it demonstrated a substantial 43% performance drop compared to Gemini in tasks requiring spatial perception, specifically identifying target objects based on relative position descriptions (e.g., "the book to the left of the lamp"). This underscores that different models have varying strengths across different cognitive and perceptual capabilities.

Error Analysis Statistics

EmbodiedBench's fine-grained approach allowed for a diagnostic breakdown of error types:

• High-level Task Errors: The majority of errors in high-level tasks stemmed from missing steps in the generated plan or the generation of invalid actions. This suggests that models struggle with the translation from a conceptual understanding of the current state to a coherent, executable sequence of operations.
• Low-level Task Errors: For low-level tasks, errors were almost exclusively attributed to perception errors (e.g., failing to correctly locate the target object) and control errors (e.g., issuing actions with imprecise parameters). This highlights limitations in accurate visual perception for object localization and the generation of precise motor commands required for fine manipulation.

Post-training for Embodied Agents

The presentation briefly touched upon promising ongoing work:

• By post-training a 3-billion parameter multi-modal embodied agent and explicitly differentiating between high-level and low-level tasks during this process, researchers achieved performance equal to or exceeding GPT-4o. Critically, this was accomplished without any in-context learning examples for the specialized agent, suggesting that targeted training can yield highly performant, specialized embodied agents.

Overall, the experimental results from EmbodiedBench paint a detailed picture of the current state of VLMs in embodied AI, pinpointing their strengths in high-level reasoning and weaknesses in low-level control and temporal visual understanding.

Practical Implications

▶ Watch: Model performance varies across diverse scenarios (6:20)

EmbodiedBench provides critical insights that have significant practical implications for practitioners, infrastructure teams, model builders, and deployers working on embodied AI.

For Model Builders and Researchers:

1. Targeted Pre-training Data: The struggle of general VLMs with low-level manipulation tasks underscores the need for more diverse and substantial atomic manipulation primitive data in pre-training datasets. Current general datasets might be rich in high-level visual concepts and text, but lack the granular, sequential data required for precise physical interaction. Future pre-training efforts for embodied agents should explicitly include such data.
2. Specialized Architectures for Temporal Vision: The consistent performance drop when models are given multi-step visual history highlights a fundamental architectural and algorithmic limitation. Model builders need to develop better mechanisms for organizing, reasoning with, and maintaining a coherent representation of visual information across the time dimension. This could involve novel multi-modal positional encodings or recurrent attention mechanisms specifically designed for sequential visual inputs.
3. Conditional Use of Visual Modality: The finding that vision can be redundant or even misleading for high-level planning suggests a need for adaptive or selective use of visual input. Models could be designed to dynamically determine the relevance of visual information for a given sub-task, potentially simplifying inputs for planning stages to avoid distraction and improve efficiency.
4. Diagnostic Evaluation is Key: Moving beyond aggregate success rates, EmbodiedBench's fine-grained error analysis (missing steps vs. perception errors) is invaluable. Model builders can use such benchmarks to diagnose specific weaknesses in their models, allowing for targeted improvements in either the planning module or the perception/control module.

For Infrastructure Teams and Deployers:

1. Compute Requirements for Multi-modal History: If future models overcome the temporal vision challenge, processing and storing extensive multi-modal interaction histories (multiple images over time) will significantly increase computational and memory requirements. Infrastructure teams should anticipate this need for more powerful GPU/TPU resources and efficient data pipelines.
2. Trade-offs in Model Deployment: The observation that even leading models have varying strengths across different scenarios implies that a single "best" model might not exist for all embodied tasks. Deployers might need to consider ensemble approaches or specialized models for different sub-tasks (e.g., one model for common-sense reasoning, another for precise spatial perception) to achieve optimal overall performance. This adds complexity to deployment and maintenance.
3. Robustness to Ambiguity: The scenario dimension of EmbodiedBench highlights the need for agents to handle implicit instructions and common-sense reasoning. Deploying agents in real-world settings will require them to interpret human commands that are often vague or casual, necessitating models that can reason beyond literal interpretations.

Tradeoffs and Limitations:

• Simulation vs. Reality: While EmbodiedBench uses advanced simulators (AI2-THOR, Habitat, Co-Pilot Sim), there is always a sim-to-real gap. Performance in simulation does not perfectly translate to the complexities, noise, and unpredictability of the physical world.
• Computational Cost of Benchmarking: Running extensive evaluations across 1128 tasks, with multiple models and ablations, is computationally intensive. This underscores the need for efficient evaluation frameworks and potentially distributed computing resources for researchers.
• Interpretability of Failures: While error categories are provided, the underlying cognitive reasons for certain failures (e.g., why a model gets "confused" by temporal visual information) still require deeper research and more interpretable model architectures.
• Current Models are Not Embodied-Native: A major limitation is that the evaluated VLMs were not purpose-built for embodied tasks. Their generalization struggles highlight that simply applying general-purpose models to embodied problems is insufficient; specialized training and architectural modifications are crucial.

In summary, EmbodiedBench serves as a crucial diagnostic tool, revealing that while general VLMs offer a strong foundation for high-level reasoning, significant work remains in developing models truly capable of robust, precise, and temporally aware low-level embodied interaction. The findings guide future research towards more specialized data, architectures, and training methodologies for embodied AI.

Key Takeaways

• EmbodiedBench is a comprehensive, fine-grained benchmark for evaluating multi-modal LLMs in vision-driven embodied agents, addressing critical gaps in task and scenario diversity in existing benchmarks.
• General VLMs generalize well to high-level planning but struggle with low-level execution, especially manipulation tasks, likely due to a lack of relevant atomic manipulation data in their pre-training.
• The role of vision is task-dependent: It is crucial for low-level spatial reasoning and control but can be redundant or even misleading for high-level abstract planning.
• Current VLMs significantly struggle with organizing visual information across the time dimension, demonstrating a consistent performance drop when provided with multi-step visual histories, even with explicit time indexing.
• Model strengths are not universal: Even state-of-the-art models like GPT-4o exhibit varying performance across different scenario types (e.g., strong in common-sense reasoning but weaker in spatial perception compared to Gemini).
• Fine-grained error analysis reveals distinct failure modes: high-level errors often involve missing steps or invalid plans, while low-level errors are predominantly perception or control related.
• Targeted post-training holds significant promise for developing highly capable embodied agents, as demonstrated by a 3B parameter model achieving SOTA performance without in-context learning by differentiating high-level and low-level task objectives.

About the Speaker(s)

The talk was primarily presented by Rui Yang. The research behind EmbodiedBench is a collaborative effort involving a large team of researchers including Hanyang(Jeremy) Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. These researchers are affiliated with prominent academic and research institutions such as the University of Illinois, the University of Washington (UW), the University of Toronto, and the Toyota Technological Institute at Chicago (TTIC). Their collective expertise spans areas of embodied AI, multi-modal large language models, and comprehensive benchmarking methodologies. Their work focuses on understanding and advancing the capabilities of AI agents in complex, vision-driven interactive environments.