Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark

Speakers: Yunzhuo Hao; Jiawei Gu; Huichen Wang; Linjie Li; Zhengyuan Yang; Lijuan Wang; Yu Cheng

Conference: ICML 2025

YouTube: https://slideslive.com/39044026

Overview

In an era where large language models (LLMs) are rapidly evolving into multimodal large language models (MLLMs), their ability to process and generate information across various modalities—text, images, audio—is increasingly critical. However, a fundamental question persists: can these models truly reason in a deeply multimodal fashion, or do they primarily rely on their strong language capabilities, with visual inputs serving merely as shallow cues? This talk, presented by Yunzhuo Hao at ICML 2025, introduces EMMA (Enhanced MultiModal ReAsoning Benchmark), a novel and rigorous benchmark designed to probe the genuine cross-modal reasoning abilities of MLLMs.

The core motivation behind EMMA stems from the observation that many existing benchmarks, while valuable, often feature problems that can be solved predominantly through text-based reasoning or by extracting superficial information from images. Such scenarios fail to test the intricate "back and forth" interaction between modalities that human experts employ when tackling complex problems, such as visualizing forces in a physics problem or mentally simulating spatial transformations. EMMA aims to fill this critical gap by providing a suite of challenging problems that inherently demand iterative cross-modal understanding and synthesis.

The significance of EMMA lies in its uncompromising focus on problems that necessitate true multimodal reasoning. By revealing the substantial performance disparity between state-of-the-art MLLMs and human experts, EMMA underscores a crucial limitation in current AI capabilities. This benchmark not only highlights the urgent need for new architectural paradigms and training methodologies but also provides a robust tool for tracking progress in developing MLLMs that can genuinely "think" in multimodalities, moving beyond language-dominant processing to achieve more human-like cognitive abilities.

Background

▶ Watch: Defining multimodal reasoning and MLLM limitations (0:00)

The rapid advancements in large language models have led to the development of multimodal variants, such as GPT-4o, Gemini 1.5 Pro, and various open-source initiatives. These MLLMs are designed to interpret and integrate information from diverse sources, most commonly text and images. The prevailing assumption is that by ingesting both modalities, these models should be capable of sophisticated reasoning that transcends the limitations of single-modality processing. However, the speaker posits that this assumption is largely unfounded for complex tasks requiring true cross-modal interaction.

The challenge lies in defining and evaluating "multimodal reasoning." As exemplified by a college physics problem involving point charges, human problem-solving often involves an iterative process: interpreting a visual (the charge configuration), applying textual knowledge (like charges repel, opposite charges attract), sketching visual aids (force vectors), combining these visual representations, and then comparing the result with textual or visual options. This "back and forth" interaction between interpreting visual inputs and applying reasoning steps across modalities is what the EMMA team defines as true multimodal reasoning. Problems requiring this type of reasoning are typically difficult to solve with text-dominant strategies or a single pass over visual information.

Existing benchmarks like MathVista and MMMU, while important for evaluating MLLMs, have been found to contain a significant number of problems solvable without genuine cross-modal reasoning. Many questions can be answered by extracting information primarily from text or by relying on shallow visual cues that don't require deep spatial or relational understanding. This creates an inflated perception of MLLMs' multimodal capabilities. The problem, therefore, is not a lack of benchmarks per se, but a lack of benchmarks that rigorously filter for and exclusively focus on tasks demanding the kind of iterative, cross-modal thinking that humans employ. This critical gap necessitates the creation of an "enhanced multimodal reasoning benchmark" like EMMA to accurately assess and drive progress in this complex domain.

Key Findings

▶ Watch: Current MLLM performance on EMMA: Large gap with humans (3:30)

The EMMA benchmark delivers a stark message: current multimodal large language models are not yet true multimodal reasoners. The key findings from the evaluation of various models on EMMA highlight significant shortcomings and point towards critical areas for future research and development.

First and foremost, state-of-the-art MLLMs significantly underperform human experts on EMMA. Even proprietary, closed-source models like O1 (a top performer at the time of initial evaluation) lagged human experts by more than 30 percentage points. The latest leaderboard update shows Gemini 1.5 Pro ranking first, but still trailing human experts by over 20 percentage points. Open-source models fare even worse, with Qwen2-VL, the top performer on the EMMA-mini subset, scoring a mere 37.25%. This substantial gap underscores that even the most advanced models struggle with the iterative, cross-modal reasoning EMMA demands.

A crucial observation from comparing human and model problem-solving strategies is that human experts frequently employ visual thinking aids, such as sketches, to visualize concepts and work through problems. In contrast, MLLMs predominantly rely on textual steps for reasoning, even when visual cues are paramount. This suggests a fundamental difference in their approach to multimodal problems, with models failing to leverage visual information as dynamically and interactively as humans do.

Further analysis of errors made by models like O1 on subjects such as math and coding revealed that most errors stem from visual reasoning challenges. This indicates that the bottleneck is not necessarily in language comprehension or basic visual recognition, but in the ability to perform complex operations like spatial imagination, visual simulation, or relational reasoning across visual elements, especially when these require iterative interaction with textual information.

The talk also explored several approaches to boost MLLM performance, with generally limited success. Chain of Thought (CoT) prompting, which encourages models to "think step by step," showed divergent and often counterproductive effects. While it sometimes improved performance for closed-source models, the gains were small, and it frequently reduced performance for open-source models, leading to more hallucinations. This detrimental effect was particularly pronounced for tasks heavily reliant on visual reasoning, such as 2D transformation, suggesting that generating more text-based "thought" tokens does not necessarily aid visual processing and can introduce erroneous textual reasoning.

Similarly, test-time compute scaling strategies, including majority voting, best-of-N selection, and tournament selection, yielded only limited improvements. Even when using stronger models as reward models or allowing models to generate more answers, the gains were marginal, indicating that simply increasing computational effort at inference time does not resolve the underlying deficiency in multimodal reasoning.

Despite these challenges, the talk acknowledged that reinforcement learning (RL) with verifiable rewards, exemplified by algorithms like GRPO and models such as VL-Rethinker and SRPO, has shown promising results on EMMA. This suggests that explicit training signals guiding models towards verifiable reasoning steps, potentially across modalities, could be a viable path forward.

In summary, the key findings from EMMA unequivocally demonstrate that current MLLMs are largely language-driven systems that struggle with genuine cross-modal reasoning. Visual reasoning stands out as a critical bottleneck, necessitating a fundamental shift in both model architectures and training paradigms to foster true cross-modal collaboration rather than just language-centric processing augmented by visual input.

Technical Deep Dive

▶ Watch: Visual reasoning identified as a key bottleneck for MLLMs (4:30)

The EMMA benchmark is meticulously designed to isolate and evaluate true multimodal reasoning, distinguishing it from tasks that can be solved by text-dominant strategies or shallow visual cues. This section delves into the technical specifics of EMMA's construction, the methodologies used for evaluation, and the various approaches explored to enhance MLLM capabilities.

EMMA Benchmark Design

EMMA comprises 2788 problems in total, including 1796 brand new ones, spanning four core subjects: math, physics, chemistry, and coding. A crucial aspect of EMMA's design is its provision of detailed category labels for each question, such as 3D simulation, 2D transformation, and multi-hop object counting. These fine-grained labels enable researchers to conduct precise analyses of model strengths and weaknesses across different types of multimodal reasoning challenges.

The distinguishing feature of EMMA is its rigorous human-in-the-loop pipeline for filtering questions. Unlike existing benchmarks, EMMA actively discards any problem that can be solved primarily by text-dominant reasoning or by a single, superficial visual pass. This strict filtering ensures that every problem in the final dataset genuinely demands cross-modal reasoning, requiring models to "jump back and forth between the image and the text" to arrive at a solution. This iterative interaction is the hallmark of true multimodal reasoning, mirroring how human experts tackle complex problems by integrating visual interpretation with abstract concepts. The new questions are manually created in collaboration with human experts, further ensuring their complexity and multimodal nature.

Evaluation Methodologies

The EMMA team evaluated a diverse set of MLLMs, encompassing both proprietary and open-source models.

• Proprietary Models: Key closed-source models assessed include O1, Gemini 1.5 Flash Thinking, GPT-4o (used in initial teaser demonstrations), and later updates featuring Gemini 1.5 Pro and Claude. These models represent the cutting edge of commercial MLLM development.
• Open-Source Models: Qwen2-VL was a prominent open-source model evaluated, particularly on the EMMA-mini subset, providing a baseline for publicly available models.

Performance was measured primarily by accuracy, comparing model outputs against ground-truth answers. The evaluation also included qualitative analysis, contrasting the reasoning steps of human experts (who often use sketches) with those of MLLMs (which tend to rely on textual descriptions).

Approaches to Boost MLLM Ability

The talk explored several strategies to improve MLLMs' multimodal reasoning, each with varying degrees of success:

1. Chain of Thought (CoT) Prompting:

• Mechanism: This technique involves instructing the model to generate intermediate reasoning steps in natural language before producing a final answer. The hypothesis is that explicit step-by-step thinking might aid complex problem-solving.
• Observed Effects: The impact of CoT was highly dependent on the model type and task. For closed-source models, CoT generally yielded small performance improvements. However, for open-source models, CoT often reduced performance and introduced more hallucinations. This was particularly evident in visually-intensive tasks like 2D transformation, where the task relies heavily on visual simulation and spatial imagination, which are "hard to express in language." In contrast, tasks like multi-hop object counting, which involve attribute recognition and relational reasoning that can be aided by language, saw some benefits for closed-source models. The conclusion drawn is that the more a task relies on visual reasoning, the less benefit (and potentially more harm) CoT brings, especially for open-source models. The speaker explained that CoT, being a text-based reasoning method, outputs more tokens for textual thought, but these textual thoughts do not inherently help visual output, often leading to hallucinations when the model cannot visually verify its textual "thinking."

1. Test-Time Compute Scaling:

• Mechanism: This approach investigates whether simply allowing models to generate multiple answers and then selecting the best one can improve performance. Three strategies were tested: majority voting (selecting the most frequent answer), best-of-N selection (picking the answer with the highest confidence score or reward), and tournament selection (an iterative process of selecting better answers). Various models were used to generate answers, and different reward models were employed for selection.
• Observed Effects: While using stronger reward models or generating more answers did lead to some improvement, the gains were consistently limited. This suggests that simply increasing computational resources during inference does not address the fundamental limitations in multimodal reasoning.

1. Reinforcement Learning (RL):

• Mechanism: RL approaches, particularly those with verifiable rewards, aim to train models to produce correct reasoning steps that can be checked against external criteria. GRPO (Generalized Policy Optimization for Reasoning) is cited as a representative algorithm.
• Observed Effects: Models that have adopted RL, such as VL-Rethinker and SRPO, have shown promising results on the EMMA dataset. This indicates that explicit feedback mechanisms that guide models towards verifiable multimodal reasoning paths could be a more effective training paradigm.

Future Directions

The talk concluded by outlining two promising directions for overcoming the current limitations:

1. Scaling up Modality-Unified Models: This paradigm focuses on developing model architectures where different modalities are naturally aligned and processed in a unified manner. The intuition is that such architectures inherently benefit multimodal reasoning by fostering seamless interaction between modalities. Examples like MVOT (Multimodal Vision-and-Language Transformer) and approaches involving visual planning have made preliminary explorations and shown promising results.

1. Tool-Augmented Models: This direction involves empowering MLLMs to generate or manipulate visual content through the use of external tools. By allowing models to interact with visual environments or create sketches, they can potentially mimic human visual thinking processes. O3 is mentioned as a representative model in this category, with open-source efforts like DeepIce and OpenThinkImage following suit. This approach seeks to enable models to perform the "back and forth" sketching and visualization that humans use, thereby bridging the gap in visual reasoning.

In essence, the technical deep dive into EMMA reveals a meticulously crafted benchmark that exposes the Achilles' heel of contemporary MLLMs: their inability to perform true, iterative cross-modal reasoning. The explored solutions highlight that simple prompting or scaling strategies are insufficient, emphasizing the need for fundamental architectural and training innovations that promote genuine visual and cross-modal collaboration.

Experimental Setup & Results

▶ Watch: Test-time compute scaling and RL show limited gains (6:50)

The experimental evaluation of MLLMs on the EMMA benchmark provides concrete evidence of their current limitations in multimodal reasoning. The setup involved a carefully curated dataset and a comparative analysis against both human performance and various model-boosting strategies.

Datasets and Baselines

The core of the experimental setup is the EMMA dataset, which consists of 2788 problems, with 1796 brand new ones specifically designed to demand cross-modal reasoning. These problems are categorized across math, physics, chemistry, and coding, and further broken down into fine-grained reasoning types such as 3D simulation, 2D transformation, and multi-hop object counting. The rigorous human-in-the-loop filtering pipeline ensures that problems solvable by text-dominant reasoning or a single visual pass are excluded, making EMMA a challenging and focused benchmark for true multimodal reasoning. An EMMA-mini subset was also used for some evaluations.

For comparison, human expert performance serves as the gold standard baseline, showcasing the target level of multimodal reasoning. Existing benchmarks like MathVista and MMMU were implicitly referenced as less rigorous in their demand for cross-modal interaction, justifying EMMA's creation.

Hardware and Metrics

The talk did not specify the exact hardware (e.g., GPU types, cluster sizes) used for training or inference of the evaluated models. This is common for conference presentations focusing on benchmark results rather than model development. However, the evaluation focused on the accuracy metric, which is a straightforward and widely accepted measure for problem-solving benchmarks.

Evaluated Models

A range of both proprietary (closed-source) and open-source MLLMs were evaluated:

• Proprietary Models:
• O1: A strong performer in the initial evaluations.
• Gemini 1.5 Flash Thinking: Another high-ranking model.
• GPT-4o: Used in the introductory physics teaser to demonstrate typical model failures.
• Gemini 1.5 Pro: The current top performer on the EMMA leaderboard (as of the talk update).
• Claude: Also evaluated, though specific performance numbers were not detailed beyond its ranking relative to Gemini 1.5 Pro.
• Open-Source Models:
• Qwen2-VL: The top open-source model evaluated on EMMA-mini.

Headline Numbers and Key Results

The performance figures reveal a significant gap between MLLMs and human capabilities:

• Human Performance: Consistently superior, serving as the benchmark for true multimodal reasoning.
• O1 (initial top closed-source): Lagged human experts by more than 30 percentage points.
• Qwen2-VL (top open-source on EMMA-mini): Scored a very low 37.25%.
• Gemini 1.5 Pro (current SOTA on leaderboard): While outperforming earlier models, it still lags human experts by more than 20 percentage points. This indicates that even with continuous model improvements, the fundamental challenge remains.

These headline numbers firmly establish that current MLLMs are far from achieving human-level multimodal reasoning.

Ablations and Analysis

The talk presented several analyses to understand the nature of model failures and the effectiveness of different strategies:

• Error Analysis (O1 on Math and Coding): The majority of errors were attributed to visual reasoning challenges. This included difficulties with tasks requiring spatial imagination, visual simulation, or understanding complex relationships within an image, even when textual instructions were clear.
• Chain of Thought (CoT) Prompting Analysis:
• Divergent Tendencies: CoT generally improved performance for closed-source models (though gains were small), but reduced performance for open-source models, often leading to hallucinations.
• Task-Specific Effects: For tasks heavily reliant on visual reasoning, such as 2D transformation (evaluating visual simulation and spatial imagination), all models experienced a performance drop with CoT. Conversely, for tasks like multi-hop object counting (testing attribute recognition and relational reasoning, which allows language to assist), closed-source models benefited from CoT. This highlights that CoT, being text-centric, is less effective and potentially harmful for purely visual reasoning tasks.
• Test-Time Compute Scaling Analysis: Strategies like majority voting, best-of-N, and tournament selection showed limited gains, even when using stronger reward models. This suggests that simply generating more text or selecting from multiple text outputs does not compensate for a lack of genuine multimodal reasoning.
• Human vs. Model Reasoning: A qualitative comparison revealed that humans instinctively use visual aids (like sketching) for problems requiring multimodal reasoning, while models primarily default to textual reasoning steps, even when visual thinking would be more efficient and accurate.

In conclusion, the experimental results from EMMA provide irrefutable evidence that current MLLMs are deficient in true multimodal reasoning. The performance gap, the nature of errors, and the limited efficacy of existing boosting techniques all point to a fundamental architectural and conceptual limitation that needs to be addressed for MLLMs to become genuinely intelligent multimodal agents.

Practical Implications

▶ Watch: Conclusion: MLLMs are not yet true cross-modal reasoners (8:00)

The findings from the EMMA benchmark carry significant practical implications for everyone involved in the development, deployment, and application of multimodal large language models. They challenge prevailing assumptions about MLLM capabilities and highlight critical areas requiring immediate attention.

For Practitioners and Model Builders

The most crucial takeaway is that current MLLMs are largely insufficient for tasks demanding deep, iterative cross-modal reasoning. Practitioners should exercise caution and avoid overestimating the true multimodal capabilities of these models. Strong performance on simpler visual question-answering tasks or those solvable by shallow visual cues does not generalize to problems requiring complex visual simulation, spatial imagination, or iterative interaction between visual and textual information. Model builders must recognize that simply scaling up existing architectures or adding more data may not bridge this fundamental reasoning gap. Instead, there's a clear mandate to develop new model architectures that can genuinely integrate and interact with different modalities at a deeper, more cognitive level, rather than treating visual input as merely another token stream for a language model.

For Infrastructure Teams and Deployers

Infrastructure teams responsible for deploying MLLMs should understand that the limitations exposed by EMMA imply that certain complex, real-world applications requiring nuanced multimodal understanding may not be feasible with current technology. This could impact resource allocation, model selection, and the design of deployment pipelines. There's a clear need for future MLLMs that are not just "language-driven" with visual augmentation, but true "cross-modal collaboration" systems. This will likely necessitate new types of hardware acceleration, software frameworks, and system designs capable of supporting dynamic, interactive processing of diverse modalities, moving beyond static token processing.

Tradeoffs and Limitations

The exploration of techniques like Chain of Thought (CoT) prompting reveals important tradeoffs. While CoT can offer marginal gains for some closed-source models on certain tasks, it often proves detrimental for open-source models and can introduce hallucinations, especially in visually-intensive scenarios. This means CoT is not a universal solution and its application requires careful consideration of the model and task at hand. Relying on CoT for visual reasoning tasks might lead to erroneous or fabricated textual "thoughts" that don't correspond to the visual reality.

Similarly, test-time compute scaling strategies offer only limited improvements. This implies that simply throwing more computational power at inference time—by generating multiple answers or using stronger reward models—does not solve the underlying problem of a model's inability to perform true multimodal reasoning. The bottleneck is not computational effort but a lack of intrinsic reasoning capability.

The current limitation of MLLMs is their tendency to default to textual reasoning, even when visual thinking is more efficient and accurate. This suggests that models struggle to form rich, internal visual representations and manipulate them effectively. The field is still nascent in developing models that can "sketch" or "visualize" internally or through external tools, much like humans do.

In essence, EMMA highlights that the current paradigm for MLLM development is hitting a ceiling for tasks requiring genuine multimodal intelligence. The practical implication is a call to action for a paradigm shift: moving away from merely augmenting language models with visual inputs towards building truly multimodal agents capable of dynamic, iterative, and collaborative reasoning across modalities. This will involve rethinking model architectures, training objectives, and evaluation metrics to foster the kind of cross-modal cognition demonstrated by human experts.

Key Takeaways

• MLLMs Lack True Multimodal Reasoning: Current state-of-the-art multimodal large language models (MLLMs) are not yet true multimodal reasoners, largely failing at tasks requiring iterative "back and forth" interaction between visual and textual information.
• EMMA Benchmark Exposes Deficiencies: The EMMA benchmark rigorously filters out problems solvable by text-dominant reasoning or shallow visual cues, providing a challenging test suite that demands genuine cross-modal understanding and synthesis.
• Significant Performance Gap: Even top-performing closed-source models like O1 and Gemini 1.5 Pro significantly underperform human experts (by over 20-30 percentage points), while open-source models like Qwen2-VL score as low as 37.25%.
• Visual Reasoning is a Core Bottleneck: Most errors made by MLLMs stem from visual reasoning challenges, such as spatial imagination or visual simulation, indicating that models struggle to effectively interpret and manipulate visual information in a cognitive manner.
• Limited Efficacy of Current Boosting Techniques: Chain of Thought (CoT) prompting offers minimal gains for some models and can even introduce hallucinations in others, particularly for visually-intensive tasks. Test-time compute scaling strategies also yield only marginal improvements.
• Need for New Paradigms: To overcome these limitations, future research must move towards modality-unified architectures that naturally align different modalities or tool-augmented models that can generate and manipulate visual content, fostering true cross-modal collaboration.

About the Speaker(s)

The talk "Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark" was presented by Yunzhuo Hao at ICML 2025. Yunzhuo Hao is the lead presenter for this work, representing a research team that includes Jiawei Gu, Huichen Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. While their specific affiliations were not detailed in the transcript, their collective authorship on this rigorous benchmark suggests a strong background in AI/ML research, likely within an academic institution or a leading technology company focused on advancing multimodal AI capabilities. Their work on EMMA contributes significantly to the understanding and evaluation of multimodal reasoning in large language models.