VideoRoPE: What Makes for Good Video Rotary Position Embedding?

VideoRoPE: What Makes for Good Video Rotary Position Embedding?

Speakers: Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, Dahua Lin

Conference: ICML 2025

YouTube: https://slideslive.com/39043889

Overview

The proliferation of video data and the increasing demand for sophisticated long-context video understanding tasks present significant challenges for current deep learning models, particularly in effectively encoding and leveraging positional information across vast temporal and spatial dimensions. This talk introduces VideoRoPE, a novel Rotary Position Embedding (RoPE) variant specifically designed to address the unique complexities of video inputs. Presented at ICML 2025 by a lab mate on behalf of the first author, Xilin Wei, and a team of researchers, VideoRoPE offers a principled solution to a critical bottleneck in video-centric AI systems: how to design position embeddings that are simultaneously scalable, robust, and effective for long-duration, high-resolution video streams.

The core problem VideoRoPE tackles lies in the limitations of existing RoPE variants, which often struggle with the inherent 2D/3D structure of video, suboptimal frequency allocation, lack of spatial symmetry, and inadequate temporal indexing for long contexts. These shortcomings manifest as models failing to distinguish between visually similar frames, misaligning attention, and ultimately hindering long-range dependency modeling. VideoRoPE distinguishes itself by systematically addressing these four essential properties, offering a comprehensive framework that dramatically improves performance across various video understanding benchmarks, including retrieval, understanding, and hallucination tasks, particularly in the presence of challenging temporal distractors.

This work is highly significant for the field of AI/ML, as robust positional encoding is fundamental to the performance of transformer-based architectures, which are now ubiquitous in multimodal learning. By providing a more effective mechanism for encoding position in video, VideoRoPE paves the way for more accurate, reliable, and scalable video analysis systems, impacting applications from surveillance and content moderation to autonomous driving and advanced human-computer interaction. Its principled design offers a blueprint for future research in spatial-temporal positional embeddings, ensuring that models can better interpret the dynamic and intricate nature of video data.

Background

▶ Watch: Introduction: Tackling long context video understanding (0:00)

The advent of transformer architectures has revolutionized natural language processing and, increasingly, computer vision and multimodal learning. A critical component of these architectures is the position embedding, which injects positional information into the input sequence, allowing the model to understand the order and relative locations of tokens. Among various position embedding techniques, Rotary Position Embeddings (RoPE) have gained prominence for their ability to encode relative positional information, which is beneficial for extrapolation to longer sequences and for maintaining a fixed-size context window during inference.

However, applying RoPE effectively to video data presents unique challenges that go beyond the one-dimensional sequence modeling common in text. Video inherently possesses a complex 2D spatial structure within each frame and a 1D temporal dimension across frames, creating a 3D spatial-temporal context. Existing RoPE variants, often designed primarily for text or simpler image tasks, frequently fall short when confronted with this multi-dimensional complexity. For instance, a purely one-dimensional RoPE, while effective for sequential data, lacks any explicit mechanism to model spatial relationships within a frame.

Prior attempts to adapt RoPE for video or image data, such as TAD-RoPE, RoPE-Tie, and M-RoPE, have addressed parts of the problem but not comprehensively. TAD-RoPE, for example, introduced temporal scaling but overlooked the crucial aspects of 2D/3D structure and spatial symmetry. RoPE-Tie focused on spatial symmetry but failed to adequately handle the temporal dimension. M-RoPE, despite incorporating a 3D structure, exhibited significant limitations, particularly in its frequency allocation and spatial symmetry, leading to issues like attention over-reliance on vertical signals and misaligned attention sinks. These shortcomings often result in models struggling with long-range temporal dependencies, exhibiting confusion when visually similar distractors are present, and failing to learn coherent spatial patterns. The fundamental issue is that without a carefully designed positional encoding, the transformer's self-attention mechanism cannot accurately discern relative positions in the intricate spatial-temporal landscape of video, leading to suboptimal performance in critical video understanding tasks. This gap highlights the necessity for a more holistic and principled approach to designing position embeddings tailored specifically for the multi-faceted nature of video data.

Key Findings

▶ Watch: M-RoPE's attention failure: spatial over-reliance, temporal neglect (2:00)

The research presented on VideoRoPE reveals several significant findings that underscore its efficacy and robustness in long-context video understanding. The core contribution lies in systematically identifying and addressing four essential properties for effective video position encoding, which existing methods only partially cover.

Firstly, VideoRoPE's holistic design, which incorporates 2D/3D structure, frequency allocation, spatial symmetry, and temporal indexing scaling, sets it apart from prior Rotary Position Embedding (RoPE) variants. This comprehensive approach ensures a more balanced and effective positional encoding framework tailored for the complex spatial-temporal dynamics of video.

A critical finding is VideoRoPE's superior robustness in challenging scenarios involving temporal distractors. Evaluated on the newly proposed V-NIA-D benchmark, which introduces periodically inserted, visually similar distractors, VideoRoPE consistently maintained high accuracy, visualized as a "mostly green" attention heatmap. In stark contrast, M-RoPE's heatmap turned "reddish," indicating frequent failures and significant struggles with distinguishing target frames from distractors. This demonstrates VideoRoPE's enhanced ability to model long-range dependencies and maintain stability under difficult conditions.

Furthermore, the study revealed that VideoRoPE facilitates more principled attention patterns. While M-RoPE often exhibited attention sinks along the Y-axis (vertical signals), leading to over-reliance on spatial cues and neglect of temporal relationships, VideoRoPE’s design allowed attention to correctly sink along the temporal axis. This enables the model to not only accurately locate the correct frame but also to reason effectively about its content and context, crucial for complex visual question answering tasks.

Empirical evaluations across three representative long video benchmarks—Long Video Bench, MLVU, and VideoMME—showcased VideoRoPE's consistent outperformance against all baselines. This broad applicability across diverse benchmarks underscores its strong adaptability and generalizability for real-world video analysis.

In the domain of video hallucination, VideoRoPE achieved major gains, demonstrating approximately a 30% improvement on temporal hallucination and about a 20% improvement on object relation hallucination. These substantial improvements highlight its strength in accurately modeling temporal dependencies and intricate spatial interactions, which are vital for generating coherent and factually grounded video descriptions.

Finally, detailed ablation studies confirmed the individual contributions of VideoRoPE's design components. Progressively adding its key modules—DL (Diagonal Layout), LTA (Low-frequency Temporal Allocation), and ATS (Adjustable Temporal Scaling)—resulted in a clear performance uplift. For instance, M-RoPE achieved around 50% on Long Video Bench and 60% on MLVU, while VideoRoPE, with all its modules, improved these scores to approximately 60% and 65% respectively. This systematic improvement validates the synergistic effect and importance of each design choice within the VideoRoPE framework.

Technical Deep Dive

▶ Watch: VideoRoPE's low-frequency temporal allocation design (5:30)

VideoRoPE is engineered to overcome the inherent limitations of existing Rotary Position Embedding (RoPE) variants when applied to the complex 3D spatial-temporal structure of video. The core of its technical innovation lies in addressing four fundamental properties: 2D/3D structure, frequency allocation, spatial symmetry, and temporal indexing scaling.

1. 2D/3D Structure

Traditional RoPE is inherently one-dimensional, designed for sequential tokens like those in text. When applied to video, this one-dimensional encoding fails to capture the intricate spatial relationships within a frame (2D) or the combined spatial-temporal dynamics (3D). Vanilla RoPE, as depicted in the talk, effectively places all tokens along a diagonal line, reflecting only sequential order. M-RoPE attempted to introduce spatial structure but suffered from misalignment issues, where some tokens collapsed along the time axis, and spatially related tokens might inadvertently share the same index. This led to a "large gap between visual and text token," introducing bias and hindering effective multimodal understanding.

VideoRoPE, in contrast, produces a clear and balanced 3D layout. It effectively preserves relative spatial relationships within frames while maintaining the incremental indexing across frames that is characteristic of RoPE. This design ensures that the distance from corners to the center of a frame remains consistent, avoiding the token collapse observed in M-RoPE. By combining robust spatial structure with RoPE's diagonal inductive bias, VideoRoPE provides a more coherent and effective 3D representation for video tokens.

2. Frequency Allocation

A critical aspect of RoPE's design is how frequencies are allocated to different dimensions. The choice of frequencies directly impacts how distinct positions are encoded and how well the model can differentiate between distant tokens. M-RoPE's approach to frequency allocation proved suboptimal for video, especially in long-context scenarios with periodic distractors. M-RoPE assigned high frequencies to the temporal dimension, specifically using the "first 16 rotary angles." This resulted in strong oscillations over time, causing distant frames, particularly those with similar visual content but different temporal positions (e.g., distractors), to end up with very similar positional encodings. This made it exceedingly difficult for the model to distinguish between the target frame and the distractors, leading to confusion and failures.

VideoRoPE adopts a more principled approach by implementing low-frequency temporal allocation. It assigns lower frequencies to the temporal dimension, utilizing the "last 16 rotary angles." This generates smoother curves over time, allowing the model to represent temporal distance more clearly and consistently. The consequence is a significant reduction in confusion from periodical distractors and enhanced robustness in long-range retrieval tasks. For the spatial dimensions (X and Y), VideoRoPE interleaves them at higher frequencies. The rationale is that fixed-resolution spatial data benefits more from high-frequency encoding, which helps preserve fine-grained spatial similarity. The interleaving strategy, as shown by experiments, maintains X and Y similarity more effectively than sequential allocation, which can make it harder for the model to learn coherent spatial patterns.

3. Spatial Symmetry

Spatial symmetry in position embeddings ensures that similar spatial relationships are encoded consistently, regardless of their absolute position on the grid. M-RoPE's spatial encoding suffered from a lack of symmetry, contributing to its over-reliance on vertical signals and causing attention sinks to appear along the Y-axis rather than the desired temporal axis. This imbalance resulted from suboptimal frequency allocation and a less-than-ideal spatial layout.

VideoRoPE addresses this with its diagonal spatial layout. By carefully arranging tokens, VideoRoPE maintains a balanced 3D structure that preserves relative spatial relationships and prevents token collapse. This design ensures that spatial attention remains localized and symmetric, allowing the model to focus on the correct frame (temporal attention) without being misled by imbalanced spatial cues. The goal is for attention sinks to appear primarily along the temporal axis, indicating that the model is effectively tracking and reasoning about events over time.

4. Temporal Indexing Scaling

In multimodal contexts, particularly when combining text and video, aligning the temporal structures across modalities is crucial. Text typically has a much slower "temporal" pace (token sequence) compared to the dense stream of visual tokens from video. Mismatched temporal indexing can introduce bias and hinder effective cross-modal understanding.

VideoRoPE introduces a scaling factor sigma (σ) to adjust time incidence and align modalities. The system defines tau as the token index in the sequence. For an initial text segment, tau directly maps to indices 0 to TS-1 for time, X, and Y. For the video segment, spanning TS to TS + TV - 1, the relative frame index is computed as tau - TS and then scaled by sigma. This sigma factor allows for fine-grained control over the temporal spacing of visual tokens, enabling a smoother alignment with the sparser text structure. Finally, for an ending text segment, tau from TS + TV to TS + TV + TE - 1 uses linear indexing across all dimensions to maintain consistency. This flexible design supports multi-modal inputs (text, video, text-video patch, text) while ensuring a smooth and balanced temporal alignment, which is critical for tasks like video captioning or visual question answering that involve both textual prompts and video content.

In summary, VideoRoPE's technical prowess stems from its holistic and principled approach to designing position embeddings for video. By meticulously considering the 3D structure, optimizing frequency allocation for temporal and spatial dimensions, ensuring spatial symmetry through a diagonal layout, and providing adjustable temporal scaling for multimodal alignment, VideoRoPE significantly enhances the ability of transformer models to process and understand long-context video data.

Experimental Setup & Results

▶ Watch: VideoRoPE's balanced 3D diagonal layout for spatial structure (8:00)

VideoRoPE's performance was rigorously evaluated across a suite of challenging long video understanding benchmarks, designed to test its robustness, adaptability, and ability to model complex spatial-temporal dependencies.

Benchmarks and Tasks

The evaluation primarily focused on three representative long video benchmarks:

1. Long Video Bench: A general benchmark for long-duration video understanding.
2. MLVU (Multi-modal Long Video Understanding): Specifically designed to test models on tasks requiring joint understanding of video and other modalities.
3. VideoMME: Another benchmark for assessing video understanding capabilities.

In addition to these, the researchers introduced a novel benchmark, V-NIA-D, a more challenging variant of the basic V-NIA (Visual Needle-in-a-Haystack) task. While V-NIA requires models to find a target frame and answer a visual question, V-NIA-D periodically inserts visually similar "distractor" frames throughout the video. This variant specifically tests the model's robustness and its ability to maintain long-range temporal understanding in the presence of confusing, repetitive visual cues.

The tasks covered included:

• Long Video Retrieval: Assessing the model's ability to locate specific frames or segments within long videos.
• Video Understanding: General comprehension of video content, often involving visual question answering.
• Video Hallucination: Evaluating the model's tendency to generate factually incorrect or inconsistent information in video descriptions or responses, specifically measuring temporal and object relation hallucination.

Baselines

VideoRoPE was benchmarked against several existing position embedding methods for comparison:

• Vanilla RoPE: The standard one-dimensional Rotary Position Embedding.
• TAD-RoPE: A variant incorporating temporal scaling.
• M-RoPE: A 3D structured RoPE variant, representing a more advanced baseline.

Metrics

Performance was measured using standard accuracy metrics for retrieval and understanding tasks. For the V-NIA and V-NIA-D tasks, accuracy heatmaps were visualized to show the model's performance across different conditions. For hallucination tasks, specific metrics for "temporal hallucination" and "object relation hallucination" were used, presumably reflecting the percentage reduction in such errors.

Headline Results

The experimental results consistently demonstrated VideoRoPE's superior performance:

1. Robustness in V-NIA-D: The accuracy heatmaps vividly illustrated the difference. M-RoPE's map turned "reddish" under V-NIA-D conditions, indicating frequent failures due to distractors. In contrast, VideoRoPE's map remained "mostly green," signifying stable and accurate performance even in the difficult V-NIA-D setting. This highlights VideoRoPE's improved temporal frequency design and its ability to handle long-range retrieval with distractors.

1. Attention Patterns: Analysis of attention maps in failure cases revealed that while both M-RoPE and VideoRoPE could sometimes locate the "needle frame," M-RoPE often gave the wrong answer due to over-reliance on vertical signals and ignoring the temporal dimension. Its attention sinks appeared along the Y-axis. VideoRoPE, with its principled frequency design, enabled attention to sink correctly along the temporal axis, leading to both correct frame localization and effective reasoning.

1. Consistent Outperformance on Long Video Benchmarks: VideoRoPE consistently outperformed all baselines across Long Video Bench, MLVU, and VideoMME, demonstrating strong robustness and adaptability. While specific absolute numbers for all baselines and benchmarks were not detailed in the transcript, the consistent "outperform" claim indicates a significant margin.

1. Major Gains in Video Hallucination: VideoRoPE achieved substantial improvements in reducing hallucination:

• Approximately 30% improvement on temporal hallucination.
• Approximately 20% improvement on object relation hallucination.

These results underscore VideoRoPE's enhanced capability in modeling precise temporal dependencies and complex spatial interactions, crucial for generating accurate video descriptions and answers.

Ablation Studies

Ablation studies were conducted to confirm the contribution of each design component within VideoRoPE:

• Starting with the M-RoPE baseline, which achieved approximately 50% accuracy on Long Video Bench and 60% on MLVU.
• By progressively adding VideoRoPE's key modules—DL (Diagonal Layout), LTA (Low-frequency Temporal Allocation), and ATS (Adjustable Temporal Scaling)—the performance steadily improved.
• The full VideoRoPE model achieved approximately 60% accuracy on Long Video Bench and 65% on MLVU. This systematic improvement validates the individual and cumulative impact of each design choice, confirming that the holistic architecture of VideoRoPE is indeed responsible for its superior performance.

The transcript did not specify the exact hardware used (e.g., GPU types, number of GPUs), dataset sizes beyond their names, or specific framework versions (e.g., PyTorch, TensorFlow). However, the consistent and significant performance gains across multiple benchmarks and the detailed ablation studies provide strong evidence for the effectiveness of VideoRoPE's design.

Practical Implications

▶ Watch: VideoRoPE's strong performance on long video benchmarks (11:00)

The development of VideoRoPE carries significant practical implications for a wide array of stakeholders in the AI/ML ecosystem, from researchers and model builders to infrastructure teams and deployers of video-centric applications. Its principled approach to spatial-temporal positional encoding addresses fundamental challenges in long-context video understanding, promising more robust, accurate, and scalable solutions.

For practitioners and model builders, VideoRoPE offers a powerful and effective drop-in replacement or enhancement for existing position embedding strategies in transformer-based video models. Its ability to accurately model long-range temporal dependencies and distinguish between visually similar distractors means that models incorporating VideoRoPE will be inherently more robust in real-world, complex video scenarios. This translates to higher accuracy in tasks such as:

• Long-form video retrieval: More precise identification of specific moments or objects within hours of footage.
• Video summarization and captioning: Generating more coherent, factually correct, and temporally consistent summaries or captions, significantly reducing temporal hallucination and object relation hallucination.
• Action recognition and anomaly detection: Improved ability to track subtle changes or rare events over extended periods.
• Multimodal AI systems: The adjustable temporal indexing scaling (sigma) is crucial for better aligning dense visual tokens with sparse text tokens, leading to more effective text-video integration in tasks like visual question answering or guided video generation.

For infrastructure teams and those responsible for deploying large-scale AI systems, VideoRoPE's focus on scalability and robustness is particularly valuable. While the talk doesn't detail specific computational overhead, a more effective positional encoding can lead to better model convergence and potentially more efficient inference by reducing the need for excessive architectural complexity to compensate for poor positional information. The ability to handle longer video contexts more effectively means that systems can process richer information without necessarily incurring proportional increases in error rates due to positional ambiguity.

Tradeoffs and Limitations:

While VideoRoPE offers substantial advantages, practitioners should consider potential tradeoffs. The design involves a more intricate spatial-temporal structure and specific frequency allocation rules, which might require a deeper understanding of its mechanics compared to simpler 1D position embeddings. The scaling factor sigma (σ) for multimodal alignment, while powerful, introduces a hyperparameter that may require careful tuning for optimal performance across different datasets and multimodal tasks. The talk does not explicitly detail the computational cost implications of VideoRoPE's more complex embedding generation compared to simpler RoPE variants, which could be a factor in extremely latency-sensitive or resource-constrained environments. However, given its significant performance gains, the benefits likely outweigh any minor increase in complexity or computational overhead for most high-value video understanding applications.

In essence, VideoRoPE moves the state-of-the-art forward by providing a principled and empirically validated solution to a critical problem in video AI. Its adoption can lead to the development of more intelligent, reliable, and capable video analysis systems, fostering advancements across numerous industries reliant on interpreting dynamic visual information.

Key Takeaways

• Four Essential Criteria for Video Position Encoding: Effective video position embeddings must address 2D/3D structure, frequency allocation, spatial symmetry, and temporal indexing scaling to accurately model complex spatial-temporal patterns.
• Limitations of Existing RoPE Variants: Prior Rotary Position Embedding (RoPE) methods like M-RoPE often struggle with temporal distractors, exhibit suboptimal frequency allocation, and lack proper spatial symmetry, hindering long-range dependency modeling in videos.
• VideoRoPE's Comprehensive Solution: VideoRoPE tackles these challenges through a novel design incorporating a robust 3D spatial-temporal structure, low-frequency temporal allocation, a diagonal spatial layout for symmetry, and an adjustable temporal indexing scaling factor (σ) for better text-video alignment.
• Superior Robustness and Performance: VideoRoPE consistently outperforms prior methods across diverse long video benchmarks (e.g., Long Video Bench, MLVU, VideoMME), demonstrating significantly enhanced robustness, especially in the presence of temporal distractors (e.g., V-NIA-D).
• Significant Gains in Accuracy and Reduced Hallucination: The model achieves substantial improvements, including approximately 30% reduction in temporal hallucination and 20% reduction in object relation hallucination, highlighting its strength in precise temporal and spatial interaction modeling.
• Validated Design Components: Ablation studies confirm the individual and cumulative contributions of VideoRoPE's design modules, validating the effectiveness of its holistic architecture in improving video understanding capabilities.

About the Speaker(s)

The research on VideoRoPE was primarily authored by Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin. The oral presentation at ICML 2025 was delivered by a lab mate of the first author, Xilin Wei, as the original authors were unable to attend in person due to visa issues. The talk did not provide specific affiliations or titles for the individual authors beyond their names. Collectively, this team has contributed a significant advancement in the field of video understanding by addressing fundamental challenges in positional encoding for long-context video data.