VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models

Speakers: Hila Chefer, Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, Shelly Sheynin

Conference: ICML 2025

YouTube: https://slideslive.com/39043892

Overview

In the rapidly evolving landscape of video generation, significant strides have been made in rendering visually stunning and high-fidelity content. However, as Hila Chefer from Meta AI and Tel Aviv University highlights in her ICML 2025 talk, "VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models," a critical challenge persists: achieving truly coherent and physically plausible motion. Despite the massive scaling of models and data, state-of-the-art video generation systems frequently falter when it comes to fundamental aspects of the temporal dimension, exhibiting jittery movements, illogical object interactions, and a general lack of understanding of real-world physics.

VideoJAM addresses this fundamental limitation by proposing a novel approach that integrates explicit motion information directly into the core learning objective and inference process of diffusion models. The key insight is that current models, primarily trained for pixel-wise reconstruction, are inherently insensitive to temporal incoherence. By encoding motion—specifically, optical flow—into an RGB format that the model already "understands," and then forcing the model to learn a joint appearance-motion latent representation, VideoJAM significantly enhances temporal consistency and motion quality. This work is particularly impactful as it achieves state-of-the-art results in motion generation with minimal architectural changes and limited fine-tuning data, challenging the prevailing notion that only massive scaling can unlock further progress in this domain.

The talk underscores a pivotal shift in addressing the limitations of generative AI: moving beyond brute-force scaling to incorporate more semantically rich priors. VideoJAM demonstrates that by thoughtfully augmenting the training objective and leveraging auxiliary signals during inference via inner guidance, it is possible to imbue video models with a deeper understanding of motion, leading to more realistic and coherent generated content. This research offers valuable insights for practitioners and researchers grappling with the complexities of temporal dynamics in generative models, providing a pathway to more intelligent and robust video synthesis.

Background

▶ Watch: Current video generation struggles with temporal coherence (0:00)

The field of video generation has witnessed an unprecedented acceleration in capabilities over the past few years. As recently as 2022, the state of the art, exemplified by models like Google's Imagine Video and Meta's Make-A-Video, produced videos that were often characterized by jitteriness and a lack of coherence. Fast forward to 2025, and models such as Meta's MovieGen and Google DeepMind's VEO 3 are capable of generating incredibly detailed and complex videos, some even incorporating audio. This rapid progress has largely been attributed to extensive scaling efforts—larger datasets, more complex model architectures, and improved backbones.

However, Chefer points out a critical, persistent flaw: despite these advancements in pixel quality and visual fidelity, even the most sophisticated models still struggle profoundly with the temporal dimension. Examples abound: MovieGen exhibits fundamental motion errors, and VEO 3, despite its impressive multimodal capabilities, fails spectacularly with basic physics, such as a screen exploding but remaining visually intact. This indicates that temporal understanding lags significantly behind the models' ability to render high-quality pixels.

The core hypothesis behind VideoJAM is that this struggle stems from the fundamental training objective of contemporary diffusion models. These models are primarily designed to reconstruct inputs pixel by pixel. When confronted with a video, their loss functions prioritize matching individual pixel values across frames. This objective proves to be remarkably insensitive to temporal inconsistencies. Consider a simple scenario: a person walking, where their legs momentarily switch places in the middle of the video. From a pixel-wise reconstruction perspective, the change might be minimal; the overall visual content of the frame remains largely similar. Yet, to a human observer, this temporal incoherence is immediately obvious and nonsensical.

To empirically validate this hypothesis, the VideoJAM team conducted a compelling experiment. They took coherent videos and randomly shuffled their frames, creating a temporally incoherent version that maintained similar appearance characteristics. They then measured the reconstruction loss of pre-trained diffusion models on both the original and shuffled videos. The expectation was that if the loss function were sensitive to temporal information, the incoherent video would yield a significantly higher loss. Surprisingly, the results showed that the loss was almost invariant to this "very strong perturbation" of frame shuffling, particularly in the critical coarse denoising steps. This striking invariance provided concrete evidence that the standard pixel-reconstruction objective indeed largely ignores temporal information, focusing predominantly on appearance. This fundamental disconnect between the training objective and the desired temporal coherence laid the groundwork for the VideoJAM solution.

Key Findings

▶ Watch: Empirical proof: Models insensitive to shuffled frames (3:00)

VideoJAM delivers several pivotal findings that challenge conventional approaches to video generation and offer a new paradigm for achieving temporal coherence:

• Inherent Insensitivity of Pixel-wise Objectives: The research empirically demonstrates that standard diffusion models, optimized for pixel-wise reconstruction, are fundamentally insensitive to temporal incoherence. The loss function shows minimal variation even when video frames are aggressively shuffled, confirming that the objective prioritizes appearance over motion logic.
• Effective Motion Encoding via RGB Optical Flow: VideoJAM successfully translates complex motion information (dense displacement fields captured by optical flow) into an RGB format. By mapping vector angles to colors and magnitudes to intensity, motion becomes a visual signal that existing pixel-reconstruction architectures can process, effectively making motion "visible" to the model.
• Joint Appearance-Motion Latent Representation: The method proposes creating a single latent representation that encapsulates both appearance and motion. This is achieved by extending the input projection layer of the diffusion model, forcing it to learn a unified representation from which both video frames and their corresponding optical flow can be predicted. This joint learning is crucial for ensuring consistency between appearance and motion.
• Powerful Inner Guidance during Inference: VideoJAM introduces inner guidance, an extension of classifier-free guidance, which leverages the predicted motion signal as an additional conditioning factor during inference. By encouraging the model to maximize the probability of the noisy motion given the noisy video, inner guidance significantly enhances temporal consistency, ensuring that the generated motion is self-consistent and plausible.
• State-of-the-Art Motion Quality with Minimal Resources: Despite fine-tuning on a mere 3 million samples (less than 3% of the original training data) and adding only two linear layers to a 30-billion parameter base model (DiT-30B), VideoJAM achieved superior motion quality. It outperformed state-of-the-art proprietary models like OpenAI's Sora and Kling 15 in human evaluations and automatic metrics (VBench) specifically for motion.
• Complementary Improvement of Motion and Appearance: Counterintuitively, improving motion quality with VideoJAM also led to improvements in text-faithfulness and overall appearance, compared to the base model. This suggests that a deeper understanding of temporal dynamics can positively impact other aspects of video quality, indicating that motion and appearance are not contradictory but rather complementary objectives.

These findings collectively demonstrate that explicit integration of temporal priors, rather than just scaling, is a highly effective strategy for overcoming the inherent limitations of current video generation models in producing coherent and physically plausible motion.

Technical Deep Dive

▶ Watch: Introducing VideoJAM: Encoding motion with optical flow (4:30)

The core innovation of VideoJAM lies in its elegant approach to embedding motion information into a framework primarily designed for static image synthesis, thereby enabling video models to "understand" and generate coherent temporal dynamics.

The Challenge of Pixel-wise Reconstruction

The initial empirical finding—that diffusion models' loss functions are largely invariant to frame shuffling—highlights the fundamental limitation of pixel-wise reconstruction for temporal tasks. A loss function designed to minimize the difference between individual pixels across frames will struggle to penalize errors where visual content remains similar but temporal order or physical consistency is violated. For instance, if a person's left and right legs swap positions in consecutive frames, the pixel-level difference might be minimal, yet the motion is entirely unphysical. This necessitates a mechanism to introduce an explicit understanding of motion.

Encoding Motion via Optical Flow

VideoJAM tackles this by introducing optical flow as the explicit representation of motion. Optical flow computes dense displacement fields between pairs of frames, essentially providing a vector for each pixel indicating its movement (direction and magnitude) from one frame to the next.

However, standard diffusion models operate on RGB pixel data. To bridge this gap, VideoJAM cleverly encodes the optical flow vectors into an RGB image format:

• Direction to Color: The angle of the optical flow vector (representing the direction of movement) is mapped to a specific color in the RGB spectrum. For example, upward motion might be green, downward red, and horizontal blue.
• Magnitude to Intensity: The magnitude of the vector (representing the speed or extent of movement) determines the intensity or saturation of that color. High movement results in vivid, intense colors, while minimal movement leads to desaturated, lighter colors, and no movement (e.g., static background) results in white.

This transformation creates an "RGB optical flow image" that visually represents motion in a format that the diffusion model is already equipped to process. The model is then trained to predict both the traditional RGB video frames and these RGB optical flow frames.

Architectural Modifications

A remarkable aspect of VideoJAM is its ability to achieve significant improvements with minimal changes to the underlying model architecture. The modifications are confined to just two linear layers within a pre-existing diffusion model (specifically, a DiT-30B model, a large transformer-based diffusion model):

1. Extended Input Projection (W_in): In a typical diffusion model, an input projection layer W_in maps the raw input (e.g., noisy video frames) into the model's latent space. VideoJAM extends this W_in layer with additional rows. This modification allows the input to include not just the noisy RGB video frames but also the noisy RGB optical flow frames. Critically, these two signals are linearly combined within this initial projection. The result is a single latent representation that is a linear combination of both appearance and motion information. This design choice is fundamental: it forces the model to learn a unified, joint representation where appearance and motion are intrinsically linked, rather than processing them separately.
2. Additional Output Projection: At the output stage, where the latent representation is projected back into the pixel space to predict the denoised RGB video, an additional projection matrix is introduced. This new projection path allows the model to simultaneously predict the RGB optical flow representation.

The core diffusion model itself—its internal layers, dimensions, and parameters—remains entirely untouched. This highlights the efficiency and surgical precision of the VideoJAM approach, adding almost no new parameters to the model.

Training Strategy

The training process for VideoJAM involves fine-tuning the entire model, including the newly added linear layers and all existing weights of the DiT-30B backbone. The objective is to simultaneously predict both the appearance (RGB video) and the motion (RGB optical flow) from the shared latent representation. This dual prediction task, guided by the encoded optical flow signal, compels the model to develop an understanding of temporal coherence.

A key practical consideration was computational efficiency. The fine-tuning was performed on a relatively small dataset of 3 million samples, representing less than 3% of the original data used for the base DiT-30B model. This demonstrates that substantial improvements in temporal quality can be achieved with limited additional data and computational resources, a significant advantage for practical deployment.

Inference with Inner Guidance

During inference, the model generates both the denoised video frames and the denoised optical flow. While one could simply discard the motion signal, VideoJAM proposes a more sophisticated approach: inner guidance. This mechanism leverages the predicted motion as an additional conditioning signal to further enhance temporal consistency.

Inner guidance is conceptualized as an extension of classifier-free guidance. In standard classifier-free guidance, the model's sampling distribution is modified to encourage generations that are highly probable given a text prompt (P(text | input)). VideoJAM extends this by adding a third guidance term:

• P(noisy motion (DT) | noisy video (XT)): This term encourages the model to generate a motion signal (DT) that is highly probable and consistent with the noisy video (XT) being generated at that step. Essentially, it acts as a self-consistency check, ensuring that the predicted motion makes sense given the visual content, and vice versa.

The full sampling distribution, therefore, seeks to maximize:

1. The vanilla probability of the model (P(video, motion | text)).
2. The text-conditional probability (P(text | input)), as in standard classifier-free guidance.
3. The inner guidance probability (P(noisy motion | noisy video)), which dynamically guides the generation towards temporally coherent outputs.

This dynamic, internal guidance loop ensures that the model's generative process is constantly nudged towards outputs where appearance and motion are mutually consistent and physically plausible. The implementation of inner guidance parallels that of classifier-free guidance, making it a readily integrable technique.

Experimental Setup & Results

▶ Watch: VideoJAM's training objective: Predict appearance and motion (6:00)

The effectiveness of VideoJAM was rigorously evaluated using both automatic metrics and human perception studies, demonstrating significant advancements in motion generation.

Base Model: The foundation for VideoJAM was a DiT-30B model. This is an internal, large transformer-based diffusion model developed by Meta AI, featuring 30 billion parameters. While not publicly available, it represents a state-of-the-art backbone for video generation.

Training Data: The VideoJAM model was fine-tuned on a relatively small dataset of 3 million samples. This constitutes less than 3% of the data typically used for training such large-scale video models, underscoring the efficiency of the proposed method in achieving substantial improvements with limited additional data.

Evaluation Metrics:

• Automatic Metrics: The primary automatic evaluation was conducted using VBench, a comprehensive benchmark suite for video generation. VideoJAM consistently outperformed all competitors across VBench metrics designed to assess motion quality and temporal coherence.
• Human Evaluations: Crucially, human evaluators were asked to compare videos generated by VideoJAM against those from leading proprietary models. The results showed a strong preference for VideoJAM's motion quality:
• 68.5% preference over OpenAI's Sora for motion quality.
• 63.8% preference over Kling 15 for motion quality.

These figures are particularly impressive given that Sora and Kling 15 were considered the state-of-the-art models at the time of the research.

Headline Numbers and Key Findings:

• Motion Quality: VideoJAM demonstrated superior motion quality across the board, as evidenced by both VBench scores and human preferences. This was the primary objective of the work, and it was achieved convincingly.
• Efficiency: The ability to achieve these results with only two additional linear layers and fine-tuning on a mere 3 million samples highlights the exceptional efficiency of the VideoJAM approach. It suggests that targeted interventions can be more impactful than simply scaling up models and data further.
• Complementary Improvements: A surprising and significant finding was that VideoJAM not only improved motion but also enhanced other aspects of video quality when compared to its base model, DiT-30B. Specifically, improvements were noted in text-faithfulness (how well the video matches the input text prompt) and overall appearance quality. This suggests that a deeper understanding of temporal dynamics, facilitated by VideoJAM, can lead to a more holistic improvement in generated video content, indicating that motion and appearance are not conflicting objectives but can be improved synergistically.
• Resolution Trade-off: The talk acknowledged a current limitation: VideoJAM's generated videos were at a lower resolution compared to those produced by models like Sora and Kling 15. The speaker expressed confidence that if the VideoJAM approach were applied at a larger scale or integrated into higher-resolution generation pipelines, this disparity could be mitigated.
• No Motion Scenarios: In cases where videos contained no motion, VideoJAM correctly predicted a "completely white" optical flow signal, and the generated video exhibited frozen, static motion as expected, demonstrating robustness across different motion profiles.

The experimental results definitively prove that explicitly guiding diffusion models with encoded motion information, coupled with an intelligent inference strategy like inner guidance, is a highly effective method for overcoming the persistent temporal coherence issues in video generation.

Practical Implications

▶ Watch: Architectural changes for joint appearance-motion representation (7:00)

VideoJAM offers profound practical implications for various stakeholders involved in the development and deployment of AI-generated video.

For Practitioners and Model Builders:

• Rethinking Training Objectives: The core finding that pixel-reconstruction objectives are insufficient for temporal coherence is a critical lesson. Practitioners should consider augmenting their training regimes with explicit temporal priors or objectives that are sensitive to motion dynamics, rather than solely relying on visual fidelity.
• Efficient Improvement: VideoJAM demonstrates that significant quality improvements, particularly in complex domains like motion, do not necessarily require massive computational resources or entirely new architectures. The approach of adding just two linear layers and fine-tuning on a small dataset (3 million samples) provides a blueprint for targeted, efficient enhancements. This means smaller teams or those with limited compute budgets can still make impactful contributions.
• Leveraging Auxiliary Signals: The concept of encoding complex information (like optical flow) into a format "understandable" by existing models (RGB) is a powerful paradigm. This can be extended to other forms of prior knowledge (e.g., physics constraints, semantic segmentation) to guide generative models more effectively.
• Inner Guidance as a General Technique: Inner guidance presents a novel and potentially generalizable technique for leveraging auxiliary predictions during inference. Model builders can explore applying this concept to other generative tasks where internal consistency across different modalities or aspects of the generation is crucial.

For Infrastructure Teams and Deployers:

• Resource Optimization: The low overhead of VideoJAM's architectural changes and fine-tuning data requirements translates to reduced computational costs for achieving higher quality. This is particularly relevant for deploying and maintaining large generative models, where every efficiency gain is valuable.
• Scalability Challenges Addressed: While scaling is important, VideoJAM shows that architectural intelligence can complement brute-force scaling. Infra teams can focus on optimizing for models that are designed to learn more efficiently from data, rather than just endlessly increasing model size or dataset volume.
• Improved User Experience: For applications relying on video generation (e.g., content creation, virtual reality, synthetic data generation), the enhanced temporal coherence and realistic motion provided by VideoJAM directly translate to a more convincing and satisfying user experience, reducing the "uncanny valley" effect often seen in AI-generated videos.

Tradeoffs and Limitations:

• Resolution: One acknowledged limitation is that VideoJAM currently operates at a lower resolution compared to some state-of-the-art models like Sora or VEO 3. While the speaker believes this can be addressed with further scaling, it's a current trade-off. Deployers might need to integrate VideoJAM's motion-enhanced outputs into a super-resolution pipeline if high-fidelity visuals are paramount.
• Physics Understanding: While optical flow significantly improves motion coherence, it is not a direct physical representation. As such, VideoJAM does not fully solve complex physics problems (e.g., fluid dynamics, rigid body interactions). For applications requiring extremely accurate physical simulations, further specialized priors or simulation engines might be necessary.
• Dependency on Optical Flow Computation: The method relies on pre-computing optical flow, which adds a processing step. While optical flow algorithms are mature and efficient, this still represents an additional component in the data pipeline.

In essence, VideoJAM offers a pragmatic and highly effective strategy for overcoming a critical bottleneck in video generation. It shifts the focus from purely quantitative scaling to a more qualitative, informed approach, promising more intelligent and robust generative AI systems.

Key Takeaways

• Current state-of-the-art video diffusion models, despite high pixel quality, fundamentally struggle with temporal coherence and physical plausibility due to their pixel-reconstruction training objectives.
• VideoJAM introduces a novel method to explicitly encode motion information using optical flow, which is then mapped into an RGB format, making it directly consumable by diffusion models.
• By extending the input projection layer, VideoJAM forces the model to learn a single latent representation that jointly encapsulates both appearance and motion, leading to a more integrated understanding of video content.
• The proposed inner guidance mechanism, an extension of classifier-free guidance, leverages the predicted motion signal during inference to dynamically enhance temporal consistency and coherence.
• VideoJAM achieves state-of-the-art motion quality, outperforming models like OpenAI's Sora and Kling 15, with minimal architectural changes (two linear layers) and limited fine-tuning data (3 million samples).
• The work demonstrates that improving motion quality can also lead to complementary improvements in text-faithfulness and overall appearance, suggesting that temporal understanding benefits holistic video generation.

About the Speaker(s)

The talk was presented by Hila Chefer, who conducted this work in collaboration between Meta AI and Tel Aviv University. The impressive results achieved with VideoJAM were highlighted as being particularly notable given that it originated as an "intern project," showcasing the significant impact that focused research can have even within resource constraints. The team of co-authors includes Uriel Singer, Amit Zohar, Yuval Kirstain, Adam Polyak, Yaniv Taigman, Lior Wolf, and Shelly Sheynin, indicating a strong collaborative effort behind this innovative research.