Sundial: A Family of Highly Capable Time Series Foundation Models

Sundial: A Family of Highly Capable Time Series Foundation Models

Speakers: Yong Liu, Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, Mingsheng Long

Conference: ICML 2025

YouTube: https://slideslive.com/39044060

Overview

Yong Liu from Tsinghua University presented Sundial, a novel family of time series foundation models designed to overcome long-standing challenges in time series forecasting. The talk delves into the inherent difficulties of applying large language model (LLM) paradigms to continuous, multi-dimensional time series data, particularly the issues of unstructured variations, ambiguous semantics, and the pervasive uncertainty in predictions. Traditional deep learning models, often relying on simplified priors like mean squared error, tend to produce over-smoothed, deterministic forecasts that fail to capture the true probabilistic nature of future events, leading to mode collapse.

Sundial addresses these limitations by introducing a prior-free generative modeling framework for time series forecasting. This innovative approach leverages patch-wise tokenization and a novel TimeFlow loss based on flow matching to learn the native and flexible "language" of time series without imposing restrictive assumptions on the underlying data distribution. The model's architecture is inspired by classical statistical methods like ARMA, yet it integrates the adaptive and in-context learning capabilities of modern transformers. This work is significant because it pushes the boundaries of foundation models beyond natural language and images, offering a scalable and generalizable solution for diverse time series applications, from weather forecasting to economic scheduling.

The importance of Sundial lies in its ability to generate diverse and statistically robust probabilistic forecasts, providing a more comprehensive understanding of future uncertainty. By achieving significant error reduction on established benchmarks and demonstrating remarkable inference speedups, Sundial paves the way for more accurate, adaptable, and deployable time series forecasting systems. Its release as out-of-the-box models on Hugging Face, capable of CPU inference, democratizes access to advanced time series AI, making foundation models a practical reality for a wider range of practitioners and applications.

Background

▶ Watch: Introduction to time series foundation models and challenges (0:00)

Time series forecasting is a critical component across numerous real-world applications, from finance and supply chain management to energy consumption and climate modeling. The core challenge lies in learning complex temporal dependencies in univariate data and intricate variable correlations in multivariate data. Historically, statistical methods like AutoRegressive Moving Average (ARMA) and its variants have provided foundational insights, excelling at capturing linear relationships. However, these methods often struggle with highly complicated, non-linear variations, require careful feature engineering, and possess limited model capacity, necessitating separate fitting for different time periods or datasets.

The advent of deep learning brought promises of learning non-linear features and offering large model capacities. Deep models, inspired by statistical theories, have indeed shown remarkable capabilities. Yet, they face their own set of hurdles: they are notoriously data-hungry, making training difficult without vast datasets, and often struggle with generalization to unseen data. This challenge is particularly acute in the context of building time series foundation models—general, scalable models pre-trained on massive, diverse datasets, capable of zero-shot forecasting and acting as versatile feature extractors.

Developing time series foundation models presents unique difficulties when drawing parallels to the success of large language models (LLMs). Unlike natural language, which has a discrete, tokenizable structure and well-defined semantics, time series data is highly unstructured, continuous-valued, and exhibits immense diversity in shapes, frequencies, and value ranges across different domains. This makes unifying pre-training and semantic alignment significantly harder. Furthermore, time series forecasting is inherently uncertain. Similar historical patterns can lead to multiple probable future outcomes. Traditional deep models, often optimized with simple priors like the Mean Squared Error (MSE), which assumes a Gaussian distribution for predictions, tend to produce deterministic, over-smooth forecasts that suffer from mode collapse. This means they fail to capture the full spectrum of possible future states, providing only a single mean estimate rather than a distribution of possibilities. While probabilistic forecasting methods using unimodal or mixture distributions (e.g., MOIRAI) exist, they still impose certain priors, limiting their flexibility. The dilemma then becomes how to achieve a flexible, prior-free optimization while enabling native tokenization for continuous time series, especially given that cross-entropy loss, common in LLMs for handling ambiguity, requires discrete categories.

Key Findings

▶ Watch: The dilemma: flexible priors and native tokenization (4:00)

The Sundial project introduces a groundbreaking approach to time series forecasting, centered on learning the "native and flexible language of time series" through prior-free generative modeling. The talk highlights several key findings and contributions:

1. Prior-Free Generative Modeling: Sundial is the first to introduce a generative modeling framework in time series foundation models that operates without prior-specific distributions. This allows the model to accumulate diverse distributions from pre-training data and generate raw predictions from which any desired statistics (e.g., mean, quantiles) can be derived during inference, effectively avoiding mode collapse.

1. Novel Architecture with Patch-wise Tokenization: Inspired by the Vision Transformer and the classical ARMA model, Sundial employs a decoder-only transformer with patch-wise tokenization. This native tokenization strategy drastically reduces context length compared to point-wise methods, leading to more efficient processing and mitigating error accumulation in long-term forecasting.

1. TimeFlow Loss based on Conditional Flow Matching: A core innovation is the TimeFlow loss function, built upon the flow matching process. This simple yet highly effective framework allows the model to transform a simple source distribution (e.g., Gaussian noise) into any complex target distribution. The integration of conditional flow matching uses learned representations from the transformer as conditions, ensuring generated predictions are coherent with the look-ahead time series.

1. TimeBench: A Large-Scale, Curated Dataset: To facilitate the training of a robust foundation model, the team curated TimeBench, a massive dataset comprising one trillion time points. This dataset, sourced from diverse domains including synthetic data and real-world systems like ERA5 (weather data), emphasizes diversity, scientific relevance, and predictability, drawing inspiration from data curation strategies in LLM development.

1. Significant Performance Gains: Sundial demonstrates significant error reduction compared to previous state-of-the-art models on well-acknowledged long-term forecasting benchmarks. It also shows strong performance on probabilistic forecasting benchmarks like Giff-Eval and surpasses statistical methods like Auto-ARIMA on in-distribution data.

1. Exceptional Inference Speed and Scalability: The patch-wise tokenization and efficient architecture enable a remarkable 35 times speed-up in inference compared to models like Chronos (which uses point-wise tokenization). This efficiency allows for deployment on CPU devices and the generation of multiple prediction samples within seconds. Furthermore, Sundial exhibits excellent scaling behaviors, benefiting from more parameters and larger pre-training scales.

1. Test-Time Calibration and Accessibility: The generative forecasting paradigm allows for test-time calibration, where increasing the number of samples or fine-tuning flow matching steps can enhance prediction precision. The immediate release of out-of-the-box Sundial models on Hugging Face, supporting CPU inference for point and quantile forecasting, underscores its practical readiness and accessibility.

Technical Deep Dive

▶ Watch: Sundial's core motivation: prior-free generative modeling (4:50)

The technical foundation of Sundial rests on addressing the unique challenges of time series as a "foreign language" to traditional deep learning paradigms, especially those inspired by large language models. The core innovations lie in its tokenization strategy, generative modeling framework, and a novel loss function.

1. Native Tokenization and Architecture Inspiration:

The speaker highlights the limitations of direct LLM-style embedding for time series:

• Point-wise tokenization (like in LLMs) leads to excessively long context inputs and multi-step auto-regression, which can cause significant error accumulation in long-term forecasting.
• Patch embedding, inspired by Vision Transformers (ViT), is adopted. This approach processes segments or "patches" of the time series rather than individual data points. This provides a more native tokenization for continuous-valued data, resulting in lossless embedding and substantially reduced context length.

The overall architecture of Sundial is ingeniously inspired by the classical ARMA model, which is considered theoretically complete:

• The auto-regression (AR) part, responsible for capturing point-wise temporal dependencies, is realized by a decoder-only transformer. This transformer processes the patch-wise tokens, leveraging its self-attention mechanisms to learn long-range dependencies within the time series history.
• The moving average (MA) part, traditionally modeling the error term and providing insights into uncertainty and distributions, is translated into Sundial's generative modules. These modules are designed to sample from a simple noise distribution (e.g., Gaussian) and gradually transform it into the final complex prediction distribution, thereby intrinsically modeling the uncertainty.

2. Prior-Free Generative Modeling with TimeFlow Loss:

A central contribution of Sundial is its prior-free optimization strategy. Unlike probabilistic forecasting methods that assume specific unimodal or mixture distributions (e.g., Gaussian, Normalizing Flows, MOIRAI), Sundial aims to learn the complex, multi-modal, and often ambiguous underlying distribution of time series data directly, without imposing any predefined parametric density.

This is achieved through a novel loss function called TimeFlow loss, which is built upon the flow matching process. Flow matching is a generative modeling technique that provides a simple yet effective framework for converting a simple source distribution (e.g., a standard Gaussian noise) into any complicated target distribution. The core idea is to learn a "velocity field" that smoothly transports samples from the source distribution to the target distribution along a continuous path.

To make this unconditional flow matching process applicable to conditional time series forecasting, Sundial proposes a conditional flow matching module. Here, the learned representations from the transformer (which encapsulate the historical context and patterns of the look-ahead time series) serve as the condition for the flow matching process. This ensures that the generated predictions are not random but are highly coherent and contextually relevant to the input time series. The process involves:

• Sampling noise from a source Gaussian distribution.
• Using the learned transformer representation as a condition.
• Applying the conditional flow matching module to predict the velocity field, pushing the noise samples towards the target prediction distribution.
• Through this iterative process, the model generates diverse prediction samples that reveal the inherent patterns and potential future trajectories of the time series, effectively capturing its true probabilistic nature.

3. Data Curation: TimeBench:

To train a foundation model capable of generalizing across diverse time series, a massive and high-quality dataset is crucial. The TimeBench dataset was curated with one trillion time points from various domains. The design principles for this dataset—diverse, scientific, and predictable—are inspired by successful data curation strategies in large language models. For instance, the inclusion of a significant amount of synthetic data helps the model learn common sense and general patterns, similar to how scientific data enhances LLMs. The dataset also includes real-world data like ERA5, a weather system time series, contributing to its scientific breadth. A pipelined data curation process, involving statistical analysis and quality improvement, ensures a unified training format, allowing high-dimensional time series to be converted into a single series sequence for consistent model input.

Experimental Setup & Results

▶ Watch: TimeFlow loss and conditional flow matching explained (6:10)

The evaluation of Sundial focused on demonstrating its superior performance in both deterministic and probabilistic forecasting, as well as its efficiency and scalability.

1. Benchmarks and Metrics:

• Long-Term Forecasting Benchmarks: Sundial was evaluated on well-acknowledged long-term forecasting benchmarks, which typically measure point forecasting accuracy using metrics like Mean Squared Error (MSE), Mean Absolute Error (MAE), etc.
• Probabilistic Forecasting Benchmark (Giff-Eval): The model was tested on Giff-Eval, a benchmark maintained by Salesforce, specifically designed for probabilistic forecasting. A notable aspect here is that Sundial provides raw generated predictions without assuming specific priors. These raw predictions can then be used to derive desired statistics, contrasting with methods that directly predict quantiles or distribution parameters (e.g., CRPS – Continuous Ranked Probability Score). The speaker emphasizes that Sundial's approach allows for flexibility in how probabilistic metrics are calculated post-generation.
• Forecasting Evaluation Leaderboard (AutoGluon): Sundial was also evaluated on the AutoGluon forecasting leaderboard, which often includes comparisons against robust statistical and machine learning baselines.

2. Headline Results and Comparisons:

• Error Reduction: Sundial achieved a significant error reduction compared to previous state-of-the-art models on long-term forecasting benchmarks. While specific percentage improvements were not detailed in the transcript, the emphasis on "significant" suggests substantial gains.
• Probabilistic Forecasting: The model demonstrated strong capabilities in probabilistic forecasting on Giff-Eval, showcasing its ability to generate diverse and statistically sound predictions without relying on predefined priors.
• Outperforming Baselines: Sundial was able to surpass traditional statistical methods like Auto-ARIMA, even when Auto-ARIMA was fitted and tuned on in-distribution data, highlighting Sundial's superior generalization and learning capacity.

3. Efficiency and Scalability:

• Inference Speed: One of the most striking results is the inference speed. Due to its native patch-wise tokenization, Sundial achieved a 35 times speed-up in inference compared to the Chronos model, which utilizes point-wise tokenization. This efficiency is critical, enabling CPU deployment and the generation of multiple prediction samples within seconds. For context, fitting and tuning an Auto-ARIMA model can take hours or even days, whereas Sundial offers near-instantaneous predictions.
• Scaling Behavior: Ablation studies and experiments confirmed that Sundial exhibits good scaling behaviors. The model's architecture design benefits from more parameters and larger pre-training scales, indicating that its performance can further improve with increased computational resources and data.

4. Test-Time Calibration:

The generative nature of Sundial allows for test-time calibration. Practitioners can increase the precision of predictions by:

• Generating more samples during inference, which helps to better approximate the underlying distribution.
• Using fine-grained flow matching steps, which refines the generative process and leads to improved performance.

5. Accessibility:

The project emphasizes accessibility by releasing out-of-the-box models on Hugging Face. These models are designed for CPU inference and currently support point forecasting, quantile forecasting, and the estimation of prediction intervals, making them immediately usable by practitioners without extensive training or specialized hardware.

Practical Implications

▶ Watch: Ablation studies confirm flow matching's efficiency and effectiveness (7:00)

The Sundial family of time series foundation models carries profound practical implications for a wide range of stakeholders, from individual practitioners to large infrastructure teams and AI researchers.

1. For Practitioners and Model Builders:

• General Toolbox with Customization Potential: Sundial offers a powerful, general-purpose toolbox for time series forecasting. Its pre-trained nature allows for immediate zero-shot forecasting capabilities on diverse datasets, significantly reducing the effort and expertise traditionally required for model selection and training.
• Enhanced Probabilistic Forecasting: The prior-free generative modeling is a game-changer for applications requiring a nuanced understanding of uncertainty. Instead of deterministic, over-smoothed predictions, practitioners gain access to a full distribution of probable outcomes, enabling more robust risk assessment and decision-making (e.g., in financial modeling, demand planning, or resource allocation).
• Domain-Specific Adaptation: While powerful, the speaker acknowledges that for highly specific tasks, fine-tuning or the integration of physical-driven knowledge or domain-specific objectives might be necessary. This suggests a hybrid approach where Sundial provides a strong foundation, which is then specialized for particular use cases, much like LLMs are fine-tuned for specific NLP tasks.
• Reinforcement Learning Integration: The inherent generative nature of Sundial opens avenues for integration with reinforcement learning (RL). Predictions can be rewarded based on desired outcomes, allowing the model to adapt and optimize its forecasting strategy in dynamic environments.

2. For Infrastructure and Deployment Teams:

• Resource Efficiency and Accessibility: The impressive 35x inference speed-up compared to models like Chronos, combined with the capability for CPU deployment, drastically lowers the computational barrier for deploying advanced time series models. This means forecasting solutions can be run on commodity hardware, edge devices, or within existing CPU-based cloud infrastructure, making them more cost-effective and scalable.
• Reduced Operational Overhead: The pre-trained, out-of-the-box models available on Hugging Face simplify deployment. Infrastructure teams can integrate these models without needing to manage complex GPU clusters or extensive training pipelines, accelerating time-to-production for new forecasting applications.
• Scalability for Large-Scale Data: The demonstrated scaling behaviors with more parameters and larger pre-training scales indicate that Sundial can effectively handle the ever-growing volumes of time series data generated in modern systems, providing a future-proof solution for enterprise-level forecasting needs.

3. Tradeoffs and Limitations:

• Data Expansion Challenges: From a model developer's perspective, a current limitation is the need to effectively expand and curate diverse data, especially synthetic time series, to capture the complex semantics inherent in this modality. This remains an active area of research for improving foundation models.
• Architectural Design: The search for "natural architectures" for time series, akin to Transformers for language, is ongoing. While Sundial makes significant strides, further exploration, potentially drawing inspiration from works like TablePFN that transform time series into tabular formulations, could yield even more effective designs.
• Generalization vs. Specialization: While Sundial is a general toolbox, highly specialized applications may still require domain experts to guide fine-tuning or inject specific domain knowledge to achieve peak performance. The balance between generalizability and task-specific accuracy is a perennial challenge in foundation models.

In essence, Sundial democratizes advanced time series forecasting by making powerful, generalizable, and efficient models accessible. It shifts the paradigm from building custom models for every time series task to leveraging a pre-trained foundation that can be adapted, reducing development cycles, improving accuracy, and providing richer insights into future uncertainties.

Key Takeaways

• Prior-Free Generative Forecasting: Sundial introduces the first prior-free generative modeling framework for time series foundation models, using TimeFlow loss based on flow matching to capture complex, diverse distributions without restrictive assumptions.
• Efficient Patch-wise Tokenization: The model employs patch-wise tokenization (inspired by Vision Transformers) and a decoder-only transformer architecture, significantly reducing context length and error accumulation compared to point-wise methods.
• ARMA-Inspired Architecture: Sundial's design draws inspiration from the classical ARMA model, with the transformer handling auto-regression and generative modules addressing the moving average component and uncertainty modeling.
• Massive TimeBench Dataset: Pre-trained on TimeBench, a curated dataset of one trillion time points from diverse, scientific, and predictable domains, enabling strong generalization capabilities.
• Superior Performance and Efficiency: Achieves significant error reduction on benchmarks, provides robust probabilistic forecasts, and boasts a 35x inference speed-up over models like Chronos, allowing for efficient CPU deployment.
• Accessible and Scalable: Demonstrates excellent scaling behaviors and is released as out-of-the-box models on Hugging Face, offering immediate utility for point and quantile forecasting with test-time calibration options.

About the Speaker(s)

Yong Liu is a researcher from Tsinghua University. He presented the work on Sundial, highlighting the contributions from a team including Guo Qin, Zhiyuan Shi, Zhi Chen, Caiyin Yang, Xiangdong Huang, Jianmin Wang, and Mingsheng Long. The presentation reflects ongoing research from their group focused on advancing time series foundation models and addressing fundamental challenges in this domain.