Has AI Finally Cracked Time Series Forecasting?

Ameet Talwalkar, professor at Carnegie Mellon and Chief Scientist at Datadog, discusses the evolution and impact of time series foundation models. Initially, there was significant skepticism regarding the necessity and efficacy of specialized foundation models for modalities like time series, genomics, or geospatial data. Drawing an analogy to the BERT moment in NLP (around 2018), where pre-training on massive unlabeled data dramatically outperformed specialized supervised models, the question was whether similar breakthroughs had occurred for other data types.

Early Skepticism and the Need for Stronger Baselines

Talwalkar, with his academic hat on, observed that many early specialized foundation models lacked strong empirical evidence of their superiority. A key issue was the use of weak baselines and inadequate benchmarks. In a paper published “last year” (referring to 2023 from the podcast’s perspective), his team demonstrated that for time series, genomics, and geospatial modeling, classical statistical or ML approaches (which were much simpler and smaller) often performed comparably to, or even significantly better than, the then-current specialized foundation models. This highlighted a critical gap: the benchmarks were too simplistic (analogous to evaluating modern CNNs on MNIST) to truly differentiate between good and bad models.

Maturation of the Field and Better Evaluation

By late 2023, the field matured rapidly. New, more rigorous benchmarks like Gifty Val (from Salesforce) emerged. Concurrently, several robust time series foundation models were released, including Amazon’s Kronos, Google’s TimesFM, Salesforce’s Moirai, and Datadog’s Toto. These newer models, trained on larger datasets and employing more sophisticated architectures, began to show significant zero-shot predictive accuracy improvements over traditional supervised baselines. The capability of these models to zero-shot (make predictions without any post-training or fine-tuning) and still outperform highly optimized classical methods was a significant breakthrough, effectively marking a “BERT moment” for time series. This implies that these models can even outperform a skilled human analyst who spends hours on a specific time series, especially given the sheer volume of time series data in enterprise settings like Datadog.

Practicality: Inference Costs and Hyperparameter Tuning

A surprising practical benefit of these foundation models, even with their larger parameter counts (e.g., Toto v1 at 100 million parameters), lies in their inference cost. Traditional ARMA-style models, while small, often require extensive on-the-fly hyperparameter tuning to achieve high-quality forecasts, making their total inference cost (tuning + prediction) comparable to, or even higher than, a zero-shot foundation model. The foundation models, being robust and pre-tuned, offer immediate utility and simplify deployment, making them highly practical for scenarios with millions of time series, such as observability data.

Datadog’s Toto: From V1 to V2

Datadog’s focus is on observability data, which refers to the telemetry collected from production software systems (e.g., CPU usage, latency, memory across distributed systems). This generates a “comical amount” of time series data (metrics). The Toto models are specifically optimized for this domain.

• Toto V1: Demonstrated the “BERT moment” for time series forecasting, showing that zero-shot performance could crush supervised baselines. It was open-weighted and saw significant downloads, indicating interest from various industries, including secretive financial services.
• Toto V2: Represented a “GPT-2 moment” by focusing on scaling. The V2 models increased significantly in size (up to 2.5 billion parameters) and demonstrated that “bigger models are significantly better.” This scaling showed no saturation at current sizes, suggesting further potential. Another critical finding for V2 was the importance of data mix. Counter-intuitively, excluding publicly available general-purpose time series data and focusing solely on observability data led to better results, even on general-purpose benchmarks. This suggests that data curation is a crucial, nascent area.

Fine-tuning and Multimodality

While zero-shot performance is strong, fine-tuning Toto on application-specific, in-distribution data can further enhance performance for specialized enterprise forecasting problems. Examples include handling unique seasonal effects not sufficiently represented in the general training data. Looking ahead, incorporating auxiliary variables (like seasonality metadata) and textual context alongside time series data is becoming increasingly important. For instance, combining text (like incident reports or notes) with time series allows for richer queries and models that can learn from both modalities. Datadog released an open-source benchmark for incident reporting, showing that specialized multimodal models can outperform frontier LLMs at integrating time series and text for observability tasks.

World Models for Observability

The ultimate goal for Datadog is to build world models for observability. Just as a world model simulates a 3D environment, an observability world model aims to simulate the behavior of a distributed software system. This involves integrating diverse data modalities: metrics (time series), logs, traces, network topology, source code, events, and alerts. The ambitious goal is to:

• Predict future states: Simulate how the system will behave.
• Enable proactive detection: Identify potential issues 5-10 minutes before they occur, leading to preemptive intervention and minimizing downtime.
• Facilitate “what if” analyses: Understand the impact of scaling services or shifting traffic.
• Generate realistic simulations: Provide environments for agentic training and reinforcement learning. This is an early, highly ambitious area of research, with benchmarks still under development to capture different aspects of system behavior and prediction.

Open Weights Models and the Future of Computer Science

The discussion shifts to the broader landscape of AI. Open weights models, especially from China, are rapidly catching up to proprietary frontier models, often lagging by only 3-12 months. While concerns exist about IP exposure and control with proprietary APIs, open weights models offer enterprises greater control over their destiny and often present a more cost-effective solution for 80% of tasks, leaving specialized tasks for proprietary services. This trend encourages companies with significant data to fine-tune open models.

Finally, the conversation touches upon the future of Computer Science and academia. While there’s concern about declining CS enrollments and the changing nature of coding due to AI, Talwalkar suggests that the role of academia will shift. The teaching aspect becomes even more critical for educating a workforce that needs to adapt to rapidly evolving AI tools. AI’s ability to act as a research partner, accelerating scientific discovery (e.g., in mathematics), will change the definition of expertise. Future Fields medalists might not just be those with raw IQ, but those who excel at leveraging these powerful AI tools to connect dots across disciplines and solve complex problems faster.

• Tags:
• Fine-Tuning
• Multimodal AI
• Observability
• Machine Learning
• World Models
• Open Weights Models
• Time Series Foundation Models
• Datadog
• Toto
• Zero-Shot Learning
• AI in Academia
• Forecasting