[arXiv 2024] Impermanent: Testing Time-Series Foundation Models Against the Arrow of Time

TL;DR

Impermanent is a paradigm-shifting benchmark that moves away from static, easily-contaminated datasets to a live, rolling-evaluation framework. Built on the volatile activity of 400 GitHub repositories, it tests whether "Foundation Models" truly generalize or simply memorize. While models like TimesFM currently lead, the benchmark proves that maintaining performance over a non-stationary stream is far harder than winning a static competition.

Background Positioning

In the current AI landscape, Time-Series Foundation Models (TSFMs) like Chronos and Moirai claim "zero-shot" mastery over any temporal data. However, most benchmarks (e.g., Monash) are static snapshots. Impermanent is to Time-Series what Chatbot Arena or LiveBench is to LLMs: a dynamic proving ground where models are judged by their ability to handle the "unknown future" as it happens.

Problem & Motivation: The Scent of Contamination

The "dirty secret" of foundation model research is data leakage. When a model is trained on "the whole internet," it has likely seen the test sets of standard benchmarks during its pre-training phase. This turns a forecasting task into a memorization task.

Moreover, real-world data is non-stationary. Trends flip, contributors leave, and external shocks (like a viral tweet) cause "bursty" behavior. Static benchmarks ignore these temporal dynamics, failing to answer the crucial question: Will this model still work six months after deployment?

Methodology: The Live Loop

The authors chose GitHub activity (Issues, PRs, Pushes, Stargazers) as their signal because it is inherently public, continuous, and highly irregular.

1. Spectral Characterization

To prove the difficulty of the task, the authors use Spectral Centroid (C) and Spectral Entropy (H).

• High C: Faster dynamics and sudden bursts.
• High H: Diffuse, broad-band behavior (less predictable).

The GitHub stream occupies a "wedge-shaped" cloud in this space, mixing smooth trends with spiky, volatile noise that requires models to be both stable and reactive.

2. The Prequential Protocol

Unlike traditional backtesting, the Impermanent evaluation loop is strictly forward-looking:

1. A "cutoff date" is set.
2. The model receives historical context and issues a forecast.
3. The "future" is only revealed after the forecast is locked.
4. Metrics (MASE and Scaled CRPS) are calculated only when the actual data arrives.

Figure 1: High-variance weekly GitHub activity across 25 repositories, illustrating the burstiness of the data.

Experiments & Results: Foundation Models vs. The Baselines

The benchmark evaluated 11 models, including classical statistical methods (AutoARIMA, Prophet) and modern TSFMs (Chronos, TimesFM, TiRex).

Key Findings:

• Foundation Dominance: TSFMs generally occupy the top 4 spots, with TimesFM showing superior probabilistic calibration (CRPS).
• The "Seasonal" Trap: A simple SeasonalNaive model achieved a better MASE rank than many sophisticated statistical models, highlighting that in high-frequency data, local history is often a stronger signal than complex global patterns.
• Unstable Stability: The rankings shift as new data arrives. A model that wins this week might fail next month if a repository undergoes a major structural break (e.g., a v2.0 release).

Table 1: Current leaderboard showing TSFMs leading, but traditional methods remaining competitive in specific metrics.

Critical Analysis & Conclusion

Impermanent exposes the fragility of the "Foundation Model" narrative in time-series. While TSFMs are powerful, their lead over optimized statistical methods is sometimes marginal when faced with real-world non-stationarity.

Limitations:

• Univariate Focus: Current evaluations are per-series. It does not yet account for cross-repository correlations (e.g., a "Rust ecosystem" trend affecting multiple repos).
• Short Evaluation Windows: The current snapshot is early; the true test will be how these models rank over 12-24 months of continuous data.

Future Outlook:

This work sets a new standard for TSFM evaluation. Future researchers should no longer be satisfied with high scores on static datasets. If a model claims to be a "Foundation Model," it must prove itself on the Impermanent Dashboard, facing data that hasn't even been generated yet.