LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

# An Open-world Living Benchmark for Time Series Foundation Models
Source: [https://arxiv.org/html/2608.17299](https://arxiv.org/html/2608.17299)
## LiveHouse\-TS: An Open\-world Living Benchmark for Time Series Foundation ModelsDOI:[XXXXXXX\.XXXXXXX](https://doi.org/XXXXXXX.XXXXXXX)Conference:Make sure to enter the correct conference title from your rights confirmation email; June 03–05, 2018; Woodstock, NYISBN:978\-1\-4503\-XXXX\-X/2018/06

Haomin Wen2,†, Ziyu Zhou1,†, Qingxiang Liu1,†, Siru Zhong1,†, Yuxuan Liang1,\*Affiliation:1The Hong Kong University of Science and Technology \(Guangzhou\) 2Shanghai Innovation Institute; †\{\\dagger\}Equal contribution;\*Corresponding author email:[wenhaomin\.whm,zziyuzhou,qingxiangliu737@gmail\.com,](mailto:wenhaomin.whm,zziyuzhou,[email protected],)email:[yuxliang@outlook\.com, szhong691@connect\.hkust\-gz\.edu\.cn](mailto:[email protected],%[email protected])

2018© , 2018;

###### Abstract\.

Time Series Foundation Models \(TSFMs\) have recently emerged as a highly promising paradigm for cross\-domain zero\-shot forecasting\. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows\. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real\-world environments characterized by seasonal variations, distribution shifts, and unexpected events\. To bridge this gap, we introduce LiveHouse\-TS, the first open\-world living benchmark infrastructure for TSFMs\. By evaluating models prequentially on real future data in open\-world environments, LiveHouse\-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity\. Rather than acting as a one\-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long\-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol\. Code, dataset, and leaderboard are available at:[https://huggingface\.co/spaces/CityMindDev/LiveHouse\-TS](https://huggingface.co/spaces/CityMindDev/LiveHouse-TS)\.

## 1\.Introduction

Time series forecasting is a foundational task across a wide spectrum of industrial and scientific domains, ranging from energy management and financial planning to climate modeling\. Time series foundation models \(TSFMs\) have shown to be highly promising paradigm for zero\-shot forecasting across domains, driven by large\-scale pretraining and the ability to perform zero\-shot inference\([20](https://arxiv.org/html/2608.17299#bib.bib12);[15](https://arxiv.org/html/2608.17299#bib.bib3);[3](https://arxiv.org/html/2608.17299#bib.bib4);[56](https://arxiv.org/html/2608.17299#bib.bib6);[24](https://arxiv.org/html/2608.17299#bib.bib9)\)\. This paradigm shift has sparked massive research interest, yielding hundreds of papers in the last two years\.

Concurrently, the rapid evolution of these models drives the demand for reliable evaluation protocols\. As shown in Figure[1](https://arxiv.org/html/2608.17299#S1.F1), current practice relies almost entirely on static benchmarks \(e\.g\., GIFT\-eval\([1](https://arxiv.org/html/2608.17299#bib.bib19)\), fev\-bench\([51](https://arxiv.org/html/2608.17299#bib.bib32)\), TSFM\-Bench\([30](https://arxiv.org/html/2608.17299#bib.bib39)\)\), where public datasets are split into predetermined, frozen train and test windows\. While static leaderboards offer a controlled environment for initial verification, they introduce a fundamental limitation: snapshot evaluation ignores the operational realities of real\-world deployment, where non\-stationarity, concept drift, and sudden exogenous shifts continuously alter the underlying data\-generating processes\. In a static benchmark, a model that captures the top spot remains there indefinitely because standard datasets yield a permanent, immutable rank once computed\. In reality, two forecasting models might achieve the exact same performance metric \(e\.g\., a Mean Absolute Error of 0\.42\) on a static split, rendering them indistinguishable offline\. Yet, when subjected to a rolling, real\-world timeline, one model might swiftly degrade under a seasonal shift while the other maintains consistent reliability\.A model’s operational superiority is not a permanent attribute; its performance and ranking must be continuously tested and earned as the world changes\.

![Refer to caption](https://arxiv.org/html/2608.17299v1/intro.png)Figure 1\.Comparison between proposed LiveHouse\-TS and static benchmark\. LiveHouse\-TS introduce a streaming evaluation protocol to capture model robustness under real\-world operational conditions\. It eliminates data leakage by requiring models to forecast at the current world time before ground truth is observed, with evaluation and the leaderboard updated continuously as new data arrives\.Table 1\.Comparison between representative TSFM benchmarks and LiveHouse\-TS\. \(Abbr; LR: Leakage\-resistant by nature; RTL: Real\-time Leaderboard; OND: Open\-to\-new\-data; FT: Forecasting Task, Prob\.: Probability Forecasting\)\. LiveHouse\-TS provides the first systematic solution for benchmarking the live performance of time series foundation models\.BenchmarkYearDatasetEvaluationLeaderboardLive/Static\#Domain\#DataLRTest zero\-shot?FTMultivariateRTLONDMetricsMonash\([22](https://arxiv.org/html/2608.17299#bib.bib20)\)2021Static720✘✘Point✘✘✘MASE, sMAPE, msMAPE, MAE, RMSEBasicTS\([50](https://arxiv.org/html/2608.17299#bib.bib30)\)2023Static520✘✘point✔✘✘MAE, RMSE, MAPE, WAPETFB\([45](https://arxiv.org/html/2608.17299#bib.bib23)\)2024Static1041✘✘Point✔✘✔MAE, MSE, MASE, MSMAPEProbTS\([60](https://arxiv.org/html/2608.17299#bib.bib24)\)2024Static612✘✔Point/Prob\.✔✘✘NMAE, CRPSCiK\([55](https://arxiv.org/html/2608.17299#bib.bib31)\)2025Static79✘✔Prob\.✘✘✘CRPSGIFT\-Eval\([1](https://arxiv.org/html/2608.17299#bib.bib19)\)2024Static723✘✔Point/Prob\.✔✘✘MAPE, CRPSfev\-bench\([51](https://arxiv.org/html/2608.17299#bib.bib32)\)2025Static796✘✔Point/Prob\.✔✘✔MASE, SQLBOOM\([11](https://arxiv.org/html/2608.17299#bib.bib33)\)2025Static5\-✘✔Point/Prob\.✔✘✘MASE, CRPSTSFM\-Bench\([30](https://arxiv.org/html/2608.17299#bib.bib39)\)2025Static1021✘✔Point✔✘✘MAE, MSEImpermanent\([21](https://arxiv.org/html/2608.17299#bib.bib26)\)2026Live11✔✔Point/Prob\.✔✘✘MASE, CRPSTS\-Arena\([41](https://arxiv.org/html/2608.17299#bib.bib8)\)2026Live13✔✔Point✔✔✔MASELiveHouse\-TS2026Live1117✔✔Point/Prob\.✔✔✔RMSE,MAPE, CRPS,Stability, Improvement

To address these limitations, we introduce LiveHouse\-TS, the first open\-world living benchmark infrastructure for TSFMs\. We define our open\-world setting as a temporally open system featuring continuous streaming observations and an extensible registry for dynamically expanding data sources\. As illustrated in Figure[1](https://arxiv.org/html/2608.17299#S1.F1), LiveHouse\-TS enforces a strict prequential evaluation protocol: predictions must be made at the current world time before the corresponding ground\-truth values exist, with metrics updated continuously as new observations arrive\. Ultimately, LiveHouse\-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity\. We list the detailed comparsion of LiveHouse\-TS and current benchmarks in Table[1](https://arxiv.org/html/2608.17299#S1.T1)\. Crucially, rather than serving as a one\-off leaderboard, LiveHouse\-TS is conceptualized as a living time series infrastructure designed to spark and systematically answer new scientific questions vital to the community\. For example, can model rankings be maintained long\-term? Which models are genuinely robust under real\-world deployment? In summary, our core contributions are:

- •New Paradigm: We identify a critical evaluation gap in the snapshot paradigm and introduce a streaming evaluation protocol centered on continuous temporal validity to capture model robustness under real\-world operational conditions\.
- •New Evaluation Infrastructure: We propose LiveHouse\-TS, a leakage\-resistant open\-world live benchmark infrastructure featuring a real\-time leaderboard, an extensible registry for streams and models, and new metrics specifically tailored for temporal stability and monotone performance improvement\.
- •New Insights: Current TSFMs generalize well for zero\-shot forecasting on real future data\. However, the rankings on LiveHouse\-TS differ from those on prior static benchmarks, suggesting strong performance on static benchmarks may not necessarily translate to practical deployment\.

## 2\.Related Work

Time Series Foundation Models\.TSFMs are pretrained on a large cross\-domain time series corpus and then applied zero\-shot or with light fine\-tuning to unseen datasets\([53](https://arxiv.org/html/2608.17299#bib.bib1);[20](https://arxiv.org/html/2608.17299#bib.bib12);[15](https://arxiv.org/html/2608.17299#bib.bib3);[3](https://arxiv.org/html/2608.17299#bib.bib4);[56](https://arxiv.org/html/2608.17299#bib.bib6);[24](https://arxiv.org/html/2608.17299#bib.bib9);[18](https://arxiv.org/html/2608.17299#bib.bib2);[7](https://arxiv.org/html/2608.17299#bib.bib27);[9](https://arxiv.org/html/2608.17299#bib.bib28)\)\. They vary in tokenization, architecture, and pre\-training objectives, including DeepAR, N\-BEATS, N\-HiTS, PatchTST, DLinear, and TimesNet\([48](https://arxiv.org/html/2608.17299#bib.bib53);[43](https://arxiv.org/html/2608.17299#bib.bib54);[10](https://arxiv.org/html/2608.17299#bib.bib55);[42](https://arxiv.org/html/2608.17299#bib.bib56);[59](https://arxiv.org/html/2608.17299#bib.bib57);[57](https://arxiv.org/html/2608.17299#bib.bib58)\); later models include Informer, Autoformer, FEDformer, Pyraformer, Crossformer, SCINet, and TiDE\([63](https://arxiv.org/html/2608.17299#bib.bib59);[58](https://arxiv.org/html/2608.17299#bib.bib60);[64](https://arxiv.org/html/2608.17299#bib.bib61);[35](https://arxiv.org/html/2608.17299#bib.bib62);[62](https://arxiv.org/html/2608.17299#bib.bib63);[34](https://arxiv.org/html/2608.17299#bib.bib64);[14](https://arxiv.org/html/2608.17299#bib.bib65)\)\. We refer the reader to[32](https://arxiv.org/html/2608.17299#bib.bib16)for a broader survey\. To name a few examples, TimesFM\([15](https://arxiv.org/html/2608.17299#bib.bib3)\)and Timer\([38](https://arxiv.org/html/2608.17299#bib.bib14)\)follow a decoder\-only design that models time series as patches, whereas Chronos\([3](https://arxiv.org/html/2608.17299#bib.bib4);[2](https://arxiv.org/html/2608.17299#bib.bib5)\)discretizes \(via scaling and quantization\) continuous values into a token vocabulary to reuse language\-model backbones\. In contrast, encoder\-style masked pretraining is adopted by MOIRAI\([56](https://arxiv.org/html/2608.17299#bib.bib6)\)\(masked any\-variate modeling\) and MOMENT\([24](https://arxiv.org/html/2608.17299#bib.bib9)\)\(masked multi\-task pretraining\), while Lag\-Llama\([46](https://arxiv.org/html/2608.17299#bib.bib11)\)produces probabilistic forecasts from lag\-based features\. Beyond modeling choices, Toto\([13](https://arxiv.org/html/2608.17299#bib.bib10)\)and TTM\([16](https://arxiv.org/html/2608.17299#bib.bib15)\)emphasize observability and lightweight deployment, and Time\-MoE\([52](https://arxiv.org/html/2608.17299#bib.bib13)\)and Moirai\-MoE\([36](https://arxiv.org/html/2608.17299#bib.bib7)\)scale up via sparse mixture\-of\-experts routing\. In parallel, another line of work reprograms or fine\-tunes frozen language models for forecasting\([29](https://arxiv.org/html/2608.17299#bib.bib17);[65](https://arxiv.org/html/2608.17299#bib.bib18);[8](https://arxiv.org/html/2608.17299#bib.bib29)\)\.

Time Series Forecasting Benchmark\.Early competitions and archives fixed the unit of comparison, as M4 and M5\([39](https://arxiv.org/html/2608.17299#bib.bib21);[40](https://arxiv.org/html/2608.17299#bib.bib22)\)standardized point and probabilistic scoring over large series collections and the Monash archive\([22](https://arxiv.org/html/2608.17299#bib.bib20)\)consolidated datasets into one format that later became a common pretraining source\. A second wave then targeted fair and reproducible comparison, where TFB\([45](https://arxiv.org/html/2608.17299#bib.bib23)\)and BasicTS\([50](https://arxiv.org/html/2608.17299#bib.bib30)\)control preprocessing and dataset heterogeneity while ProbTS\([60](https://arxiv.org/html/2608.17299#bib.bib24)\)and CiK\([55](https://arxiv.org/html/2608.17299#bib.bib31)\)broaden the evaluation axis to distributional and context\-aware forecasting\. Most recently, GIFT\-Eval\([1](https://arxiv.org/html/2608.17299#bib.bib19);[31](https://arxiv.org/html/2608.17299#bib.bib25)\), TSFM\-Bench\([30](https://arxiv.org/html/2608.17299#bib.bib39)\), fev\-bench\([51](https://arxiv.org/html/2608.17299#bib.bib32)\), and BOOM\([11](https://arxiv.org/html/2608.17299#bib.bib33)\)target foundation models directly to test their zero\-shot ability\. Across all three waves, a curated set of public series is frozen with predetermined train/test splits, and models are scored once over the held\-out windows\. Such a design introduces the potential data leakage and may not reflect model performance in real deployment \(as discussed in Sec[1](https://arxiv.org/html/2608.17299#S1)\), which motivates our live and open\-world benchmark\. Overall, we refer to Table[1](https://arxiv.org/html/2608.17299#S1.T1)for a detailed comparison with LiveHouse\-TS and related ones\.

Live Benchmark\.A growing number of works in language and code evaluation address data contamination by making the benchmarks themselves*live*, continuously refreshing test data or gating them by release date so that every scored example is released*after*the model training\. LiveBench\([54](https://arxiv.org/html/2608.17299#bib.bib34)\), LiveCodeBench\([28](https://arxiv.org/html/2608.17299#bib.bib35)\), and the multimodal LiveXiv\([49](https://arxiv.org/html/2608.17299#bib.bib38)\)follow this principle and are supported by literature that analyzes why static benchmarks fail once their data leaks into pretraining\([47](https://arxiv.org/html/2608.17299#bib.bib36);[23](https://arxiv.org/html/2608.17299#bib.bib37)\)\. The methodological basis comes from the stream\-learning literature, where prequential evaluation enforces that predictions are always made before observing the corresponding labels, thereby preventing look\-ahead bias\([19](https://arxiv.org/html/2608.17299#bib.bib40)\)and enabling adaptive performance tracking under concept drift\([5](https://arxiv.org/html/2608.17299#bib.bib41)\)\. This live paradigm is well\-suited for benchmarking general time series forecasting, since real\-world series are generated continuously and provide a natural supply of strictly post\-cutoff evaluation data\.

![Refer to caption](https://arxiv.org/html/2608.17299v1/img/overall_architecture.png)Figure 2\.Overall architecture of LiveHouse\-TS, as an analogy in which models are performers and the benchmark is a*live house*\. It contains three decoupled yet coordinated components:OPENi\)i\)a*Model Entrance*that “checks tickets” and standardizes heterogeneous forecasters before they take the stage;OPENi​i\)ii\)*Live Data*that turns public streams into a rolling set of forecasting tasks according to the dataset description \(e\.g\., frequency, context length, and forecast horizon\); andOPENi​i​i\)iii\)an*Evaluation House*\(the live house\) that enforces the future\-only rule, and updates the leaderboard\.
## 3\.Benchmark Details

### 3\.1\.Task Formulation

We consider a time series\{𝐲t\}t≥1\\\{\\mathbf\{y\}\_\{t\}\\\}\_\{t\\geq 1\}observed at regular time steps, where𝐲t∈ℝD\\mathbf\{y\}\_\{t\}\\in\\mathbb\{R\}^\{D\}collectsDDtarget variables\. At timett, the goal is to forecast the nextHHvectors given the most recentLLobservations \(with optional covariates/metadata\)\. In this setting, a point forecasting model outputs

\(1\)𝐲^t\+1:t\+H=f\(𝐲t−L\+1:t,𝐱t−L\+1:t,𝐦\),\\hat\{\\mathbf\{y\}\}\_\{t\+1:t\+H\}=f\\big\(\\mathbf\{y\}\_\{t\-L\+1:t\},\\,\\mathbf\{x\}\_\{t\-L\+1:t\},\\,\\mathbf\{m\}\\big\),where𝐱\\mathbf\{x\}denotes covariates \(e\.g\., calendar features\) and𝐦\\mathbf\{m\}denotes metadata such as frequency or horizon\. Probabilistic forecasting instead targets a full predictive distribution over future trajectories,

\(2\)P\(𝐲t\+1:t\+H∣𝐲t−L\+1:t,𝐱t−L\+1:t,𝐦\)\.P\\big\(\\mathbf\{y\}\_\{t\+1:t\+H\}\\mid\\mathbf\{y\}\_\{t\-L\+1:t\},\\,\\mathbf\{x\}\_\{t\-L\+1:t\},\\,\\mathbf\{m\}\\big\)\.*Zero\-shot*time series forecasting refers to applying a pretrained model to a novel dataset or unseen series without fine\-tuning, using only the provided context window at inference time\.

### 3\.2\.Overall Architecture

LiveHouse\-TS is guided by the following three design principles:

Remark: Design principles\. \(ii\)Leakge\-resistant:Tasks are constructed from continually arriving public streams, and designed to test model’s ability on the real future to prevent the potential data leakage\. \(i​iii\)Fairness:Ensuring fair comparisons across methods over time, since methods that join at different times may be evaluated over different time spans\. \(i​i​iiii\)Easy\-to\-scale:It should be easy for researchers and practitioners to join the leaderboard or contribute a new data source\. Since we hope LiveHouse\-TS serves as an infrastructure to evaluate the model’s generalizability in the open\-world environment\.

As in Figure[2](https://arxiv.org/html/2608.17299#S2.F2), we realize these principles with three decoupled yet coordinated components—as an analogy in which models are performers and the benchmark is a*live house*:OPENi\)i\)a*Model Entrance*that “checks tickets” and standardizes heterogeneous forecasters before they take the stage;OPENi​i\)ii\)*Live Data*that turns public streams into a rolling set of forecasting tasks according to a dataset description \(e\.g\., frequency, context length, and horizon\); andOPENi​i​i\)iii\)an*Evaluation House*\(the live house\) that enforces the future\-only rule, and updates the leaderboard\. Details are provided inAppx\.[B](https://arxiv.org/html/2608.17299#A2)\.

Model Entrance,which exposes a unified forecasting interface for forecasters, including hosted TSFMs and lightweight statistical baselines\. Given a context window and dataset metadata \(e\.g\., sampling frequency and horizon\), each predictor is required to return forecasts aligned with the requested prediction horizon\. The entrance adapter then validates the output shape and converts heterogeneous model outputs into a common scoring representation: a mean forecast for point\-error metrics such as MSE/RMSE, a median forecast for quantile\-based point metrics such as MAPE\. And a fixed set of quantile forecasts at predefined levels for probabilistic metrics such as CRPS when available\. If a method only provides point forecasts, we treat the point prediction as a degenerate predictive distribution for the evaluator\. This canonical representation ensures that all methods, regardless of whether they are local TSFMs, user\-submitted models, or statistical baselines, are scored by the same metric implementation under the same horizon and target alignment\. Details inAppx\.[B\.1](https://arxiv.org/html/2608.17299#A2.SS1)\.

Live Data,where collectors periodically retrieve fresh observations from multiple domains \(see Sec[3\.3](https://arxiv.org/html/2608.17299#S3.SS3)for more details\)\. Each stream is cleaned and mapped into a shared schema before being windowed into tasks\. Rather than imposing a single global setting, task construction follows the per\-dataset specification \(context length, prediction horizon, and frequency\), ensuring that all models evaluated on a given dataset receive identical inputs and targets while respecting the natural time scale of each stream\.

Evaluation House\.For each newly created task, the evaluation engine retrieves the historical context available at issue time and packages it into a standardized forecasting instance\. A future\-only gate then compares the task timestamp with each model’s admission time, filtering out any tasks issued before the model entered the leaderboard\. The remaining eligible tasks are dispatched through the unified forecasting interface, and their forecasts are evaluated once the corresponding future targets become observable\. Details of the evaluation house are provided inAppx\.[B\.3](https://arxiv.org/html/2608.17299#A2.SS3)\.

### 3\.3\.Streaming Data

#### 3\.3\.1\.Data Curation

LiveHouse\-TS builds its evaluation stream from public, continuously updated time series rather than from a frozen archive\. Figure[3](https://arxiv.org/html/2608.17299#S3.F3)showcases representative examples from various domains\. The current registry contains17 benchmark datasetsacross15 public sources,11 domains, and8 native frequencies, as detailed in Table[2](https://arxiv.org/html/2608.17299#S3.T2)\. The registry contains both directly reported time series \(e\.g\., sensor readings, market prices and macro indicators\) and event\-derived time series, where timestamped events are aggregated into regular buckets, such as GDELT document volume and USGS earthquake counts\. Each dataset is described by a single registry entry containing its source identifier, entity granularity, native data frequency, recommended evaluation frequency, history length, forecast horizon, target variable, and optional covariates\. Fourteen of the seventeen datasets include covariates for multivariate evaluation\. History and forecast windows are measured in native\-frequency steps\. High\-rate and daily operational streams provide short live contexts from minutes to weeks; while monthly and annual series preserve the longer seasonal and structural context \(Appx\.[C\.5](https://arxiv.org/html/2608.17299#A3.SS5)Table[5](https://arxiv.org/html/2608.17299#A3.T5)\)\. We treat the native data frequency as a property of the series\. This design choice ensures that changing how often we fetch data does not alter the forecasting problem itself—only the native frequency does\.

Figure 3\.Dataset examples with representative temporal patterns\. Each panel shows one selected dataset window at its native cadence; axes are normalized independently to highlight temporal shape rather than absolute magnitude\. The upper\-right label gives the displayed window span, and the lower\-right label is the native sampling frequency\.Table 2\.Overview of streaming data in LiveHouse\-TS \(n=17n\{=\}17datasets, 15 public sources\)\.*Native freq\.*is the sampling rate;*Eval freq\.*is the recommended evaluation frequency\. Both exclude crawler polling frequency\.
#### 3\.3\.2\.Data Characteristics

The curated datasets have two properties: diversity \(covering qualitatively different forecasting regimes\) and liveness \(streaming coming data\)\.

![Refer to caption](https://arxiv.org/html/2608.17299v1/img/data.png)Figure 4\.Four measurements for dataset diversity\.LiveHouse\-TS is diverse enough both in terms of domain breadth, temporal scale, statistical variability, and task structure\.Diversity\.To avoid a high score being driven by matching a single domain, sampling rate, or smoothness pattern, we curate datasets along four complementary axes:*source/domain breadth*,*temporal\-scale breadth*,*task\-structure breadth*, and*dynamic richness*\(Figure[4](https://arxiv.org/html/2608.17299#S3.F4)\)\. Concretely, the current registry spans 15 public sources, 11 domains, and 8 native frequencies from one second to one year\. Figure[5](https://arxiv.org/html/2608.17299#S3.F5)summarizes this coverage by domain, native sampling frequency, and their joint distribution across the 17 datasets\. Hourly series are the most common \(7 datasets\), followed by 15\-minute \(3\) and daily \(2\); the remaining datasets cover 1s, 6min, 10min, 1mo, and 1y regimes\. This range allows LiveHouse\-TS to evaluate short\-horizon high\-rate forecasting, ordinary sensor forecasting, event\-volume forecasting, and slow low\-frequency forecasting under one protocol\. Moreover, LiveHouse\-TS is paried with dynamic richness diverse temporal behaviors such as seasonality, bursts, and regime shifts, and task\-structure breadth that spans distinct forecasting setups \(e\.g\., horizons, targets, and available covariates\)\.

Figure 5\.Coverage by domain and native sampling frequency\. LiveHouse\-TS evaluation covers from short\-horizon high\-rate forecast to long\-horizon low\-rate forecast\.Liveness\.Observations are collected from public sources as they are released, which enables evaluation on values that post\-date a model’s participation in the leaderboard\. The leaderboard therefore evolves as new collection rounds complete and previously issued forecasts become scoreable\. Every canonical observation row links back to a raw response record and parser version, and the verification run parsed 2,672 observations across all 17 datasets\. To keep the main paper focused, per\-dataset registry fields and verification\-slice statistics are reported inAppx\.[C\.5](https://arxiv.org/html/2608.17299#A3.SS5)\.

#### 3\.3\.3\.Data pipeline

The crawled raw data are archived with request metadata, and converted into the forecasting task used for model evaluation\. A future\-only filter drops any context row whose observation was not yet available at forecast issue time\.Appx\.[C\.1](https://arxiv.org/html/2608.17299#A3.SS1)details collection, canonical parsing, and task filtering\.

### 3\.4\.Evaluation Mechanism

#### 3\.4\.1\.Metrics\.

Like most static benchmark, for point accuracy we use RMSE \(↓\\downarrow\) computed onzz\-normalized series to make magnitudes comparable across datasets, and we additionally report MAPE \(↓\\downarrow\) when targets are bounded away from zero\. For probabilistic forecasts we use CRPS \(↓\\downarrow\), estimated from the quantiles emitted by each model\. To summarize performance across datasets, we report Average Rank\([61](https://arxiv.org/html/2608.17299#bib.bib43)\)\(↓\\downarrow\) across datasets, Win Rate\([61](https://arxiv.org/html/2608.17299#bib.bib43)\)\(↑\\uparrow\) from pairwise wins, and an Elo rating\([17](https://arxiv.org/html/2608.17299#bib.bib42)\)\(↑\\uparrow\) that weights wins over stronger opponents more heavily and yields a robust leaderboard score\. We refer toAppx\.[B\.4](https://arxiv.org/html/2608.17299#A2.SS4)for more details\.

Moreover, since LiveHouse\-TS is a live benchmark, we additionally introduce live\-specific metrics that capture what static leaderboards cannot\. Letsts\_\{t\}denote a base metric computed within a time window \(i\.e\., a day\) indexed byt∈\{1,…,T\}t\\in\\\{1,\\dots,T\\\}on a forecasting task\. We define*Temporal Stability*\(↓\\downarrow\) as the standard deviation over evaluations; it tests how stable a model’s performance is over time:

\(3\)Stability=1T−1​∑t=1T\(st−s¯\)2,s¯=1T​∑t=1Tst,\\mathrm\{Stability\}=\\sqrt\{\\frac\{1\}\{T\-1\}\\sum\_\{t=1\}^\{T\}\(s\_\{t\}\-\\bar\{s\}\)^\{2\}\},\\qquad\\bar\{s\}=\\frac\{1\}\{T\}\\sum\_\{t=1\}^\{T\}s\_\{t\},We next introduce*Improvement*\(↓\\downarrow\) to test how reliably a model improves as new targets are revealed\. Fix a target timeuu\(which will be realized later\)\. As the benchmark evolves, the model may issue multiple forecasts for this sameuuat different issue timest1<⋯<tTut\_\{1\}<\\cdots<t\_\{T\_\{u\}\}; letsu,ks\_\{u,k\}be the resulting error \(lower\-is\-better\) onceyuy\_\{u\}is revealed\. We quantify the monotone trend of\{su,k\}k=1Tu\\\{s\_\{u,k\}\\\}\_\{k=1\}^\{T\_\{u\}\}with Kendall’sτ\\tau,

\(4\)τu=2Tu​\(Tu−1\)​∑1≤i<j≤Tusign⁡\(su,j−su,i\),Improvement=1\|𝒰\|​∑u∈𝒰τu,\\begin\{gathered\}\\tau\_\{u\}=\\frac\{2\}\{T\_\{u\}\(T\_\{u\}\-1\)\}\\sum\_\{1\\leq i<j\\leq T\_\{u\}\}\\mathrm\{sign\}\(s\_\{u,j\}\-s\_\{u,i\}\),\\\\ \\mathrm\{Improvement\}=\\frac\{1\}\{\|\\mathcal\{U\}\|\}\\sum\_\{u\\in\\mathcal\{U\}\}\\tau\_\{u\},\\end\{gathered\}where more negative values indicate a consistently decreasing \(i\.e\., improving\) error sequence as the issue time approaches the target\. We choose Kendall’sτ\\taubecause it is non\-linear and robust to spikes, capturing whether performance predominantly improves as new information arrives \(details inAppx\.[B\.4](https://arxiv.org/html/2608.17299#A2.SS4)\)\.

![Refer to caption](https://arxiv.org/html/2608.17299v1/liverelease.png)Figure 6\.Future\-only evaluation under asynchronous model entry: models are scored only on targets released after they join the live leaderboard, to avoid leakage\.Table 3\.Overall performance on LiveHouse\-TS evaluated by RMSE, MAPE, and CRPS using Average Rank \(↓\\downarrow\), Win Rate \(↑\\uparrow\), and Elo Rating \(↑\\uparrow\)\. All baselines are ordered by their average rank across the three metrics\. The best and second\-best results in each row are highlighted inred boldandblue underline, respectively\. TSFMs consistently outperform classical statistical methods\. Moirai\-2\.0 dominates probabilistic forecasting, while TimesFM\-2\.5 and Chronos–2 excel primarily in point accuracy\.
#### 3\.4\.2\.Fairness

Note that in the real case, the model can join the leaderboard at different times, which introduces a significant challenge to fair comparison for the live benchmark: As shown in Figure[6](https://arxiv.org/html/2608.17299#S3.F6), models that join later may have already observed the ground\-truth outcomes from earlier forecasting rounds, so scoring them on earlier periods would give an information advantage and introduce potential data leakage\. To address this challenge, we propose evaluating each model only on targets released after it joins the leaderboard\. This keeps comparisons fair across entry times\.

However, models can therefore have different evaluation horizons under the future\-only evaluation paradigm; we propose two solutions\(i\)\(i\)fixed\-horizon leaderboards \(daily, weekly, and monthly\) so that models are compared over the same evaluation window\.\(i​i\)\(ii\)pair\-wise historical ranking ranks models using only the forecasting tasks on which they were evaluated together \(see more details inAppx\.[E](https://arxiv.org/html/2608.17299#A5)\)\. For each eligible model pair, it computes a dataset\-balanced win rate from per\-release comparisons \(using both RMSE and CRPS\), then aggregates these pairwise win rates into a model\-level score by macro\-averaging over eligible opponents\.

## 4\.Experiments

We conduct extensive experiments to answer the following four research questions:RQ1 \(Zero\-shot ability\)\. Do TSFMs genuinely have strong zero\-shot forecasting ability when evaluated on the real future?RQ2 \(Static vs\. live rankings\)\.Do rankings of TSFMs on LiveHouse\-TS change significantly compared to static ones?RQ3 \(Drift robustness\)\.How does TSFM performance evolve as the live distribution drifts?RQ4 \(Ranking Stability\.\)Can model rankings be maintained long\-term?

Models\.We evaluate six TSFMs in frozen zero\-shot mode, namely TiRex\([4](https://arxiv.org/html/2608.17299#bib.bib44)\), Chronos\-2\([2](https://arxiv.org/html/2608.17299#bib.bib5)\), and TimesFM\-2\.5\([15](https://arxiv.org/html/2608.17299#bib.bib3)\)\. Toto\-1\.0\([12](https://arxiv.org/html/2608.17299#bib.bib49)\), Moirai\-2\.0\([33](https://arxiv.org/html/2608.17299#bib.bib50)\)Chronos\-Bolt\([3](https://arxiv.org/html/2608.17299#bib.bib4)\), TabPFN\-TS\([25](https://arxiv.org/html/2608.17299#bib.bib51)\), Sundial\([37](https://arxiv.org/html/2608.17299#bib.bib52)\)\. We select these models as the union of the TSFMs compared in widely used benchmarks, including GIFT\-Eval\([1](https://arxiv.org/html/2608.17299#bib.bib19)\), fev\-bench\([51](https://arxiv.org/html/2608.17299#bib.bib32)\), and TIME\([44](https://arxiv.org/html/2608.17299#bib.bib45)\)\. Collectively, these models represent major paradigms of modern TSFMs, including tokenization\-based forecasting \(Chronos\-2\), direct continuous\-value prediction \(Chronos\-Bolt\), decoder\-only large\-scale forecasting \(TimesFM\-2\.5\), retrieval\-enhanced forecasting \(TiRex\), probabilistic modeling \(Moirai\-2\.0\), diffusion\-based forecasting \(Sundial\), large\-scale autoregressive pre\-training \(Toto\-1\.0\), and table foundation model adaptation for time series forecasting \(TabPFN\-TS\)\. We also include four classical baselines, namely Seasonal Naive\([26](https://arxiv.org/html/2608.17299#bib.bib46)\), Moving Average\([6](https://arxiv.org/html/2608.17299#bib.bib47)\), ARIMA\([6](https://arxiv.org/html/2608.17299#bib.bib47)\), and ETS\([27](https://arxiv.org/html/2608.17299#bib.bib48)\), which provide the evaluation baselines\. The foundation models are run with frozen weights and a fixed context window\. To keep the result tables compact we abbreviate the models as Chr2 for Chronos\-2, TFM for TimesFM\-2\.5, Toto1 for Toto\-1\.0, Moi2 for Moirai\-2\.0, MovAvg for Moving\-Average, TabPFN for TabPFN\-TS, Chr2 for Chronos\-2, and SNaive for Seasonal\-Naive, while TiRex, ARIMA, and ETS keep their origianl names\.

Settings\.All models are scored with a rolling mode, so that a forecast at timettuses only observations up tottand is graded once the ground truth arrives\. The detailed forecasting settings of each dataset are presented inAppx\.[C\.4](https://arxiv.org/html/2608.17299#A3.SS4)Table[5](https://arxiv.org/html/2608.17299#A3.T5)\. Performance is assessed with the point metrics RMSE, and MAPE and the probabilistic metric CRPS, together with the aggregate measures Average Rank, Win Rate, and Elo detailed inAppx\.[B\.4](https://arxiv.org/html/2608.17299#A2.SS4)\. Following GIFT\-Eval\([1](https://arxiv.org/html/2608.17299#bib.bib19)\), each Rank reported in the tables assigns every model a per\-dataset rank by the metric \(best=1\{\}=\{\}1\) and averages these ranks within the reported group\. The evaluation horizon reported in the results is from 01/06/2026 to 01/07/2026\. Among the 17 datasets, 10 are consistently available; the remaining 7 occasionally have missing observations during data collection\. We therefore report results on the 10 stable datasets and reserve the other 7 for future benchmarks\.

### 4\.1\.RQ1\. Zero\-shot ability

We evaluate whether TSFMs deliver strong zero\-shot forecasts by comparing their predictions across the 10 eligible datasets in the live snapshot\. We summarize the overall results in Table[3](https://arxiv.org/html/2608.17299#S3.T3)and report per\-dataset scores aggregated by domain, sampling frequency, and forecasting horizon inAppx\.[D](https://arxiv.org/html/2608.17299#A4)Table[14](https://arxiv.org/html/2608.17299#A4.T14), Table[15](https://arxiv.org/html/2608.17299#A4.T15), and Table[16](https://arxiv.org/html/2608.17299#A4.T16)\.

Overall\.Table[3](https://arxiv.org/html/2608.17299#S3.T3)presents the overall performance of all baselines across three ranking metrics computed from MAE, MAPE, and CRPS, with detailed per\-dataset results reported inAppx\.[D](https://arxiv.org/html/2608.17299#A4)Table[13](https://arxiv.org/html/2608.17299#A4.T13)\. Overall, TSFMs consistently outperform statistical baselines\. Specifically, Moirai\-2\.0 and TimesFM\-2\.5 achieve the best Average Rank, whereas Toto\-1\.0 performs consistently worse\. The Elo and Win Rate results further reveal substantial differences among TSFMs\. Moirai\-2\.0 dominates probabilistic forecasting, while TimesFM\-2\.5 excels primarily in point accuracy \(also seeAppx\.[E](https://arxiv.org/html/2608.17299#A5)Figure[17](https://arxiv.org/html/2608.17299#A5.F17)and Figure[18](https://arxiv.org/html/2608.17299#A5.F18)for more results\)\.

![Refer to caption](https://arxiv.org/html/2608.17299v1/rq1_crps_rank_heatmap.png)Figure 7\.CRPS\-based average rank across seven domains\. The results show that different TSFMs excel in distinct domains\. Notably, the top\-ranked Moirai\-2\.0 performs poorly on weather\-related domains \(Weather, Air Quality, and Ocean\)\.By different domains\.Figure[7](https://arxiv.org/html/2608.17299#S4.F7)summarizes CRPS\-based ranks across the domains; detailed results for other metrics, sampling frequencies, and forecasting horizons are reported inAppx\.[D](https://arxiv.org/html/2608.17299#A4)Table[14](https://arxiv.org/html/2608.17299#A4.T14), Table[15](https://arxiv.org/html/2608.17299#A4.T15), and Table[16](https://arxiv.org/html/2608.17299#A4.T16)\. Moirai\-2\.0 stays in the Top\-2 CRPS rank on most domains, and different TSFMs lead under different domains, frequencies, and forecast horizons\. Notably, the top\-ranked Moirai\-2\.0 performs poorly on weather\-related domains \(Weather, Air Quality, and Ocean\), which highlights a clear opportunity to improve performance on such domains\.

Findings\.TSFMs consistently outperform classical statistical baselines in the zero\-shot setting, yet no single model dominates both point forecasting and probabilistic forecasting\. Among tested TSFMs, Moirai\-2\.0 achives the best in probabilistic forecasting while TimesFM\-2\.5 and TiRex are good at point forecasting\.

### 4\.2\.RQ2\. Static versus live rankings

To avoid biases in a single leaderboard, we compare LiveHouse\-TS with three popular static benchmarks, i\.e\., GIFT\-Eval\([1](https://arxiv.org/html/2608.17299#bib.bib19)\), fev\-bench\([51](https://arxiv.org/html/2608.17299#bib.bib32)\), and TIME\([44](https://arxiv.org/html/2608.17299#bib.bib45)\)\. The comparison is restricted to the eight shared TSFMs with identical model versions \(TabPFN\-TS is unavailable in TIME\)\. All benchmarks are ranked by CRPS\. Figure[8](https://arxiv.org/html/2608.17299#S4.F8)presents the ranking comparison\.

Findings\.The three static benchmarks exhibit remarkable agreement\. Chronos\-2 consistently ranks first, followed by TiRex and TimesFM\-2\.5, while Chronos\-Bolt and Sundial remain near the bottom\. In contrast, LiveHouse\-TS produces a quite different ranking\. Moirai\-2\.0 and Toto\-1\.0 rise to the top, while Chronos\-2 and TimesFM\-2\.5 fall to the bottom\. Together with the RQ1 observation that Chronos\-2 and TimesFM\-2\.5 achieve strong point accuracy but poor probabilistic calibration, these results suggest that static benchmarks fail to capture aspects of robustness that become apparent only under continuous live evaluation\.

Figure 8\.CRPS ranking of the shared TSFMs across three static benchmarks and the live benchmark LiveHouse\-TS\. The dotted segment for TabPFN\-TS indicates that it is not evaluated in TIME\. The three static benchmarks reach a similar consensus, whereas the live benchmark produces a pronounced ranking inversion, showing that offline \(static\) rankings can diverge from online performance and that a live benchmark is necessary to assess models under real deployment conditions\.
### 4\.3\.RQ3\. Drift robustness

Unlike static benchmarks, which evaluate a fixed test set, the live benchmark continuously assesses baselines on newly arriving observations and therefore reveals their robustness to temporal distribution shift\. Figure[9](https://arxiv.org/html/2608.17299#S4.F9)reports the Average Rank of the proposed Temporal Stability \(↓\\downarrow\) and Improvement \(↓\\downarrow\) metrics\. The detailed numderical results are provided inAppx\.[D](https://arxiv.org/html/2608.17299#A4)Table[11](https://arxiv.org/html/2608.17299#A4.T11)and Table[12](https://arxiv.org/html/2608.17299#A4.T12)\. Moirai\-2\.0 achieves the best performance on both metrics, consistent with its top ranking in the live benchmark\. In contrast, Chronos\-2 and TimesFM\-2\.5, which consistently lead the static benchmarks, rank near the bottom among TSFMs, indicating substantial degradation under temporal drift\. Toto\-1\.0 further illustrates the difference between average accuracy and robustness\. Despite ranking second in live CRPS, its Stability rank is only 10\.50, revealing large performance fluctuations over time\.

Findings\.Drift robustness largely explains the ranking inversion observed in RQ2\. Models that remain stable under evolving data also achieve stronger live benchmarking performance, while strong static accuracy alone does not guarantee robust deployment\. These complementary metrics provide aspects of forecasting quality that static evaluations cannot capture\.

Figure 9\.Average Rank of Temporal Stability \(blue\) and Improvement \(yellow\)\. Moirai\-2\.0 ranks highest on both drift\-robustness metrics, while several top static performers exhibit substantially lower temporal stability\.
### 4\.4\.RQ4\. Ranking Stability

Can model rankings be maintained long\-term? In this subsection, we investigate whether model rankings remain stable across consecutive weekly snapshots of the live benchmark\. Unlike RQ2, which compares static leaderboards with LiveHouse\-TS, this analysis focuses on temporal ranking dynamics under the same evaluation protocol\. Figure[11](https://arxiv.org/html/2608.17299#S4.F11)shows CRPS rankings from W27 to W30 for all twelve baselines\.

\(a\)Wiki and Wave\.\(b\)PM2\.5 and Potomac\.
Figure 10\.Forecasting visualization on Wiki, Wave, PM 2\.5, and Potomac datasets\. The left of the dashed line denotes the historical context and the right the forecasting horizon\. Per\-model visualizations are provided in Figure[13](https://arxiv.org/html/2608.17299#A4.F13)and Figure[14](https://arxiv.org/html/2608.17299#A4.F14)\. Although different TSFMs perform well on different datasets, they share common failure modes under evolving data distributions, including oversmoothing, delayed adaptation, and underestimated distribution shifts\.Finding\.Rankings continue to evolve even over consecutive weekly snapshots\. Rather than converging to a fixed ordering, the leading position alternates among Chronos\-2, TiRex, TimesFM\-2\.5, and TabPFN\-TS, while several mid\-ranked models exchange positions across weeks\. These observations suggest that no single model can dominated the live benchmark all the way, even though it is the best method in the static benchmark\. This further highlights the necessity of the proposed LiveHouse\-TS and its supporting infrastructure for establishing a realistic testbed to constantly assess model performance under real\-world deployment\.

Figure 11\.CRPS rankings over four consecutive weekly snapshots\. The dynamic rankings demonstrate that model performance evolves with the incoming data stream, highlighting the need for continuous live evaluation rather than a single leaderboard snapshot\.
### 4\.5\.Case study

We visualize the predictions on four representative datasets from different domains and with diverse temporal characteristics: Wiki \(daily Wikimedia page views\), PM2\.5 \(hourly air quality\), Wave \(10\-minute buoy wave height\), and Potomac river flow \(15\-minute hydrology\)\. Figure[10](https://arxiv.org/html/2608.17299#S4.F10)shows the visualization comparison of these TSFMs together with ARIMA, and we provide the per\-model showcases inAppx\.[D](https://arxiv.org/html/2608.17299#A4)Figure[13](https://arxiv.org/html/2608.17299#A4.F13)and[14](https://arxiv.org/html/2608.17299#A4.F14)\.

As shown in Figure[10](https://arxiv.org/html/2608.17299#S4.F10)\(a\), on Wiki dataset, Moirai\-2\.0 and Chronos\-2 can yield similar predictions with the ground truth throughout the forecasting horizon, whereas Toto\-1\.0 exhibits unstable oscillations with wider prediction intervals\. Wave further reveals the complexity of long\-term temporal dynamics\. Chronos\-2, TimesFM\-2\.5, and TiRex initially produce less fluctuating predictions, but their predictions gradually weaken and tend towards constant trajectories rather than maintaining the underlying periodicity\. As shown in Figure[10](https://arxiv.org/html/2608.17299#S4.F10)\(b\), although TiRex, TabPFN\-TS, Chronos\-Bolt and Sundial capture the overall upeard trend of PM2\.5 in later stages, they consistently underestimate the sharp increase in the earlier stages, while ARIMA remaines anchored near historical levels and failed to predict the shift\. On the Potomac River dataset, the river flow drops rapidly after the predicted boundary, but all the highlighted methods react too slowly, consistently exceeding the true trajectory despite varying rates of decline\.

Findings\.Across these datasets, TSFM failures exhibit consistent patterns rather than isolated errors\. Models smooth out abrupt PM 2\.5 spikes, bias toward historical levels in Potomac, or collapse long\-term forecasts into smoothed trajectories in Wave when periodic dynamics are under\-observed\. Crucially, these errors often occur simultaneously, indicating that prediction consistency does not imply reliability\. Static benchmarks struggle to identify this behavior, merely averaging results over a fixed horizon\. In contrast, live benchmarks iteratively evaluate TSFMs across evolving sources, revealing real\-world performance degradation\.

## 5\.Conclusion

We presented a benchmark and live leaderboard for evaluating time series foundation models across diverse datasets and forecasting horizons\. By standardizing data processing, evaluation protocols, this work aims to make comparisons more transparent and reproducible for the community\. Our results highlight both the strengths of modern TSFMs and the remaining gaps in robustness and generalization when conditions shift across domains\. Future work will expand the dataset coverage and tasks, and incorporate richer modalities to better reflect real\-world deployment needs\.

Limitations and Ethical Considerations\. LiveHouse\-TS evaluates forecasting models using publicly accessible time series streams and is not designed to collect private or personally identifiable information; therefore, individual consent is generally not applicable to the current datasets\. Data sources are reviewed for accessibility, licensing, and provenance, and future contributors are expected to exclude sensitive personal data\. Nevertheless, geographic, domain, availability, and measurement biases in the selected streams may affect model scores and rankings, which should not be interpreted as evidence of universal superiority or downstream fairness\. Finally, the reported forecasts and rankings are research artifacts rather than operational advice and should not be used directly for high\-stakes financial, environmental, or public\-safety decisions without domain\-specific validation and human oversight\.

## References

- T\. Aksu, G\. Woo, J\. Liu, X\. Liu, C\. Liu, S\. Savarese, C\. Xiong, and D\. SahooGIFT\-Eval: a benchmark for general time series forecasting model evaluation\.arXiv preprint arXiv:2410\.10393\.Note:NeurIPS 2024 Workshop on Time Series in the Age of Large Models \(TSALM\)Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.8.1),[§1](https://arxiv.org/html/2608.17299#S1.p2.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1),[§4\.2](https://arxiv.org/html/2608.17299#S4.SS2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1),[§4](https://arxiv.org/html/2608.17299#S4.p3.1)\.
- Ansariet al\.\(2025\)A\. F\. Ansari, O\. Shchur, J\. Küken, A\. Auer, B\. Han, P\. Mercado, S\. S\. Rangapuram, H\. Shen, L\. Stella, X\. Zhang, M\. Goswami, S\. Kapoor, D\. C\. Maddix, Y\. Wang, and M\. Bohlke\-SchneiderChronos\-2: from univariate to universal forecasting\.arXiv preprint arXiv:2510\.15821\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Ansariet al\.\(2024\)A\. F\. Ansari, L\. Stella, C\. Turkmen, X\. Zhang, P\. Mercado, H\. Shen, O\. Shchur, S\. S\. Rangapuram, S\. Pineda Arango, S\. Kapoor, J\. Zschiegner, D\. C\. Maddix, H\. Wang, M\. W\. Mahoney, K\. Torkkola, A\. Gordon Wilson, M\. Bohlke\-Schneider, and Y\. WangChronos: learning the language of time series\.Transactions on Machine Learning Research \(TMLR\)\.Note:arXiv:2403\.07815Cited by:[§1](https://arxiv.org/html/2608.17299#S1.p1.1),[§2](https://arxiv.org/html/2608.17299#S2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Aueret al\.\(2025\)A\. Auer, P\. Podest, D\. Klotz, S\. Böck, G\. Klambauer, and S\. HochreiterTiRex: zero\-shot forecasting across long and short horizons with enhanced in\-context learning\.External Links:2505\.23719,[Link](https://arxiv.org/abs/2505.23719)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Bifet and Gavaldà \(2007\)A\. Bifet and R\. GavaldàLearning from time\-changing data with adaptive windowing\.InProceedings of the 2007 SIAM International Conference on Data Mining \(SDM\),pp\. 443–448\.External Links:[Document](https://dx.doi.org/10.1137/1.9781611972771.42)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Boxet al\.\(2015\)G\. E\. P\. Box, G\. M\. Jenkins, G\. C\. Reinsel, and G\. M\. LjungTime series analysis: forecasting and control\.5 edition,John Wiley & Sons\.Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Caoet al\.\(2025a\)D\. Cao, M\. Gee, J\. Liu, H\. Wang, W\. Yang, R\. Wang, and Y\. LiuConversational time series foundation models: towards explainable and effective forecasting\.External Links:2512\.16022Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Caoet al\.\(2026\)D\. Cao, Z\. Lei, M\. Weng, J\. Sun, and Y\. LiuSpeaking numbers to llms: multi\-wavelet number embeddings for time series forecasting\.External Links:2606\.26487Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Caoet al\.\(2025b\)D\. Cao, W\. Ye, Y\. Zhang, and Y\. LiuTimeDiT: general\-purpose diffusion transformers for time series foundation model\.External Links:2409\.02322,[Link](https://arxiv.org/abs/2409.02322)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Challuet al\.\(2022\)C\. Challu, K\. G\. Olivares, B\. N\. Oreshkin, F\. Garza, M\. Mergenthaler\-Canseco, and A\. DubrawskiN\-hits: neural hierarchical interpolation for time series forecasting\.arXiv preprint arXiv:2201\.12886\.External Links:[Link](https://arxiv.org/abs/2201.12886)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Cohenet al\.\(2025\)B\. Cohen, E\. Khwaja, Y\. Doubli, S\. Lemaachi, C\. Lettieri, C\. Masson, H\. Miccinilli, E\. Ramé, Q\. Ren, A\. Rostamizadeh, J\. O\. du Terrail, A\. Toon, K\. Wang, S\. Xie, Z\. Xu, V\. Zhukova, D\. Asker, A\. Talwalkar, and O\. Abou\-AmalThis time is different: an observability perspective on time series foundation models\.arXiv preprint arXiv:2505\.14766\.Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.10.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Cohenet al\.\(2026\)B\. Cohen, E\. Khwaja, Y\. Doubli, S\. Lemaachi, C\. Lettieri, C\. Masson, H\. Miccinilli, E\. Ramé, Q\. Ren, A\. Rostamizadeh, J\. O\. du Terrail, A\. Toon, K\. Wang, S\. Xie, Z\. Xu, V\. Zhukova, D\. Asker, A\. Talwalkar, and O\. Abou\-AmalThis time is different: an observability perspective on time series foundation models\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=1jDAYXfcS2)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Cohenet al\.\(2024\)B\. Cohen, E\. Khwaja, K\. Wang, C\. Masson, E\. Ramé, Y\. Doubli, and O\. Abou\-AmalToto: time series optimized transformer for observability\.arXiv preprint arXiv:2407\.07874\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Daset al\.\(2023\)A\. Das, W\. Kong, A\. Leach, S\. Mathur, R\. Sen, and R\. YuLong\-term forecasting with tide: time\-series dense encoder\.arXiv preprint arXiv:2304\.08424\.External Links:[Link](https://arxiv.org/abs/2304.08424)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Daset al\.\(2024\)A\. Das, W\. Kong, R\. Sen, and Y\. ZhouA decoder\-only foundation model for time\-series forecasting\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2310\.10688Cited by:[§1](https://arxiv.org/html/2608.17299#S1.p1.1),[§2](https://arxiv.org/html/2608.17299#S2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Ekambaramet al\.\(2024\)V\. Ekambaram, A\. Jati, P\. Dayama, S\. Mukherjee, N\. H\. Nguyen, W\. M\. Gifford, C\. Reddy, and J\. KalagnanamTiny time mixers \(ttms\): fast pre\-trained models for enhanced zero/few\-shot forecasting of multivariate time series\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2401\.03955Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Elo \(1978\)A\. E\. EloThe rating of chessplayers, past and present\.Arco Publishing,New York\.Cited by:[§B\.4](https://arxiv.org/html/2608.17299#A2.SS4.p7.1),[§3\.4\.1](https://arxiv.org/html/2608.17299#S3.SS4.SSS1.p1.1)\.
- Fanget al\.\(2026\)Y\. Fang, H\. Miao, Y\. Liang, L\. Deng, Y\. Cui, X\. Zeng, Y\. Xia, Y\. Zhao, T\. B\. Pedersen, C\. S\. Jensen, X\. Zhou, and K\. ZhengUnraveling spatio\-temporal foundation models via the pipeline lens: a comprehensive review\.IEEE Transactions on Knowledge and Data Engineering38\(3\),pp\. 2040–2063\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Gamaet al\.\(2013\)J\. Gama, R\. Sebastião, and P\. P\. RodriguesOn evaluating stream learning algorithms\.Machine Learning90\(3\),pp\. 317–346\.External Links:[Document](https://dx.doi.org/10.1007/s10994-012-5320-9)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Garzaet al\.\(2023\)A\. Garza, C\. Challu, and M\. Mergenthaler\-CansecoTimeGPT\-1\.arXiv preprint arXiv:2310\.03589\.Cited by:[§1](https://arxiv.org/html/2608.17299#S1.p1.1),[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Garzaet al\.\(2026\)A\. Garza, R\. Rosillo, R\. Mendoza\-Smith, D\. Salinas, A\. R\. Williams, A\. Ashok, M\. Goswami, and J\. M\. JuárezImpermanent: a live benchmark for temporal generalization in time series forecasting\.arXiv preprint arXiv:2603\.08707\.Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.12.1)\.
- Godahewaet al\.\(2021\)R\. Godahewa, C\. Bergmeir, G\. I\. Webb, R\. J\. Hyndman, and P\. Montero\-MansoMonash time series forecasting archive\.InProceedings of the NeurIPS Track on Datasets and Benchmarks,Note:arXiv:2105\.06643Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.3.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Golchin and Surdeanu \(2024\)S\. Golchin and M\. SurdeanuTime travel in llms: tracing data contamination in large language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Note:arXiv:2308\.08493Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Goswamiet al\.\(2024\)M\. Goswami, K\. Szafer, A\. Choudhry, Y\. Cai, S\. Li, and A\. DubrawskiMOMENT: a family of open time\-series foundation models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2402\.03885Cited by:[§1](https://arxiv.org/html/2608.17299#S1.p1.1),[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Hooet al\.\(2026\)S\. B\. Hoo, S\. Müller, D\. Salinas, and F\. HutterFrom tables to time: extending tabpfn\-v2 to time series forecasting\.External Links:2501\.02945,[Link](https://arxiv.org/abs/2501.02945)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Hyndman and Athanasopoulos \(2018\)R\. J\. Hyndman and G\. AthanasopoulosForecasting: principles and practice\.2 edition,OTexts,Melbourne, Australia\.External Links:ISBN 9780987507112,[Link](https://otexts.com/fpp2/)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Hyndmanet al\.\(2002\)R\. J\. Hyndman, A\. B\. Koehler, J\. K\. Ord, and R\. D\. SnyderA state space framework for automatic forecasting using exponential smoothing methods\.International Journal of Forecasting18\(3\),pp\. 439–454\.Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Jainet al\.\(2025\)N\. Jain, K\. Han, A\. Gu, W\. Li, F\. Yan, T\. Zhang, S\. Wang, A\. Solar\-Lezama, K\. Sen, and I\. StoicaLiveCodeBench: holistic and contamination free evaluation of large language models for code\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:arXiv:2403\.07974Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Jinet al\.\(2024\)M\. Jin, S\. Wang, L\. Ma, Z\. Chu, J\. Y\. Zhang, X\. Shi, P\. Chen, Y\. Liang, Y\. Li, S\. Pan, and Q\. WenTime\-llm: time series forecasting by reprogramming large language models\.InThe Twelfth International Conference on Learning Representations \(ICLR\),Note:arXiv:2310\.01728Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Liet al\.\(2025\)Z\. Li, X\. Qiu, P\. Chen, Y\. Wang, H\. Cheng, Y\. Shu, J\. Hu, C\. Guo, A\. Zhou, C\. S\. Jensen,et al\.Tsfm\-bench: a comprehensive and unified benchmark of foundation models for time series forecasting\.InProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\. 2,pp\. 5595–5606\.Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.11.1),[§1](https://arxiv.org/html/2608.17299#S1.p2.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Liet al\.\(2024\)Z\. Li, X\. Qiu, P\. Chen, Y\. Wang, H\. Cheng, Y\. Shu, J\. Hu, C\. Guo, A\. Zhou, C\. S\. Jensen, and B\. YangFoundTS: comprehensive and unified benchmarking of foundation models for time series forecasting\.arXiv preprint arXiv:2410\.11802\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Lianget al\.\(2024\)Y\. Liang, H\. Wen, Y\. Nie, Y\. Jiang, M\. Jin, D\. Song, S\. Pan, and Q\. WenFoundation models for time series analysis: a tutorial and survey\.InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining \(KDD\),pp\. 6555–6565\.Note:arXiv:2403\.14735External Links:[Document](https://dx.doi.org/10.1145/3637528.3671451)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Liuet al\.\(2026\)C\. Liu, T\. Aksu, J\. Liu, X\. Liu, H\. Yan, Q\. Pham, S\. Savarese, D\. Sahoo, C\. Xiong, and J\. LiMoirai 2\.0: when less is more for time series forecasting\.External Links:2511\.11698,[Link](https://arxiv.org/abs/2511.11698)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Liuet al\.\(2022a\)M\. Liu, A\. Zeng, M\. Chen, Z\. Xu, Q\. Lai, L\. Ma, and Q\. XuSCINet: time series modeling and forecasting with sample convolution and interaction\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2106.09305)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Liuet al\.\(2022b\)S\. Liu, H\. Yu, C\. Liao, J\. Li, W\. Lin, A\. X\. Liu, and S\. DustdarPyraformer: low\-complexity pyramidal attention for long\-range time series modeling and forecasting\.InInternational Conference on Learning Representations,External Links:[Link](https://iclr.cc/virtual/2022/poster/6827)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Liuet al\.\(2024a\)X\. Liu, J\. Liu, G\. Woo, T\. Aksu, Y\. Liang, R\. Zimmermann, C\. Liu, S\. Savarese, C\. Xiong, and D\. SahooMoirai\-moe: empowering time series foundation models with sparse mixture of experts\.arXiv preprint arXiv:2410\.10469\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Liuet al\.\(2025\)Y\. Liu, G\. Qin, Z\. Shi, Z\. Chen, C\. Yang, X\. Huang, J\. Wang, and M\. LongSundial: a family of highly capable time series foundation models\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=LO7ciRpjI5)Cited by:[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Liuet al\.\(2024b\)Y\. Liu, H\. Zhang, C\. Li, X\. Huang, J\. Wang, and M\. LongTimer: generative pre\-trained transformers are large time series models\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2402\.02368Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Makridakiset al\.\(2020\)S\. Makridakis, E\. Spiliotis, and V\. AssimakopoulosThe M4 competition: 100,000 time series and 61 forecasting methods\.International Journal of Forecasting36\(1\),pp\. 54–74\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Makridakiset al\.\(2022\)S\. Makridakis, E\. Spiliotis, and V\. AssimakopoulosThe M5 competition: background, organization, and implementation\.International Journal of Forecasting38\(4\),pp\. 1325–1336\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Meyeret al\.\(2026\)M\. Meyer, S\. Kaltenpoth, H\. Albers, K\. Zalipski, and O\. MüllerTS\-arena – a live forecast pre\-registration platform\.InProceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V\.2,pp\. 9558–9568\.External Links:[Link](http://dx.doi.org/10.1145/3770855.3817515),[Document](https://dx.doi.org/10.1145/3770855.3817515)Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.13.1)\.
- Nieet al\.\(2023\)Y\. Nie, N\. H\. Nguyen, P\. Sinthong, and J\. KalagnanamA time series is worth 64 words: long\-term forecasting with transformers\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2211.14730)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Oreshkinet al\.\(2020\)B\. N\. Oreshkin, D\. Carpov, N\. Chapados, and Y\. BengioN\-beats: neural basis expansion analysis for interpretable time series forecasting\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1905.10437)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Qiaoet al\.\(2026\)Z\. Qiao, S\. Pan, A\. Wang, V\. Zhukova, Y\. Liu, X\. Jiang, Q\. Wen, M\. Long, M\. Jin, and C\. LiuIt’s time: towards the next generation of time series forecasting benchmarks\.arXiv preprint arXiv:2602\.12147\.Cited by:[§4\.2](https://arxiv.org/html/2608.17299#S4.SS2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Qiuet al\.\(2024\)X\. Qiu, J\. Hu, L\. Zhou, X\. Wu, J\. Du, B\. Zhang, C\. Guo, A\. Zhou, C\. S\. Jensen, Z\. Sheng, and B\. YangTFB: towards comprehensive and fair benchmarking of time series forecasting methods\.Proceedings of the VLDB Endowment \(PVLDB\)17\(9\),pp\. 2363–2377\.Note:arXiv:2403\.20150Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.5.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Rasulet al\.\(2023\)K\. Rasul, A\. Ashok, A\. R\. Williams, H\. Ghonia, R\. Bhagwatkar, A\. Khorasani, M\. J\. D\. Bayazi, G\. Adamopoulos, R\. Riachi, N\. Hassen, M\. Biloš, S\. Garg, A\. Schneider, N\. Chapados, A\. Drouin, V\. Zantedeschi, Y\. Nevmyvaka, and I\. RishLag\-llama: towards foundation models for probabilistic time series forecasting\.arXiv preprint arXiv:2310\.08278\.Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Sainzet al\.\(2023\)O\. Sainz, J\. A\. Campos, I\. García\-Ferrero, J\. Etxaniz, O\. L\. de Lacalle, and E\. AgirreNLP evaluation in trouble: on the need to measure llm data contamination for each benchmark\.InFindings of the Association for Computational Linguistics: EMNLP 2023,pp\. 10776–10787\.Note:arXiv:2310\.18018Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Salinaset al\.\(2017\)D\. Salinas, V\. Flunkert, and J\. GasthausDeepAR: probabilistic forecasting with autoregressive recurrent networks\.arXiv preprint arXiv:1704\.04110\.External Links:[Link](https://arxiv.org/abs/1704.04110)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Shabtayet al\.\(2025\)N\. Shabtay, E\. Schwartz, A\. Arbelle, P\. Staar, S\. Doveh, K\. Saenko, L\. Karlinsky, and R\. GiryesLiveXiv – a multi\-modal live benchmark based on arxiv papers content\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:arXiv:2410\.10783Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Shaoet al\.\(2023\)Z\. Shao, F\. Wang, Y\. Xu, W\. Wei, C\. Yu, Z\. Zhang, D\. Yao, T\. Sun, G\. Jin, X\. Cao, G\. Cong, C\. S\. Jensen, and X\. ChengExploring progress in multivariate time series forecasting: comprehensive benchmarking and heterogeneity analysis\.arXiv preprint arXiv:2310\.06119\.Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.4.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Shchuret al\.\(2025\)O\. Shchur, A\. F\. Ansari, C\. Turkmen, L\. Stella, N\. Erickson, P\. Guerron, M\. Bohlke\-Schneider, and Y\. WangFev\-bench: a realistic benchmark for time series forecasting\.arXiv preprint arXiv:2509\.26468\.Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.9.1),[§1](https://arxiv.org/html/2608.17299#S1.p2.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1),[§4\.2](https://arxiv.org/html/2608.17299#S4.SS2.p1.1),[§4](https://arxiv.org/html/2608.17299#S4.p2.1)\.
- Shiet al\.\(2025\)X\. Shi, S\. Wang, Y\. Nie, D\. Li, Z\. Ye, Q\. Wen, and M\. JinTime\-moe: billion\-scale time series foundation models with mixture of experts\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:arXiv:2409\.16040Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Wanget al\.\(2026\)S\. Wang, J\. Ni, Z\. Zhang, B\. Mo, X\. Zhong, C\. Wang, Y\. Fang, Z\. Ye, and Y\. XiangConFlux: multivariate time series in flux, one unified forecast in confluence\.InInternational Conference on Machine Learning \(ICML\),Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Whiteet al\.\(2025\)C\. White, S\. Dooley, M\. Roberts, A\. Pal, B\. Feuer, S\. Jain, R\. Shwartz\-Ziv, N\. Jain, K\. Saifullah, S\. Naidu, C\. Hegde, Y\. LeCun, T\. Goldstein, W\. Neiswanger, and M\. GoldblumLiveBench: a challenging, contamination\-limited llm benchmark\.InThe Thirteenth International Conference on Learning Representations \(ICLR\),Note:Spotlight; arXiv:2406\.19314Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p3.1)\.
- Williamset al\.\(2025\)A\. R\. Williams, A\. Ashok, É\. Marcotte, V\. Zantedeschi, J\. Subramanian, R\. Riachi, J\. Requeima, A\. Lacoste, I\. Rish, N\. Chapados, and A\. DrouinContext is key: a benchmark for forecasting with essential textual information\.InProceedings of the 42nd International Conference on Machine Learning \(ICML\),Note:arXiv:2410\.18959Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.7.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Wooet al\.\(2024\)G\. Woo, C\. Liu, A\. Kumar, C\. Xiong, S\. Savarese, and D\. SahooUnified training of universal time series forecasting transformers\.InProceedings of the 41st International Conference on Machine Learning \(ICML\),Note:arXiv:2402\.02592Cited by:[§1](https://arxiv.org/html/2608.17299#S1.p1.1),[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Wuet al\.\(2023\)H\. Wu, T\. Hu, Y\. Liu, H\. Zhou, J\. Wang, and M\. LongTimesNet: temporal 2d\-variation modeling for general time series analysis\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/2210.02186)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Wuet al\.\(2021\)H\. Wu, J\. Xu, J\. Wang, and M\. LongAutoformer: decomposition transformers with auto\-correlation for long\-term series forecasting\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2106.13008)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Zenget al\.\(2023\)A\. Zeng, M\. Chen, L\. Zhang, and Q\. XuAre transformers effective for time series forecasting?\.InProceedings of the AAAI Conference on Artificial Intelligence,External Links:[Link](https://arxiv.org/abs/2205.13504)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Zhanget al\.\(2024\)J\. Zhang, X\. Wen, Z\. Zhang, S\. Zheng, J\. Li, and J\. BianProbTS: benchmarking point and distributional forecasting across diverse prediction horizons\.InAdvances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track,Note:arXiv:2310\.07446Cited by:[Table 1](https://arxiv.org/html/2608.17299#S1.T1.7.1.6.1),[§2](https://arxiv.org/html/2608.17299#S2.p2.1)\.
- Zhanget al\.\(2026\)X\. Zhang, D\. Maddix Robinson, J\. Yin, N\. Erickson, A\. F\. Ansari, B\. Han, S\. Zhang, L\. Akoglu, C\. Faloutsos, M\. Mahoney,et al\.Mitra: mixed synthetic priors for enhancing tabular foundation models\.Advances in neural information processing systems38,pp\. 15795–15840\.Cited by:[§3\.4\.1](https://arxiv.org/html/2608.17299#S3.SS4.SSS1.p1.1)\.
- Zhang and Yan \(2023\)Y\. Zhang and J\. YanCrossformer: transformer utilizing cross\-dimension dependency for multivariate time series forecasting\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=vSVLM2j9eie)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Zhouet al\.\(2021\)H\. Zhou, S\. Zhang, J\. Peng, S\. Zhang, J\. Li, H\. Xiong, and W\. ZhangInformer: beyond efficient transformer for long sequence time\-series forecasting\.InProceedings of the AAAI Conference on Artificial Intelligence,External Links:[Link](https://arxiv.org/abs/2012.07436)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Zhouet al\.\(2022\)T\. Zhou, Z\. Ma, Q\. Wen, X\. Wang, L\. Sun, and R\. JinFEDformer: frequency enhanced decomposed transformer for long\-term series forecasting\.InInternational Conference on Machine Learning,External Links:[Link](https://arxiv.org/abs/2201.12740)Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.
- Zhouet al\.\(2023\)T\. Zhou, P\. Niu, X\. Wang, L\. Sun, and R\. JinOne fits all: power general time series analysis by pretrained lm\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Note:arXiv:2302\.11939Cited by:[§2](https://arxiv.org/html/2608.17299#S2.p1.1)\.

## Appendix AGetting Started

We introduce how external participants can connect their TSFMs and contribute datasets to the live leaderboard\. Because the benchmark updates in real time, each model must support sustainable repeated inference\. Therefore, the leaderboard does not download model weights or execute user code\. Instead, participants host a forecasting endpoint—such as a Hugging Face Space or Inference Endpoint—which the leaderboard calls via a standardized API\. Participants manage inference resources, while the leaderboard handles data collection, evaluation, aggregation, and display\.

### A\.1\.How to join the leaderboard

To join the leaderboard, participants provide a public Hugging Face model repository, a public URL for the endpoint implementation, and a stable HTTPS forecasting endpoint\. Inference runs on participant\-controlled infrastructure: the model owner supplies the inference compute, while the leaderboard handles live task generation, evaluation, aggregation, and display\. A paid Hugging Face Space, a participant\-owned domain, and a public server IP are not required\.

The recommended workflow is:

1. \(1\)Initialize the portable endpoint template, replace itsforecast\_onefunction with the model inference logic, and start the service on the participant’s inference server\.
2. \(2\)Validate the local endpoint using a complete forecasting request\.
3. \(3\)Run the publishing helper\. By default, it exposes the local service through a persistent Tailscale Funnel with managed HTTPS and validates the resulting public route\. A stable institutional HTTPS endpoint may be supplied instead\.
4. \(4\)Submit the generated metadata and validation receipt after both the local and public endpoints pass validation\.

The default deployment requires Python 3, Docker Engine, and Docker Compose v2\. When the Tailscale route is used, Tailscale 1\.52 or later must also be installed on the inference server\. The complete setup is:

gitclonehttps://github\.com/zhouziyu02/TS\-Live\.git

cdTS\-Live

python3\-mvenv\.venv

source\.venv/bin/activate

python\-mpipinstall\-rrequirements\.txt

exportMODEL\_ID="your\-hf\-username/your\-model"

exportDISPLAY\_NAME="YourModelName"

exportCODE\_URL="https://github\.com/your\-org/your\-endpoint"

dockerversion

dockercomposeversion

pythonscripts/community\_model\_wizard\.pyinit\\

\-\-output\-dirforecast\-service

dockercompose\-fforecast\-service/compose\.yaml\\

up\-d\-\-build

dockercompose\-fforecast\-service/compose\.yamlps

curl\-\-fail\-\-show\-error\\

\-\-retry30\-\-retry\-delay2\-\-retry\-connrefused\\

http://127\.0\.0\.1:7860/health

pythonscripts/validate\_external\_model\_endpoint\.py\\

\-\-endpoint\-urlhttp://127\.0\.0\.1:7860/forecast\\

\-\-model\-id"$\{MODEL\_ID\}"\\

\-\-allow\-http

tailscaleversion

sudotailscaleup

sudotailscaleset\-\-operator="$\{USER\}"

tailscalestatus

pythonscripts/community\_model\_wizard\.pypublish\\

\-\-model\-id"$\{MODEL\_ID\}"\\

\-\-display\-name"$\{DISPLAY\_NAME\}"\\

\-\-code\-url"$\{CODE\_URL\}"\\

\-\-output\-dircommunity\-submission

The first Funnel activation may require the participant or a tailnet administrator to approve Funnel and HTTPS access in a browser\. If the publishing helper displays an approval URL, the participant completes this one\-time authorization and reruns the samepublishcommand\. The helper allows up to ten minutes for the public DNS record and HTTPS route to become available\.

Participants who already operate a stable institutional HTTPS endpoint, or a named Cloudflare Tunnel with a stable hostname, may skip the Tailscale setup and run:

pythonscripts/community\_model\_wizard\.pypublish\\

\-\-model\-id"$\{MODEL\_ID\}"\\

\-\-display\-name"$\{DISPLAY\_NAME\}"\\

\-\-code\-url"$\{CODE\_URL\}"\\

\-\-endpoint\-url\\

https://forecast\.your\-domain\.example/forecast\\

\-\-output\-dircommunity\-submission

For each evaluation request, the endpoint receives only the causal target history, an opaque series identifier, the forecast horizon, frequency, and requested quantile levels\. It does not receive future observations, ground truth, raw dataset names, private metric values, or predictions from other models\. The validator checksGET /healthand sends a complete request toPOST /forecast\. It verifies the response length, numerical finiteness, requested quantiles, response\-size limit, and HTTPS requirement before producing a successful receipt\.

The default Funnel is initiated from the inference server and therefore requires no inbound firewall rule, participant\-owned domain, or manually managed TLS certificate\. Its\*\.ts\.netaddress remains stable while the Tailscale device identity and MagicDNS name are retained\. Tailscale Funnel is a low\-friction option but remains a beta service with provider\-defined bandwidth limits\. Participants requiring a custom domain or stronger ingress guarantees should use an institutional HTTPS service or a named Cloudflare Tunnel\. Temporary tunnel URLs are not accepted\.

After successful validation, the participant pastes the contents ofcommunity\-submission/community\_model\.yamlandcommunity\-submission/validator\_receipt\.jsoninto the community model request form at[https://github\.com/zhouziyu02/TS\-Live/issues/new?template=community\-model\.yml](https://github.com/zhouziyu02/TS-Live/issues/new?template=community-model.yml)\. The submitted endpoint is reviewed before being enabled\.

An accepted model is admitted only to future live evaluation rounds and is never backfilled on releases preceding its admission time\. Its results appear after the next successful evaluation cycle\. Detailed release snapshots may be retained privately for auditing and metric recomputation, while only metric summaries and aggregate leaderboard tables are published\.

### A\.2\.How to contribute new dataset

As an open online benchmark, LiveHouse\-TS accepts new*public*time series streams from the community\. A contribution plugs into the same live loop as the built\-in sources in Sec\.[3\.3](https://arxiv.org/html/2608.17299#S3.SS3): register the dataset semantics, ingest fresh observations on a recurring schedule, and let the evaluation house form and score forecasting tasks automatically\. Contributors provide \(i\) a dataset\-level specification that defines the forecasting problem and \(ii\) an ingestion adapter that fetches the public source and emits normalized observations\. The full attribute contract and preprocessing requirements are given in Appx\.[B\.2](https://arxiv.org/html/2608.17299#A2.SS2); accepted sources are reviewed for licensing, stability, and schema compliance before entering the public registry\.

## Appendix BImplementation Details of LiveHouse\-TS

### B\.1\.Model Entrance\.

The model entrance is the component through which all forecasting methods are registered, adapted, and admitted into LiveHouse\-TS\. Its primary role is to decouple model\-specific inference logic from the live data and evaluation pipeline\. A model may be a hosted TSFM, an external service, a local predictor, or a statistical baseline; after passing through the entrance, however, all methods expose a uniform forecasting contract to the evaluation house\.

Model registry\.Each model is specified by a registry entry containing a stable model identifier, a display name, the model type, organization metadata, links to the model and replication code, and the information needed to instantiate its adapter\. For hosted TSFMs, this includes the remote model identifier and API backend\. For community submissions, the entry may instead specify a validated HTTPS forecast endpoint\. For statistical baselines, the entry records the algorithm family and its fixed hyperparameters, such as the ARIMA order, moving\-average window, or season length\. The registry also records whether a model is enabled for public evaluation and the time from which its future\-only evaluation window begins\. This start time is important: LiveHouse\-TS never backfills a newly added model on targets that were already observable before the model entered the benchmark\.

Unified forecasting contract\.For every admitted evaluation task, the entrance presents the model with a causal context window, the sampling frequency, the prediction horizon, and minimal task metadata\. Formally, the model receives𝐱1:T\\mathbf\{x\}\_\{1:T\}, a frequency descriptorff, and a horizonHH, and is asked to return forecasts for𝐱T\+1:T\+H\\mathbf\{x\}\_\{T\+1:T\+H\}\. The standardized output contains a point forecast and, when available, predictive quantiles:

\(𝝁^1:H,\{𝐪^τ,1:H\}τ∈𝒬\),\\left\(\\hat\{\\boldsymbol\{\\mu\}\}\_\{1:H\},\\\{\\hat\{\\mathbf\{q\}\}\_\{\\tau,1:H\}\\\}\_\{\\tau\\in\\mathcal\{Q\}\}\\right\),where𝒬\\mathcal\{Q\}denotes the requested quantile levels\. This contract is implemented through a common predictor abstraction, so the downstream evaluator does not need method\-specific logic for TSFM APIs, local predictors, endpoints, or baselines\. Forecast arrays are checked for shape, horizon length, and finite numeric values before they are converted into the common evaluation format\.

Adapters for heterogeneous model outputs\.Different TSFMs expose various native output formats\. Some return means and quantile forecasts; others return samples, prediction intervals, or point forecasts\. The model entrance normalizes these outputs into a uniform mean\-plus\-quantiles representation\. If a model provides samples, empirical quantiles are computed from the sample paths\. If a model exposes only point forecasts, the adapter constructs an approximate predictive distribution via repeated stochastic calls or residual\-bootstrap perturbations\. This normalization allows point and distributional metrics, such as the quantile\-loss approximation to CRPS, to be computed by a single metric engine\.

Hosted and external models\.Built\-in hosted TSFMs are queried through authenticated forecast APIs with fixed request fields: historical target values, frequency, prediction length, and requested quantile levels\. Community models can be integrated through a self\-hosted HTTPS endpoint\. In this mode, LiveHouse\-TS treats the submitted model as a black box: it does not import user code, download user weights, or execute third\-party dependencies inside the leaderboard process\. By default, the endpoint receives only the causal context, an opaque series identifier, the frequency, the horizon, and the requested quantiles\. Future targets, metric values, authentication tokens, raw leaderboard internals, and other models’ predictions are never sent to the endpoint\. The adapter enforces request timeouts, retry limits, maximum context lengths, maximum response sizes, and forecast\-validity checks\.

Statistical baselines\.The same entrance also hosts non\-pretrained reference methods, including seasonal naive, moving average, ARIMA, and exponential smoothing\. These baselines are fit or instantiated only from the context window of the current task and therefore obey the same causal restriction as TSFMs\. Since several classical methods naturally produce point forecasts, the entrance derives quantile forecasts using residual\-bootstrap samples when probabilistic outputs are required\. This makes the baselines compatible with both point\-accuracy and distributional metrics without giving them any special treatment in the evaluator\.

##### Failure handling and auditability\.

The entrance isolates model failures from the rest of the benchmark\. Transient API failures can be retried, while invalid responses, non\-finite forecasts, short horizons, malformed quantiles, or endpoint errors are rejected before scoring\. For each successful run, LiveHouse\-TS stores model metadata, task metadata, metric rows, and evaluation timestamps as persistent artifacts\. These records make it possible to audit which adapter, model identifier, context window, horizon, and evaluation time produced each leaderboard entry\. Thus, the model entrance provides both a practical integration layer for heterogeneous forecasting systems and a reproducibility boundary that keeps the live evaluation protocol model agnostic\.

### B\.2\.Live Data\.

As a public online benchmark, LiveHouse\-TS is designed to grow beyond its initial registry\. Researchers and practitioners can contribute additional*public*streams—for example, a weather station feed, an open mobility API, or a government statistics portal—as long as the source is openly accessible and can be mapped into the shared contract below\. Each contribution reuses the same live loop as the built\-in sources: raw responses are archived for provenance, observations are normalized into a common schema, forecast tasks are generated from per\-dataset window settings, and the evaluation house scores only targets that become observable after a model joins the leaderboard\.

Contribution contract\.A new source requires two complementary pieces\. First, a*dataset specification*that semantically defines the forecasting problem \(domain, entity type, native frequency, evaluation frequency, context and horizon lengths, targets, and covariates\)\. Second, an*ingestion adapter*that periodically retrieves the public endpoint, preserves immutable raw evidence, and emits normalized observation records\. The dataset specification is the single source of semantic truth; polling cadence is an operational scheduling choice and is kept separate from the forecasting definition itself\.

Required dataset attributes\.Each contributed dataset must declare the attributes below\.

- •Identity:dataset identifier, source identifier, human\-readable name, domain, entity granularity, and short English description/background for downstream users\.
- •Forecasting setup:nativedata\_frequency, recommendedeval\_frequency,history\_length\_stepsandforecast\_horizon\_steps\(both measured in native\-frequency steps\),target\_variables, and optionalcovariate\_variables\.
- •Behavior flags:whether the series is ready for direct zero\-shot leaderboard scoring, and whether raw events must be aggregated to a regular grid before forecasting\.

History and horizon windows are always expressed in the native series frequency, so a high\-rate stream and a monthly macro series can coexist without forcing a single global window length\.

Storage model\.The data layer separates three concerns rather than relying on one monolithic training file\.

- •Raw archive:immutable copies of public responses, together with request metadata, fetch time, content hash, and parser version, so every normalized value remains auditable\.
- •Canonical store:relational long\-format tables for datasets, entities, variables, observations, forecast tasks, and quality/run logs\. This layer is the source of truth\.
- •Task export:ephemeral model\-facing bundles that pair a context window with dataset metadata; future targets are exported separately for the evaluator only\.

Preprocessing requirements\.After each fetch, an ingestion adapter must satisfy the following requirements\.

1. \(1\)Preserve provenance\.Store the untouched response before parsing; never overwrite prior raw evidence\.
2. \(2\)Normalize entities and variables\.Map each forecastable object \(station, market, fleet, grid point, country, etc\.\) to a stable entity identifier, and map measurable quantities to variable identifiers with units and frequency hints\.
3. \(3\)Emit long\-format observations\.Represent each value as one record keyed by entity, timestamp, and variable\. Every record must carry: - •timestamp: when the measurement refers to; - •available\_time: when the value became knowable to a forecaster; - •ingest\_time: when LiveHouse\-TS retrieved it; - •numericvalue,frequency,unit, and a link back to the raw evidence\. Event streams \(e\.g\., news documents or earthquake catalogs\) must be bucketed to the declared native frequency when aggregation is required\.
4. \(4\)Generate forecast tasks\.Instantiate tasks from the dataset specification: forecast issue time, context window, horizon window, covariate set, and frequency must be identical for all models evaluated on the same dataset\.
5. \(5\)Report data health\.Record missingness, duplicates, freshness delay, outliers, and ingestion failures so unstable or stale sources do not silently enter the leaderboard\.

Leakage rule\.Model inputs must never contain information that was unavailable at prediction time\. For every context observation,

available\_time≤forecast\_issue\_time\.\\texttt\{available\\\_time\}\\leq\\texttt\{forecast\\\_issue\\\_time\}\.A measurement timestamp alone is therefore insufficient: the value must also have been publicly knowable before the forecast was issued\. Future targets are kept evaluator\-only and are never exposed as model inputs\. This rule is the same leakage\-resistant contract used by the built\-in streams described in Appx\.[C\.1](https://arxiv.org/html/2608.17299#A3.SS1)\. Once accepted, a contributed source inherits the same task generation, future\-only gating, and leaderboard update loop as the core registry entries\.

### B\.3\.Evaluation House\.

The evaluation house is the component in LiveHouse\-TS that turns standardized model forecasts into auditable leaderboard records\. It receives forecasting tasks from the live data stage and model predictions from the model entrance, then applies a fixed evaluation protocol to all admitted model–task pairs\. Its design goal is to ensure that every reported score is computed from the same target window, the same metric implementation, and the same aggregation rule, regardless of the model’s inference backend or output format\.

Task admission and future\-only scoring\.For each refreshed data stream, the live data stage materializes a forecasting task consisting of a historical context window𝐱1:T\\mathbf\{x\}\_\{1:T\}, a prediction horizonHH, dataset metadata, and the future target𝐱T\+1:T\+H\\mathbf\{x\}\_\{T\+1:T\+H\}\. The evaluation house first checks whether the task is eligible for a given model\. Letama\_\{m\}denote the accepted entrance time of modelmm\. A task is scored formmonly if its target window is generated from observations that become available afterama\_\{m\}\. This future\-only gate prevents newly submitted models from being evaluated retrospectively on data that could have been inspected during model development or endpoint debugging\. The same gate is applied to hosted TSFMs, external community endpoints, local wrappers, and statistical baselines\.

Forecast execution\.After admission, the evaluation house dispatches the task through the unified predictor interface\. The model receives only the causal context, frequency, and horizon; the target values are held out until scoring\. The returned forecast is converted into a common representation containing a mean or median forecast and, when available, a set of predictive quantiles\. The evaluator validates that all forecast arrays have the required horizon length and contain finite numeric values\. Invalid, malformed, or incomplete outputs are rejected before metrics are computed, so downstream ranking is never based on partially parsed forecasts\.

Metric computation\.The metric engine evaluates both point accuracy and probabilistic quality\. For point forecasts, LiveHouse\-TS reports metrics such as MSE, RMSE, MAE, MASE, MAPE, sMAPE, NRMSE, and ND when they are well defined\. For probabilistic forecasts, the evaluator computes interval and quantile\-based scores, including MSIS and the mean weighted sum quantile loss, which is used as the CRPS\-style distributional metric in the leaderboard\. All metrics are computed on the same held\-out target window for all models admitted to that task\. Metrics that are undefined for a particular target, such as percentage errors near zero, are marked as unavailable rather than silently imputed\.

Reference baselines and relative gain\.The evaluation house also maintains matched baseline scores for lightweight reference methods\. In particular, Seasonal\-Naive is used as a causal reference because it requires no pretraining and can be instantiated from the same context window as every other method\. For live aggregate reporting, LiveHouse\-TS computes a relative real\-time gain \(RTG\) from matched MSE values:

RTG⁡\(m\)=100⋅MSESNaive−MSEmMSESNaive\+MSEm\.\\mathrm\{RTG\}\(m\)=100\\cdot\\frac\{\\mathrm\{MSE\}\_\{\\mathrm\{SNaive\}\}\-\\mathrm\{MSE\}\_\{m\}\}\{\\mathrm\{MSE\}\_\{\\mathrm\{SNaive\}\}\+\\mathrm\{MSE\}\_\{m\}\}\.This bounded form gives positive values to models that improve over the seasonal\-naive reference and negative values to models that underperform it, while avoiding instability when absolute errors are small\.

Aggregation and ranking\.The evaluation house produces fine\-grained and aggregate views\. At the lowest level, it stores a metric row for each model–dataset–release combination\. Dataset\-level scores are obtained by averaging over releases from the same stream\. Overall live scores assign equal weight to datasets, preventing frequently refreshed streams from dominating\. For grouped analysis, LiveHouse\-TS reports GIFT\-Eval\-style aggregates by domain, frequency, and prediction length\. These grouped tables normalize MSE and CRPS against matched Seasonal\-Naive scores, then aggregate across configurations\.

Ranks are computed from matched comparisons rather than from incomparable partial records\. For compact overall presentation, LiveHouse\-TS reports average rank, pairwise win rate, and an Elo\-style score based on shared releases\. The rank computation uses the lower\-is\-better ordering of MSE and CRPS and only compares models on tasks where both models have valid scores\. This preserves a fair comparison when some models join later or when an admitted forecast fails validation for a particular release\.

Persistence and reproducibility\.Every successful evaluation produces persistent artifacts: the model name and identifier, dataset configuration, release timestamp, prediction horizon, metric values, and aggregation metadata\. These artifacts are stored separately from the model execution code and are sufficient to reconstruct the public leaderboard tables\. The system also records metadata for aggregate tables, including generation time, the normalization baseline, the aggregation rule, and the number of contributing models and configurations\. This separation allows LiveHouse\-TS to audit individual leaderboard entries, regenerate aggregate tables, and distinguish changes caused by new data from changes caused by model or code updates\.

Robustness of the online loop\.Because LiveHouse\-TS operates continuously, the evaluation house is designed to handle partial failures without interrupting the full benchmark\. A failed model call, invalid endpoint response, missing quantile, or undefined metric affects only the corresponding model–task pair\. Other models evaluated on the same task remain valid, and later releases can still contribute new evidence for the failed model\. As new observations arrive, the same admission, inference, scoring, persistence, and aggregation steps are repeated\. The evaluation house therefore serves as the reproducible boundary between live forecasting execution and public leaderboard publication\.

### B\.4\.Evaluation Metrics

We evaluate each models along two complementary perspectives\. Basic metrics quantify the absolute quality and cost of a forecast on each dataset, while performance\-rank metrics aggregate these per\-dataset scores into a single comparable measure of relative standing across the whole benchmark\. Throughout we let𝐲t∈ℝC\\mathbf\{y\}\_\{t\}\\in\\mathbb\{R\}^\{C\}denote the ground\-truth vector at forecast stepttand𝐲^t∈ℝC\\hat\{\\mathbf\{y\}\}\_\{t\}\\in\\mathbb\{R\}^\{C\}the corresponding prediction over a horizon of lengthHH, whereCCis the forecast dimension and∥⋅∥\\\|\\cdot\\\|denotes a vector norm\.

MSE and RMSE\.The mean squared error \(MSE\) and root mean squared error \(RMSE\) arelower\-is\-betterpoint\-accuracy metrics\. MSE penalizes large deviations quadratically and RMSE reports the same quantity in the original data scale,

\(5\)MSE=1C​H​∑t=1H‖𝐲t−𝐲^t‖22,RMSE=MSE\.\\mathrm\{MSE\}=\\frac\{1\}\{CH\}\\sum\_\{t=1\}^\{H\}\\bigl\\\|\\mathbf\{y\}\_\{t\}\-\\hat\{\\mathbf\{y\}\}\_\{t\}\\bigr\\\|\_\{2\}^\{2\},\\qquad\\mathrm\{RMSE\}=\\sqrt\{\\mathrm\{MSE\}\}\.We compute both onzz\-normalized series so that datasets with different magnitudes contribute comparably\.

MAPE\.The mean absolute percentage error \(MAPE\) is a*lower\-is\-better*metric that expresses error relative to the magnitude of each target, which makes it scale\-free and comparable across datasets,

\(6\)MAPE=1C​H​∑t=1H‖\(𝐲t−𝐲^t\)/𝐲t‖1,\\mathrm\{MAPE\}=\\frac\{1\}\{CH\}\\sum\_\{t=1\}^\{H\}\\bigl\\\|\(\\mathbf\{y\}\_\{t\}\-\\hat\{\\mathbf\{y\}\}\_\{t\}\)/\\mathbf\{y\}\_\{t\}\\bigr\\\|\_\{1\},Because MAPE is undefined when a target is zero and inflates near small targets, we report it on series whose values stay bounded away from zero and rely on MSE and RMSE elsewhere\.

CRPS\.The continuous ranked probability score \(CRPS\) is a*lower\-is\-better*metric that assesses probabilistic forecasts\. For a single coordinate with predicted cumulative distributionFFand realized valueyyit is

\(7\)CRPS\(F,y\)=∫−∞∞\(F\(z\)−𝟏\{z≥y\}\)2dz,\\mathrm\{CRPS\}\(F,y\)=\\int\_\{\-\\infty\}^\{\\infty\}\\bigl\(F\(z\)\-\\mathbf\{1\}\\\{z\\geq y\\\}\\bigr\)^\{2\}\\,\\mathrm\{d\}z,and we average it over theCCcoordinates of𝐲^t\\hat\{\\mathbf\{y\}\}\_\{t\}and over theHHforecast steps\. CRPS is a strictly proper scoring rule that jointly rewards calibration and sharpness, and it reduces to absolute error when the forecast is a point mass\. We estimate it from the predicted quantiles emitted by each model\.

Average Rank\.This*lower\-is\-better*metric ranks models on each dataset by a basic metric and averages these ranks across datasets, providing a simple scale\-free indicator of consistent standing\.

Win Rate\.This*higher\-is\-better*metric is the fraction of pairwise comparisons a model wins\. For each dataset and each opposing model it scores a win when its metric is better, and we report the proportion of wins over all such comparisons\.

Elo\.Elo rating\([17](https://arxiv.org/html/2608.17299#bib.bib42)\)is a*higher\-is\-better*metric that treats per\-dataset head\-to\-head outcomes as matches and fits a ratingRmR\_\{m\}to each model, where expected score of modelmmagainst modelnnis

\(8\)𝔼⁡\[Sm,n\]=11\+10\(Rn−Rm\)/400\.\\mathbb\{E\}\[S\_\{m,n\}\]=\\frac\{1\}\{1\+10^\{\(R\_\{n\}\-R\_\{m\}\)/400\}\}\.Ratings are updated from observed wins and losses, so Elo rewards beating strong competitors more than weak ones and yields a single rating robust to the inclusion or removal of individual models\.

## Appendix CDataset details

The main paper summarizes benchmark coverage and task configuration\. This appendix provides additional background on series origins, physical or social quantities represented, and public acquisition methods\. All sources are openly accessible without proprietary API keys\. Table[4](https://arxiv.org/html/2608.17299#A3.T4)documents each dataset’s monitoring entity, forecasting target, public endpoint, and background notes from our diversity verification samples\.

### C\.1\.Data pipeline

Collection\.At each collection round, the pipeline loads the registry, queries the corresponding public endpoints, and stores the response body without modification under a source\-specific raw directory\. Collection runs on a source\-dependent schedule, from frequent polls for high\-rate markets to daily polls for environmental and macroeconomic feeds\. Each round appends new raw response records rather than overwriting prior responses; when sufficient observations arrive, the pipeline exports fresh forecasting tasks whose future windows can be scored against newly observed ground truth\. The raw layer keeps JSON, TXT, or ZIP responses together with request URL, request parameters, HTTP status, fetch time, content hash, parser version, and error messages when applicable\. This raw response archive makes every benchmark observation traceable to the exact public response from which it was parsed\.

Canonical parsing\.Successful raw responses are parsed into a small relational schema rather than a single monolithic CSV\. The metadata tables store dataset definitions, entities, variables, raw response records, quality reports, and run logs\. The central table is a long\-format observation table:

> dataset\_id, source\_id, entity\_id, variable\_id, timestamp, available\_time, ingest\_time, value, frequency, unit, raw\_id

This design preserves provenance and supports heterogeneous sources with different entities, units, and update mechanisms\. For event sources, such as GDELT and USGS Earthquake, raw events are converted into regular aggregate time series, e\.g\., document volume or earthquake counts per time bucket\.

Task export\.Forecasting tasks are generated from the canonical observation table using the history and horizon lengths specified in the registry\. Each task records a target entity, target variable, forecast issue time, context window, horizon window, covariate list, and frequency\. The model\-facing export containstask\.jsonandcontext\.csv; labels are isolated infuture\_target\.csvfor the evaluator\. To prevent look\-ahead leakage, the exporter enforces

available\_time≤forecast\_issue\_time\\texttt\{available\\\_time\}\\leq\\texttt\{forecast\\\_issue\\\_time\}for every context row\. Thus, a timestamp is insufficient for inclusion: values must also be available prior to prediction issuance\.

### C\.2\.Per\-dataset background and access

Table 4\.Background and public access information for each benchmark dataset\. Endpoints are the base URLs used for collection; query parameters depend on entity, time window, and variables\. Covariates may be omitted in univariate runs\. We use the following abbreviations in subsequent tables and figures: BTC for Binance BTCUSDT, PM2\.5 for Open\-Meteo air quality, Quake for USGS earthquake aggregates, Potomac for USGS Potomac discharge, Water for NOAA CO\-OPS water level, Wave for NOAA NDBC buoy observations, T2M for NASA POWER meteorology, KSFO for NWS KSFO observations, Temp2m for Open\-Meteo weather, and Wiki for Wikimedia pageviews\.
### C\.3\.Event\-derived series

Two datasets arrive as*events*rather than natively regular measurements\.GDELTreturns a timeline of relative news\-document volume for the query “climate change” at 15\-minute resolution\.USGS earthquakespublishes a rolling one\-week GeoJSON feed of global events\. For both sources, we aggregate timestamped events into regular time buckets before forming forecasting tasks, so that event\-derived streams follow the same leaderboard protocol as directly reported time series\.

### C\.4\.Diversity verification samples

Before live deployment we collected short public samples from every provider to confirm that each source can be downloaded and converted into a regular numeric series\. The verification covered 17 datasets, 11 domains, and 2,672 representative observations in total; every source in Table[4](https://arxiv.org/html/2608.17299#A3.T4)formed a usable time series in this check\. Table[5](https://arxiv.org/html/2608.17299#A3.T5)reports the resulting per\-dataset registry windows and descriptive statistics; those numbers are*verification slices*, not fixed train/test sizes for the live leaderboard\.

These statistics are generated from the canonical observation rows produced by the data pipeline, not from manual measurements\. For each dataset, the diversity\-report script selects the configured target variable \(or a documented display variable for visualization\), chooses one representative entity or aggregate, deduplicates observations by dataset, entity, variable, and timestamp, and then computesNN, mean, range, and standard deviation on the selected target values\. The underlying verification slices come from source\-dependent sample collection runs rather than from one identical calendar period imposed on every domain\-frequency pair: high\-rate feeds use short recent windows, backfillable daily or hourly sources use multi\-day verification windows, and monthly or annual sources use their available historical records\. This design lets the verification check whether each source is parseable and behaviorally distinct, while leaving the live benchmark free to keep collecting future observations\.

### C\.5\.Dataset inventory and verification statistics

Table 5\.Per\-dataset registry and verification statistics\.*Data freq\.*and*Eval freq\.*are native series frequency and recommended scoring frequency\. Hist\. and Hor\. are history and forecast horizon in native\-frequency steps\. Type distinguishes directly reported time series from event\-derived time series\.NNisrow\_countin the diversity verification slice \(not a fixed train/test size\)\. Mean, Std, and Range summarize the selected target variable in that slice\.Table[6](https://arxiv.org/html/2608.17299#A3.T6)summarizes the behavioral patterns seen in the verification charts\. They illustrate why the registry mixes smooth environmental signals, volatile financial series, sparse attention counts, and event\-driven streams under one benchmark\.

Table 6\.Representative dynamics observed in diversity verification samples \(June 2026\)\.Table 7\.Per\-dataset verification variability used to support the dynamic\-richness criterion in Figure[4](https://arxiv.org/html/2608.17299#S3.F4)\. CV is computed on the selected target variable in the verification slice asstd/\|mean\|\\mathrm\{std\}/\|\\mathrm\{mean\}\|\. These short slices illustrate temporal behavior but are not fixed train/test windows\.
### C\.6\.Per\-dataset verification charts

Figure[12](https://arxiv.org/html/2608.17299#A3.F12)shows the verification\-slice target trajectories used to summarize the dynamic patterns in Table[6](https://arxiv.org/html/2608.17299#A3.T6)\. Each panel corresponds to one registry dataset; together they illustrate the diversity of observed temporal behavior\. These are not fixed test windows but short public samples used to check that each source can be parsed into a regular numeric series\. For the Citi Bike verification chart, each point averages available bikes across 50 stations before connecting the sequence; live evaluation can instead track individual stations or GBFS systems using the same public feed\.

Figure 12\.Verification\-slice target trajectories for all 17 registry datasets\. Each panel plots the locally parsed public sample for one dataset and target variable; panel titles give the dataset identifier and selected target\.
### C\.7\.Data attribution and usage

We gratefully acknowledge Open\-Meteo, NASA POWER, USGS, NOAA, GBFS/Citi Bike, Binance, CoinGecko, Wikimedia Foundation, World Bank Open Data, and GDELT as the public data providers summarized in Table[4](https://arxiv.org/html/2608.17299#A3.T4)\. Users operating a live deployment should respect each provider’s terms of use, attribution requirements, and request\-rate limits\.

## Appendix DDetailed Forecasting results

For completeness, we report per\-dataset performance in Tables[8](https://arxiv.org/html/2608.17299#A4.T8),[9](https://arxiv.org/html/2608.17299#A4.T9), and[10](https://arxiv.org/html/2608.17299#A4.T10), which present the RMSE, MAPE, and CRPS of every model in the frozen online benchmark snapshot on each dataset\.

Table 8\.Per\-dataset RMSE \(↓\\downarrow\) on the online benchmark\. In each column the best available model is inred boldand the second best is inblue underline\.Table 9\.Per\-dataset MAPE \(↓\\downarrow\) on the online benchmark\. In each column the best available model is inred boldand the second best is inblue underline\.Table 10\.Per\-dataset CRPS \(↓\\downarrow\) on the online benchmark\. In each column the best available model is inred boldand the second best is inblue underline\.Table 11\.Per\-dataset Temporal Stability \(↓\\downarrow\) on the online benchmark, computed from release\-level MSE histories\. Cells with fewer than two releases are shown as –\. In each column the best available model is inred boldand the second best is inblue underline\.Table 12\.Per\-dataset Improvement \(↓\\downarrow; more negative is better\) on the online benchmark, computed as Kendallτ\\tauon release\-level MSE histories\. Cells with fewer than two releases are shown as –\. In each column the best available model is inred boldand the second best is inblue underline\.Table 13\.Overall zero\-shot standing on the online benchmark aggregated over all datasets\. For each metric we report Average Rank \(↓\\downarrow\), Win Rate \(↑\\uparrow\), and Elo \(↑\\uparrow\) computed from the per\-dataset values\. In each row the best available model is inred boldand the second best is inblue underline\.Table 14\.Zero\-shot rankings on the online benchmark aggregated by domain\. For each metric we report Average Rank \(↓\\downarrow\), Win Rate \(↑\\uparrow\), and Elo \(↑\\uparrow\) computed from the per\-dataset values within each domain\. In each row the best available model is inred boldand the second best is inblue underline\.Table 15\.Zero\-shot rankings on the online benchmark aggregated by sampling frequency\. For each metric we report Average Rank \(↓\\downarrow\), Win Rate \(↑\\uparrow\), and Elo \(↑\\uparrow\) computed from the per\-dataset values within each frequency group\. In each row the best available model is inred boldand the second best is inblue underline\.Table 16\.Zero\-shot rankings on the online benchmark aggregated by forecast horizon\. For each prediction length and metric, models are first ranked within each dataset and then ranked by their average rank across datasets sharing that prediction length \(best=1\{\}=\{\}1\)\. In each row the best available model is inred boldand the second best is inblue underline\.Figure 13\.Forecasting visualizations on Wiki \(left column\) and Wave \(right column\) for all baselines\. Each row corresponds to one baseline\. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons\.Figure 14\.Forecasting visualizations on PM 2\.5 \(left column\) and Potomac \(right column\) for all baselines\. Each row corresponds to one baseline\. The left side of the dash line shows the context windows, and the right side shows the forecasting horizons\.![[Uncaptioned image]](https://arxiv.org/html/2608.17299v1/pairwise1.png)

Figure 15\.Pairwise historical ranking under asynchronous model admission\. The top panel shows representative resolved releases and model admission times\. Models A–D have sufficient shared evidence, whereas Model E joined later and remains below the eligibility thresholds\. The lower\-left panel shows dataset\-balanced pairwise win ratesWi​jW\_\{ij\}; unavailable comparisons involving Model E are omitted rather than treated as ties\. The lower\-right panel aggregates eligible pairwise comparisons into model\-level scores and official ranks\. Numerical values are illustrative\.

## Appendix EPairwise Historical Ranking

Models may enter a live leaderboard at different times\. Directly comparing their average errors since admission can therefore be misleading: an early model may have experienced both volatile and calm periods, whereas a recently admitted model may have been evaluated only under the latest conditions\. We address this cohort mismatch by comparing each pair of models only on releases that they completed in common\.

A*release*is one resolved causal forecasting task for a particular dataset and future target window\. At issue time, all admitted models receive the same historical context, sampling frequency, and forecast horizon\. Their forecasts are frozen before the future target becomes available\. Once the complete target window has been observed, the forecasts are scored and the task becomes a resolved release\. Models evaluated on the same release therefore share the same context, target values, timestamps, normalization statistics, and metric implementation\.

Figure[15](https://arxiv.org/html/2608.17299#A4.F15)connects the three stages of the ranking procedure\. In the top panel, the horizontal axis represents historical time up to cutoffTT\. Each colored dot denotes a representative resolved dataset–target\-window release, with colors indicating different datasets\. The dots are illustrative; the actual ranking uses every resolved release\. The blue triangle on each model row marks its admission time\. A model is evaluated only on releases issued after admission\. Models A–D entered sufficiently early to accumulate overlapping evaluations from at least 30 releases, 5 datasets, and 7 days\. Model E entered later and has only 12 shared releases from 3 datasets over 2 days\. Its overlap is therefore insufficient for an official comparison\. The lower\-left panel records the pairwise win rateWi​jW\_\{ij\}of row modeliiagainst column modeljj\. For example,WA​B=0\.62W\_\{AB\}=0\.62means that Model A obtains a dataset\-balanced win rate of62%62\\%against Model B\. The reverse comparison isWB​A=0\.38W\_\{BA\}=0\.38\. Green cells indicate values above0\.50\.5, red cells indicate values below0\.50\.5, and gray cells are unavailable\. In particular, missing comparisons involving Model E are omitted; they are not assigned a neutral value of0\.50\.5\. The lower\-right panel averages each model’s eligible pairwise win rates with equal weight per opponent\. This produces the model\-level scoreWiW\_\{i\}used for ranking\. Models A–D receive official ranks, while Model E remains*Provisional*until it accumulates sufficient shared evidence\.

### E\.1\.Shared releases and pair eligibility

Lettrt\_\{r\}denote the end time of the target window for releaserr\. At historical cutoffTT, define

\(9\)Ri\(T\)=\{r:tr≤T,MSEi​randCRPSi​rare valid\}R\_\{i\}\(T\)=\\left\\\{r:t\_\{r\}\\leq T,\\;\\mathrm\{MSE\}\_\{ir\}\\ \\text\{and\}\\ \\mathrm\{CRPS\}\_\{ir\}\\ \\text\{are valid\}\\right\\\}
as the valid release history of modelii\. The shared release history of modelsiiandjjis

\(10\)Ri​j​\(T\)=Ri​\(T\)∩Rj​\(T\)\.R\_\{ij\}\(T\)=R\_\{i\}\(T\)\\cap R\_\{j\}\(T\)\.
Thus, a release completed by only one member of the pair does not enter their comparison\. Releases issued before the later model’s admission are automatically excluded\. Letdrd\_\{r\}be the dataset associated with releaserr\. The shared dataset set is

\(11\)Di​j​\(T\)=\{dr:r∈Ri​j​\(T\)\},D\_\{ij\}\(T\)=\\left\\\{d\_\{r\}:r\\in R\_\{ij\}\(T\)\\right\\\},
and the temporal coverage of the pair is

\(12\)span⁡\(Ri​j​\(T\)\)=maxr∈Ri​j​\(T\)⁡tr−minr∈Ri​j​\(T\)⁡tr\.\\operatorname\{span\}\\\!\\left\(R\_\{ij\}\(T\)\\right\)=\\max\_\{r\\in R\_\{ij\}\(T\)\}t\_\{r\}\-\\min\_\{r\\in R\_\{ij\}\(T\)\}t\_\{r\}\.
A pair is eligible only if

\(13\)\|Ri​j​\(T\)\|≥30,\|Di​j​\(T\)\|≥5,span⁡\(Ri​j​\(T\)\)≥7​days\.\|R\_\{ij\}\(T\)\|\\geq 30,\\qquad\|D\_\{ij\}\(T\)\|\\geq 5,\\qquad\\operatorname\{span\}\\\!\\left\(R\_\{ij\}\(T\)\\right\)\\geq 7\\ \\text\{days\}\.
These requirements prevent a model from receiving an official comparison based on a short period, a small number of releases, or a narrow selection of datasets\. A pair that fails any requirement is treated as unavailable rather than as a tie\.

### E\.2\.Comparing point and probabilistic forecasts

The leaderboard includes both probabilistic models and models that produce only point forecasts\. Both model types are evaluated using MSE and CRPS under the same context\-only normalization\.

Letzr​hz\_\{rh\}be the normalized target at horizon stephh\. A probabilistic model provides a predictive meanμi​r​h\\mu\_\{irh\}and quantilesQi​r​h​\(τ\)Q\_\{irh\}\(\\tau\)for

\(14\)𝒬=\{0\.1,0\.2,…,0\.9\}\.\\mathcal\{Q\}=\\\{0\.1,0\.2,\\ldots,0\.9\\\}\.
Its MSE is computed from the predictive mean, while its CRPS is approximated using the quantile forecasts:

\(15\)CRPSi​r=2Hr​\|𝒬\|​∑h=1Hr∑τ∈𝒬ρτ​\(zr​h−Qi​r​h​\(τ\)\),\\mathrm\{CRPS\}\_\{ir\}=\\frac\{2\}\{H\_\{r\}\|\\mathcal\{Q\}\|\}\\sum\_\{h=1\}^\{H\_\{r\}\}\\sum\_\{\\tau\\in\\mathcal\{Q\}\}\\rho\_\{\\tau\}\\\!\\left\(z\_\{rh\}\-Q\_\{irh\}\(\\tau\)\\right\),
where

\(16\)ρτ\(u\)=u\(τ−𝕀\{u<0\}\)\\rho\_\{\\tau\}\(u\)=u\\left\(\\tau\-\\mathbb\{I\}\\\{u<0\\\}\\right\)
is the pinball loss\.

A point\-only model provides one forecastz^i​r​h\\hat\{z\}\_\{irh\}per horizon step\. We represent it as a degenerate predictive distribution by assigning the point forecast to every required quantile:

\(17\)Qi​r​hpoint​\(τ\)=z^i​r​h,∀τ∈𝒬\.Q^\{\\mathrm\{point\}\}\_\{irh\}\(\\tau\)=\\hat\{z\}\_\{irh\},\\qquad\\forall\\tau\\in\\mathcal\{Q\}\.
This conversion does not add artificial uncertainty: the model assigns all predictive mass to its point forecast\. Because the quantile grid is symmetric around0\.50\.5, substituting Eq\. \([17](https://arxiv.org/html/2608.17299#A5.E17)\) into Eq\. \([15](https://arxiv.org/html/2608.17299#A5.E15)\) gives

\(18\)CRPSi​rpoint=1Hr​∑h=1Hr\|zr​h−z^i​r​h\|\.\\mathrm\{CRPS\}^\{\\mathrm\{point\}\}\_\{ir\}=\\frac\{1\}\{H\_\{r\}\}\\sum\_\{h=1\}^\{H\_\{r\}\}\\left\|z\_\{rh\}\-\\hat\{z\}\_\{irh\}\\right\|\.
Hence, the CRPS of a point\-only model reduces to its normalized MAE\. Probabilistic models are evaluated on both the location and dispersion of their predictive distributions, while point models are evaluated as zero\-uncertainty distributions\. Since CRPS is defined for both ordinary and degenerate predictive distributions, their scores remain directly comparable\. Model output type and any benchmark\-side procedure used to construct quantiles are fixed before the target is observed\. Native quantiles are used when available; a declared point\-only output is converted using Eq\. \([17](https://arxiv.org/html/2608.17299#A5.E17)\)\.

Figure 16\.Overall model ranking in the latest leaderboard snapshot\. Bars show the dataset\-balanced pairwise win rate aggregated over eligible metric\-specific comparisons, historical releases, and opponents\. Chronos\-2 achieves the highest overall score, followed closely by TiRex\. Higher values are better\.Figure 17\.MSE\-based ranking within the point\-forecast track\. Bars report dataset\-balanced pairwise win rates over eligible historical releases and opponents\. TimesFM\-2\.5 achieves the highest score\. Higher values are better\.Figure 18\.CRPS\-based ranking restricted to the eight TSFMs whose model families natively support probabilistic forecasting through quantiles or forecast samples\. Chronos\-2 achieves the highest score\. Higher values are better\.
### E\.3\.Release\-level pairwise score

Both MSE and CRPS are lower\-is\-better\. For two error valuesaaandbb, let

\(19\)c⁡\(a,b\)=\{1,a<b,0\.5,a=b,0,a\>b\.c\(a,b\)=\\begin\{cases\}1,&a<b,\\\\ 0\.5,&a=b,\\\\ 0,&a\>b\.\\end\{cases\}
On shared releaserr, the score of modeliiagainst modeljjis

\(20\)si​j​r=12​\[c⁡\(MSEi​r,MSEj​r\)\+c⁡\(CRPSi​r,CRPSj​r\)\]\.s\_\{ijr\}=\\frac\{1\}\{2\}\\left\[c\\\!\\left\(\\mathrm\{MSE\}\_\{ir\},\\mathrm\{MSE\}\_\{jr\}\\right\)\+c\\\!\\left\(\\mathrm\{CRPS\}\_\{ir\},\\mathrm\{CRPS\}\_\{jr\}\\right\)\\right\]\.
The two metrics receive equal weight\. A model receivessi​j​r=1s\_\{ijr\}=1if it wins on both metrics andsi​j​r=0s\_\{ijr\}=0if it loses on both\. A split decision gives0\.50\.5; a win and a tie give0\.750\.75; and a loss and a tie give0\.250\.25\. Therefore,

\(21\)si​j​r∈\{0,0\.25,0\.5,0\.75,1\},sj​i​r=1−si​j​r\.s\_\{ijr\}\\in\\\{0,0\.25,0\.5,0\.75,1\\\},\\qquad s\_\{jir\}=1\-s\_\{ijr\}\.
The same rule applies to point–point, probabilistic–probabilistic, and point–probabilistic comparisons\. The MSE component compares central forecast accuracy, while the CRPS component compares the corresponding predictive distributions\.

### E\.4\.Dataset\-balanced pairwise win rate

Datasets resolve releases at different rates\. Pooling all releases directly would allow high\-frequency streams to dominate the ranking\. We therefore average scores within each shared dataset before averaging across datasets\.

Let

\(22\)Ri​j,d​\(T\)=\{r∈Ri​j​\(T\):dr=d\}R\_\{ij,d\}\(T\)=\\left\\\{r\\in R\_\{ij\}\(T\):d\_\{r\}=d\\right\\\}
be the releases shared by modelsiiandjjfor datasetdd\. Their dataset\-specific win rate is

\(23\)Wi​j,d​\(T\)=1\|Ri​j,d​\(T\)\|​∑r∈Ri​j,d​\(T\)si​j​r\.W\_\{ij,d\}\(T\)=\\frac\{1\}\{\|R\_\{ij,d\}\(T\)\|\}\\sum\_\{r\\in R\_\{ij,d\}\(T\)\}s\_\{ijr\}\.
The dataset\-balanced pairwise win rate is

\(24\)Wi​j​\(T\)=1\|Di​j​\(T\)\|​∑d∈Di​j​\(T\)Wi​j,d​\(T\)\.W\_\{ij\}\(T\)=\\frac\{1\}\{\|D\_\{ij\}\(T\)\|\}\\sum\_\{d\\in D\_\{ij\}\(T\)\}W\_\{ij,d\}\(T\)\.
Every shared dataset therefore receives equal weight, regardless of how many releases it produces\. Complementarity of the release\-level score implies

\(25\)Wj​i​\(T\)=1−Wi​j​\(T\)\.W\_\{ji\}\(T\)=1\-W\_\{ij\}\(T\)\.
This is visible in the lower\-left panel of Figure[15](https://arxiv.org/html/2608.17299#A4.F15): for example,WA​B=0\.62W\_\{AB\}=0\.62andWB​A=0\.38W\_\{BA\}=0\.38\. A value above0\.50\.5means that modeliiwins more often than modeljjafter dataset balancing; it is a ranking score, not a statistical significance test\.

### E\.5\.Model\-level ranking and status

LetNi​\(T\)N\_\{i\}\(T\)be the set of opponents for which modeliisatisfies Eq\. \([13](https://arxiv.org/html/2608.17299#A5.E13)\)\. The model\-level score is the macro\-average

\(26\)Wi​\(T\)=1\|Ni​\(T\)\|​∑j∈Ni​\(T\)Wi​j​\(T\)\.W\_\{i\}\(T\)=\\frac\{1\}\{\|N\_\{i\}\(T\)\|\}\\sum\_\{j\\in N\_\{i\}\(T\)\}W\_\{ij\}\(T\)\.
Each eligible opponent receives equal weight\. Thus, an opponent with a longer shared history does not dominate the final score merely because the pair has more releases\.

In Figure[15](https://arxiv.org/html/2608.17299#A4.F15), Model A has

\(27\)WA=WA​B\+WA​C\+WA​D3=0\.62\+0\.55\+0\.583=0\.583,W\_\{A\}=\\frac\{W\_\{AB\}\+W\_\{AC\}\+W\_\{AD\}\}\{3\}=\\frac\{0\.62\+0\.55\+0\.58\}\{3\}=0\.583,
which gives it rank 1\. Models are ordered by decreasingWi​\(T\)W\_\{i\}\(T\)\.

We construct an undirected comparison graph whose vertices are models and whose edges are eligible model pairs\. A model receives an official rank only if

1. \(1\)it has at least three eligible opponents; and
2. \(2\)it belongs to the main connected component of the graph\.

Models that fail either condition remain*Provisional*\. Missing pairs are omitted from Eq\. \([26](https://arxiv.org/html/2608.17299#A5.E26)\); they are never imputed as0\.50\.5\. This is why Model E in the figure has neither a model\-level score nor a rank, despite having completed some releases\.

Alongside the rank, the leaderboard reports the number of eligible opponents, shared releases, shared datasets, and covered time span\. These fields expose the amount of evidence supporting each result and distinguish an established ranking from a provisional one\.

## Appendix FOverall Model Ranking

Figures[16](https://arxiv.org/html/2608.17299#A5.F16)–[18](https://arxiv.org/html/2608.17299#A5.F18)provide three complementary views of the latest leaderboard snapshot up to cutoff timeTT\. Each score is a dataset\-balanced pairwise win rate, averaged with equal weight over eligible opponents and historical releases; higher values indicate better relative performance\. Figure[16](https://arxiv.org/html/2608.17299#A5.F16)shows the overall eligibility\-aware aggregation, where Chronos\-2 ranks first \(0\.6710\.671\), narrowly ahead of TiRex \(0\.6680\.668\), followed by TimesFM\-2\.5 \(0\.6090\.609\)\.

To separate point accuracy from distributional forecast quality, Figure[17](https://arxiv.org/html/2608.17299#A5.F17)ranks models using MSE within the point\-forecast track\. TimesFM\-2\.5 achieves the highest MSE\-based win rate \(0\.6800\.680\), followed by TiRex \(0\.6730\.673\) and Chronos\-2 \(0\.6680\.668\)\. Figure[18](https://arxiv.org/html/2608.17299#A5.F18)restricts the comparison to the eight TSFMs whose model families natively support probabilistic forecasts through quantiles or samples\. Under CRPS, Chronos\-2 ranks first \(0\.6340\.634\), followed by TiRex \(0\.6150\.615\) and Moirai\-2\.0 \(0\.5460\.546\)\.

Together, the three views show that point\-forecast accuracy and probabilistic forecast quality need not produce the same ordering: TimesFM\-2\.5 leads the MSE track, whereas Chronos\-2 leads the CRPS track and the overall aggregation\. TiRex remains consistently competitive across all three views\. Because the MSE and CRPS rankings use different eligible model sets and opponent groups, their absolute scores should be interpreted within each figure rather than compared directly across tracks\. These rankings summarize relative historical performance and do not by themselves establish statistical significance or long\-term rank stability\.