Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

# Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting
Source: [https://arxiv.org/html/2606.18367](https://arxiv.org/html/2606.18367)
Xian SunLingdong KongWei GaoYanhang LiZhichao FanZexin Zhuang

###### Abstract

Standard benchmarks evaluate time series foundation models \(TSFMs\) using aggregate metrics, but these can mask severe failures in critical operating regimes\. We introduce*regime\-stratified evaluation*and apply it to three TSFMs on two standard traffic speed benchmarks\. Traffic exhibits abrupt regime switching between free\-flow and congested states, producing bimodal speed distributions during transitions\. When we stratify by traffic regime, both accuracy and prediction\-interval coverage degrade sharply during transitions: transition\-regime MAE reaches 11 mph \(versus 3 mph overall\), and empirical coverage of 90% prediction intervals drops as low as 55%\. These failures are invisible in aggregate metrics because free\-flow observations dominate the sample\. A simple historical conditional baseline \(sampling from per\-sensor training distributions\) achieves better transition coverage than any TSFM, but has far worse overall accuracy\. We propose*bimodal mixture augmentation*\(BMA\), a post\-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline’s transition coverage while preserving the TSFM’s accuracy\. Our results suggest that TSFM benchmarks should incorporate regime\-aware evaluation to surface failures that aggregate metrics hide\.

time series foundation models, probabilistic forecasting, regime switching, benchmark evaluation

## 1Introduction

Traffic speed forecasting matters most during transitions between free\-flow and congestion\. In stable conditions, real\-time data suffices; during onset or dissipation of congestion, traffic can break down or recover, and systems that consume future speeds \(estimated times of arrival \(ETAs\), freight windows, emergency dispatch\) need forecasts with reliable uncertainty\. This is where forecasting earns its value\.

Time series foundation models \(TSFMs\) are positioned as general\-purpose probabilistic forecasters\(Ansariet al\.,[2024](https://arxiv.org/html/2606.18367#bib.bib1); Wooet al\.,[2024](https://arxiv.org/html/2606.18367#bib.bib2)\)\. TSFM\-Bench\(Liet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib3)\)and GIFT\-Eval\(Aksuet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib4)\)include METR\-LA \(207 loop detectors, Los Angeles\) and PEMS\-BAY \(325 detectors, San Francisco Bay Area\)\(Liet al\.,[2018](https://arxiv.org/html/2606.18367#bib.bib11)\), the standard benchmarks used by DCRNN\(Liet al\.,[2018](https://arxiv.org/html/2606.18367#bib.bib11)\), STGCN\(Yuet al\.,[2018](https://arxiv.org/html/2606.18367#bib.bib10)\), and subsequent graph neural network models\. These leaderboards stratify by domain and frequency but not by operating regime within a domain\. A model can appear accurate on average while failing in the regimes that matter most\.Adleret al\.\([2025](https://arxiv.org/html/2606.18367#bib.bib6)\)studied TSFM prediction\-interval coverage on six general datasets and found foundation models “consistently better calibrated,” but did not include high\-frequency traffic data\. Traffic at 5\-minute resolution exhibits*abrupt regime switching*between free\-flow \(∼\\sim65 mph\) and congested \(∼\\sim10–20 mph\) states\(Greenshields,[1935](https://arxiv.org/html/2606.18367#bib.bib7)\), driven by sharp capacity thresholds\. This is a property of traffic physics, not of any particular sensor\. During transitions, the true future speed distribution is bimodal \(Figure[1](https://arxiv.org/html/2606.18367#S2.F1)\): speed will either remain high or drop sharply\. TSFMs produce unimodal intervals centered between the modes \(∼\\sim30–45 mph\), a range that is transient and rarely sustained\.

#### Related work\.

Conformal methods have been extended to non\-stationary series:Gibbs and Candès \([2021](https://arxiv.org/html/2606.18367#bib.bib8)\)introduced adaptive conformal inference \(ACI\); recent work addresses regime switching\(Luet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib15)\)and correlated series\(Ciniet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib16)\)\. All adjust interval*width*but cannot correct distributional*shape*\. On the traffic side,Wuet al\.\([2023](https://arxiv.org/html/2606.18367#bib.bib17)\)andZhenget al\.\([2025](https://arxiv.org/html/2606.18367#bib.bib18)\)proposed uncertainty quantification for deep traffic models, but train domain\-specific architectures from scratch and do not evaluate zero\-shot TSFMs\. Widening a unimodal interval centered on 40 mph still misses outcomes near 15 or 65 mph; a different approach is needed\.

We present the first*regime\-stratified*evaluation of TSFMs on standard traffic speed benchmarks\. Our contributions are:\(1\)We show that aggregate metrics mask severe regime\-dependent failures in*both*accuracy and prediction\-interval coverage during traffic regime transitions\.\(2\)We show that a simple historical conditional baseline achieves better transition coverage than any TSFM but worse overall accuracy, revealing complementary strengths\.\(3\)We propose*bimodal mixture augmentation*\(BMA\), which combines TSFM forecasts with historical distributional knowledge, improving transition coverage by 3–19 pp without retraining\.

## 2Experimental Setup

#### Datasets\.

We use METR\-LA \(207 sensors, Los Angeles freeways, Mar–Jun 2012\) and PEMS\-BAY \(325 sensors, San Francisco Bay Area, Jan–May 2017\)\(Liet al\.,[2018](https://arxiv.org/html/2606.18367#bib.bib11)\), the standard benchmarks for spatio\-temporal traffic forecasting\. Both record speed at 5\-minute intervals\. We use the standard 70/10/20 train/validation/test split\.

#### Sensor selection\.

To focus on the regimes where coverage failures matter most, we rank sensors by their training\-set congestion frequency \(fraction of readings below 25 mph\) and form a pool of the top 50 per dataset\. Each random seed then subsamples 30 sensors from this pool, introducing variability across runs\. This uses only training data, avoiding test\-set leakage\.

#### Models\.

We evaluate three TSFMs in zero\-shot mode:Chronos\-T5\-Base\(Ansariet al\.,[2024](https://arxiv.org/html/2606.18367#bib.bib1)\), a T5\-based encoder\-decoder that tokenizes time series values and generates 100 distributional samples;Chronos\-Bolt\-Small\(Ansariet al\.,[2024](https://arxiv.org/html/2606.18367#bib.bib1)\), an efficient quantile\-based variant from which we draw pseudo\-samples via interpolation; andMoirai\-1\.1\-R\-Base\(Wooet al\.,[2024](https://arxiv.org/html/2606.18367#bib.bib2)\), which uses any\-variate attention with mixture outputs and generates 100 samples\. Baselines includeACI\-LR\(linear regression wrapped with adaptive conformal inference\(Gibbs and Candès,[2021](https://arxiv.org/html/2606.18367#bib.bib8)\)\) and ahistorical conditionalbaseline that draws 100 samples directly from the per\-sensor empirical distributionP​\(speedt\+h∣speedt\)P\(\\text\{speed\}\_\{t\+h\}\\mid\\text\{speed\}\_\{t\}\)computed from training data, with no forecasting model involved\. The historical conditional tests whether BMA’s TSFM anchor adds value beyond the historical lookup it uses\.

#### Evaluation protocol\.

From the pool of 50 congestion\-prone sensors, we draw three different random subsets of 30 sensors each \(using seeds 42, 43, 44\) so that means and standard deviations reflect variability across sensor selections\. For each subset, we evaluate on 50 test windows at horizonsh∈\{3,6,12\}h\\in\\\{3,6,12\\\}steps \(15, 30, 60 minutes ahead\), with 14 hours of context for foundation models\. We report*mean absolute error*\(MAE\) and*empirical coverage*: the fraction of true observations that fall within the model’s 90% prediction interval\. Differences between coverage rates are reported in percentage points \(pp\)\.

#### Regime detection\.

We classify each \(window, sensor\) target period using Highway Capacity Manual\(Transportation Research Board,[2022](https://arxiv.org/html/2606.18367#bib.bib14)\)thresholds:*free\-flow*\(all steps\>\>55 mph, LOS A/B\),*congested*\(all steps<<25 mph, LOS E/F\), or*transition*\(mixed/intermediate, LOS C/D\)\. These thresholds align with the bimodal speed distributions in Figure[1](https://arxiv.org/html/2606.18367#S2.F1): individual congestion\-prone sensors show modes near∼\\sim18 and∼\\sim65 mph with a valley near 30–40 mph\.

![Refer to caption](https://arxiv.org/html/2606.18367v1/x1.png)Figure 1:Speed distributions from training data\.Top:pooled across all sensors, the distribution appears unimodal near 60–65 mph\.Bottom:individual congestion\-prone sensors reveal strong bimodality with modes near∼\\sim18 mph and∼\\sim64–66 mph\. Red and green lines mark the regime thresholds \(25 and 55 mph\); shaded regions indicate the congested \(red\) and free\-flow \(green\) regimes\.
#### Post\-hoc correction methods\.

We compare four post\-hoc correction strategies, all applied to the cached TSFM samples without retraining:

*Global ACI \(G\-ACI\)*: Adaptive conformal inference\(Gibbs and Candès,[2021](https://arxiv.org/html/2606.18367#bib.bib8)\)tracks a miscoverage rateαt\\alpha\_\{t\}updated sequentially: when recent observations fall outside the interval,αt\\alpha\_\{t\}shrinks, widening future intervals\. Becauseαt\\alpha\_\{t\}is shared across regimes, free\-flow observations \(near\-nominal\) dilute the transition\-regime signal\.

*Regime\-Conditional ACI \(R\-ACI\)*: Maintains separateα\\alphaparameters per regime, so transition\-regime errors drive wider transition intervals without affecting free\-flow\. Both ACI variants adjust interval*width*only and cannot shift probability mass toward a missing mode\.

*Bimodal Mixture Augmentation \(BMA\)*: Our proposed method\. From training data, we precompute per\-sensor historical transition probabilities:P​\(speedt\+h∈R∣speedt\)P\(\\text\{speed\}\_\{t\+h\}\\in R\\mid\\text\{speed\}\_\{t\}\)for each regimeRRand horizonhh\. At test time, we replace a fraction of the TSFM’s 100 forecast samples with draws from this historical conditional distribution, scaled by the transition probability\. This injects the missing mode \(e\.g\., congested speeds∼\\sim15 mph\) when historical data indicates a non\-trivial chance of regime switching\. The mixing weightw∈\[0\.1,0\.5\]w\\in\[0\.1,0\.5\]is selected on 10 held\-out windows and evaluated on all remaining data\.

*BMA \+ ACI*: BMA fixes the distributional*shape*by injecting the missing mode, but may leave residual over\- or undercoverage because the mixing weightwwis fixed per configuration\. Global ACI then adjusts the interval*width*sequentially to close any remaining coverage gap\. The two corrections are complementary: shape first, then width\.

## 3Results

### 3\.1Aggregate Metrics Hide Regime\-Dependent Failures

Table[1](https://arxiv.org/html/2606.18367#S3.T1)reports MAE stratified by traffic regime athh=12\. TSFMs achieve much lower overall MAE than the historical conditional baseline \(5\.8 vs 12\.9 mph on METR\-LA\), confirming that the TSFM provides genuine forecasting value\. But during transitions, all methods converge to similar error \(∼\\sim10–11 mph\) because neither the TSFM nor the historical lookup can predict which direction traffic will move\. Free\-flow dominates the sample, so the aggregate is pulled toward the easy regime\.

Table 1:MAE \(mph\) athh=12 stratified by traffic regime\. TSFMs achieve much lower overall MAE than the historical conditional baseline, but all methods fail similarly during transitions\.
### 3\.2Calibration Failures Concentrate in Transitions

We construct 90% prediction intervals for each model \(the standard operating point in TSFM benchmarks\(Adleret al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib6); Aksuet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib4)\)\), meaning the interval should contain the true speed 90% of the time, and measure how often it actually does \(empirical coverage\)\. We then stratify by traffic regime\. Figure[2](https://arxiv.org/html/2606.18367#S3.F2)reveals a striking pattern\. Duringfree\-flow\(\>\>55 mph\), TSFMs achieve coverage near the 90% target \(bars near zero\)\. Duringcongestion\(<<25 mph\), coverage drops moderately\. Thetransitionregime \(25–55 mph\) is the worst: Chronos\-Bolt achieves only 54\.9% on PEMS\-BAY athh=12, a 35 pp gap\. The ACI\-LR baseline \(linear regression with adaptive conformal inference\) is even worse during transitions on PEMS\-BAY despite overcovering in free\-flow, confirming that width\-only adjustment cannot fix this problem\.

![Refer to caption](https://arxiv.org/html/2606.18367v1/x2.png)Figure 2:Coverage gap by traffic regime at three horizons\. Each bar shows empirical coverage minus the 90% nominal target; the zero line means the model’s intervals achieve exactly 90% coverage\. Negative bars indicate undercoverage \(intervals too narrow\)\. All models fail duringtransitions, with gaps reaching−\-35 pp \(Chronos\-Bolt\) and−\-48 pp \(ACI\-LR\) on PEMS\-BAY\. ACI\-LR overcorrects in free\-flow \(bars above zero\) but has the worst transition coverage, illustrating that widening intervals alone cannot fix the shape mismatch\.The transition regime’s severe undercoverage matters most when operators need to act\. Consider a freeway segment at 55 mph at 6:45 AM as morning demand builds: it will either hold near free\-flow or collapse to 15 mph once a downstream bottleneck activates\. During stable conditions real\-time observations suffice, but at the edge of breakdown only a forward\-looking prediction can distinguish a 4\-minute from a 16\-minute traversal\. The root cause is a shape mismatch: the true future speed distribution during transitions is bimodal \(Figure[1](https://arxiv.org/html/2606.18367#S2.F1)\), but in zero\-shot mode all three TSFMs produce effectively unimodal prediction intervals centered between the modes\. The congested regime is itself unimodal \(concentrated near 10–20 mph\), so the problem there is merely one of interval width; transitions are fundamentally bimodal, making this a problem of distributional*shape*\.

### 3\.3Post\-Hoc Coverage Correction

Table[2](https://arxiv.org/html/2606.18367#S3.T2)compares the four post\-hoc methods on transition\-regime coverage athh=12\.

Global ACIbarely moves the needle: \+0\.5–1\.2 pp on transition coverage\. This is expected: widening a unimodal interval centered at the wrong location cannot recover the missing mode\.

Regime\-Conditional ACIdoes slightly better \(\+2\.3–2\.9 pp on PEMS\-BAY\) because the transition\-specificα\\alphaadapts more aggressively\. But it still only adjusts width, not shape\.

BMAproduces the largest improvements, ranging from \+2\.6 pp \(Moirai, PEMS\-BAY\) to \+16\.3 pp \(Chronos\-Bolt, PEMS\-BAY\)\. The improvement is largest for Chronos\-Bolt, which has the worst native coverage: BMA lifts its PEMS\-BAY transition coverage from 54\.9% to 71\.2%\.BMA \+ ACIadds a further 1–2 pp on top of BMA by adjusting residual width, achieving the best overall results\. The two methods are complementary: BMA corrects distributional*shape*\(injecting the missing mode\) while ACI corrects residual*width*\(scaling the interval to account for remaining miscoverage\)\. We use global ACI rather than regime\-conditional ACI in the combination because once BMA has corrected the shape mismatch, the residual coverage error is uniform enough across regimes that per\-regime width adjustment offers no additional benefit\.

Table 2:Transition\-regime coverage \(%\) athh=12, 90% nominal\. The historical conditional baseline \(Hist\.\) achieves high transition coverage by sampling from training data\. BMA approaches this level while preserving the TSFM’s superior point accuracy\.The historical conditional baseline \(Table[2](https://arxiv.org/html/2606.18367#S3.T2), “Hist\.”\) achieves 81–82% transition coverage by construction, since it samples directly from the bimodal historical distribution\. But it has poor overall MAE \(Table[1](https://arxiv.org/html/2606.18367#S3.T1)\), because it is a lookup table, not a forecaster\. TSFMs provide the opposite: good overall accuracy but poor transition coverage\. BMA combines both: it keeps the TSFM’s point predictions while borrowing the historical distribution’s shape, approaching the historical baseline’s transition coverage without sacrificing overall accuracy\.

BMA uses per\-sensor historical data, just as ACI uses past prediction residuals\. The TSFM itself remains zero\-shot; BMA is a post\-hoc correction applied to its outputs, analogous to how conformal methods adjust intervals using domain data\. BMA does not degrade coverage in other regimes because the replacement fraction is modulated by the transition probability: in stable free\-flow \(P​\(congested\)≈0P\(\\text\{congested\}\)\\approx 0\), no samples are replaced\. Results are stable across mixing weights \(w∈\[0\.2,0\.5\]w\\in\[0\.2,0\.5\]\) and degrade only beloww=0\.1w=0\.1\.

## 4Discussion and Limitations

#### Implications for benchmarks\.

Our results suggest that aggregate evaluation metrics are insufficient for domains with regime switching\. Regime\-stratified evaluation is straightforward to implement and could be applied to any domain where physical thresholds produce distinct operating regimes: electricity spot prices switch between normal and spike regimes\(Huisman and Mahieu,[2003](https://arxiv.org/html/2606.18367#bib.bib13); Lehnaet al\.,[2025](https://arxiv.org/html/2606.18367#bib.bib9)\), and wind power near turbine cut\-in speeds exhibits similar bimodality\.

#### BMA and the zero\-shot boundary\.

BMA uses per\-sensor historical data from the training set, just as conformal methods use past residuals\. The TSFM itself remains zero\-shot; BMA is applied to its outputs\. An alternative is to fine\-tune the TSFM on traffic data directly, which may teach it to produce bimodal predictions natively\. Whether fine\-tuning closes the gap is an open empirical question\. BMA remains relevant when fine\-tuning is impractical or when using closed\-source models\.

#### Other limitations\.

Our evaluation uses 30 congestion\-prone sensors per dataset with univariate forecasts; spatial information from upstream and downstream sensors could improve both accuracy and coverage\. BMA’s mixing weight is tuned on 10 held\-out windows; a validation\-set approach would be more principled\. The interval width increase \(∼\\sim50–80%\) trades sharpness for coverage; whether this is acceptable depends on the application\.

## References

- C\. Adler, Y\. Chang, F\. Draxler, S\. Abdi, and P\. Smyth \(2025\)Beyond accuracy: are time series foundation models well\-calibrated?\.arXiv preprint arXiv:2510\.16060\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3),[§3\.2](https://arxiv.org/html/2606.18367#S3.SS2.p1.3)\.
- T\. Aksu, G\. Woo, J\. Liu, X\. Liu, C\. Liu, S\. Savarese, C\. Xiong, and D\. Sahoo \(2025\)GIFT\-Eval: a benchmark for general time series forecasting model evaluation\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3),[§3\.2](https://arxiv.org/html/2606.18367#S3.SS2.p1.3)\.
- A\. F\. Ansari, L\. Stella, C\. Turkmen, X\. Zhang, P\. Mercado, H\. Sez, O\. Shchur, S\. S\. Rangapuram, S\. P\. Arango, S\. Kapoor,et al\.\(2024\)Chronos: learning the language of time series\.Transactions on Machine Learning Research\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3),[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px3.p1.1)\.
- A\. Cini, A\. Jenkins, D\. Mandic, C\. Alippi, and F\. M\. Bianchi \(2025\)Relational conformal prediction for correlated time series\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.SS0.SSS0.Px1.p1.1)\.
- I\. Gibbs and E\. Candès \(2021\)Adaptive conformal inference under distribution shift\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px3.p1.1),[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px6.p2.3)\.
- B\. D\. Greenshields \(1935\)A study of traffic capacity\.Highway Research Board Proceedings14,pp\. 448–477\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3)\.
- R\. Huisman and R\. Mahieu \(2003\)Regime jumps in electricity prices\.Energy Economics25\(5\),pp\. 425–434\.Cited by:[§4](https://arxiv.org/html/2606.18367#S4.SS0.SSS0.Px1.p1.1)\.
- M\. Lehna, C\. Sieberichs, and C\. Scholz \(2025\)Assessing time series foundation models for probabilistic electricity price forecasting\.Energies18\(23\),pp\. 6269\.Cited by:[§4](https://arxiv.org/html/2606.18367#S4.SS0.SSS0.Px1.p1.1)\.
- Y\. Li, R\. Yu, C\. Shahabi, and Y\. Liu \(2018\)Diffusion convolutional recurrent neural network: data\-driven traffic forecasting\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3),[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px1.p1.1)\.
- Z\. Li, X\. Qiu, P\. Chen, Y\. Wang, H\. Cheng, Y\. Shu, J\. Hu, C\. Guo, A\. Zhou, Q\. Wen, C\. S\. Jensen, and B\. Yang \(2025\)TSFM\-Bench: a comprehensive and unified benchmark of foundation models for time series forecasting\.InACM SIGKDD Conference on Knowledge Discovery and Data Mining,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3)\.
- E\. D\. Lu, C\. Findling, M\. Clausel, A\. Leite, W\. Gong, and P\. Kersaudy \(2025\)Adaptive regime\-switching forecasts with distribution\-free uncertainty: deep switching state\-space models meet conformal prediction\.arXiv preprint arXiv:2512\.03298\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.SS0.SSS0.Px1.p1.1)\.
- Transportation Research Board \(2022\)Highway capacity manual\.7th edition,National Academies of Sciences, Engineering, and Medicine,Washington, DC\.Cited by:[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px5.p1.4)\.
- G\. Woo, C\. Liu, A\. Kumar, C\. Xiong, S\. Savarese, and D\. Sahoo \(2024\)Unified training of universal time series forecasting transformers\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3),[§2](https://arxiv.org/html/2606.18367#S2.SS0.SSS0.Px3.p1.1)\.
- Y\. Wu, Y\. Ye, A\. Zeb, J\. J\.Q\. Yu, and Z\. Wang \(2023\)Adaptive modeling of uncertainties for traffic forecasting\.IEEE Transactions on Intelligent Transportation Systems\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.SS0.SSS0.Px1.p1.1)\.
- B\. Yu, H\. Yin, and Z\. Zhu \(2018\)Spatio\-temporal graph convolutional networks: a deep learning framework for traffic forecasting\.InInternational Joint Conference on Artificial Intelligence,pp\. 3634–3640\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.p2.3)\.
- V\. Z\. Zheng, S\. Choi, and L\. Sun \(2025\)Probabilistic traffic forecasting with dynamic regression\.Transportation Science59\(4\),pp\. 689–707\.Cited by:[§1](https://arxiv.org/html/2606.18367#S1.SS0.SSS0.Px1.p1.1)\.