HART：混合注意力-RWKV 时间序列基础模型

# HART: Hybrid Attention-RWKV Time series Foundation Model
Source: [https://arxiv.org/html/2610.12883](https://arxiv.org/html/2610.12883)
Qingwen LinEmail:[qingwenlin@foxmail\.com](mailto:[email protected])Affiliation:School of Computer Science, Guangdong University of TechnologyBoyan XuEmail:[hpakyim@gmail\.com](mailto:[email protected])Affiliation:School of Computer Science, Guangdong University of TechnologyShifeng XieEmail:[shifeng\.xie@telecom\-paris\.fr](mailto:[email protected])Affiliation:Huawei Noah’s Ark LabAffiliation:LIPADE, Université Paris Cité, Paris, FranceZhifeng HaoEmail:[haozhifeng@stu\.edu\.cn](mailto:[email protected])Affiliation:School of Computer Science, Guangdong University of TechnologyAffiliation:College of Science, Shantou UniversityRuichu Cai††thanks:˜˜Corresponding authorEmail:[cairuichu@gmail\.com](mailto:[email protected])Affiliation:School of Computer Science, Guangdong University of TechnologyAffiliation:Peng Cheng Laboratory

###### Abstract

Non\-autoregressive multi\-step forecasting \(NAF\) enables time series foundation models to predict an entire future horizon in a single forward pass\. Despite its strong sequence modeling capabilities, RWKV\-7 exhibits long\-horizon degradation in our NAF experiments\. In this setup, future patches are represented as missing inputs, requiring the recurrent state to evolve across the prediction horizon without new observations\. Our analysis of RWKV\-7’s generalized delta rule suggests a mechanism we term Placeholder Idling Decay \(PID\): repeated state transitions can attenuate historical information when placeholder\-driven writes provide insufficient compensation\. Motivated by this analysis, we propose HART, a Hybrid Attention\-RWKV Time\-series Foundation Model that combines direct access to historical tokens with recurrent temporal modeling by alternating attention and RWKV blocks\. Layer\-wise analysis reveals reduced state\-injection norms in most layers, while transition\-matrix norms remain similar between historical and prediction regions, consistent with PID\. Controlled experiments on GIFT\-Eval and TIME\-Benchmark show that HART improves over pure RWKV at medium and long horizons while maintaining comparable short\-horizon performance, and achieves better aggregate performance than a comparably sized pure Transformer\. Using 36M parameters, approximately 1/28 of Toto\-2\.0\-1B’s, HART achieves comparable long\-horizon MASE on GIFT\-Eval \(0\.752 vs\. 0\.754\)\.

## 1Introduction

Figure 1:Qualitative NAF predictions on a synthetic signal\. Gray: history and ground truth; colored: median predictions with quantile\.Time series forecasting underpins applications such as energy dispatch, traffic control, and financial risk management[Rizki et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib1);[Lee and Kim \(2026\)](https://arxiv.org/html/2610.12883#bib.bib2);[Wang et al\. \(2026b\)](https://arxiv.org/html/2610.12883#bib.bib4);[Liao et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib3);[Wang et al\. \(2026a\)](https://arxiv.org/html/2610.12883#bib.bib5);[Kaleem et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib18);[Xu et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib19)\. Non\-autoregressive multi\-step forecasting \(NAF\) generates all future steps in a single forward pass, avoiding the error accumulation of autoregressive decoding, and has become the dominant inference paradigm in modern time series foundation models[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6);[Feng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib8);[Ansari et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib12);[Khwaja et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib13)\. NAF, however, imposes one constraint: the prediction region contains no observed input, and the model must rely on its internal state or on retrieval from the historical region\.

Recurrent neural networks \(RNNs\) carry a temporal inductive bias, modeling temporal dependencies through stepwise state updates\. RWKV is one of the strongest pure\-RNN language models, matching Transformers of similar scale in language modeling[Peng et al\. \(2023\)](https://arxiv.org/html/2610.12883#bib.bib14);[Peng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib16);[Peng et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib15), with long\-range modeling ability validated at scale\. On these grounds, it should be competitive in long\-horizon time series forecasting\. However, our NAF experiments show that RWKV\-7 exhibits systematic degradation at long horizons\.

Figure[1](https://arxiv.org/html/2610.12883#S1.F1)shows that RWKV\-7 collapses in amplitude over the prediction region on a synthetic signal, while the Transformer shows no comparable collapse under the same data and training configuration\. This contrast motivates examining how recurrent state updates interact with placeholder inputs\. In the NAF prediction region, placeholders carry positional information but contain no observed future values\. When placeholder\-driven writes provide insufficient compensation for repeated state transitions, historical representations in a fixed\-capacity recurrent state can be progressively attenuated, and predictions can lose their dynamic structure and drift toward the sequence mean\. We term this mechanism*Placeholder Idling Decay*\(PID\)\.

Attention can directly query historical patches without traversing the recurrent state’s stepwise decay path\. Based on this distinction, we proposeHART\(Hybrid Attention\-RWKV Time\-series Foundation Model\): RWKV blocks perform temporal compression and local modeling in the historical region, while Transformer blocks retrieve historical information in the prediction region through global attention, bypassing the stepwise decay path\. The two are stacked alternately, providing both local modeling and global access\.

Layer\-wise analysis shows reduced state\-injection norms in most layers, while transition\-matrix norms remain similar between historical and prediction regions, consistent with PID\. On GIFT\-Eval and TIME\-Benchmark, HART significantly improves over pure RWKV at medium and long horizons while maintaining comparable short\-horizon performance\. HART also outperforms the controlled pure Transformer baseline on aggregate metrics, supporting the benefit of combining attention with recurrence\. TiRex also shows a similar collapse in the placeholder region in Figure[1](https://arxiv.org/html/2610.12883#S1.F1); HART\-36M, with a parameter count and pre\-training task aligned with TiRex, achieves better long\-horizon and overall metrics than TiRex \(GIFT Long: 0\.752 vs 0\.767, All: 0\.710 vs 0\.716; TIME Long: 0\.7233 vs 0\.7608, All: 0\.6658 vs 0\.6831\), with comparable short\-horizon performance\.

Our main contributions are as follows:

1. 1\.We characterize Placeholder Idling Decay \(PID\) in RWKV\-7 under NAF through generalized delta\-rule analysis and layer\-wise norm measurements\.
2. 2\.We proposeHART, which bypasses the recurrent decay path by alternately stacking RWKV and Transformer blocks\.
3. 3\.On GIFT\-Eval and TIME\-Benchmark, HART achieves better long\-horizon and overall MASE than comparably sized baselines\.

## 2Related Work

Time series forecasting is widely applied in energy, transportation, finance, and healthcare, where deep learning has become the mainstream approach[Asghar et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib27);[Zhu et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib28);[Suresh \(2025\)](https://arxiv.org/html/2610.12883#bib.bib29);[Zhu et al\. \(2023\)](https://arxiv.org/html/2610.12883#bib.bib30);[Lu et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib26);[Shao et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib31);[Cao et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib32)\. We review time series foundation models, with an emphasis on recurrent architectures, followed by hybrid recurrent\-attention models\.

### 2\.1Time Series Foundation Models

Early forecasting was dominated by autoregressive models, such as Lag\-Llama[Rasul et al\. \(2023\)](https://arxiv.org/html/2610.12883#bib.bib23), TimesFM[Das et al\. \(2023\)](https://arxiv.org/html/2610.12883#bib.bib22), and YingLong[Xu et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib19), but their error accumulates with the forecast horizon\. NAF generates all future steps in a single forward pass and substantially alleviates this problem, and has been widely adopted by TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6), Toto[Khwaja et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib13), Chronos[Ansari et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib12), Xihe[Sun et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib10), TempoPFN[Moroshan et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib33), TabPFN\-TS[Hoo et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib11), Moirai[Liu et al\. \(2025a\)](https://arxiv.org/html/2610.12883#bib.bib7), Sundial[Liu et al\. \(2025c\)](https://arxiv.org/html/2610.12883#bib.bib9), and Kairos[Feng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib8)\.

TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6)is the most closely related baseline: it is built on xLSTM, has a parameter count comparable to our 36M model, and uses the same CPM pre\-training task and Pinball loss\. It is among the strongest fixed\-capacity recurrent models on NAF benchmarks\. We compare HART with TiRex using results from the official benchmark leaderboards, as reported in Section[4\.4](https://arxiv.org/html/2610.12883#S4.SS4)\.

RWKV is a pure RNN architecture that matches Transformers of similar scale in language modeling[Peng et al\. \(2023\)](https://arxiv.org/html/2610.12883#bib.bib14);[Peng et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib15);[Peng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib16)\. Several works have applied RWKV to time series[Rizki et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib1);[Lee and Kim \(2026\)](https://arxiv.org/html/2610.12883#bib.bib2);[Wang et al\. \(2026b\)](https://arxiv.org/html/2610.12883#bib.bib4), but they target specific applications and do not develop it into a time series foundation model; other linear recurrent models such as xLSTM\-Mixer[Kraus et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib25)show similar potential\. We examine how NAF inference affects recurrent state updates in RWKV\-7, particularly in the placeholder\-driven prediction region\. This question motivates our analysis of PID in Section[3\.1](https://arxiv.org/html/2610.12883#S3.SS1)\.

### 2\.2Hybrid Recurrent\-Attention Architectures

Mixing recurrent models with attention is a well\-established architecture family in the language domain, including Kimi K3[Team et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib36)and Jamba[Lenz et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib35); ST\-MambaSync[Shao et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib34)applies a similar idea to time series\.

Hybrid architectures have been developed to balance modeling capacity and inference efficiency\. Our work uses the PID analysis of RWKV\-7 under NAF to motivate hybrid design: attention bypasses the stepwise decay path in the prediction region, while RWKV preserves the local temporal inductive bias in the historical region\. The detailed design is given in Section[3\.2](https://arxiv.org/html/2610.12883#S3.SS2)\.

## 3Method

This section first formulates the NAF setting and analyzes PID in RWKV\-7, then introduces theHARTarchitecture motivated by this analysis, and finally presents the training objective and inference pipeline\.

### 3\.1Non\-Autoregressive Forecasting and Placeholder Idling Decay

##### Formalization of Non\-Autoregressive Multi\-Step Forecasting

Given a historical sequenceX1:C=\(x1,…,xC\)X\_\{1:C\}=\(x\_\{1\},\\ldots,x\_\{C\}\), the goal of NAF is to predict the nextHHstepsX^C\+1:C\+H\\hat\{X\}\_\{C\+1:C\+H\}in a single forward pass\. The model input is the concatenation of historical patches and prediction placeholders: for each position in the prediction region, the original scalar input is set to00and the missingness indicator is set to11, marking the position as a placeholder to be predicted\. The original input, observation mask, and missingness mask are concatenated and passed through the patch embedding layer to obtain the vector representation of that position\. The placeholder embedding incorporates the masks and is not constrained to be zero; the prediction region contains no observed future values\.PPdenotes the patch length,Th​i​s​t=⌈C/P⌉T\_\{hist\}=\\lceil C/P\\rceilandTp​r​e​d=⌈H/P⌉T\_\{pred\}=\\lceil H/P\\rceilthe numbers of historical and prediction patches,DDthe hidden dimension,hhthe number of heads, anddh=D/hd\_\{h\}=D/hthe per\-head dimension\. This placeholder construction is used throughout the paper\. NAF excludes feeding predicted values back into later forecast inputs\. A recurrent backbone can still perform successive state updates within the single forecasting pass\.

##### State Update under the Generalized Delta Rule

RWKV\-7[Peng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib16)maintains a recurrent state matrixSt∈ℝdh×dhS\_\{t\}\\in\\mathbb\{R\}^\{d\_\{h\}\\times d\_\{h\}\}as a fast\-weight memory for key\-value associations and updates it via the generalized delta rule \(GDR\):

St=St−1​\(diag⁡\(wt\)−κ^t⊤​\(at⊙κ^t\)\)⏟Mt\+vt⊤​k~t⏟KtS\_\{t\}=S\_\{t\-1\}\\underbrace\{\\bigl\(\\mathrm\{diag\}\(w\_\{t\}\)\-\\hat\{\\kappa\}\_\{t\}^\{\\top\}\(a\_\{t\}\\odot\\hat\{\\kappa\}\_\{t\}\)\\bigr\)\}\_\{M\_\{t\}\}\+\\underbrace\{v\_\{t\}^\{\\top\}\\tilde\{k\}\_\{t\}\}\_\{K\_\{t\}\}\(1\)
All vectors in this update are per\-head row vectors:ata\_\{t\}is the in\-context learning\-rate vector,κ^t\\hat\{\\kappa\}\_\{t\}is the normalized removal key, andk~t\\tilde\{k\}\_\{t\}is the replacement key\. The transition matrixMtM\_\{t\}propagates and transforms the previous state, while the outer\-product termKt=vt⊤​k~tK\_\{t\}=v\_\{t\}^\{\\top\}\\tilde\{k\}\_\{t\}writes new key\-value associations\. These quantities are computed from the layer’s token representations through token shift, learned projections, and nonlinear transformations; their parameterization is given in Appendix[E](https://arxiv.org/html/2610.12883#A5)\.

##### Historical\-State Propagation across the Prediction Region

Lett0=Th​i​s​tt\_\{0\}=T\_\{hist\}denote the last historical patch and define the ordered productΠi:j=MiMi\+1⋯Mj\\Pi\_\{i:j\}=M\_\{i\}M\_\{i\+1\}\\cdots M\_\{j\}, withΠi:j=I\\Pi\_\{i:j\}=Iwheni\>ji\>j\. Unrolling Equation[1](https://arxiv.org/html/2610.12883#S3.E1)overmmprediction patches gives

St0\+m=St0Πt0\+1:t0\+m\+∑j=1mKt0\+jΠt0\+j\+1:t0\+m\.S\_\{t\_\{0\}\+m\}=S\_\{t\_\{0\}\}\\Pi\_\{t\_\{0\}\+1:t\_\{0\}\+m\}\+\\sum\_\{j=1\}^\{m\}K\_\{t\_\{0\}\+j\}\\Pi\_\{t\_\{0\}\+j\+1:t\_\{0\}\+m\}\.\(2\)The first term propagates the boundary state through all prediction transitions; each later write passes through the remaining transitions\. This identity alone implies no contraction, and later writes can also carry history supplied by preceding layers\. Equal transition norms can nevertheless act on different state directions: for example,diag⁡\(1,0\)\\operatorname\{diag\}\(1,0\)anddiag⁡\(0,1\)\\operatorname\{diag\}\(0,1\)both have unit Frobenius norm but preserve different columns of a right\-multiplied state\. The product in Equation[2](https://arxiv.org/html/2610.12883#S3.E2)therefore matters beyond individual matrix magnitudes\.

##### Degradation under NAF and PID

Under NAF, the model receives observed values in the historical region and placeholders in the prediction region\. Although placeholder representations can carry historical context from preceding layers, no new observations enter during prediction\. Equation[2](https://arxiv.org/html/2610.12883#S3.E2)separates this historical\-state propagation from subsequent writes\. If these transitions attenuate that contribution and placeholder\-driven writes provide insufficient compensation, useful historical information can diminish asTp​r​e​dT\_\{pred\}grows\. We term this potential failure mode*Placeholder Idling Decay*\(PID\)\. PID concerns insufficient write compensation during recurrent state evolution and does not require accelerated decay in the prediction region\.

To examine this hypothesis, we compare the Frobenius norms ofKtK\_\{t\}andMtM\_\{t\}between the historical and prediction regions in Section[4\.2](https://arxiv.org/html/2610.12883#S4.SS2)\. Reduced write norms alongside similar transition norms would be consistent with PID\. These measurements characterize matrix magnitudes; they do not directly establish historical information loss or the contraction rate of transition products\.

Our mechanistic analysis is restricted to RWKV\-7\. The xLSTM\-based TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6)exhibits qualitatively similar degradation in Figure[1](https://arxiv.org/html/2610.12883#S1.F1), but whether the same mechanism applies to TiRex or other recurrent architectures remains unverified\.

### 3\.2HART: A Mechanism\-Driven Hybrid Architecture

Figure 2:Architecture of the proposed hybrid backbone\. \(a\) Overall pipeline: time\-patch embedding feeds into an alternating stack of Attention and RWKV blocks \(LLtimes\), followed by the output head\. \(b\) Attention block: RMSNorm, gated self\-attention, and feed\-forward with residual connections\. \(c\) RWKV block: LayerNorm, RWKV\-TimeMIX \(generalized delta rule\), and feed\-forward with residual connections\.The PID analysis in Section[3\.1](https://arxiv.org/html/2610.12883#S3.SS1)motivates direct access to historical tokens during NAF prediction\. Attention provides this access without propagating information through every intervening recurrent state; the Transformer in Figure[1](https://arxiv.org/html/2610.12883#S1.F1)also shows no comparable collapse in the illustrated example\.HARTcombines recurrent temporal modeling with global information retrieval by alternating RWKV and Transformer blocks\. Both block types process the full token sequence: RWKV updates a recurrent representation, while attention allows prediction tokens to retrieve historical information directly\.

##### Overall Architecture

As shown in Figure[2](https://arxiv.org/html/2610.12883#S3.F2),HARTconsists of three components: a patch embedding layer, an alternately stacked backbone, and an output head\. The backbone containsLLalternately stacked module groups, each consisting of one Transformer block followed by one RWKV block, giving2​L2Llayers in total\. This order gives each RWKV block attention\-conditioned representations at prediction positions\. Its writes can therefore depend on retrieved history even when the original future values are masked\. Repeating the pair makes this route available at successive depths, supplementing the historical information carried by the recurrent state\. We make only minimal modifications to the Transformer and RWKV blocks; the complete structure is given in Appendix[E](https://arxiv.org/html/2610.12883#A5)\.

##### Patch Embedding Layer

The input series is first normalized byzz\-score and then segmented into patches of lengthPP\. Each patch is concatenated with the observation mask and the missingness mask, and then projected to dimensionDDthrough the skip embedding of Moirai[Liu et al\. \(2025a\)](https://arxiv.org/html/2610.12883#bib.bib7)\. The resulting embeddings serve as tokens for the subsequent Attention and RWKV blocks; the complete formulation is given in Appendix[E](https://arxiv.org/html/2610.12883#A5.SS0.SSS0.Px1)\.

##### Responsibilities of Transformer and RWKV Blocks

Transformer blocks perform global information retrieval\. In the prediction region, the attention mechanism can associate placeholder positions with all historical tokens, bypassing the stepwise decay path of the recurrent state and retrieving relevant historical patterns\. These blocks adopt gated self\-attention and SwiGLU feed\-forward networks; the complete formulation is given in Appendix[E\.1](https://arxiv.org/html/2610.12883#A5.SS1)\.

RWKV blocks provide recurrent temporal modeling\. In the historical region, the GDR accumulates and updates observational information in the state matrix\. In the prediction region, the inputs to each RWKV block have been contextualized by the preceding Transformer block, allowing historical information retrieved by attention to enter the recurrent computation\. These blocks adopt RWKV\-TimeMIX \(GDR\) and SwiGLU feed\-forward networks; the complete formulation is given in Appendix[E\.2](https://arxiv.org/html/2610.12883#A5.SS2)\.

##### Output Head

The backbone output is passed through LayerNorm, mapped by a residual MLP to\|Qquantile\|=9\|Q\_\{\\text\{quantile\}\}\|=9quantile predictions, and restored to the original scale by inversezz\-score normalization\.

### 3\.3Training Objective and Pipeline

##### Pre\-training Task

We adopt contiguous patch masking \(CPM\) with next\-patch prediction, following TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6); related NAF pre\-training approaches include TiRex, Toto, and TempoPFN[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6);[Khwaja et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib13);[Moroshan et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib33)\. During training, contiguous groups of patches are randomly masked, and the model reconstructs the masked regions from the available context\. Masked patches are replaced by placeholders in the input, and their ground\-truth values serve as supervision in the loss; unmasked positions also contribute through next\-patch prediction\. CPM exposes the model to the NAF inference mode during training: it learns to reconstruct a masked future from arbitrary historical positions rather than extrapolating only at the sequence terminus\. The detailed construction is given in Appendix[D\.2](https://arxiv.org/html/2610.12883#A4.SS2)\.

##### Quantile Loss

The model outputs\|Qquantile\|=9\|Q\_\{\\text\{quantile\}\}\|=9equidistant quantiles\{0\.1,0\.2,…,0\.9\}\\\{0\.1,0\.2,\\ldots,0\.9\\\}\. Given the ground truthyty\_\{t\}and the quantile predictionsy^tq\\hat\{y\}\_\{t\}^\{q\}, we adopt the Pinball loss:

ℒ=∑t∈ℳ∑q∈Qquantile\{q⁡\(yt−y^tq\)if​y^tq≤yt\(1−q\)​\(y^tq−yt\)otherwise\\mathcal\{L\}=\\sum\_\{t\\in\\mathcal\{M\}\}\\sum\_\{q\\in Q\_\{\\text\{quantile\}\}\}\\begin\{cases\}q\\,\(y\_\{t\}\-\\hat\{y\}\_\{t\}^\{q\}\)&\\text\{if \}\\hat\{y\}\_\{t\}^\{q\}\\leq y\_\{t\}\\\\ \(1\-q\)\\,\(\\hat\{y\}\_\{t\}^\{q\}\-y\_\{t\}\)&\\text\{otherwise\}\\end\{cases\}\(3\)
whereℳ\\mathcal\{M\}denotes the set of positions whose original values exist, so that training combines both the CPM and autoregressive tasks\. The loss is computed on the standardized scale, reducing its dependence on the original magnitude of each series\.

##### Inference Pipeline

At inference, the model switches to the NAF mode: the input is the concatenation of historical tokens and prediction placeholders, and all prediction steps are generated in a single forward pass\. After inference, the mean and standard deviation of the historical sequence are used for inversezz\-score normalization to restore the output to the original scale\. Training hyperparameters, pre\-training data composition, and processing details are given in Appendix[D](https://arxiv.org/html/2610.12883#A4)\.

## 4Experiments

### 4\.1Evaluation Setup

All models are evaluated on two benchmarks: GIFT\-Eval[Aksu et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib17)and TIME\-Benchmark[Qiao et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib39)\. The GIFT\-Eval results cover 97 evaluation entries across Short, Medium, and Long horizon groups; per\-dataset results and horizon details are provided in Appendix[A](https://arxiv.org/html/2610.12883#A1)\. TIME\-Benchmark contains 98 task\-frequency combinations \(39 unique datasets after deduplication\), comprising 50 Short tasks \(averaging 30 steps\), 24 Medium tasks \(averaging 122 steps\), and 24 Long tasks \(averaging 367 steps\)\. We adopt MASE \(Mean Absolute Scaled Error\) as the primary metric, computed relative to the seasonal naïve forecast following the official protocol, and aggregate results using the geometric mean\.

We train three models under strictly controlled conditions: HART, pure RWKV, and pure Transformer\. All operate in the NAF mode: the input is the concatenation of historical patches and prediction placeholders, and all prediction steps are produced in a single forward pass\. Except for the core blocks, all other components remain identical across the three models; detailed training configurations are provided in Appendix[D](https://arxiv.org/html/2610.12883#A4)\. Results for external baselines are obtained from the official leaderboards of GIFT\-Eval and TIME\-Benchmark\.

### 4\.2Mechanistic Evidence of PID

Following the diagnostic analysis in Section[3\.1](https://arxiv.org/html/2610.12883#S3.SS1), we compare write and transition magnitudes across the historical and prediction regions\. Table[1](https://arxiv.org/html/2610.12883#S4.T1)reports the layer\-wise mean Frobenius norms of a pure RWKV model, averaged over 4,096 synthetic sequences\. For this diagnostic, the history contains 16,384 time steps and the prediction region contains 1,024, using patches of length 16\. Norms are averaged over heads and sequences before the regional comparison; the temporal traces in the appendix show variation within these averages\.

Table 1:Layer\-wise mean Frobenius norms ofKtK\_\{t\}andMtM\_\{t\}in the historical and prediction regions, averaged over 4,096 sequences\. Lower ratios indicate smaller prediction\-region norms relative to the historical region\.The write norms exhibit a clear layer\-wise pattern\. In the first seven layers, the prediction\-to\-history ratio ofKtK\_\{t\}falls between 0\.60 and 0\.87; in Layers 8 to 11, the ratios range from 0\.84 to 0\.97\. Layer 12 exhibits a reversal, with a ratio of 1\.34\. Thus, prediction\-region mean write norms are lower in the first eleven layers, with the strongest relative reductions occurring in the earlier layers\.

Across all layers, theMtM\_\{t\}ratio stays within 1\.00 to 1\.03, indicating similar transition\-matrix norms in the two regions\. Together with the lowerKtK\_\{t\}norms in most layers, this pattern is consistent with the write\-compensation hypothesis of PID\. SimilarMtM\_\{t\}norms alone do not establish equal decay rates or equal retention of historical information\.

One possible explanation for the layer\-wise variation is that deeper layers receive representations already transformed by preceding layers, so their responses to placeholders can differ from those of the initial layers\. The Layer 12 reversal shows that the change in write magnitude is not uniform across depth\.

Reduced write norms do not imply a complete loss of forecasting capability\. RWKV remains comparable to Toto\-2\.0\-22M on Long \(GIFT Long: 0\.770 vs 0\.781; TIME Long: 0\.755 vs 0\.736\), while HART improves over RWKV on both benchmarks as reported in Section[4\.3](https://arxiv.org/html/2610.12883#S4.SS3)\. These performance comparisons complement the layer\-wise norm diagnostics\. The full experimental setup and additional visualizations are provided in Appendix[B](https://arxiv.org/html/2610.12883#A2)\.

### 4\.3Main Results

Table[2](https://arxiv.org/html/2610.12883#S4.T2)reports geometric\-mean MASE for HART, pure RWKV, and pure Transformer on GIFT\-Eval and TIME\-Benchmark\. We assess the paired median difference between HART and each baseline using bootstrap confidence intervals\. Cells marked with †indicate that the 95% interval excludes zero\. These intervals describe paired median differences, rather than uncertainty in the reported geometric means\.

Table 2:Self\-developed architectures on GIFT\-Eval and TIME\-Benchmark \(MASE, geometric mean\)\. Lower is better; bold marks the lowest displayed value in each column, including ties\. †indicates that the paired median difference between HART and the baseline has a 95% bootstrap confidence interval excluding zero\.##### HART versus Pure RWKV

HART improves over pure RWKV on Medium and Long horizons while maintaining comparable reported MASE on Short\. Neither comparison between HART and a baseline on Short is marked as significant; the reported geometric means range from 0\.689 to 0\.695 on GIFT and from 0\.618 to 0\.620 on TIME\. On GIFT Medium, HART reduces MASE from 0\.749 for pure RWKV to 0\.727 \(a gain of 0\.022\); on GIFT Long, it reduces MASE from 0\.770 to 0\.752 \(a gain of 0\.018\)\. TIME\-Benchmark shows gains of 0\.011 on Medium and 0\.032 on Long\. The paired median comparisons favor HART on Medium and Long in both benchmarks, as indicated by the bootstrap intervals\.

The Medium and Long improvements are consistent with the motivation for providing direct access to history during placeholder\-based prediction\. The gain is larger on Medium than Long for GIFT, and larger on Long than Medium for TIME\. The shared training configuration supports a comparison of the architectures under the reported setup, but does not isolate PID as the sole cause of the gains\.

##### HART versus Pure Transformer

The comparison with the pure Transformer evaluates the benefit of combining attention with recurrent blocks\. On aggregate metrics, HART reduces MASE relative to the pure Transformer on both benchmarks: from 0\.721 to 0\.710 on GIFT All, and from 0\.670 to 0\.666 on TIME All; in both cases the 95% bootstrap confidence intervals for the paired median differences exclude zero, supporting the aggregate comparison in favor of HART under the reported evaluation\.

At the horizon level, the HART–Transformer paired median difference is significant on GIFT Long and TIME Medium\. The other horizon\-specific Transformer comparisons are not marked as significant\. This variation suggests that the benefit of the hybrid depends on the evaluated tasks; the aggregate results alone do not identify the contribution of a particular signal property\.

The synthetic experiments in Appendix[C](https://arxiv.org/html/2610.12883#A3)illustrate this task dependence\. On the multi\-component SE4 signal, RWKV outperforms both HART and Transformer under NAF \(Long: 1\.058, 1\.148, and 1\.151, respectively\)\. On SE3, HART and Transformer outperform RWKV \(Long: 0\.265, 0\.306, and 0\.581, respectively\)\. These results support combining the two modeling approaches while showing that HART does not dominate every signal regime\.

##### Sensitivity to Available Future Observations

The same pre\-trained checkpoints, without fine\-tuning, also compare NAF with teacher\-forced AR supplied with real future inputs\. On Long, RWKV’s NAF–AR gaps for SE1, SE2, and SE3 are 0\.060, 0\.095, and 0\.316; HART’s corresponding gaps are 0\.001, 0\.030, and 0\.096, and the Transformer’s are 0\.002, 0\.016, and 0\.134\. RWKV has the largest gap for each of these signals; SE5 and SE6 have absolute gaps of at most 0\.001 at the reported precision\. This signal\-dependent sensitivity complements the SE4 comparison of absolute forecasting accuracy\. Because teacher forcing changes the available information, this control neither isolates PID nor represents deployable autoregressive forecasting\.

### 4\.4Benchmark Comparison

Table[3](https://arxiv.org/html/2610.12883#S4.T3)compares HART with mainstream foundation models on GIFT\-Eval and TIME\-Benchmark\.

Table 3:External baselines on GIFT\-Eval and TIME\-Benchmark \(MASE, geometric mean\)\. Lower is better; bold marks the lowest displayed value in each column, including ties\.TiRex is a closely related recurrent baseline: it has a parameter count comparable to HART \(35M vs 36M\) and uses CPM pre\-training and Pinball loss\. HART supplements recurrence with attention, whereas TiRex uses xLSTM\. Figure[1](https://arxiv.org/html/2610.12883#S1.F1)shows qualitatively similar long\-horizon degradation in TiRex, without establishing a shared mechanism\. Among TiRex, RWKV, and HART, TiRex has the lowest Short MASE \(GIFT Short: 0\.685 vs RWKV 0\.689 and HART 0\.689; TIME Short: 0\.617 vs 0\.620 and 0\.618\)\. Its difference between the reported Short and Long MASE is larger than HART’s \(GIFT: 0\.082 vs 0\.063; TIME: 0\.144 vs 0\.106\)\. On Long, HART outperforms both recurrent baselines \(GIFT Long: TiRex 0\.767, HART 0\.752, and RWKV 0\.770; TIME Long: 0\.761, 0\.723, and 0\.755, respectively\)\. These leaderboard comparisons establish the observed ranking across horizon groups, but do not isolate the effects of architecture, training data, or inference design\.

In the cross\-scale comparison, HART\-36M uses approximately 1/28 of Toto\-2\.0\-1B’s parameters while performing on par on GIFT Long \(0\.752 vs 0\.754\)\. Xihe\-1\.5B uses direct multi\-horizon prediction heads and outperforms HART on GIFT Medium \(0\.718 vs 0\.727\), while HART achieves lower MASE on Long \(0\.752 vs 0\.763\)\. These results show competitive long\-horizon accuracy at a smaller parameter count under the reported benchmark evaluation\.

## 5Conclusion

We investigate RWKV\-7 under non\-autoregressive multi\-step forecasting, where the recurrent state evolves through a prediction region containing placeholders rather than new observations\. Our generalized delta\-rule analysis motivates Placeholder Idling Decay \(PID\): historical information can be attenuated when repeated state transitions are not sufficiently compensated by placeholder\-driven writes\. Layer\-wise measurements show lower write norms in most layers and similar transition norms across the historical and prediction regions, consistent with this hypothesis\.

Motivated by this analysis,HARTalternates Transformer and RWKV blocks to combine direct access to historical tokens with recurrent temporal modeling\. Controlled experiments on GIFT\-Eval and TIME\-Benchmark show improvements over pure RWKV on Medium and Long horizons, comparable Short performance, and lower aggregate MASE than a comparably sized pure Transformer\. Our mechanistic analysis is restricted to RWKV\-7; applying the transition–injection decomposition to other recurrent architectures requires separate validation\.

## 6Limitations and Future Work

We analyze the PID hypothesis in RWKV\-7 only\. The norm diagnostics do not directly measure retained historical information, and the forecasting comparisons do not isolate PID from all other architectural effects\. Figure[1](https://arxiv.org/html/2610.12883#S1.F1)shows that the xLSTM\-based TiRex also exhibits similar long\-horizon amplitude degradation, but whether it shares the same write\-compensation mechanism requires per\-architecture analysis\. We adopt a 1:1 stacking configuration; whether other ratios or orderings yield further gains remains open\.

## 7AI Use Statement

We used DeepSeek\-V4 Pro to assist with writing, polishing, and translation of this manuscript\. We also used OpenAI Codex for rhetorical editing, consistency checks, and revision suggestions\. The authors take full responsibility for the content and final version\.

## References

- Aksuet al\.\(2024\)T\. Aksu, G\. Woo, J\. Liu, X\. Liu, C\. Liu, S\. Savarese, C\. Xiong, and D\. SahooGift\-eval: a benchmark for general time series forecasting model evaluation\.arXiv preprint arXiv:2410\.10393\.Cited by:[1st item](https://arxiv.org/html/2610.12883#A4.I1.i1.p1.1),[§4\.1](https://arxiv.org/html/2610.12883#S4.SS1.p1.1)\.
- Ansariet al\.\(2025\)A\. F\. Ansari, O\. Shchur, J\. Küken, A\. Auer, B\. Han, P\. Mercado, S\. S\. Rangapuram, H\. Shen, L\. Stella, X\. Zhang,et al\.Chronos\-2: from univariate to universal forecasting\.arXiv preprint arXiv:2510\.15821\.Cited by:[2nd item](https://arxiv.org/html/2610.12883#A4.I1.i2.p1.1),[item 1](https://arxiv.org/html/2610.12883#A4.I2.i1.p1.1),[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1)\.
- Asgharet al\.\(2024\)R\. Asghar, F\. R\. Fulginei, M\. Quercio, and A\. MahrouchArtificial neural networks for photovoltaic power forecasting: a review of five promising models\.IEEe Access12,pp\. 90461–90485\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Aueret al\.\(2026\)A\. Auer, P\. Podest, D\. Klotz, S\. Böck, G\. Klambauer, and S\. HochreiterTirex: zero\-shot forecasting across long and short horizons with enhanced in\-context learning\.Advances in Neural Information Processing Systems38,pp\. 57529–57580\.Cited by:[2nd item](https://arxiv.org/html/2610.12883#A4.I1.i2.p1.1),[§D\.2](https://arxiv.org/html/2610.12883#A4.SS2.p1.1),[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p2.1),[§3\.1](https://arxiv.org/html/2610.12883#S3.SS1.SSS0.Px4.p3.1),[§3\.3](https://arxiv.org/html/2610.12883#S3.SS3.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.13.1)\.
- Caoet al\.\(2026\)J\. Cao, X\. Sheng, and Z\. DuanMSTF: a multi\-scale long\-term spatio\-temporal traffic flow forecasting framework\.IEEE Internet of Things Journal\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Daset al\.\(2023\)A\. Das, W\. Kong, R\. Sen, and Y\. ZhouA decoder\-only foundation model for time\-series forecasting\.arXiv preprint arXiv:2310\.10688\.Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.5.1)\.
- Fenget al\.\(2025\)K\. Feng, S\. Lan, Y\. Fang, W\. He, L\. Ma, X\. Lu, and K\. RenKairos: towards adaptive and generalizable time series foundation models\.arXiv e\-prints,pp\. arXiv–2509\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.10.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.9.1)\.
- Hooet al\.\(2025\)S\. B\. Hoo, S\. Müller, D\. Salinas, and F\. HutterFrom tables to time: extending tabpfn\-v2 to time series forecasting\.arXiv preprint arXiv:2501\.02945\.Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1)\.
- Kaleemet al\.\(2026\)M\. B\. Kaleem, H\. Li, Z\. Zhu, X\. Chen, Y\. Fan, L\. Yan, and W\. LiuLag\-llama\-based remaining useful life prediction for lithium\-ion batteries of electric vehicles with auto\-correlation analysis\.IEEE Transactions on Transportation Electrification\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1)\.
- Khwajaet al\.\(2026\)E\. Khwaja, C\. Lettieri, G\. Woo, E\. Belouadah, M\. Cenac, G\. Jarry, E\. Paquin, X\. Zhao, V\. Zhukov, O\. Abou\-Amal,et al\.Toto 2\.0: time series forecasting enters the scaling era\.arXiv preprint arXiv:2605\.20119\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12883#S3.SS3.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.14.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.15.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.16.1)\.
- Kimet al\.\(2021\)T\. Kim, J\. Kim, Y\. Tae, C\. Park, J\. Choi, and J\. ChooReversible instance normalization for accurate time\-series forecasting against distribution shift\.InInternational conference on learning representations,Cited by:[Appendix E](https://arxiv.org/html/2610.12883#A5.SS0.SSS0.Px1.p1.1)\.
- Krauset al\.\(2026\)M\. Kraus, F\. Divo, D\. S\. Dhami, and K\. KerstingXlstm\-mixer: multivariate time series forecasting by mixing via scalar memories\.Advances in Neural Information Processing Systems38,pp\. 6547–6581\.Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Lee and Kim \(2026\)J\. Lee and J\. KimFast and effective prediction of spatiotemporal nonlinear dynamics using the tailored rwkv architecture\.JPhys Photonics8\(3\),pp\. 035002\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Lenzet al\.\(2025\)B\. Lenz, O\. Lieber, A\. Arazi, A\. Bergman, A\. Manevich, B\. Peleg, B\. Aviram, C\. Almagor, C\. Fridman, D\. Padnos,et al\.Jamba: hybrid transformer\-mamba language models\.InInternational Conference on Learning Representations,Vol\.2025,pp\. 67959–67984\.Cited by:[§2\.2](https://arxiv.org/html/2610.12883#S2.SS2.p1.1)\.
- Liaoet al\.\(2026\)L\. Liao, C\. Wang, J\. Wang, Y\. Liao, and Y\. LaiWavelet\-decoupled spatiotemporal network for stock return prediction\.Entropy28\(5\),pp\. 548\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1)\.
- Liuet al\.\(2025a\)C\. Liu, T\. Aksu, J\. Liu, X\. Liu, H\. Yan, Q\. Pham, S\. Savarese, D\. Sahoo, C\. Xiong, and J\. LiMoirai 2\.0: when less is more for time series forecasting\.arXiv preprint arXiv:2511\.11698\.Cited by:[Appendix E](https://arxiv.org/html/2610.12883#A5.SS0.SSS0.Px1.p4.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[§3\.2](https://arxiv.org/html/2610.12883#S3.SS2.SSS0.Px2.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.11.1)\.
- Liuet al\.\(2025b\)J\. Liu, J\. Su, X\. Yao, Z\. Jiang, G\. Lai, Y\. Du, Y\. Qin, W\. Xu, E\. Lu, J\. Yan,et al\.Muon is scalable for llm training\.arXiv preprint arXiv:2502\.16982\.Cited by:[§D\.3](https://arxiv.org/html/2610.12883#A4.SS3.p1.1)\.
- Liuet al\.\(2025c\)Y\. Liu, G\. Qin, Z\. Shi, Z\. Chen, C\. Yang, X\. Huang, J\. Wang, and M\. LongSundial: a family of highly capable time series foundation models\.InProceedings of the 42nd International Conference on Machine Learning,A\. Singh, M\. Fazel, D\. Hsu, S\. Lacoste\-Julien, F\. Berkenkamp, T\. Maharaj, K\. Wagstaff, and J\. Zhu \(Eds\.\),Proceedings of Machine Learning Research, Vol\.267,pp\. 39295–39317\.External Links:[Link](https://proceedings.mlr.press/v267/liu25be.html)Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.6.1)\.
- Luet al\.\(2025\)K\. Lu, M\. Huo, Y\. Li, Q\. Zhu, and Z\. ChenCt\-patchtst: channel\-time patch time\-series transformer for long\-term renewable energy forecasting\.In2025 10th International Conference on Computer and Information Processing Technology \(ISCIPT\),pp\. 86–95\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Moroshanet al\.\(2025\)V\. Moroshan, J\. Siems, A\. Zela, T\. Carstensen, and F\. HutterTempoPFN: towards synthetic pre\-training of linear rnns for zero\-shot time series forecasting\.InEurIPS 2025 Workshop: AI for Tabular Data,Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12883#S3.SS3.SSS0.Px1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.8.1)\.
- Naouret al\.\(2026\)E\. L\. Naour, T\. Nabil, and A\. PetraliaTS\-icl: a flexible time\-indexed foundation model for time series via in\-context learning\.arXiv preprint arXiv:2606\.05878\.Cited by:[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.12.1)\.
- Penget al\.\(2023\)B\. Peng, E\. Alcaide, Q\. Anthony, A\. Albalak, S\. Arcadinho, S\. Biderman, H\. Cao, X\. Cheng, M\. Chung, L\. Derczynski,et al\.Rwkv: reinventing rnns for the transformer era\.InFindings of the association for computational linguistics: EMNLP 2023,pp\. 14048–14077\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Penget al\.\(2024\)B\. Peng, D\. Goldstein, Q\. Anthony, A\. Albalak, E\. Alcaide, S\. Biderman, E\. Cheah, X\. Du, T\. Ferdinan, H\. Hou,et al\.Eagle and finch: rwkv with matrix\-valued states and dynamic recurrence\.arXiv preprint arXiv:2404\.05892\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Penget al\.\(2025\)B\. Peng, R\. Zhang, D\. Goldstein, E\. Alcaide, X\. Du, H\. Hou, J\. Lin, J\. Liu, J\. Lu, W\. Merrill,et al\.Rwkv\-7" goose" with expressive dynamic state evolution\.arXiv preprint arXiv:2503\.14456\.Cited by:[§E\.2](https://arxiv.org/html/2610.12883#A5.SS2.p1.1),[§1](https://arxiv.org/html/2610.12883#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1),[§3\.1](https://arxiv.org/html/2610.12883#S3.SS1.SSS0.Px2.p1.1)\.
- Qiaoet al\.\(2026\)Z\. Qiao, S\. Pan, A\. Wang, V\. Zhukova, Y\. Liu, X\. Jiang, Q\. Wen, M\. Long, M\. Jin, and C\. LiuIt’s time: towards the next generation of time series forecasting benchmarks\.arXiv preprint arXiv:2602\.12147\.Cited by:[§4\.1](https://arxiv.org/html/2610.12883#S4.SS1.p1.1)\.
- Qiuet al\.\(2026\)Z\. Qiu, Z\. Wang, B\. Zheng, Z\. Huang, K\. Wen, S\. Yang, R\. Men, L\. Yu, F\. Huang, S\. Huang,et al\.Gated attention for large language models: non\-linearity, sparsity, and attention\-sink\-free\.Advances in Neural Information Processing Systems38,pp\. 100092–100118\.Cited by:[§E\.1](https://arxiv.org/html/2610.12883#A5.SS1.p1.1)\.
- Rasulet al\.\(2023\)K\. Rasul, A\. Ashok, A\. R\. Williams, A\. Khorasani, G\. Adamopoulos, R\. Bhagwatkar, M\. Biloš, H\. Ghonia, N\. Hassen, A\. Schneider,et al\.Lag\-llama: towards foundation models for time series forecasting\.InR0\-FoMo: Robustness of Few\-shot and Zero\-shot Learning in Large Foundation Models,Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1)\.
- Ribeiroet al\.\(2021\)A\. H\. Ribeiro, G\. Paixao, E\. M\. Lima, M\. Horta Ribeiro, M\. M\. Pinto Filho, P\. R\. Gomes, D\. M\. Oliveira, W\. Meira Jr, T\. B\. Schon, and A\. L\. P\. RibeiroCODE\-15%: a large scale annotated dataset of 12\-lead ecgs, june 2021\.URL https://doi\. org/10\.5281/zenodo4916206\.Cited by:[3rd item](https://arxiv.org/html/2610.12883#A4.I1.i3.p1.1)\.
- Rizkiet al\.\(2026\)A\. Rizki, A\. Echchatbi, and H\. YantourRWKV\-cvm: gated cross\-variate mixing for multivariate power load forecasting\.Electricity7\(2\),pp\. 58\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Shaoet al\.\(2024\)Z\. Shao, M\. G\. Bell, Z\. Wang, D\. G\. Geers, H\. Xi, and J\. GaoST\-mamba: spatial\-temporal selective state space model for traffic flow prediction\.arXiv preprint arXiv:2404\.13257\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Shaoet al\.\(2025\)Z\. Shao, Z\. Wang, X\. Yao, M\. G\. Bell, and J\. GaoST\-mambasync: complement the power of mamba and transformer fusion for less computational cost in spatial–temporal traffic forecasting\.Information Fusion117,pp\. 102872\.Cited by:[§2\.2](https://arxiv.org/html/2610.12883#S2.SS2.p1.1)\.
- Suet al\.\(2024\)J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. LiuRoformer: enhanced transformer with rotary position embedding\.Neurocomputing568,pp\. 127063\.Cited by:[§E\.1\.1](https://arxiv.org/html/2610.12883#A5.SS1.SSS1.p3.1)\.
- Sunet al\.\(2025\)Y\. Sun, Y\. Fang, Z\. Zhu, J\. Li, Y\. Liu, Q\. Deng, J\. Zhou, H\. Yu, X\. Lu, and L\. MaXihe: scalable zero\-shot time series learner via hierarchical interleaved block attention\.arXiv preprint arXiv:2510\.21795\.Cited by:[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.7.1)\.
- Suresh \(2025\)V\. SureshBenchmarking transformer variants for hour\-ahead pv forecasting: patchtst with adaptive conformal inference\.Energies18\(18\),pp\. 5000\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Teamet al\.\(2026\)K\. Team, T\. Bai, Y\. Bai, Y\. Bao, J\. Cai, X\. Cai, P\. Cao, Y\. Cao, Z\. Chai, Y\. Charles,et al\.Kimi k3: open frontier intelligence\.arXiv preprint arXiv:2607\.24653\.Cited by:[§2\.2](https://arxiv.org/html/2610.12883#S2.SS2.p1.1)\.
- Wanget al\.\(2026a\)W\. Wang, L\. Li, D\. Su, X\. Zhang, H\. Tong, T\. Shao, and J\. FanEnhancing daily runoff prediction via uniform design and meta\-learning integrated hyperparameter optimization embedded in transformer\.Hydrology\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1)\.
- Wanget al\.\(2025\)X\. Wang, T\. Zhou, J\. Gao, B\. Ding, and J\. ZhouOutput scaling: yinglong\-delayed chain of thought in a large pretrained time series forecasting model\.arXiv preprint arXiv:2506\.11029\.Cited by:[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.3.1),[Table 3](https://arxiv.org/html/2610.12883#S4.T3.2.1.4.1)\.
- Wanget al\.\(2026b\)Z\. Wang, Z\. Chen, R\. Li, Z\. Zhang, and Z\. YangSynergizing global pattern learning and time order characterization in mobile channel prediction: an rwkv\-based approach\.InICC 2026\-IEEE International Conference on Communications,pp\. 1–6\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p3.1)\.
- Xuet al\.\(2024\)P\. Xu, X\. Zheng, T\. Gao, Y\. Wang, J\. Yin, J\. Zhang, X\. Zhang, S\. Luo, Z\. Wang, Z\. Zhang,et al\.YingLong\-weather: ai\-based limited area models for forecasting of non\-precipitation surface meteorological variables\.arXiv preprint arXiv:2401\.16254\.Cited by:[§1](https://arxiv.org/html/2610.12883#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12883#S2.SS1.p1.1)\.
- Zhuet al\.\(2023\)J\. Zhu, Z\. Zhao, X\. Zheng, Z\. An, Q\. Guo, Z\. Li, J\. Sun, and Y\. GuoTime\-series power forecasting for wind and solar energy based on the sl\-transformer\.Energies16\(22\),pp\. 7610\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.
- Zhuet al\.\(2025\)Z\. Zhu, T\. Wu, K\. Xie, M\. Wang, J\. Li, and A\. AhmedLong\-term forecasting of solar and wind energy patterns using deep learning\.PeerJ Computer Science11,pp\. e3114\.Cited by:[§2](https://arxiv.org/html/2610.12883#S2.p1.1)\.

## Appendix ADetailed Per\-Dataset Results

This appendix provides per\-dataset MASE scores on GIFT\-Eval and TIME\-Benchmark for all models reported in the main benchmark table\. Lower MASE indicates better accuracy\. All values are rounded to three decimal places\. Results for external baselines are taken from the official leaderboards of the two benchmarks; HART, RWKV, and Transformer are trained and evaluated by us under identical settings\.

Table 4:Per\-dataset MASE on GIFT\-Eval long\-term datasets \(21 datasets\)\.Table 5:Per\-dataset MASE on GIFT\-Eval medium\-term datasets \(21 datasets\)\.Table 6:Per\-dataset MASE on GIFT\-Eval short\-term datasets \(Part 1: A to M, 28 datasets\)\.Table 7:Per\-dataset MASE on GIFT\-Eval short\-term datasets \(Part 2: M–Z, 27 datasets\)\.Table 8:Per\-dataset MASE on TIME\-Benchmark long\-term tasks \(24 tasks\)\.Table 9:Per\-dataset MASE on TIME\-Benchmark medium\-term tasks \(24 tasks\)\.Table 10:Per\-dataset MASE on TIME\-Benchmark short\-term tasks \(Part 1: A to M, 25 tasks\)\.Table 11:Per\-dataset MASE on TIME\-Benchmark short\-term tasks \(Part 2: O–Z, 25 tasks\)\.### A\.1GIFT\-Eval

Table[4](https://arxiv.org/html/2610.12883#A1.T4)reports MASE on all 21 long\-term datasets\. Long\-term tasks use prediction lengths between 720 and 900 time steps\.

Table[5](https://arxiv.org/html/2610.12883#A1.T5)reports MASE on all 21 medium\-term datasets\. Medium\-term tasks use prediction lengths of approximately 500 time steps\.

Table[6](https://arxiv.org/html/2610.12883#A1.T6)and Table[7](https://arxiv.org/html/2610.12883#A1.T7)report MASE on all 55 short\-term datasets\. Short\-term tasks use prediction lengths of approximately 30 time steps\.

### A\.2TIME\-Benchmark

Table[8](https://arxiv.org/html/2610.12883#A1.T8)reports MASE on all 24 long\-term tasks\. Table[9](https://arxiv.org/html/2610.12883#A1.T9)reports MASE on all 24 medium\-term tasks\. Table[10](https://arxiv.org/html/2610.12883#A1.T10)and Table[11](https://arxiv.org/html/2610.12883#A1.T11)report MASE on all 50 short\-term tasks\.

## Appendix BMechanistic Evidence of PID

This section provides the full experimental setup and temporal evolution evidence for the norm analysis in Section[4\.2](https://arxiv.org/html/2610.12883#S4.SS2)\. The table in the main text reports region\-averaged ratios between the historical and prediction regions; here we present two figures showing the temporal evolution of the corresponding norms near the history–prediction boundary\.

### B\.1Experimental Setup

The synthetic signal consists of three consecutive sinusoidal segments designed to mimic structural changes commonly found in real\-world time series\. The three segments have periods of 100, 250, and 80 time steps, with amplitudes sampled from𝒰⁡\(0\.8,1\.2\)\\mathcal\{U\}\(0\.8,1\.2\),𝒰⁡\(1\.0,1\.5\)\\mathcal\{U\}\(1\.0,1\.5\), and𝒰⁡\(0\.6,1\.0\)\\mathcal\{U\}\(0\.6,1\.0\), respectively, and phases sampled from𝒰⁡\(0,2​π\)\\mathcal\{U\}\(0,2\\pi\)\. Gaussian noise with a standard deviation of 0\.1 is added to the entire signal\. Each sequence has a total length of 17,408 time steps\. With patch sizeP=16P=16, the historical region contains 1,024 patches and the prediction region contains 64 patches, yielding 1,088 patches in total\.

Inference follows the NAF paradigm: the input is the concatenation of historical patches and prediction placeholder patches, and all prediction steps are produced in a single forward pass\. We generate 4,096 independent sequences, each with randomly sampled phases and amplitudes\. During layer\-by\-layer forward propagation, we record‖Kt‖F\\\|K\_\{t\}\\\|\_\{F\}and‖Mt‖F\\\|M\_\{t\}\\\|\_\{F\}at every patch position, average them across the 8 heads, and then average across all 4,096 sequences, yielding a norm matrix of dimension 12 layers×\\times1,088 patches\. The visualization focuses on the boundary region: we display the last 64 historical patches and all 64 prediction patches, with a red dashed line marking the boundary\.

### B\.2Temporal Evolution ofKtK\_\{t\}andMtM\_\{t\}

Figures[3](https://arxiv.org/html/2610.12883#A2.F3)and[4](https://arxiv.org/html/2610.12883#A2.F4)show the temporal evolution of theKtK\_\{t\}andMtM\_\{t\}norms near the boundary, respectively\.

##### KtK\_\{t\}: lower prediction\-region averages with layer\-dependent dynamics\.

Figure 3:Temporal evolution of‖Kt‖F\\\|K\_\{t\}\\\|\_\{F\}across all 12 layers\. Blue: historical region; orange: prediction region; red dashed line marks the boundary\. Prediction\-region averages are lower in Layers 1 to 11, with transient increases in some layers; Layer 12 has a higher prediction\-region average\.The prediction\-to\-history mean ratios are below one in Layers 1 to 11\. The largest relative reductions occur in Layers 2, 3, and 5, with ratios from 0\.60 to 0\.66; Layers 8 to 11 have ratios from 0\.84 to 0\.97\. The temporal traces show that this regional pattern is not a uniform pointwise decrease: the earlier layers generally maintain lower norms, while several later layers show transient recoveries toward or above the historical level\. Thus, the region\-averaged reduction should be distinguished from the within\-region dynamics\.

Layer 12 has a higher prediction\-region mean norm, rising from approximately 3\.2 to 7\.2 before settling around 5\.8\. This reversal highlights variation across depth\. The norms alone do not establish whether this layer restores information affected by preceding updates\.

##### MtM\_\{t\}: similar transition\-matrix norms\.

Figure 4:Temporal evolution of‖Mt‖F\\\|M\_\{t\}\\\|\_\{F\}across all 12 layers\. Blue: historical region; orange: prediction region; red dashed line marks the boundary\. TheMtM\_\{t\}norms remain similar across the boundary and throughout the prediction region, with small increases in some layers\.TheMtM\_\{t\}norm is nearly constant within the displayed historical region across all 12 layers\. After the boundary, it remains similar or increases slightly; Layers 10 to 12 rise gradually from about 7\.3 to 7\.5\. Figure[4](https://arxiv.org/html/2610.12883#A2.F4)shows that these small norm differences persist through the displayed prediction interval\. They do not, by themselves, determine the contraction rate of successive transitions or the amount of historical information retained\.

Together, the two figures extend the region\-averaged observations in Section[4\.2](https://arxiv.org/html/2610.12883#S4.SS2)by showing their temporal structure\. Lower write norms in most layers coexist with similar transition norms, while transient recoveries and the Layer 12 reversal qualify the pattern\. These observations are consistent with PID at the level of update magnitudes, but do not directly measure net information loss\.

## Appendix CSynthetic Evidence and Teacher\-Forced Comparisons

We use synthetic signals to examine how architecture rankings and sensitivity to missing future observations vary with signal structure\. The comparison pairs NAF with a teacher\-forced condition that receives ground\-truth future inputs, denoted AR in this appendix\. The NAF–AR gap is a diagnostic of sensitivity to the available observations, rather than a comparison with free\-running autoregressive forecasting\. Full results are reported in Table[12](https://arxiv.org/html/2610.12883#A3.T12)\.

### C\.1Experimental Setup

We design six synthetic experiments \(SE1 to SE6\), each corresponding to a specific signal structure\. All experiments are conducted with a 16k context length: the historical length is 16,384 time steps, and the prediction length has three levels: Short \(32 steps\), Medium \(480 steps\), and Long \(720 steps\)\. Each experiment generates 256 independent sequences, with MASE as the evaluation metric\. The AR control uses real future values as input in the prediction region under teacher forcing, whereas the NAF condition uses placeholders\. Only the NAF condition matches the forecasting setup used for the main benchmark comparisons\. All experiments reuse the pre\-trained checkpoints from Section[4](https://arxiv.org/html/2610.12883#S4), without additional fine\-tuning on the synthetic signals\. The signal designs are described below\.

##### SE1 \(Regime Switch\)\.

The signal consists of three consecutive sinusoidal segments designed to mimic structural changes in real time series\. The three segments have periods of 100, 250, and 80 time steps, with amplitudes sampled from𝒰⁡\(0\.8,1\.2\)\\mathcal\{U\}\(0\.8,1\.2\),𝒰⁡\(1\.0,1\.5\)\\mathcal\{U\}\(1\.0,1\.5\), and𝒰⁡\(0\.6,1\.0\)\\mathcal\{U\}\(0\.6,1\.0\), respectively, and phases sampled from𝒰⁡\(0,2​π\)\\mathcal\{U\}\(0,2\\pi\)\. Gaussian noise with a standard deviation of 0\.1 is added to the entire signal\. Each segment is locally stationary, but the periodic patterns change across segments, and the history contains clearly learnable local structures\.

##### SE2 \(Ultra\-Long Cycle\)\.

The signal is composed of a sinusoid with a dominant period of 50,000 time steps superimposed with a local high\-frequency component with a period of 200 time steps, plus Gaussian noise with a standard deviation of 0\.12\. The dominant period far exceeds the historical length, so the model can only observe an approximately linear drift of the slow baseline; accurate prediction requires the model to maintain an estimate of this drift over a long time span and combine it with the local high\-frequency component\.

##### SE3 \(AM \+ Phase Jumps\)\.

The signal is a fast sinusoid with a period of 50 time steps, modulated by an amplitude envelope with a period of 800 time steps, with phase jumps of magnitudeπ/3\\pi/3introduced at fixed positions every 800 steps\. The amplitude is sampled from𝒰⁡\(0\.8,1\.2\)\\mathcal\{U\}\(0\.8,1\.2\), and the noise standard deviation is 0\.15\. This signal requires the model to simultaneously parse fast local oscillations, slow global envelopes, and discrete phase jumps\.

##### SE4 \(Ultimate Mix\)\.

The signal is composed of multiple components: eight frequency components \(periods 30, 80, 150, 300, 600, 1200, 2500, and 5000\), a chirp component \(period 100, drift rate 0\.02\), an amplitude\-modulated component \(period 400, modulation period 600\), two structural breaks \(periods 200 and 500\), and a linear trend \(slope sampled from𝒰⁡\(0\.0005,0\.005\)\\mathcal\{U\}\(0\.0005,0\.005\)\)\. The noise standard deviation is 0\.25\. This is the most complex signal among the six, containing both predictable deterministic structures and non\-stationary components that require online denoising\.

##### SE5 \(Noise Floor\)\.

The signal is a weak sinusoid with a period of 200 and amplitude sampled from𝒰⁡\(0\.2,0\.4\)\\mathcal\{U\}\(0\.2,0\.4\), superimposed with Gaussian noise of standard deviation 0\.8\. The weak periodic component is obscured by strong noise, providing a low\-signal\-to\-noise control\.

##### SE6 \(Constant\)\.

The signal is a constant value of 100\.0 plus Gaussian noise with a standard deviation of 1\.0\. The underlying signal level is constant, with observed fluctuations introduced by Gaussian noise\.

### C\.2Signal\-Dependent Architecture Performance

The synthetic designs vary signal structure, noise, and non\-stationarity, allowing us to compare the three trained architectures across these regimes\.

##### RWKV on the noisy multi\-component signal\.

SE4 is the most complex signal among the six\. It superimposes multiple dynamically changing components: multi\-frequency oscillations, chirp, amplitude modulation, structural breaks, and a linear trend, under strong noise \(σ=0\.25\\sigma=0\.25\)\. Online denoising and non\-stationary modeling are both required for this type of signal: the model needs to continuously track multiple simultaneously varying dynamic components in noise\.

Results are reported in Table[12](https://arxiv.org/html/2610.12883#A3.T12)\. Under NAF, RWKV achieves the lowest Long MASE \(1\.058\), compared with HART \(1\.148\) and Transformer \(1\.151\)\. RWKV also outperforms Transformer under the teacher\-forced AR condition \(0\.631 vs\. 0\.641\)\. This ranking suggests that recurrent modeling can be useful on this noisy multi\-component signal, but the comparison does not isolate a particular filtering mechanism\. HART slightly improves on Transformer under NAF while remaining worse than pure RWKV on SE4\.

##### Attention\-based models on amplitude modulation and phase jumps\.

The SE3 signal requires structural parsing at three levels simultaneously: fast local oscillations \(period 50\), slow global envelope \(period 800\), and discrete phase jumps\. The coupling of fast oscillation and slow envelope requires the model to see local details and global shape simultaneously\.

Under the teacher\-forced AR condition, RWKV has higher Long MASE than HART and Transformer \(0\.265 vs\. 0\.169 and 0\.172\)\. Thus, the architecture ranking is already present when real future inputs are supplied\. Under NAF, HART achieves the lowest Long MASE \(0\.265\), followed by Transformer \(0\.306\) and RWKV \(0\.581\)\. These results favor the attention\-based architectures on SE3, without identifying a unique cause for RWKV’s gap\.

RWKV’s Long NAF–AR gap on SE3 is 0\.316\. The SE3 comparison is compatible with both task\-dependent architectural differences and sensitivity to placeholder inputs\.

##### Complementarity and balance\.

SE4 and SE3 yield different architecture rankings: RWKV performs best under NAF on SE4, whereas HART performs best on SE3\. This variation supports examining hybrid architectures across diverse signal regimes rather than expecting universal dominance\. On the real benchmarks in Table[2](https://arxiv.org/html/2610.12883#S4.T2), HART improves over the pure Transformer on aggregate MASE and over pure RWKV on Medium and Long horizons\.

### C\.3Interpreting Sensitivity to Missing Future Observations

We next examine how the NAF–AR gap changes across signal regimes and what this comparison can establish about PID\.

The teacher\-forced control supplies real future inputs instead of recursively generated predictions\. Switching to NAF therefore changes the information available to the model as well as its input pattern\. The resulting gap measures sensitivity to this change; it does not isolate PID from every other source of forecasting error\.

##### Structured\-signal comparisons\.

SE1 to SE3 provide three different structured signals on which to compare the effect of replacing real future inputs with placeholders\.

On Long, RWKV has a larger NAF–AR gap than both other architectures on each signal: SE1, 0\.060 \(HART 0\.001, Transformer 0\.002\); SE2, 0\.095 \(HART 0\.030, Transformer 0\.016\); and SE3, 0\.316 \(HART 0\.096, Transformer 0\.134\)\. The consistent ranking indicates greater sensitivity of the evaluated RWKV model to the removal of future observations on these signals\. It is consistent with the motivation for PID analysis, but does not rule out architecture\-specific sensitivity to task difficulty or changes in the input distribution\.

##### Low\-signal controls\.

SE5 and SE6 examine whether replacing future inputs with placeholders causes a similar loss on signals dominated by noise or a constant level\.

SE5 contains a weak sinusoid under strong noise, whereas SE6 has a constant underlying level plus noise\. Their limited temporal variation contrasts with the richer structures in SE1 to SE4\.

The two conditions yield nearly identical scores on these controls\. At the reported precision, the absolute NAF–AR gaps on SE5 are at most 0\.001, while those on SE6 are zero\. This shows that the loss observed on structured signals is not uniform across all tested regimes\. These controls do not, by themselves, exclude overfitting, distribution shift, or other mechanisms contributing to the structured\-signal gaps\.

Table 12:NAF versus teacher\-forced AR at 16k context length \(MASE\)\. AR uses real future inputs as a diagnostic control\. Lower is better\.

## Appendix DPre\-training Setup

This section details the composition and processing of pre\-training data and the training hyperparameters\. All experiments are conducted under identical pre\-training conditions: the three architectures, HART, pure RWKV, and pure Transformer, are trained with exactly the same data, optimization strategy, and random seed\.

### D\.1Pre\-training Data

The pre\-training corpus consists of real\-world and synthetic time series, totaling approximately 86\.7M sequences\. Roughly half are real \(43\.7M\) and half synthetic \(43M\), so that the model sees both natural distributions and controlled generative signals\.

Real data are drawn from three public sources:

- •GIFT\-Eval pre\-training split[Aksu et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib17): the same collection from which the evaluation benchmark is constructed, covering energy, transportation, meteorology, finance, and other domains\. For multivariate datasets, each variate is extracted as an independent univariate series\.
- •Chronos corpus[Ansari et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib12): the large\-scale time series pre\-training collection released by the Chronos project\. Following the data\-cleaning protocol of TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6), we remove sequences that overlap temporally with the GIFT\-Eval test sets to prevent evaluation leakage\.
- •Code15[Ribeiro et al\. \(2021\)](https://arxiv.org/html/2610.12883#bib.bib21): a public healthcare dataset used as an additional real\-world source\.

Every real series is truncated to a maximum length of 32k time steps\. For longer series, sliding windows are applied\. The first 32k steps form the first sample, and subsequent windows are constructed by extending backwards from the tail of the remaining portion\. This yields approximately 43\.7M real training sequences\.

Synthetic data are generated to cover diverse patterns observed in real time series while keeping sequence lengths within the pre\-training window\. Three types are produced, totaling roughly 43M sequences:

1. 1\.Chronos\-style synthesis \(15M sequences\): following the protocol of Chronos[Ansari et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib12), 4096\-step sequences are drawn from Gaussian processes with randomly sampled kernel parameters, introducing multi\-scale fluctuation patterns\.
2. 2\.Long\-range GP\-interpolated synthesis \(15M sequences\): direct GP sampling at length 32k is computationally prohibitive due to the𝒪⁡\(n3\)\\mathcal\{O\}\(n^\{3\}\)time complexity of dense Gaussian process sampling \(cubic in the number of conditioning points\)\. We therefore first sample coarse 4096\-step sequences and then interpolate groups of eight windows to 32k steps using GP\-based probabilistic infilling, which preserves fine\-grained stochastic fluctuations rather than performing naive linear interpolation\.
3. 3\.Non\-uniformly sampled synthesis \(13M sequences\): real\-world sensor data often contain random gaps caused by device failure or transmission interruptions\. To mimic this, we generate 4096\-step GP sequences, interpolate groups of four windows to approximately 16k steps via GP infilling, and then randomly drop roughly 10% of the time points\. The dropped points are not recorded as missing values with a placeholder; they are simply absent from the sequence, requiring the model to infer full dynamics from incomplete observations\.

Together, the three synthetic families cover uniform high\-frequency signals, long\-range continuous patterns, and non\-uniformly missing data, complementing those in real data\.

### D\.2Contiguous Patch Masking

Following TiRex[Auer et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib6), we adopt Contiguous Patch Masking \(CPM\) as the pre\-training task, so that the model is exposed to the NAF inference mode already during training\.

For each training sample, we uniformly sample a contiguous patch countcmask∈\{1,…,cmaskmax\}c\_\{\\text\{mask\}\}\\in\\\{1,\\ldots,c\_\{\\text\{mask\}\}^\{\\max\}\\\}and a masking probabilitypmask∼U⁡\(0,pmaskmax\)p\_\{\\text\{mask\}\}\\sim U\(0,p\_\{\\text\{mask\}\}^\{\\max\}\), partition theTTpatches of the sequence into⌈T/cmask⌉\\lceil T/c\_\{\\text\{mask\}\}\\rceilgroups ofcmaskc\_\{\\text\{mask\}\}patches each, draw an independent Bernoulli variable with parameterpmaskp\_\{\\text\{mask\}\}for each group, and extend it to all patches and time steps within that group\. Adjacent groups may be masked simultaneously, so the effective masked span can exceedcmaskmaxc\_\{\\text\{mask\}\}^\{\\max\}\.

Masked patches are replaced by placeholders in the input, and their ground\-truth values serve only as supervision in the loss; unmasked positions also contribute to the loss\. This design exposes the model to an input form in which the prediction region is filled with placeholders, so that it learns to reconstruct a contiguous future from arbitrary historical positions rather than extrapolating only at the sequence terminus, aligning training with the input distribution at NAF inference\.

### D\.3Optimizer and Training Schedule

All three architectures use identical optimization settings\. The learning rate follows a two\-phase schedule: it warms up linearly from zero to1×10−31\\times 10^\{\-3\}over the first 8k steps, then decays via a cosine schedule to3\.3×10−63\.3\\times 10^\{\-6\}over the subsequent 212k steps\. The total of 220k steps corresponds to exactly one pass over the full training corpus, so each sample is visited only once\. This schedule limits repeated exposure to the same samples, but does not by itself guarantee the absence of overfitting\. We use the Muon optimizer[Liu et al\. \(2025b\)](https://arxiv.org/html/2610.12883#bib.bib20)with weight decay 0\.1 and gradient clipping norm 1\.0\. We do not apply dropout during training\. Complementary to CPM, next\-patch prediction ensures that every patch directly contributes to the loss, so that training signals are fully utilized\.

HART\-36M, pure RWKV\-36M, and pure Transformer\-35M are pre\-trained under exactly the same data, training strategy, and random seed\. This shared setup supports comparison of the architectures under the reported training configuration\. All models are trained on two A800\-80G GPUs using Keras 3 and JAX\.

## Appendix EDetailed Computational Formulae for Each Block

Figure[2](https://arxiv.org/html/2610.12883#S3.F2)illustrates the internal structure of the two building blocks that compose the HART backbone\. This appendix provides the complete computational formulae for each sub\-module shown therein\. HART introduces minimal modifications to the established architectures of both RWKV and Attention\.

##### Patch Embedding Layer

The input time series first undergoeszz\-score normalization[Kim et al\. \(2021\)](https://arxiv.org/html/2610.12883#bib.bib38)to unify sequences of different scales into a comparable numerical range:

y~1:C=y1:C−y¯1:Cσy1:C\\tilde\{y\}\_\{1:C\}=\\frac\{y\_\{1:C\}\-\\bar\{y\}\_\{1:C\}\}\{\\sigma\_\{y\_\{1:C\}\}\}\(4\)
wherey¯\\bar\{y\}andσy\\sigma\_\{y\}are the mean and standard deviation of the historical series, respectively; these two statistics are retained during inference for inverse normalization in the output head\.

The normalized series is segmented into patches of lengthPP, yielding a patch sequence𝐗∈ℝB×T×P\\mathbf\{X\}\\in\\mathbb\{R\}^\{B\\times T\\times P\}\. For each position, we construct an observation mask𝐎∈\{0,1\}B×T×P\\mathbf\{O\}\\in\\\{0,1\\\}^\{B\\times T\\times P\}indicating whether the value is observed, and a missingness mask𝐍∈\{0,1\}B×T×P\\mathbf\{N\}\\in\\\{0,1\\\}^\{B\\times T\\times P\}indicating whether the position is a placeholder to be predicted\. The three tensors are concatenated along the last dimension and then projected to dimensionDDthrough a residual MLP\. We follow the skip embedding design of Moirai[Liu et al\. \(2025a\)](https://arxiv.org/html/2610.12883#bib.bib7):

𝐙=SiLU⁡\(𝐗cat​Win\)​Wout\+𝐗cat​Wskip\\mathbf\{Z\}=\\mathrm\{SiLU\}\(\\mathbf\{X\}\_\{\\text\{cat\}\}W\_\{\\text\{in\}\}\)W\_\{\\text\{out\}\}\+\\mathbf\{X\}\_\{\\text\{cat\}\}W\_\{\\text\{skip\}\}\(5\)
where𝐗cat=concat⁡\(𝐗,𝐎,𝐍\)∈ℝB×T×3​P\\mathbf\{X\}\_\{\\text\{cat\}\}=\\mathrm\{concat\}\(\\mathbf\{X\},\\mathbf\{O\},\\mathbf\{N\}\)\\in\\mathbb\{R\}^\{B\\times T\\times 3P\},Win∈ℝ3​P×4​DW\_\{\\text\{in\}\}\\in\\mathbb\{R\}^\{3P\\times 4D\},Wout∈ℝ4​D×DW\_\{\\text\{out\}\}\\in\\mathbb\{R\}^\{4D\\times D\}, andWskip∈ℝ3​P×DW\_\{\\text\{skip\}\}\\in\\mathbb\{R\}^\{3P\\times D\}\.T=Th​i​s​t\+Tp​r​e​dT=T\_\{hist\}\+T\_\{pred\}is the total number of patches, whereTh​i​s​t=⌈C/P⌉T\_\{hist\}=\\lceil C/P\\rceilis the number of historical patches andTp​r​e​d=⌈H/P⌉T\_\{pred\}=\\lceil H/P\\rceilis the number of prediction placeholder patches\. The original inputs in the prediction region are processed as described in Section[3\.1](https://arxiv.org/html/2610.12883#S3.SS1); their embeddings incorporate the masks and are not constrained to be zero, while the prediction region contains no observed future values\. The patch embedding layer shares weights between historical patches and prediction placeholder patches\.

### E\.1Attention Block

Figure[2](https://arxiv.org/html/2610.12883#S3.F2)\(b\) depicts the Attention Block, which comprises RMSNorm, gated self\-attention, and a SwiGLU feed\-forward network with residual connections\. It applies sigmoid gating to the self\-attention output, following the gated\-attention formulation of[Qiu et al\. \(2026\)](https://arxiv.org/html/2610.12883#bib.bib37)\.

#### E\.1\.1Gated Attention

Given the inputxx, the query, key, value, and gate are obtained via linear projections:

Q=x​WQ,K=x​WK,V=x​WV,𝐆=x​WG\\displaystyle Q=xW\_\{Q\},\\quad K=xW\_\{K\},\\quad V=xW\_\{V\},\\quad\\mathbf\{G\}=xW\_\{G\}\(6\)
All projection matrices are∈ℝD×D\\in\\mathbb\{R\}^\{D\\times D\}\. After applying RoPE positional encoding[Su et al\. \(2024\)](https://arxiv.org/html/2610.12883#bib.bib41)toQQandKK, causal self\-attention is computed and gated\.𝐂\\mathbf\{C\}denotes the causal mask:

A\\displaystyle A=softmax⁡\(Q​K⊤dh\+𝐂\)​V\\displaystyle=\\mathrm\{softmax\}\\left\(\\frac\{QK^\{\\top\}\}\{\\sqrt\{d\_\{h\}\}\}\+\\mathbf\{C\}\\right\)V\(7\)O\\displaystyle O=A⊙sigmoid⁡\(𝐆\)\\displaystyle=A\\odot\\mathrm\{sigmoid\}\(\\mathbf\{G\}\)\(8\)Oout\\displaystyle O\_\{\\text\{out\}\}=O​WO,WO∈ℝD×D\\displaystyle=OW\_\{O\},\\quad W\_\{O\}\\in\\mathbb\{R\}^\{D\\times D\}\(9\)

#### E\.1\.2Feed\-Forward Network

The FFN adopts SwiGLU activation\. The only difference from the RWKV\-FFN is the absence of token shift; all other aspects remain consistent:

FFN⁡\(x\)\\displaystyle\\mathrm\{FFN\}\(x\)=\(Swish⁡\(x​Wgate\)⊙x​Wkey\)​Wout\\displaystyle=\\bigl\(\\mathrm\{Swish\}\(xW\_\{\\text\{gate\}\}\)\\odot xW\_\{\\text\{key\}\}\\bigr\)W\_\{\\text\{out\}\}\(10\)
whereWgate,Wkey∈ℝD×4​DW\_\{\\text\{gate\}\},W\_\{\\text\{key\}\}\\in\\mathbb\{R\}^\{D\\times 4D\}andWout∈ℝ4​D×DW\_\{\\text\{out\}\}\\in\\mathbb\{R\}^\{4D\\times D\}\.

### E\.2RWKV Block

Figure[2](https://arxiv.org/html/2610.12883#S3.F2)\(c\) depicts the RWKV Block, which consists of LayerNorm, RWKV\-TimeMIX \(generalized delta rule\), and a SwiGLU channel mixer with residual connections\. The RWKV Block inherits the full design of RWKV\-7[Peng et al\. \(2025\)](https://arxiv.org/html/2610.12883#bib.bib16), with the only modification being the replacement of the activation function in RWKV\-FFN from ReLU2to SwiGLU\. Following the standard RWKV\-7 architecture, the first RWKV block in the backbone prepends an extra LayerNorm before the TimeMix sub\-block for input preprocessing\.

#### E\.2\.1Token Shift

Both TimeMix and RWKV\-FFN employ the token shift mechanism, which takes place after layer normalization and before the respective submodules\. Token shift linearly interpolates the current token with the previous token:

lerp⁡\(a,b,x\)=a\+\(b−a\)⊙x\\mathrm\{lerp\}\(a,b,x\)=a\+\(b\-a\)\\odot x\(11\)
The shifted inputs for TimeMix arext□=lerp⁡\(xt,xt−1,μ□\)x^\{\\square\}\_\{t\}=\\mathrm\{lerp\}\(x\_\{t\},x\_\{t\-1\},\\mu\_\{\\square\}\)for□∈\{r,k,v,d,a,g\}\\square\\in\\\{r,k,v,d,a,g\\\}, where eachμ□∈ℝD\\mu\_\{\\square\}\\in\\mathbb\{R\}^\{D\}is a learnable mixing factor\. RWKV\-FFN follows the same pattern, using independent mixing factorsμk′,μg′∈ℝD\\mu^\{\\prime\}\_\{k\},\\mu^\{\\prime\}\_\{g\}\\in\\mathbb\{R\}^\{D\}\.

#### E\.2\.2LoRA Parameterization in TimeMix

The TimeMix quantities are computed from the shifted inputs using learned projections, LoRA adapters, and nonlinear transformations:

at\\displaystyle a\_\{t\}=sigmoid⁡\(LoRAa​\(xta\)\)\\displaystyle=\\mathrm\{sigmoid\}\(\\mathrm\{LoRA\}\_\{a\}\(x^\{a\}\_\{t\}\)\)\(12\)νt\\displaystyle\\nu\_\{t\}=sigmoid⁡\(LoRAν​\(xtv\)\)\\displaystyle=\\mathrm\{sigmoid\}\(\\mathrm\{LoRA\}\_\{\\nu\}\(x^\{v\}\_\{t\}\)\)\(13\)dt\\displaystyle d\_\{t\}=LoRAd​\(tanh,xtd\)\\displaystyle=\\mathrm\{LoRA\}\_\{d\}\(\\tanh,x^\{d\}\_\{t\}\)\(14\)wt\\displaystyle w\_\{t\}=exp⁡\(−e−0\.5​sigmoid​\(dt\)\)\\displaystyle=\\exp\(\-e^\{\-0\.5\}\\mathrm\{sigmoid\}\(d\_\{t\}\)\)\(15\)gt\\displaystyle g\_\{t\}=LoRAg​\(sigmoid,xtg\)\\displaystyle=\\mathrm\{LoRA\}\_\{g\}\(\\mathrm\{sigmoid\},x^\{g\}\_\{t\}\)\(16\)kt\\displaystyle k\_\{t\}=xtk​Wk\\displaystyle=x^\{k\}\_\{t\}W\_\{k\}\(17\)vt,l\\displaystyle v\_\{t,l\}=xtv​Wv\\displaystyle=x^\{v\}\_\{t\}W\_\{v\}\(18\)rt\\displaystyle r\_\{t\}=xtr​Wr\\displaystyle=x^\{r\}\_\{t\}W\_\{r\}\(19\)
whereWk,Wv,Wr∈ℝD×DW\_\{k\},W\_\{v\},W\_\{r\}\\in\\mathbb\{R\}^\{D\\times D\}\. Before head splitting, the removal and replacement keys areκt=kt⊙ξ\\kappa\_\{t\}=k\_\{t\}\\odot\\xiandk~t=kt⊙lerp⁡\(1,at,α\)\\tilde\{k\}\_\{t\}=k\_\{t\}\\odot\\mathrm\{lerp\}\(1,a\_\{t\},\\alpha\), with trainable vectorsξ,α∈ℝD\\xi,\\alpha\\in\\mathbb\{R\}^\{D\}\. Following RWKV\-7, value residual learning interpolates the final value asvt=vt,0v\_\{t\}=v\_\{t,0\}forl=0l=0andvt=lerp⁡\(vt,0,vt,l,νt\)v\_\{t\}=\\mathrm\{lerp\}\(v\_\{t,0\},v\_\{t,l\},\\nu\_\{t\}\)forl≥1l\\geq 1\.

#### E\.2\.3GDR State Update and Attention Readout

We split\{rt,wt,k~t,vt,κt,at\}\\\{r\_\{t\},w\_\{t\},\\tilde\{k\}\_\{t\},v\_\{t\},\\kappa\_\{t\},a\_\{t\}\\\}intohhheads, each of dimensiondh=D/hd\_\{h\}=D/h, and normalize the removal key within each head asκ^t=κt/∥κt∥2\\hat\{\\kappa\}\_\{t\}=\\kappa\_\{t\}/\\lVert\\kappa\_\{t\}\\rVert\_\{2\}\. All subsequent state\-update and readout operations are per\-head, with the head index omitted\. The state follows Equation[1](https://arxiv.org/html/2610.12883#S3.E1)\. In a generic kernel using a plus\-sign rank\-one correction, the corresponding effective vectors areatker=−κ^ta\_\{t\}^\{\\mathrm\{ker\}\}=\-\\hat\{\\kappa\}\_\{t\},btker=at⊙κ^tb\_\{t\}^\{\\mathrm\{ker\}\}=a\_\{t\}\\odot\\hat\{\\kappa\}\_\{t\}, andktker=k~tk\_\{t\}^\{\\mathrm\{ker\}\}=\\tilde\{k\}\_\{t\};btkerb\_\{t\}^\{\\mathrm\{ker\}\}is therefore not an independently projected vector\. The attention resultptp\_\{t\}is read from the state via the receptance vector:

ut\\displaystyle u\_\{t\}=\(rt⋅\(ρ⊙k~t\)⊤\)​vt,bonus term\\displaystyle=\\bigl\(r\_\{t\}\\cdot\(\\rho\\odot\\tilde\{k\}\_\{t\}\)^\{\\top\}\\bigr\)v\_\{t\},\\quad\\text\{bonus term\}\(20\)pt\\displaystyle p\_\{t\}=LayerNorm⁡\(rt​St⊤\)\+ut\\displaystyle=\\mathrm\{LayerNorm\}\(r\_\{t\}S\_\{t\}^\{\\top\}\)\+u\_\{t\}\(21\)
whereρ∈ℝdh\\rho\\in\\mathbb\{R\}^\{d\_\{h\}\}is the bonus weight\. After concatenating all heads, the final output of TimeMix is obtained by gating and output projection:

ot=\(gt⊙pt\)​Wo,Wo∈ℝD×Do\_\{t\}=\(g\_\{t\}\\odot p\_\{t\}\)W\_\{o\},\\quad W\_\{o\}\\in\\mathbb\{R\}^\{D\\times D\}\(22\)

#### E\.2\.4RWKV\-FFN \(SwiGLU Version\)

RWKV\-FFN employs token shift and SwiGLU activation\. The only difference from the original RWKV\-7 is the replacement of ReLU2with SwiGLU:

kt′\\displaystyle k^\{\\prime\}\_\{t\}=lerp⁡\(xt′,xt−1′,μk′\)​Wk′∈ℝ4​D\\displaystyle=\\mathrm\{lerp\}\(x^\{\\prime\}\_\{t\},x^\{\\prime\}\_\{t\-1\},\\mu^\{\\prime\}\_\{k\}\)W\_\{k^\{\\prime\}\}\\in\\mathbb\{R\}^\{4D\}\(23\)gt′\\displaystyle g^\{\\prime\}\_\{t\}=lerp⁡\(xt′,xt−1′,μg′\)​Wg′∈ℝ4​D\\displaystyle=\\mathrm\{lerp\}\(x^\{\\prime\}\_\{t\},x^\{\\prime\}\_\{t\-1\},\\mu^\{\\prime\}\_\{g\}\)W\_\{g^\{\\prime\}\}\\in\\mathbb\{R\}^\{4D\}\(24\)ot′\\displaystyle o^\{\\prime\}\_\{t\}=\(Swish⁡\(gt′\)⊙kt′\)​Wv′∈ℝD\\displaystyle=\\bigl\(\\mathrm\{Swish\}\(g^\{\\prime\}\_\{t\}\)\\odot k^\{\\prime\}\_\{t\}\\bigr\)W\_\{v^\{\\prime\}\}\\in\\mathbb\{R\}^\{D\}\(25\)
whereSwish⁡\(x\)=x⋅sigmoid⁡\(x\)\\mathrm\{Swish\}\(x\)=x\\cdot\\mathrm\{sigmoid\}\(x\), andWk′,Wg′∈ℝD×4​DW\_\{k^\{\\prime\}\},W\_\{g^\{\\prime\}\}\\in\\mathbb\{R\}^\{D\\times 4D\},Wv′∈ℝ4​D×DW\_\{v^\{\\prime\}\}\\in\\mathbb\{R\}^\{4D\\times D\}\.

### E\.3Summary of Architectural Differences from the Originals

HART modifies existing modules only in the following respects:

1. 1\.RWKV Block: Only the activation function in RWKV\-FFN is changed from ReLU2to SwiGLU to maintain a consistent activation function choice with the Attention Block\. The entire TimeMix \(including GDR, token shift, LoRA parameterization, value residual learning, and attention readout\) retains the original design of RWKV\-7\.
2. 2\.Attention Block: We use gated self\-attention and a SwiGLU FFN, with the formulations specified above\.

Based on these components, we assemble them into our proposed HART architecture\.