Scale-Aware Pretraining of Time Series Foundation Models via Multi-Patch Token Alignment and Hybrid Masking

# Scale-Aware Pretraining of Time SeriesFoundation Models via Multi-PatchToken Alignment and Hybrid Masking
Source: [https://arxiv.org/html/2608.20005](https://arxiv.org/html/2608.20005)
## Scale\-Aware Pretraining of Time Series Foundation Models via Multi\-Patch Token Alignment and Hybrid Masking\*Corresponding author: Lizhen Cui \(clz@sdu\.edu\.cn\)\.

Taihua Chen1,2, Xiang Ma1, Yixin Zhang3, Tailin Zhan1, Manyu Sun1, and Lizhen Cui1,2,\*Affiliation:1School of Software, Shandong University, Jinan 250101, China 2Joint SDU–NTU Centre for Artificial Intelligence Research \(C\-FAIR\), Shandong University, Jinan 250101, China 3Nanyang Technological University, Singapore cfair\-cth@mail\.sdu\.edu\.cn,xiangma@sdu\.edu\.cn,zhangyixin9610@gmail\.com, tailinzhan@mail\.sdu\.edu\.cn,sjxyctcsusud6383@gmail\.com,clz@sdu\.edu\.cn

###### Abstract

Pretraining time series foundation models across heterogeneous datasets necessitates effective handling of varying sampling frequencies\. Current methods either employ dataset\-specific patch sizes and separate FFNs, leading to fragmented representations, or enforce a fixed patch size that neglects inherent temporal variations\. To address this, we propose SATS, featuring a scale\-aware token alignment mechanism that treats patch size as an explicit notion of scale\. By incorporating a contrastive\-inspired alignment regularizer, SATS aligns representation spaces across scales while preserving distinct modeling capacities\. Furthermore, a hybrid masking strategy combining random and contiguous masking is introduced to capture multi\-scale temporal structures\. Experimental results on LSTF benchmarks demonstrate that SATS achieves a 9\.2% improvement in MSE and an 8\.3% gain in GIFT\-Eval MASE compared to competitive baselines\. Notably, SATS consistently delivers SOTA performance while achieving a 65\.6% increase in model efficiency over advanced baselines, highlighting its effectiveness and scalability in time series pretraining\.

###### Index Terms:

Time series foundation models, Time series forecasting, Heterogeneous time series, Zero\-shot forecasting\.

## IIntroduction

The recent emergence of foundation models has significantly advanced various domains such as natural language processing\[[1](https://arxiv.org/html/2608.20005#bib.bib1),[2](https://arxiv.org/html/2608.20005#bib.bib2)\], computer vision\[[3](https://arxiv.org/html/2608.20005#bib.bib3),[4](https://arxiv.org/html/2608.20005#bib.bib4)\], and speech understanding\[[5](https://arxiv.org/html/2608.20005#bib.bib5),[6](https://arxiv.org/html/2608.20005#bib.bib6)\]\. Inspired by their success, growing efforts have been devoted to developing foundation models for time series, aiming to produce general\-purpose representations transferable across diverse downstream tasks\. An early line of work adapts pretrained language models to time series tasks, leveraging their sequence modeling capabilities in hopes of achieving strong generalization\[[7](https://arxiv.org/html/2608.20005#bib.bib7),[8](https://arxiv.org/html/2608.20005#bib.bib8),[9](https://arxiv.org/html/2608.20005#bib.bib9)\]\. However, the modality gap often hinders their performance on temporally structured data, resulting in suboptimal generalization across diverse time series tasks\. Moreover, their black\-box nature further exacerbates the issue, raising concerns about interpretability and the lack of alignment with intrinsic temporal characteristics\[[10](https://arxiv.org/html/2608.20005#bib.bib10)\]\. To address these challenges, a second line of work has emerged that trains foundation models from scratch on large\-scale, heterogeneous time series datasets\[[11](https://arxiv.org/html/2608.20005#bib.bib11),[12](https://arxiv.org/html/2608.20005#bib.bib12),[13](https://arxiv.org/html/2608.20005#bib.bib13)\]\. These models aim to capture universal temporal dynamics in a data\-driven and domain\-adaptive manner, thereby enhancing robustness to distribution shifts and improving transferability across domains with varying sampling rates, modalities, and sequence lengths \(e\.g\., finance, healthcare, meteorology, IoT\)\.

Despite the promise of the latter direction, it presents unique challenges— particularly in how to effectively segment and tokenize continuous signals for cross\-dataset pretraining\. Unlike language, where discrete word units naturally serve as stable tokens\[[14](https://arxiv.org/html/2608.20005#bib.bib14)\], or vision, where uniform patch sizes are viable due to consistent spatial resolution and semantic robustness\[[15](https://arxiv.org/html/2608.20005#bib.bib15),[16](https://arxiv.org/html/2608.20005#bib.bib16)\], time series data exhibit irregular sampling and variable sequence lengths, making fixed\-size downsampling ineffective\. These characteristics necessitate the use of small, adaptive patch sizes to preserve fine\-grained temporal patterns\.

As shown in Figure[1](https://arxiv.org/html/2608.20005#S1.F1), recent studies have explored two main strategies for time series tokenization, each with inherent limitations\. \(1\)Dataset\-specific patchingadopts variable patch sizes tailored to local sampling rates, combined with independent FFNs for token projection\[[17](https://arxiv.org/html/2608.20005#bib.bib17),[12](https://arxiv.org/html/2608.20005#bib.bib12)\]\. While this design aligns well with the granularity of each dataset, it results in fragmented token spaces that hinder the learning of generalizable temporal patterns and compromise training stability\[[18](https://arxiv.org/html/2608.20005#bib.bib18)\]\. \(2\)Uniform patchingenforces a globally small patch size across datasets to promote representational consistency\[[19](https://arxiv.org/html/2608.20005#bib.bib19),[20](https://arxiv.org/html/2608.20005#bib.bib20)\]\. However, this strategy introduces information bottlenecks and often misaligns local dynamics, as it fails to accommodate the diverse temporal structures inherent in different datasets\[[21](https://arxiv.org/html/2608.20005#bib.bib21)\]\. Both strategies, therefore, face a trade\-off between dataset adaptability and representational generality, limiting their effectiveness in scalable pretraining\.

![Refer to caption](https://arxiv.org/html/2608.20005v1/Fig1_intro.png)

Fig\. 1:\(a\) Dataset\-specific patch sizes and independent FFNs for varying sampling rates lead to fragmented token spaces\. \(b\) Using a unified patch size and FFN risks information bottlenecks and misaligned local dynamics\. \(c\) SATS adopts dataset\-specific patch sizes and enforces scale\-aware alignment across FFN\-projected spaces, yielding semantically rich and consistent representations\.To bridge the gap between fragmented token spaces introduced by adaptive patching and the representational rigidity of fixed segmentation, we propose a scale\-aware token alignment mechanism tailored for time series pretraining\. By treating the patch size as an explicit notion of scale, our method aligns the representation spaces induced by scale\-specific FFNs\. This is achieved by minimizing the distance between mean token embeddings across scales to encourage semantic alignment, while simultaneously maximizing the distance between their maximal embeddings to preserve the scale\-specific modeling capacity\. The resulting token space offers a unified yet expressive foundation for downstream tasks\.

Building on this aligned representation space, a remaining challenge lies in the diverse temporal structures inherent to different datasets\. Even with aligned embeddings, temporal variations may manifest within individual tokens or span across multiple tokens, depending on the dynamics of the underlying sequence\. To capture such variability, we introduce a hybrid masking strategy that enhances multi\-scale temporal modeling during masked reconstruction\. This strategy combines random masking, which promotes fine\-grained inference, with contiguous masking, which facilitates the modeling of long\-range dependencies\. By jointly optimizing across these complementary patterns, the model learns to recover temporal structures at varying resolutions, improving its robustness and generalization\.

Our main contributions are summarized as follows:

- •We proposeSATS, aScale\-Aware foundation model forTimeSeries, which achieves superior generalization across heterogeneous time series datasets\.
- •We introduce a scale\-aware alignment mechanism based on scale\-specific FFNs, unifying token spaces across patch scales while preserving scale\-specific expressiveness\.
- •We design a hybrid masking strategy that enables the model to capture both fine\-grained and long\-range temporal dependencies across multiple resolutions\.
- •Extensive experiments demonstrate the effectiveness of SATS in both zero\-shot and in\-distribution forecasting settings, establishing its potential as a strong pretraining paradigm for time series foundation models\.

## IIRelated Work

### II\-ATime Series Foundation Models

Traditional forecasting models are typically optimized for specific datasets, horizons, and sampling resolutions, so their scale handling is often encoded through architectural or hyperparameter choices\. Multi\-scale models such as Pathformer\[[22](https://arxiv.org/html/2608.20005#bib.bib22)\]and TimeMixer\[[23](https://arxiv.org/html/2608.20005#bib.bib23)\]strengthen this line by modeling temporal patterns at multiple resolutions through scale\-specific pathways, decomposition, pooling, patching, or dilation\. While these methods capture multi\-resolution patterns within individual datasets, foundation\-model pretraining mixes datasets with different sampling frequencies and patch scales, making cross\-frequency representation compatibility necessary for transferable temporal representations\. Time series foundation models have therefore gained increasing attention\. To meet forecasting\-specific demands, decoder\-only models such as Timer\[[24](https://arxiv.org/html/2608.20005#bib.bib24)\]and Lag\-Llama\[[25](https://arxiv.org/html/2608.20005#bib.bib25)\]adopt causal architectures, while sparse MoE variants such as Time\-MoE\[[11](https://arxiv.org/html/2608.20005#bib.bib11)\]and Moirai\-MoE\[[20](https://arxiv.org/html/2608.20005#bib.bib20)\]improve scalability\. Encoder\-decoder models such as LightGTS\[[19](https://arxiv.org/html/2608.20005#bib.bib19)\]and Chronos\[[13](https://arxiv.org/html/2608.20005#bib.bib13)\]use parallel decoding or discretized objectives\. Encoder\-only architectures remain less explored for time series foundation models, although recent experimental analyses\[[26](https://arxiv.org/html/2608.20005#bib.bib26)\]suggest stronger representational capacity under limited compute, motivating further investigation into their architecture and pretraining strategies\[[12](https://arxiv.org/html/2608.20005#bib.bib12),[27](https://arxiv.org/html/2608.20005#bib.bib27)\]\. Following this encoder\-only pretraining direction, SATS addresses the overlooked need to align scale\-specific token spaces induced by heterogeneous patch sizes\.

### II\-BContrastive Learning in Pretraining

Contrastive learning has emerged as a powerful paradigm in large\-scale pretraining across various domains\. In NLP, methods such as SimCSE\[[28](https://arxiv.org/html/2608.20005#bib.bib28)\]leverage contrastive objectives to learn semantically meaningful sentence embeddings without supervision\. In computer vision, CLIP\[[4](https://arxiv.org/html/2608.20005#bib.bib4)\]and ALIGN\[[29](https://arxiv.org/html/2608.20005#bib.bib29)\]jointly embed images and texts by maximizing the similarity of paired modalities while contrasting unpaired ones, achieving impressive zero\-shot performance\. While contrastive learning in time series remains relatively underexplored, recent works like TS\-TCC\[[30](https://arxiv.org/html/2608.20005#bib.bib30)\]and CoST\[[31](https://arxiv.org/html/2608.20005#bib.bib31)\]demonstrate its potential in learning transferable representations by aligning augmented views of temporal data\. These methods generally seek a balance between aligning semantically related representations and maintaining sufficient dispersion in the embedding space\. Such an attraction–repulsion structure not only improves feature discriminability but also helps prevent representations from collapsing into an uninformative configuration\. A key advantage of contrastive learning lies in its ability to preserve embedding diversity—by pulling semantically similar instances closer and pushing dissimilar ones apart, it structures the latent space in a discriminative and robust manner\. Inspired by contrastive learning’s structured divergence, we adopt an InfoNCE\-motivated objective to enhance distinctiveness among multi\-scale features—without explicit negative samples—thus inheriting its regularization benefits\.

## IIIMethodology

### III\-AProblem Formulation

Let𝒮=\{\(𝐗\(i\),𝐂\(i\)\)\}i=1N\\mathcal\{S\}=\\\{\(\\mathbf\{X\}^\{\(i\)\},\\mathbf\{C\}^\{\(i\)\}\)\\\}\_\{i=1\}^\{N\}denote a dataset of multivariate time series, where𝐗\(i\)∈ℝdx×Ti\\mathbf\{X\}^\{\(i\)\}\\in\\mathbb\{R\}^\{d\_\{x\}\\times T\_\{i\}\}are target sequences and𝐂\(i\)∈ℝdc×Ti\\mathbf\{C\}^\{\(i\)\}\\in\\mathbb\{R\}^\{d\_\{c\}\\times T\_\{i\}\}are associated covariates\. Given the unmasked observations𝐗obs\\mathbf\{X\}\_\{\\text\{obs\}\}and the corresponding covariates𝐂\\mathbf\{C\}, the objective is to learn model parametersθ\\thetasuch that the modelfθf\_\{\\theta\}predicts the distribution parameters𝝍^\\hat\{\\boldsymbol\{\\psi\}\}for the masked subset𝐗ℳ\\mathbf\{X\}\_\{\\mathcal\{M\}\}of the target sequence\.

This leads to the following optimization problem:

minθ\\displaystyle\\min\_\{\\theta\}𝔼\(𝐗,𝐂\)∼p⁡\(𝒮\)​𝔼ℳ∼p⁡\(𝒯∣𝒮\)​\[ℒnll​\(𝐗ℳ,𝝍^\)\]\\displaystyle\\mathbb\{E\}\_\{\(\\mathbf\{X\},\\mathbf\{C\}\)\\sim p\(\\mathcal\{S\}\)\}\\mathbb\{E\}\_\{\\mathcal\{M\}\\sim p\(\\mathcal\{T\}\\mid\\mathcal\{S\}\)\}\\left\[\\mathcal\{L\}\_\{\\text\{nll\}\}\\left\(\\mathbf\{X\}\_\{\\mathcal\{M\}\},\\hat\{\\boldsymbol\{\\psi\}\}\\right\)\\right\]\(1\)s\.t\.\\displaystyle\\text\{s\.t\.\}𝝍^=fθ​\(𝐗obs,𝐂\)\\displaystyle\\hat\{\\boldsymbol\{\\psi\}\}=f\_\{\\theta\}\(\\mathbf\{X\}\_\{\\text\{obs\}\},\\mathbf\{C\}\)Here,ℒnll\\mathcal\{L\}\_\{\\text\{nll\}\}denotes thenegative log\-likelihood loss:

ℒnll​\(𝐗ℳ,𝝍^\)=−log⁡p⁡\(𝐗ℳ∣𝝍^\)\\mathcal\{L\}\_\{\\text\{nll\}\}\(\\mathbf\{X\}\_\{\\mathcal\{M\}\},\\hat\{\\boldsymbol\{\\psi\}\}\)=\-\\log p\(\\mathbf\{X\}\_\{\\mathcal\{M\}\}\\mid\\hat\{\\boldsymbol\{\\psi\}\}\)\(2\)wherep⁡\(𝒮\)p\(\\mathcal\{S\}\)is the data\-generating distribution over time series instances\(𝐗,𝐂\)\(\\mathbf\{X\},\\mathbf\{C\}\), andp⁡\(𝒯∣𝒮\)p\(\\mathcal\{T\}\\mid\\mathcal\{S\}\)defines the task sampling distribution that governs the selection of masked positionsℳ⊂\{1,…,T\}\\mathcal\{M\}\\subset\\\{1,\\dots,T\\\}for prediction\. Classical forecasting corresponds to the special case where the masked regionℳ\\mathcal\{M\}is located at the end of the sequence\.

![Refer to caption](https://arxiv.org/html/2608.20005v1/Fig2_frame.png)

Fig\. 2:Overview of the SATS framework\. Tokens from multiple patch sizes are projected via separate FFNs\. SATS employsScale\-aware Alignmentmechanism to promote proximity of mean\-pooled representations within each scale, while enforcing separation of max\-pooled representations across scales—balancing consistency and scale\-specific expressiveness\.Hybrid masking strategy, integrating Random Masking and Continuous Masking, is further applied to capture both fine\-grained and long\-range temporal dependencies\.
### III\-BModel Architecture

As shown in Figure[2](https://arxiv.org/html/2608.20005#S3.F2), SATS adopts a non\-overlapping patch\-based, encoder\-only Transformer\[[32](https://arxiv.org/html/2608.20005#bib.bib32)\]\. The multivariate time series is first flattened and, following Moirai\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\], mapped into patches of varying sizes based on the dataset\. To improve efficiency, we adopt packing as a default setting\[[33](https://arxiv.org/html/2608.20005#bib.bib33),[2](https://arxiv.org/html/2608.20005#bib.bib2)\], enabling tokens with different patch sizes from multiple datasets to be packed into a single sequence\. This multi\-scale design introduces inconsistencies in the token space; while packing is not the direct cause, it is an indispensable component of modern scalable training, making it both practical and necessary to develop solutions within this paradigm\.

To mitigate such inconsistencies while embracing the packing paradigm, SATS employs a scale\-aware alignment mechanism: it pulls closer the mean\-pooled representations within the same scale, while pushing apart the max\-pooled ones across scales, ensuring consistency while preserving scale\-specific expressiveness\. Based on this aligned space, a hybrid masking strategy combining random and contiguous patterns is applied to capture both fine\-grained and long\-range dependencies\.

Although not shown, the encoder incorporates key techniques from foundational model pretraining—such as RoPE\[[34](https://arxiv.org/html/2608.20005#bib.bib34)\], SwiGLU\[[35](https://arxiv.org/html/2608.20005#bib.bib35)\], and RMSNorm\[[36](https://arxiv.org/html/2608.20005#bib.bib36)\]—as well as inductive biases specific to time\-series pretraining, including Any\-Variate Bias, Mixture Distribution Output\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\]and RevIN\[[37](https://arxiv.org/html/2608.20005#bib.bib37)\]for modeling inter\-variable dependencies and normalization under distribution shifts\.

#### III\-B1Scale\-aware Alignment

To enhance the effectiveness of universal temporal modeling, especially when dealing with subsequences of varying scales, it is crucial to design a robust embedding space alignment strategy\. Given token sequencesℐ∈ℝL×D\\mathcal\{I\}\\in\\mathbb\{R\}^\{L\\times D\}, whereLLrepresents the maximum input length during training andDDis the hidden layer dimension of the encoder, the challenge arises from the coexistence of tokens originating fromn≤Nn\\leq Ndifferent patch sizes, whereNNdenotes the total number of distinct patch sizes\. A direct approach could be to minimize the feature space distance, such as cosine similarity, between subsequences, encouraging their proximity\. However, this approach faces several challenges: first, the varying lengths of subsequences make it difficult to quantify alignment; second, different samples within the same batch may contain different numbers of subsequences, complicating the application of proximity constraints both within and across samples\. Furthermore, a structured information constraint is necessary to discourage scale\-wise anchor collapse and maintain diverse temporal representations\.

In response to these challenges, we propose the Scale\-aware Alignment \(SA\) method, which integrates two key components\. First, we introduce a pooling mechanism to address the issues of variable subsequence lengths and differing numbers of subsequences across samples\. Specifically, we pool tokens based on their patch sizes to generate the scale\-wise embedding representationY∈ℝN×DY\\in\\mathbb\{R\}^\{N\\times D\}\. In cases where a patch size is absent in a given sample, the corresponding embedding positionYiY\_\{i\}is set to zero \(i≤Ni\\leq N\) to maintain fixed tensor dimensions for batching\. Crucially, a binary validity mask is applied during the computation of the alignment loss to explicitly exclude these zeroed positions from the similarity summation, thereby preventing gradient propagation from missing patch sizes\. Second, inspired by the principles of contrastive learning, we design a structured information constraint: the mean embeddings from different patch sizes are pulled closer to establish neighboring centers in the token space, while the maximal embeddings are repelled to encode scale\-specific information, ensuring richer and more diverse token semantics\. More theoretical analysis is provided in Section[IV\-A](https://arxiv.org/html/2608.20005#S4.SS1)\. To operationalize this constraint, we adopt the InfoNCE framework, as detailed in Equation[3](https://arxiv.org/html/2608.20005#S3.E3)and Equation[4](https://arxiv.org/html/2608.20005#S3.E4), wherecos⁡\(⋅\)\\cos\(\\cdot\)denotes the cosine similarity function andτ\\tauis the temperature parameter\.

ℒclose=−𝔼⁡\[log⁡∑j≠iexp⁡\(cos⁡\(Yi⋅Yj\)/τ\)∑j=1Nexp⁡\(cos⁡\(Yi⋅Yj\)/τ\)\]\\displaystyle\\mathcal\{L\}\_\{\\text\{close\}\}=\-\\mathbb\{E\}\\\!\\left\[\\log\\frac\{\\sum\_\{j\\neq i\}\\exp\\left\(\\cos\(Y\_\{i\}\\cdot Y\_\{j\}\)/\\tau\\right\)\}\{\\sum\_\{j=1\}^\{N\}\\exp\\left\(\\cos\(Y\_\{i\}\\cdot Y\_\{j\}\)/\\tau\\right\)\}\\right\]\(3\)ℒfar=−𝔼⁡\[1−log⁡\(∑j=1Nexp⁡\(cos⁡\(Yi⋅Yj\)/τ\)\)\]\\displaystyle\\mathcal\{L\}\_\{\\text\{far\}\}=\-\\mathbb\{E\}\\\!\\left\[1\-\\log\\left\(\\sum\_\{j=1\}^\{N\}\\exp\\left\(\\cos\(Y\_\{i\}\\cdot Y\_\{j\}\)/\\tau\\right\)\\right\)\\right\]\(4\)
In practice,Yi∈YmeanY\_\{i\}\\in Y^\{\\text\{mean\}\}is sequentially substituted into Equation[3](https://arxiv.org/html/2608.20005#S3.E3), whileYi∈YmaxY\_\{i\}\\in Y^\{\\text\{max\}\}is substituted into Equation[4](https://arxiv.org/html/2608.20005#S3.E4)\. Although both equations follow the InfoNCE form, they do not involve true negative samples\. We therefore combine these two losses to form the final scale\-aware alignment constraint in Equation[5](https://arxiv.org/html/2608.20005#S3.E5)\. This design ensures that the loss function conforms to the geometric structure of contrastive learning, as detailed in Section[IV\-B](https://arxiv.org/html/2608.20005#S4.SS2)\. Consequently, it provides structured regularization that aligns feature representations across different patch sizes, enhances cross\-scale consistency, and discourages scale\-wise anchor collapse\. The hyperparameterβ\\betacontrols the relative weight of the maximal embedding pull\-away term, balancing the overall objective\.

ℒsa=ℒclose\+β​ℒfar\\mathcal\{L\}\_\{\\text\{sa\}\}=\\mathcal\{L\}\_\{\\text\{close\}\}\+\\beta\\mathcal\{L\}\_\{\\text\{far\}\}\(5\)

#### III\-B2Hybrid Masking Strategy

On top of the aligned token space, the intrinsic heterogeneity and complexity of temporal dynamics across datasets continue to challenge effective representation learning\. Although alignment mitigates certain variations, temporal dependencies inherently span multiple scales: some manifest as fine\-grained, localized fluctuations within individual tokens, while others emerge as extended, structured patterns across contiguous token segments\. To comprehensively capture these diverse temporal scales and improve the robustness of learned representations, we therefore propose a Hybrid Masking Strategy \(HM\) that synergistically combines Random Masking \(RM\) with Continuous Masking \(CM\), implemented as contiguous\-span masking during pretraining\.

Concretely, given each token subsequenceℐj∈ℝLj×D\\mathcal\{I\}\_\{j\}\\in\\mathbb\{R\}^\{L\_\{j\}\\times D\}extracted from the full sequenceℐ\\mathcal\{I\}, whereLjL\_\{j\}denotes the length of thejj\-th subsequence, a masking ratior∈\[0\.15,0\.5\]r\\in\[0\.15,0\.5\]is applied\. For each subsequence, a predefined probabilityp∈\[0,1\]p\\in\[0,1\]determines whether RM or CM is used\. With probabilitypp, RM uniformly selectsmjm\_\{j\}token positions, wheremj=⌈r⋅Lj⌉m\_\{j\}=\\lceil r\\cdot L\_\{j\}\\rceiland∑i=0Lj−1ℳr\(j\)​\(i\)=mj\\sum\_\{i=0\}^\{L\_\{j\}\-1\}\\mathcal\{M\}\_\{r\}^\{\(j\)\}\(i\)=m\_\{j\}, producing a binary maskℳr\(j\)\\mathcal\{M\}\_\{r\}^\{\(j\)\}:

ℳr\(j\)​\(i\)=\{1,if token​i​is randomly selected0,otherwise\.\\mathcal\{M\}\_\{r\}^\{\(j\)\}\(i\)=\\begin\{cases\}1,&\\text\{if token \}i\\text\{ is randomly selected\}\\\\ 0,&\\text\{otherwise\}\\end\{cases\}\.\(6\)Note thatmj≤Ljm\_\{j\}\\leq L\_\{j\}is inherently satisfied givenr≤0\.5r\\leq 0\.5\. Alternatively, with probability1−p1\-p, CM samples a start indexsj∈\{0,…,Lj−mj\}s\_\{j\}\\in\\\{0,\\ldots,L\_\{j\}\-m\_\{j\}\\\}, masking a continuous block of tokens:

ℳc\(j\)​\(i\)=\{1,sj≤i<sj\+mj0,otherwise\.\\mathcal\{M\}\_\{c\}^\{\(j\)\}\(i\)=\\begin\{cases\}1,&s\_\{j\}\\leq i<s\_\{j\}\+m\_\{j\}\\\\ 0,&\\text\{otherwise\}\.\\end\{cases\}\(7\)The final maskℳ\(j\)\\mathcal\{M\}^\{\(j\)\}applied to each subsequenceℐj\\mathcal\{I\}\_\{j\}is sampled as

ℳ\(j\)=\{ℳr\(j\),with probability​pℳc\(j\),with probability​1−p\.\\mathcal\{M\}^\{\(j\)\}=\\begin\{cases\}\\mathcal\{M\}\_\{r\}^\{\(j\)\},&\\text\{with probability \}p\\\\ \\mathcal\{M\}\_\{c\}^\{\(j\)\},&\\text\{with probability \}1\-p\.\\end\{cases\}\(8\)
By guiding the model to recover masked tokens across both RM and CM patterns, HM balances fine\-grained local inference and long\-range dependency learning\. Consequently, it enhances the robustness and generalizability of learned representations for diverse temporal modeling tasks\.

### III\-CModel Training

#### III\-C1Unified Learning Objective

The proposed alignment and masking strategies introduce no additional learnable parameters, enabling their integration into a unified learning objective without parameter overhead\. In practice, the maskℳ\\mathcal\{M\}obtained from Equation[8](https://arxiv.org/html/2608.20005#S3.E8)is applied to Equation[2](https://arxiv.org/html/2608.20005#S3.E2)to compute the primary training loss\. Simultaneously, Equation[5](https://arxiv.org/html/2608.20005#S3.E5)is employed as an auxiliary training loss to enforce SA\. We combine them into the total loss function as follows:

ℒ=ℒnll\+α​ℒsa\\mathcal\{L\}=\\mathcal\{L\}\_\{\\text\{nll\}\}\+\\alpha\\mathcal\{L\}\_\{\\text\{sa\}\}\(9\)whereα\\alphais a weighting coefficient balancing the two objectives\.

#### III\-C2SATS Setup

We pretrain the SATS models on the LOTSA dataset\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\]in two configurations—small and base—with detailed model specifications provided in Table[I](https://arxiv.org/html/2608.20005#S3.T1)\. The small model is trained for 100,000 steps with a batch size of 256, while the base model is trained for 200,000 steps with a batch size of 128\. All experiments use the following fixed hyperparameters unless otherwise specified:

- •Optimizer: AdamW with learning rate1×10−31\\times 10^\{\-3\}, weight decay1×10−11\\times 10^\{\-1\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.98\\beta\_\{2\}=0\.98\.
- •SA: Temperatureτmean=0\.1\\tau\_\{\\text\{mean\}\}=0\.1\(Eq\.[3](https://arxiv.org/html/2608.20005#S3.E3)\),τmax=0\.2\\tau\_\{\\text\{max\}\}=0\.2\(Eq\.[4](https://arxiv.org/html/2608.20005#S3.E4)\)\.
- •HM: Masking probabilityp=0\.5p=0\.5for balanced RM and CM\.
- •Loss Weights: Primary objective weightα=0\.1\\alpha=0\.1, auxiliary objective weightβ=0\.3\\beta=0\.3\.

Due to limited computational resources and empirical evidence suggesting that large\-scale language model pretraining is relatively robust to hyperparameter choices within reasonable ranges — as performance is primarily governed by scale rather than fine\-tuned hyperparameters\[[38](https://arxiv.org/html/2608.20005#bib.bib38),[39](https://arxiv.org/html/2608.20005#bib.bib39)\]— no further hyperparameter tuning was performed beyond the values listed above\.

TABLE I:Key parameter details of SATS model sizes\.

## IVTheoretical Analysis

### IV\-AMean Consistency and Max Discriminability

We provide a concise justification for the two pooling choices in SA: mean\-pooled representations are pulled together to encourage cross\-scale semantic consistency, whereas max\-pooled representations are pushed apart to preserve scale\-specific information\.

Consider a time series patch decomposed as

x⁡\[n\]=ℓ⁡\[n\]\+h⁡\[n\],x\[n\]=\\ell\[n\]\+h\[n\],whereℓ⁡\[n\]\\ell\[n\]is a low\-frequency trend satisfying the Lipschitz condition\|ℓ⁡\[n\]−ℓ⁡\[m\]\|≤K​\|n−m\|\|\\ell\[n\]\-\\ell\[m\]\|\\leq K\|n\-m\|\[[40](https://arxiv.org/html/2608.20005#bib.bib40),[41](https://arxiv.org/html/2608.20005#bib.bib41)\], andh⁡\[n\]h\[n\]is a zero\-mean high\-frequency component with bounded variance and short\-range dependence\. For a patch of lengthPP, define

μP=1P​∑n=1Px⁡\[n\],MP=max1≤n≤P⁡x⁡\[n\]\.\\mu\_\{P\}=\\frac\{1\}\{P\}\\sum\_\{n=1\}^\{P\}x\[n\],\\qquad M\_\{P\}=\\max\_\{1\\leq n\\leq P\}x\[n\]\.
#### IV\-A1Mean consistency

The mean statistic suppresses high\-frequency fluctuations\. Under short\-range dependence,

Var⁡\(μP\)=1P2​∑n,m=1PCov⁡\(h⁡\[n\],h⁡\[m\]\)≤Ccov​σmax2P,\\mathrm\{Var\}\(\\mu\_\{P\}\)=\\frac\{1\}\{P^\{2\}\}\\sum\_\{n,m=1\}^\{P\}\\mathrm\{Cov\}\(h\[n\],h\[m\]\)\\leq\\frac\{C\_\{\\mathrm\{cov\}\}\\sigma\_\{\\max\}^\{2\}\}\{P\},which shows that the variance of the high\-frequency component decays asO⁡\(1/P\)O\(1/P\)\. Moreover, since the trend component is Lipschitz smooth, the expectation gap between two patch lengths is controlled by their scale difference:

\|𝔼⁡\[μP1\]−𝔼⁡\[μP2\]\|≤K2​\|P1−P2\|\.\\big\|\\mathbb\{E\}\[\\mu\_\{P\_\{1\}\}\]\-\\mathbb\{E\}\[\\mu\_\{P\_\{2\}\}\]\\big\|\\leq\\frac\{K\}\{2\}\|P\_\{1\}\-P\_\{2\}\|\.Thus, mean pooling yields stable scale\-wise prototypes that mainly retain shared low\-frequency semantics\.

#### IV\-A2Max discriminability

In contrast, the max statistic remains sensitive to local high\-frequency responses\. Under mild tail conditions for the high\-frequency component, classical extreme value theory\[[42](https://arxiv.org/html/2608.20005#bib.bib42)\]gives

𝔼⁡\[maxn≤P⁡h⁡\[n\]\]≍σmax​2​log⁡P,\\mathbb\{E\}\\left\[\\max\_\{n\\leq P\}h\[n\]\\right\]\\asymp\\sigma\_\{\\max\}\\sqrt\{2\\log P\},up to lower\-order terms\. Therefore, max\-pooled anchors emphasize scale\-dependent peaks whose magnitude varies with the patch length\. This makes them suitable for preserving scale\-specific information that would be weakened if all scale representations were only pulled together\.

#### IV\-A3Summary

These complementary properties motivate the design of SA\. Pulling mean\-pooled embeddings together promotes a coherent cross\-scale semantic space, while pushing max\-pooled embeddings apart preserves scale\-sensitive anchor diversity and mitigates scale\-wise representation collapse\.

### IV\-BGeometric Correspondence with Contrastive Learning

We further examine the intrinsic connection between SA and contrastive representation learning\. Although our objective does not rely on conventional instance\-level negative sampling, it preserves the core contrastive geometry: aligning related representations while dispersing informative anchors\. This perspective explains why mean\-attraction and max\-repulsion are consistent with contrastive learning principles and can promote cross\-scale consistency while reducing collapse risk\.

#### IV\-B1Structured contrastive geometry

We make this geometric prior explicit through a normalized L2 analytic surrogate:

ℒsaanalytic=∑i=1N‖hi−h¯‖2⏟ℒalign−β​∑i≠t‖mi−mt‖2⏟ℒsep,\\mathcal\{L\}\_\{\\text\{sa\}\}^\{\\text\{analytic\}\}=\\underbrace\{\\sum\_\{i=1\}^\{N\}\\\|h\_\{i\}\-\\bar\{h\}\\\|^\{2\}\}\_\{\\mathcal\{L\}\_\{\\text\{align\}\}\}\-\\beta\\underbrace\{\\sum\_\{i\\neq t\}\\\|m\_\{i\}\-m\_\{t\}\\\|^\{2\}\}\_\{\\mathcal\{L\}\_\{\\text\{sep\}\}\},where\{mi\}i=1N⊂𝕊D−1\\\{m\_\{i\}\\\}\_\{i=1\}^\{N\}\\subset\\mathbb\{S\}^\{D\-1\}are normalized scale\-wise max anchors,hih\_\{i\}are scale\-wise mean prototypes, andh¯\\bar\{h\}is their centroid\. For normalized embeddings,‖x−y‖2=2−2​x⊤​y\\\|x\-y\\\|^\{2\}=2\-2x^\{\\top\}y; hence the first term increases similarity between mean prototypes and a shared semantic center, while the second decreases similarity among max anchors\. The surrogate does not replace the InfoNCE\-style training loss; it exposes the same normalized attraction–repulsion geometry\. Equations[3](https://arxiv.org/html/2608.20005#S3.E3)–[4](https://arxiv.org/html/2608.20005#S3.E4)implement this geometry in smooth softmax form: mean\-pooled prototypes are aligned, and scale\-dominant max anchors are dispersed\. This correspondence yields a structured alignment–uniformity variant\[[43](https://arxiv.org/html/2608.20005#bib.bib43)\], where uniformity acts only on the compact anchor set\{mi\}\\\{m\_\{i\}\\\}rather than all tokens\.

#### IV\-B2Mitigation of Representation Collapse

The separation term provides an anti\-collapse bias for scale\-wise max anchors\. Since these anchors capture scale\-sensitive high\-variation responses, angular separation preserves diversity in the scale\-anchor subspace\. This connects to dimensional collapse, commonly characterized by degeneration of the embedding covariance spectrum\[[44](https://arxiv.org/html/2608.20005#bib.bib44)\]\.

Formally, let\{mi\}i=1N⊂𝕊D−1\\\{m\_\{i\}\\\}\_\{i=1\}^\{N\}\\subset\\mathbb\{S\}^\{D\-1\}denote the normalized max\-pooled scale anchors, withm¯=N−1​∑imi\\bar\{m\}=N^\{\-1\}\\sum\_\{i\}m\_\{i\}and

Cm=1N​∑i=1N\(mi−m¯\)​\(mi−m¯\)⊤\.C\_\{m\}=\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\(m\_\{i\}\-\\bar\{m\}\)\(m\_\{i\}\-\\bar\{m\}\)^\{\\top\}\.Then

tr⁡\(Cm\)=1−‖m¯‖2=12​N2​∑i,j=1N‖mi−mj‖2\.\\mathrm\{tr\}\(C\_\{m\}\)=1\-\\\|\\bar\{m\}\\\|^\{2\}=\\frac\{1\}\{2N^\{2\}\}\\sum\_\{i,j=1\}^\{N\}\\\|m\_\{i\}\-m\_\{j\}\\\|^\{2\}\.Thus, increasing pairwise separation directly increases total anchor variance\. Consequently,ℒsep\\mathcal\{L\}\_\{\\text\{sep\}\}complementsℒalign\\mathcal\{L\}\_\{\\text\{align\}\}by discouraging scale\-wise point collapse and preserving non\-zero variance in the anchor subspace\.

#### IV\-B3Summary

Overall,ℒsa\\mathcal\{L\}\_\{\\text\{sa\}\}is a lightweight anchor\-level contrastive regularizer\. It aligns mean\-pooled prototypes for cross\-scale consistency, disperses max\-pooled anchors for scale\-specific diversity, and reduces scale\-wise collapse risk without large\-batch negative sampling\.

## VExperiments

### V\-ABenchmark Description

We evaluate SATS on three complementary benchmark collections: LSTF\[[45](https://arxiv.org/html/2608.20005#bib.bib45)\], GIFT\-Eval\[[46](https://arxiv.org/html/2608.20005#bib.bib46)\], and Monash\[[47](https://arxiv.org/html/2608.20005#bib.bib47)\]\. Together, they cover long\-horizon forecasting, zero\-shot cross\-dataset generalization, and in\-distribution evaluation across diverse domains, frequencies, and horizons\.

- •LSTF: A collection of widely used multivariate datasets from domains such as electricity, traffic, and weather, used here to evaluate long\-term forecasting\.
- •GIFT\-Eval: A general forecasting benchmark for evaluating transfer to unseen heterogeneous datasets\. It spans diverse domains, frequencies, variable types, and prediction horizons, and reports both point and probabilistic forecasting quality through metrics such as MASE and CRPS\.
- •Monash: A large\-scale collection of univariate real\-world datasets, used here for in\-distribution evaluation under heterogeneous data distributions\.

TABLE II:Full results of zero\-shot forecasting across all evaluated models\. Lower values of MSE and MAE indicate superior performance\. As TimesFM incorporates Weather data during pretraining, it is excluded from evaluation on this dataset \(denoted by “–”\)\.Redhighlights the best result, whileBluemarks the second best\.
### V\-BBenchmarking Setup

Baselines\.We conduct extensive comparisons with widely adopted time series foundation models, including Timer\-XL\[[48](https://arxiv.org/html/2608.20005#bib.bib48)\], Time\-MoE\[[11](https://arxiv.org/html/2608.20005#bib.bib11)\], Moirai\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\], Chronos\[[13](https://arxiv.org/html/2608.20005#bib.bib13)\], Moment\[[27](https://arxiv.org/html/2608.20005#bib.bib27)\], TimesFM\[[49](https://arxiv.org/html/2608.20005#bib.bib49)\], and TTM\-R2\[[50](https://arxiv.org/html/2608.20005#bib.bib50)\]\. To broaden the comparative scope, we additionally incorporate two methods adapted from foundation models of other modalities: VisionTS\[[51](https://arxiv.org/html/2608.20005#bib.bib51)\]and LLMTime\[[52](https://arxiv.org/html/2608.20005#bib.bib52)\]\. Furthermore, following the recommendations of Bergmeir\[[53](https://arxiv.org/html/2608.20005#bib.bib53)\], we extend our in\-distribution evaluation to encompass traditional baselines, including Naive, ETS\[[54](https://arxiv.org/html/2608.20005#bib.bib54)\], and DeepAR\[[55](https://arxiv.org/html/2608.20005#bib.bib55)\]\.

Evaluation Setup\.Baselines follow their original configurations\. For SATS, we follow Moirai, search context lengths in \{1000, 2000, 3000, 4000, 5000\}, and use Woo et al\.’s frequency\-specific patch candidates\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\]:

- •Yearly/Quarterly: 8
- •Monthly: 8, 16, 32
- •Weekly/Daily: 16, 32
- •Hourly: 32, 64
- •Minute\-level: 32, 64, 128
- •Second\-level: 64, 128

Empirically, we choose the largest feasible patch size with a lookback of at least 3000\. Metrics use 100 predictive samples and report results based on the mean prediction\. As pretrained baselines often impose model\-specific input lengths, such as Time\-MoE’s four\-horizon input\[[11](https://arxiv.org/html/2608.20005#bib.bib11)\]and Timer\-XL’s dataset\-dependent lengths\[[48](https://arxiv.org/html/2608.20005#bib.bib48)\], we follow each model’s recommended setting for fairness\.

![Refer to caption](https://arxiv.org/html/2608.20005v1/Fig3_GE.png)

Fig\. 3:Zero\-shot forecasting performance evaluated on 23 datasets from GIFT\-Eval benchmark\[[46](https://arxiv.org/html/2608.20005#bib.bib46)\]\. Lower values of MASE and CRPS indicate superior performance\. Methods trained with access to these evaluation datasets during pretraining are denoted with asterisks \(\*\)\. Results were aggregated in accordance with the standard method of GIFT\-Eval\.
### V\-CZero\-shot Forecasting

We evaluate zero\-shot forecasting on LSTF and GIFT\-Eval\. LSTF tests long\-horizon forecasting on datasets excluded from pretraining, while GIFT\-Eval tests cross\-dataset generalization under a leakage\-controlled protocol\.

#### V\-C1LSTF Benchmark

We evaluate five LSTF datasets excluded from LOTSA, using horizons\{96,192,336,720\}\\\{96,192,336,720\\\}and reporting MSE and MAE\. For baselines with multiple variants, we exclude models above 1B parameters and report the variant with the best average performance\. Electricity, Traffic, and PEMS are excluded because they appear in the pretraining corpora of most models\[[48](https://arxiv.org/html/2608.20005#bib.bib48),[11](https://arxiv.org/html/2608.20005#bib.bib11),[12](https://arxiv.org/html/2608.20005#bib.bib12)\]\.

Result\.The detailed zero\-shot results are presented in Table[II](https://arxiv.org/html/2608.20005#S5.T2), whereSATSB\\text\{SATS\}\_\{\\text\{B\}\}consistently achieves state\-of\-the\-art performance\. Compared toMoiraiB\\text\{Moirai\}\_\{\\text\{B\}\}\(encoder\-only\), the strongest multi\-scale FFNs\-embedded baseline,SATSB\\text\{SATS\}\_\{\\text\{B\}\}achieves a9\.2%improvement in MSE\. It also outperforms Timer\-XL \(decoder\-only\) andChronosL\\text\{Chronos\}\_\{\\text\{L\}\}\(encoder\-decoder\) with MSE improvements of4\.2%and27\.4%, respectively\. Notably,SATSB\\text\{SATS\}\_\{\\text\{B\}\}contains only 70M parameters, which is substantially fewer than those of the compared baselines\. Moreover, even the lightweightSATSS\\text\{SATS\}\_\{\\text\{S\}\}with 14M parameters surpasses all other baselines in overall average performance, highlighting its efficiency\. In addition, SATS exhibits a clear performance gain as model size increases, revealing strong scalability\. This trend contrasts with models such as Time\-MoE\[[11](https://arxiv.org/html/2608.20005#bib.bib11)\]and Moirai\[[12](https://arxiv.org/html/2608.20005#bib.bib12)\], whose performance plateaus or even degrades with larger model configurations\.

#### V\-C2GIFT\-Eval Benchmark

We follow the standard GIFT\-Eval protocol on 23 datasets spanning economics, energy, healthcare, nature, sales, transportation, and cloud operations\. To enforce a strict zero\-shot setting, we remove all GIFT\-Eval overlaps from LOTSA and retrain SATS\. Performance is aggregated by the official protocol, covering both point and probabilistic forecasting metrics\.

Result\.The detailed zero\-shot results are shown in Figure[3](https://arxiv.org/html/2608.20005#S5.F3)\. Overall,SATSB\\text\{SATS\}\_\{\\text\{B\}\}delivers consistently state\-of\-the\-art performance across metrics\. Against the strongest encoder\-only baseline,MoiraiL\\text\{Moirai\}\_\{\\text\{L\}\}, it yields substantial gains—8\.3%in MASE and8\.2%in CRPS\. Notably, even when set against the runner\-up,Chronos\-BB\\text\{Chronos\-B\}\_\{\\text\{B\}\}, SATS maintains a clear lead in CRPS under a strictly leakage\-free evaluation protocol while requiring only35%of its parameters, underscoring its ability to learn highly transferable temporal representations during pretraining\. Beyond encoder\-based models, SATS also surpasses VisionTS—which converts sequences into images for training—with consistently more stable improvements on both MASE and CRPS\. This suggests that SATS captures the underlying data distribution more faithfully, rather than merely improving point forecasts as VisionTS tends to do\. Together, these findings highlight SATS’s robust cross\-dataset generalization under both point and probabilistic forecasting metrics\.

### V\-DIn\-distribution Forecasting

We evaluate in\-distribution forecasting on 29 Monash datasets\[[47](https://arxiv.org/html/2608.20005#bib.bib47)\]\. Only training portions are included in LOTSA, while test sets are reserved for evaluation\. Results are reported as normalized MAE relative to a naive forecast and aggregated by the geometric mean across datasets\.

Fig\. 4:In\-distribution forecasting performance evaluated on 29 datasets from the Monash benchmark\[[47](https://arxiv.org/html/2608.20005#bib.bib47)\]\. Methods trained with access to these evaluation datasets during pretraining are denoted with asterisks \(\*\)\. Results are normalized using the naive forecast and summarized with the geometric mean\.TABLE III:Module ablation under zero\-shot evaluation\.RedandBlueindicate the best and second\-best results\.
TABLE IV:Alignment ablation under zero\-shot evaluation\. A dash denotes a removed objective;RedandBlueindicate the best and second\-best results\.

Result\.As shown in Figure[4](https://arxiv.org/html/2608.20005#S5.F4),SATSconsistently outperforms all competing methods\. Compared toMoiraiL\{\}\_\{\\text\{L\}\}, the best baseline trained on clean data,SATSB\{\}\_\{\\text\{B\}\}achieves a6\.9%improvement while usingonly 22\.6%of its parameters\. Similarly, againstChronosS\{\}\_\{\\text\{S\}\}, the strongest baseline under data contamination,SATSS\{\}\_\{\\text\{S\}\}achieves superior performance with just30\.4%of its parameter count\. Notably, the gain fromSATSS\{\}\_\{\\text\{S\}\}toSATSB\{\}\_\{\\text\{B\}\}is modest, likely because in\-distribution forecasting involves limited temporal complexity, where increasing model size yields diminishing returns\.

### V\-EAblation Studies11footnotemark:1

22footnotetext:ETTh/ETTm are means of ETTh1/ETTh2 and ETTm1/ETTm2 after horizon averaging\.#### V\-E1Module Design

We begin by performing ablation studies on the components ofSATSB\{\}\_\{\\text\{B\}\}, with results summarized in Table[IV](https://arxiv.org/html/2608.20005#S5.T4)\. On ETT, removing SA consistently degrades performance, while discarding either HM component causes larger drops in most cases; Weather shows a similar MSE pattern, with relatively small MAE differences\. These results indicate that HM is central to long\-term forecasting, where random and contiguous masking jointly encourage robust temporal representations, while SA provides complementary gains by regularizing cross\-scale token spaces\. On GIFT\-Eval, removing SA yields the worst results, whereas removing individual HM components has a smaller impact\. This contrast suggests that HM mainly strengthens long\-horizon dependency learning on LSTF, while SA is more critical for cross\-dataset generalization\. Together with the t\-SNE visualizations in Section[V\-F1](https://arxiv.org/html/2608.20005#S5.SS6.SSS1), these findings show that SA forms a stable, structured embedding space and that the two components provide complementary benefits\.

#### V\-E2Alignment Mechanism

The key design of SA is to align mean embeddings while separating maximal embeddings, thereby preserving both cross\-scale consistency and scale\-anchor diversity\. We therefore vary the pooling choices in Table[IV](https://arxiv.org/html/2608.20005#S5.T4)to examine whether these two roles are necessary\. Removing the max\-embedding repulsion term causes a clear performance drop, confirming the importance of scale\-anchor separation\. Replacing max pooling with min or random pooling generally yields degraded or less stable results, indicating that maximal embeddings better capture scale\-specific information\. Applying alignment only to maximal embeddings also remains suboptimal in most cases, suggesting that this constraint is too weak to prevent scale\-wise anchor collapse\. Overall, bringing mean embeddings closer while pushing maximal embeddings apart is a robust design choice, consistent with the theoretical analysis in Section[IV\-A](https://arxiv.org/html/2608.20005#S4.SS1)\.

#### V\-E3Transformer Types

TABLE V:Transformer\-type ablation under zero\-shot evaluation\. Values are average MSE or MASE; “\-” denotes an inapplicable setting, andRedmarks the best result\.We further assess architectural generality by integrating the proposed modules into three Transformer variants, omitting HM for the decoder\-only model because its pre\-training objective is incompatible with hierarchical masking\. As shown in Table[V](https://arxiv.org/html/2608.20005#S5.T5), SA and HM bring consistent gains across applicable architectures: HM mainly improves long\-horizon forecasting, while SA benefits cross\-dataset generalization\. Under the same setting, the decoder\-only model performs best, likely due to denser training signals\[[39](https://arxiv.org/html/2608.20005#bib.bib39)\], though its incompatibility with HM limits further gains\. By contrast, the larger encoder\-decoder variant often trails the encoder\-only model, possibly because additional cross\-attention layers introduce optimization instability and gradient dilution\[[56](https://arxiv.org/html/2608.20005#bib.bib56)\]\. Overall, these results suggest that SA and HM provide architecture\-agnostic improvements\.

### V\-FModel Analysis

#### V\-F1Mechanism Analysis

To understand what SA learns beyond forecasting gains, we examine whether it organizes multi\-resolution tokens into a coherent yet scale\-discriminative embedding space\. We consider both a sufficient\-scale regime with representative tokens at each scale and a scarce\-scale regime with underrepresented patch sizes\. In Figure[5](https://arxiv.org/html/2608.20005#S5.F5), SATS forms compact yet separable clusters across patch sizes, even when size\-8 tokens are scarce, whereas w/o SA exhibits confusion between patch sizes 16 and 32 and nearly absorbs scarce size\-8 tokens\. This visual pattern is consistent with Table[VI](https://arxiv.org/html/2608.20005#S5.T6), where downsampled patches at resolutions \{8, 16, 32, 64\} are compared with the original size\-128 embeddings\. SATS achieves very high mean\-pooling similarity \(0\.979–0\.998\), indicating cross\-resolution alignment, while retaining lower max\-pooling similarity \(0\.424–0\.639\), indicating preserved scale\-specific anchors\. Conversely, w/o SA shows weak or negative mean similarity \(\-0\.054–0\.293\) but high max similarity \(0\.721–0\.889\), revealing fragmented global semantics and a tendency toward local\-anchor collapse\. Thus, the qualitative clusters and quantitative geometry tell a consistent story: SA pulls semantically matched resolutions toward a shared center while separating scale\-specific maxima, producing coherent representations that ease Transformer pretraining and support generalization\.

![Refer to caption](https://arxiv.org/html/2608.20005v1/Fig5_tsne.png)

Fig\. 5:T\-SNE token distributions in sufficient\- and scarce\-scale regimes\. Colors denote patch\-size origins\.TABLE VI:Cosine similarity between multi\-resolution patch embeddings \(\{8, 16, 32, 64\}\) and size\-128 representations under
mean and max pooling\.
#### V\-F2Patch Protocol Sensitivity

![Refer to caption](https://arxiv.org/html/2608.20005v1/Fig6_sen.png)

Fig\. 6:Sensitivity analysis of patch protocols on the LSTF benchmark\. The charts report average MSE \(lower is better\) across different patch sizes\.Fig\. 7:Model efficiency under zero\-shot LSTF evaluation, measured as1MSE⋅log⁡\(N\+1\)\\frac\{1\}\{\\text\{MSE\}\\cdot\\log\(N\+1\)\}\. Higher values indicate better accuracy\-parameter trade\-offs for foundation models, whose parameter count meaningfully proxies capacity\.To further understand the mechanism of SATS and guide patch\-size selection when sampling\-frequency metadata is noisy or unavailable, we evaluate its sensitivity to patch protocols on the LSTF benchmark\. As shown in Figure[7](https://arxiv.org/html/2608.20005#S5.F7), the prescribed protocol consistently yields the best performance, while deviations exhibit an asymmetric effect: using patch sizes smaller than the protocol causes substantial degradation, whereas using larger patch sizes leads to only mild performance loss\. This suggests that excessively small patches fragment temporal semantics and fail to capture complete patterns\[[32](https://arxiv.org/html/2608.20005#bib.bib32)\], while larger patches can still retain macroscopic trends through effective downsampling\[[57](https://arxiv.org/html/2608.20005#bib.bib57)\]\. Therefore, in zero\-shot scenarios with unreliable metadata, a conservative strategy is to prefer larger patch sizes within overlapping protocol ranges; when metadata is absent, spectral analysis can be used to select the closest frequency\-based protocol\.

#### V\-F3Model Efficiency

To examine whether the gains of SATS translate into a practical accuracy\-parameter trade\-off, we evaluate model efficiency under the zero\-shot LSTF setting\. Since the proposed alignment and masking strategies introduce no additional learnable parameters to the multi\-FFNs backbone, we measure efficiency as the inverse of the product between zero\-shot error and the logarithm of model size\. As shown in Figure[7](https://arxiv.org/html/2608.20005#S5.F7), SATS achieves a strong balance between accuracy and compactness:SATSB\\text\{SATS\}\_\{\\text\{B\}\}attains SOTA accuracy and surpasses the runner\-up model, Timer\-XL, by8\.9%in efficiency, while the lightweightSATSS\\text\{SATS\}\_\{\\text\{S\}\}delivers a65\.6%efficiency improvement despite only slightly outperforming Timer\-XL in accuracy\. These results indicate that SATS improves predictive performance without relying on parameter expansion, making it suitable for resource\-constrained or real\-time deployment\.

## VIConclusion

This paper presents SATS, aScale\-Aware foundation model forTimeSeries that addresses the challenge of fragmented token spaces and misaligned representations in time series pretraining\. SA is introduced to unify representations across patch sizes by jointly minimizing inter\-scale embedding discrepancies and preserving scale\-specific modeling capacity\. Furthermore, HM combines RM and CM to capture temporal dependencies at multiple resolutions\. Extensive experiments demonstrate that SATS achieves superior generalization and robustness while maintaining high efficiency, as its proposed alignment and masking strategies introduce no additional learnable parameters\.

## References

- \[1\]T\. Brown, B\. Mann, N\. Ryder, M\. Subbiah, J\. D\. Kaplan, P\. Dhariwal, A\. Neelakantan, P\. Shyam, G\. Sastry, A\. Askell*et al\.*, “Language models are few\-shot learners,”*Advances in Neural Information Processing Systems*, vol\. 33, pp\. 1877–1901, 2020\.
- \[2\]A\. Dubey, A\. Jauhri, A\. Pandey, A\. Kadian, A\. Al\-Dahle, A\. Letman, A\. Mathur, A\. Schelten, A\. Yang, A\. Fan*et al\.*, “The llama 3 herd of models,” 2024, arXiv:2407\.21783\.
- \[3\]M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\.\-Y\. Huang, S\.\-W\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jegou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. Bojanowski, “Dinov2: Learning robust visual features without supervision,”*Transactions on Machine Learning Research*, 2024\.
- \[4\]A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever, “Learning transferable visual models from natural language supervision,” in*Proceedings of the 38th International Conference on Machine Learning*, vol\. 139, 2021, pp\. 8748–8763\.
- \[5\]A\. Baevski, Y\. Zhou, A\. Mohamed, and M\. Auli, “wav2vec 2\.0: A framework for self\-supervised learning of speech representations,”*Advances in Neural Information Processing Systems*, vol\. 33, pp\. 12 449–12 460, 2020\.
- \[6\]A\. Radford, J\. W\. Kim, T\. Xu, G\. Brockman, C\. Mcleavey, and I\. Sutskever, “Robust speech recognition via large\-scale weak supervision,” in*Proceedings of the 40th International Conference on Machine Learning*, vol\. 202, 2023, pp\. 28 492–28 518\.
- \[7\]D\. Cao, F\. Jia, S\. O\. Arik, T\. Pfister, Y\. Zheng, W\. Ye, and Y\. Liu, “Tempo: Prompt\-based generative pre\-trained transformer for time series forecasting,” in*International Conference on Learning Representations*, 2024\.
- \[8\]M\. Jin, S\. Wang, L\. Ma, Z\. Chu, J\. Y\. Zhang, X\. Shi, P\.\-Y\. Chen, Y\. Liang, Y\.\-F\. Li, S\. Pan, and Q\. Wen, “Time\-llm: Time series forecasting by reprogramming large language models,” in*International Conference on Learning Representations*, 2024\.
- \[9\]Z\. Pan, Y\. Jiang, S\. Garg, A\. Schneider, Y\. Nevmyvaka, and D\. Song, “s2s^\{2\}IP\-LLM: Semantic space informed prompt learning with LLM for time series forecasting,” in*Proceedings of the 41st International Conference on Machine Learning*, vol\. 235, 2024, pp\. 39 135–39 153\.
- \[10\]M\. Tan, M\. Merrill, V\. Gupta, T\. Althoff, and T\. Hartvigsen, “Are language models actually useful for time series forecasting?”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 60 162–60 191, 2024\.
- \[11\]X\. Shi, S\. Wang, Y\. Nie, D\. Li, Z\. Ye, Q\. Wen, and M\. Jin, “Time\-moe: Billion\-scale time series foundation models with mixture of experts,” in*International Conference on Learning Representations*, 2025\.
- \[12\]G\. Woo, C\. Liu, A\. Kumar, C\. Xiong, S\. Savarese, and D\. Sahoo, “Unified training of universal time series forecasting transformers,” in*Proceedings of the 41st International Conference on Machine Learning*, vol\. 235, 2024, pp\. 53 140–53 164\.
- \[13\]A\. F\. Ansari, L\. Stella, C\. Turkmen, X\. Zhang, P\. Mercado, H\. Shen, O\. Shchur, S\. S\. Rangapuram, S\. P\. Arango, S\. Kapoor, J\. Zschiegner, D\. C\. Maddix, H\. Wang, M\. W\. Mahoney, K\. Torkkola, A\. G\. Wilson, M\. Bohlke\-Schneider, and Y\. Wang, “Chronos: Learning the language of time series,”*Transactions on Machine Learning Research*, 2024\.
- \[14\]R\. Sennrich, B\. Haddow, and A\. Birch, “Neural machine translation of rare words with subword units,” in*Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, 2016, pp\. 1715–1725\.
- \[15\]H\. Touvron, M\. Cord, M\. Douze, F\. Massa, A\. Sablayrolles, and H\. Jegou, “Training data\-efficient image transformers & distillation through attention,” in*Proceedings of the 38th International Conference on Machine Learning*, vol\. 139, 2021, pp\. 10 347–10 357\.
- \[16\]A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” in*International Conference on Learning Representations*, 2021\.
- \[17\]J\. Zhang, S\. Zheng, X\. Wen, X\. Zhou, J\. Bian, and J\. Li, “Elastst: Towards robust varied\-horizon forecasting with elastic time\-series transformer,”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 119 174–119 197, 2024\.
- \[18\]T\. Ronen, O\. Levy, and A\. Golbert, “Vision transformers with mixed\-resolution tokenization,” in*Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops*, 2023, pp\. 4613–4622\.
- \[19\]Y\. Wang, Y\. Qiu, P\. Chen, Y\. Shu, Z\. Rao, L\. Pan, B\. Yang, and C\. Guo, “LightGTS: A lightweight general time series forecasting model,” in*Proceedings of the 42nd International Conference on Machine Learning*, vol\. 267, 2025, pp\. 64 109–64 126\.
- \[20\]X\. Liu, J\. Liu, G\. Woo, T\. Aksu, Y\. Liang, R\. Zimmermann, C\. Liu, S\. Savarese, C\. Xiong, and D\. Sahoo, “Moirai\-moe: Empowering time series foundation models with sparse mixture of experts,” in*Proceedings of the 42nd International Conference on Machine Learning*, vol\. 267, 2025, pp\. 38 940–38 962\.
- \[21\]K\. Ding, F\. Fan, C\. Hou, Z\. Wang, L\. Wang, Z\. Yang, and J\. Zhan, “Timemosaic: Temporal heterogeneity guided time series forecasting via adaptive granularity patch and segment\-wise decoding,”*Proceedings of the AAAI Conference on Artificial Intelligence*, vol\. 40, no\. 25, pp\. 20 790–20 798, 2026\.
- \[22\]P\. Chen, Y\. Zhang, Y\. Cheng, Y\. Shu, Y\. Wang, Q\. Wen, B\. Yang, and C\. Guo, “Pathformer: Multi\-scale transformers with adaptive pathways for time series forecasting,” in*International Conference on Learning Representations*, 2024\.
- \[23\]S\. Wang, H\. Wu, X\. Shi, T\. Hu, H\. Luo, L\. Ma, J\. Y\. Zhang, and J\. Zhou, “Timemixer: Decomposable multiscale mixing for time series forecasting,” in*International Conference on Learning Representations*, 2024\.
- \[24\]Y\. Liu, H\. Zhang, C\. Li, X\. Huang, J\. Wang, and M\. Long, “Timer: Generative pre\-trained transformers are large time series models,” in*Proceedings of the 41st International Conference on Machine Learning*, vol\. 235, 2024, pp\. 32 369–32 399\.
- \[25\]K\. Rasul, A\. Ashok, A\. R\. Williams, H\. Ghonia, R\. Bhagwatkar, A\. Khorasani, M\. J\. D\. Bayazi, G\. Adamopoulos, R\. Riachi, N\. Hassen, M\. Biloš, S\. Garg, A\. Schneider, N\. Chapados, A\. Drouin, V\. Zantedeschi, Y\. Nevmyvaka, and I\. Rish, “Lag\-llama: Towards foundation models for probabilistic time series forecasting,” 2024, arXiv:2310\.08278\.
- \[26\]Q\. Yao, C\.\-H\. H\. Yang, R\. Jiang, Y\. Liang, M\. Jin, and S\. Pan, “Towards neural scaling laws for time series foundation models,” 2025, arXiv:2410\.12360\.
- \[27\]M\. Goswami, K\. Szafer, A\. Choudhry, Y\. Cai, S\. Li, and A\. Dubrawski, “Moment: A family of open time\-series foundation models,” in*Proceedings of the 41st International Conference on Machine Learning*, vol\. 235, 2024, pp\. 16 115–16 152\.
- \[28\]T\. Gao, X\. Yao, and D\. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” in*Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing*, 2021, pp\. 6894–6910\.
- \[29\]C\. Jia, Y\. Yang, Y\. Xia, Y\.\-T\. Chen, Z\. Parekh, H\. Pham, Q\. Le, Y\.\-H\. Sung, Z\. Li, and T\. Duerig, “Scaling up visual and vision\-language representation learning with noisy text supervision,” in*Proceedings of the 38th International Conference on Machine Learning*, vol\. 139, 2021, pp\. 4904–4916\.
- \[30\]E\. Eldele, M\. Ragab, Z\. Chen, M\. Wu, C\. K\. Kwoh, X\. Li, and C\. Guan, “Time\-series representation learning via temporal and contextual contrasting,” in*Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI\-21*, 2021, pp\. 2352–2359\.
- \[31\]G\. Woo, C\. Liu, D\. Sahoo, A\. Kumar, and S\. Hoi, “Cost: Contrastive learning of disentangled seasonal\-trend representations for time series forecasting,” in*International Conference on Learning Representations*, 2022\.
- \[32\]Y\. Nie, N\. H\. Nguyen, P\. Sinthong, and J\. Kalagnanam, “A time series is worth 64 words: Long\-term forecasting with transformers,” in*International Conference on Learning Representations*, 2023\.
- \[33\]M\. M\. Krell, M\. Kosec, S\. P\. Perez, and A\. Fitzgibbon, “Efficient sequence packing without cross\-contamination: Accelerating large language models without impacting performance,” 2022, arXiv:2107\.02027\.
- \[34\]J\. Su, M\. Ahmed, Y\. Lu, S\. Pan, W\. Bo, and Y\. Liu, “RoFormer: Enhanced transformer with rotary position embedding,”*Neurocomputing*, vol\. 568, p\. 127063, 2024\.
- \[35\]N\. Shazeer, “Glu variants improve transformer,” 2020, arXiv:2002\.05202\.
- \[36\]B\. Zhang and R\. Sennrich, “Root mean square layer normalization,”*Advances in Neural Information Processing Systems*, vol\. 32, 2019\.
- \[37\]T\. Kim, J\. Kim, Y\. Tae, C\. Park, J\.\-H\. Choi, and J\. Choo, “Reversible instance normalization for accurate time\-series forecasting against distribution shift,” in*International Conference on Learning Representations*, 2022\.
- \[38\]Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov, “Roberta: A robustly optimized bert pretraining approach,” 2019, arXiv:1907\.11692\.
- \[39\]J\. Kaplan, S\. McCandlish, T\. Henighan, T\. B\. Brown, B\. Chess, R\. Child, S\. Gray, A\. Radford, J\. Wu, and D\. Amodei, “Scaling laws for neural language models,” 2020, arXiv:2001\.08361\.
- \[40\]C\. Giraud, F\. Roueff, and A\. Sanchez\-Perez, “Aggregation of predictors for nonstationary sub\-linear processes and online adaptive forecasting of time varying autoregressive processes,”*The Annals of Statistics*, pp\. 2412–2450, 2015\.
- \[41\]S\. Yakowitz, L\. Györfi, J\. Kieffer, and G\. Morvai, “Strongly consistent nonparametric forecasting and regression for stationary ergodic sequences,”*Journal of multivariate analysis*, vol\. 71, no\. 1, pp\. 24–41, 1999\.
- \[42\]M\. R\. Leadbetter, G\. Lindgren, and H\. Rootzén,*Extremes and related properties of random sequences and processes*\. Springer Science & Business Media, 2012\.
- \[43\]T\. Wang and P\. Isola, “Understanding contrastive representation learning through alignment and uniformity on the hypersphere,” in*Proceedings of the 37th International Conference on Machine Learning*, vol\. 119, 2020, pp\. 9929–9939\.
- \[44\]L\. Jing, P\. Vincent, Y\. LeCun, and Y\. Tian, “Understanding dimensional collapse in contrastive self\-supervised learning,” 2022, arXiv:2110\.09348\.
- \[45\]H\. Zhou, S\. Zhang, J\. Peng, S\. Zhang, J\. Li, H\. Xiong, and W\. Zhang, “Informer: Beyond efficient transformer for long sequence time\-series forecasting,” in*Proceedings of the AAAI conference on artificial intelligence*, vol\. 35, 2021, pp\. 11 106–11 115\.
- \[46\]T\. Aksu, G\. Woo, J\. Liu, X\. Liu, C\. Liu, S\. Savarese, C\. Xiong, and D\. Sahoo, “Gift\-eval: A benchmark for general time series forecasting model evaluation,” 2024, arXiv:2410\.10393\.
- \[47\]R\. Godahewa, C\. Bergmeir, G\. I\. Webb, R\. J\. Hyndman, and P\. Montero\-Manso, “Monash time series forecasting archive,” 2021, arXiv:2105\.06643\.
- \[48\]Y\. Liu, G\. Qin, X\. Huang, J\. Wang, and M\. Long, “Timer\-xl: Long\-context transformers for unified time series forecasting,” in*International Conference on Learning Representations*, 2025\.
- \[49\]A\. Das, W\. Kong, R\. Sen, and Y\. Zhou, “A decoder\-only foundation model for time\-series forecasting,” in*Proceedings of the 41st International Conference on Machine Learning*, vol\. 235, 2024, pp\. 10 148–10 167\.
- \[50\]V\. Ekambaram, A\. Jati, P\. Dayama, S\. Mukherjee, N\. H\. Nguyen, W\. M\. Gifford, C\. Reddy, and J\. Kalagnanam, “Tiny time mixers \(TTMs\): Fast pre\-trained models for enhanced zero/few\-shot forecasting of multivariate time series,”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 74 147–74 181, 2024\.
- \[51\]M\. Chen, L\. Shen, Z\. Li, X\. J\. Wang, J\. Sun, and C\. Liu, “VisionTS: Visual masked autoencoders are free\-lunch zero\-shot time series forecasters,” in*Proceedings of the 42nd International Conference on Machine Learning*, vol\. 267, 2025, pp\. 8979–9007\.
- \[52\]N\. Gruver, M\. Finzi, S\. Qiu, and A\. G\. Wilson, “Large language models are zero\-shot time series forecasters,”*Advances in Neural Information Processing Systems*, vol\. 36, pp\. 19 622–19 635, 2023\.
- \[53\]C\. Bergmeir, “Fundamental limitations of foundational forecasting models: The need for multimodality and rigorous evaluation,” Vancouver, Canada, Dec\. 2024\. \[Online\]\. Available:https://cbergmeir\.com/talks/neurips2024/
- \[54\]R\. Hyndman, A\. Koehler, K\. Ord, and R\. Snyder,*Forecasting with Exponential Smoothing: The State Space Approach*\. Springer, 2008\.
- \[55\]D\. Salinas, V\. Flunkert, J\. Gasthaus, and T\. Januschowski, “DeepAR: Probabilistic forecasting with autoregressive recurrent networks,”*International journal of forecasting*, vol\. 36, no\. 3, pp\. 1181–1191, 2020\.
- \[56\]J\. Hong and S\. Lee, “Variance sensitivity induces attention entropy collapse and instability in transformers,” in*Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing*, 2025, pp\. 8360–8378\.
- \[57\]H\. Wang, J\. Peng, F\. Huang, J\. Wang, J\. Chen, and Y\. Xiao, “MICN: Multi\-scale local and global context modeling for long\-term series forecasting,” in*International Conference on Learning Representations*, 2023\.