Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift
Summary
This paper presents an empirical study comparing how different neural architectures (MLPs, CNNs, RNNs, pretrained transformers) degrade under temporal distribution shift across image and text domains, finding that models exploiting localized features degrade fastest while pretrained encoders drift more gradually.
arXiv:2607.05908v1 Announce Type: new
Abstract: Real-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood.
We present a systematic empirical comparison of temporal robustness across three heterogeneous, time-indexed domains encompassing image classification, multi-label text classification, and text regression tasks. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross-temporal generalization. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer-based encoders.
Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually. These observations offer practical guidance for selecting architectures for real-world systems subject to temporal drift.
View Cached Full Text
Cached at: 07/08/26, 04:45 AM
# 1 Introduction
Source: [https://arxiv.org/html/2607.05908](https://arxiv.org/html/2607.05908)
marginparsep has been altered\. topmargin has been altered\. marginparpush has been altered\. The page layout violates the ICML style\.Please do not change the page layout, or include packages like geometry, savetrees, or fullpage, which change it for you\. We’re not able to reliably undo arbitrary changes to the style\. Please remove the offending package\(s\), or layout\-changing commands and try again\.
Drift Happens: An Empirical Study of Neural Architecture Robustness to Temporal Distribution Shift
Robin Holzinger111Both authors contributed equally\.Riccardo Colletti111Both authors contributed equally\.
Department of Electrical Engineering and Computer Sciences, University of California, Berkeley, USA
Extended version\. Accepted at QCDS 2026; the proceedings version will appear in Springer LNCS\.
###### Abstract
Real\-world data distributions evolve over time, inducing temporal distribution shift that can substantially degrade the reliability of deployed machine learning systems\. However, the extent to which architectural choices and their associated inductive biases affect temporal robustness remains insufficiently understood\.
We present a systematic empirical comparison of temporal robustness across three heterogeneous, time\-indexed domains encompassing image classification, multi\-label text classification, and text regression tasks\. Using a unified evaluation framework based on temporal drift matrices, we train models on cumulative historical data and evaluate their performance on both earlier and later time periods, thereby quantifying cross\-temporal generalization\. Our study spans model families ranging from simple multilayer perceptrons and convolutional networks to recurrent networks and pretrained Transformer\-based encoders\.
Collectively, the results show that architectural inductive biases systematically shape temporal robustness: models whose inductive biases lead them to exploit localized, highly discriminative features attain the highest in\-distribution accuracy, yet those features are often the ones that change most over time, so these models degrade fastest, while pretrained encoders that draw on coarser, more stable representations drift more gradually\. These observations offer practical guidance for selecting architectures for real\-world systems subject to temporal drift\.
Machine learning models are typically trained under the assumption that training and test data are drawn from the same distribution\. In practice, this assumption rarely holds: real\-world data evolves over time, exhibitingtemporal distribution shiftthat can degrade model performance in ways that standard held\-out evaluation fails to capture\. A model that achieves high accuracy on held\-out data from the same time period may fail when deployed on future inputs\.
While distribution shift is well\-documented, less understood is how architectural choices influence a model’s robustness to temporal drift\.
> *Do different inductive biases \(the translation invariance of convolutions, the sequential modeling of recurrent nets, the attention of Transformers\) lead to different rates of temporal degradation?* *Do frozen, pretrained encoders resist temporal drift better than models trained end to end?*
These questions have practical implications for model selection, yet systematic comparisons across architectures and domains remain scarce\.
This work investigates the temporal robustness of neural classifiers across three domains: image classification \(Yearbook\), text regression \(Amazon Reviews\), and multi\-label text classification \(arXiv\)\. For each domain, we evaluate diverse architectures, from simple baselines to pretrained transformers, using a unified framework based on temporal drift matrices and provide qualitative explanations for model degradation via gradient saliency maps\. Our contributions are threefold:
- •We provide a unified empirical assessment of temporal robustness across three long\-range, time\-indexed domains, enabling direct comparison of how neural architectures behave under temporal distribution shift\.
- •We organize time\-indexed evaluation in the spirit of Wild\-Time\[[39](https://arxiv.org/html/2607.05908#bib.bib13)\]into temporal drift matrices, a compact representation that quantifies cross\-temporal generalization by measuring performance when training on cumulative historical data and testing on both earlier and later time periods\.
- •We systematically compare a broad spectrum of model families, from multilayer perceptrons, convolutional and residual networks, and recurrent networks to transformers trained on the data and frozen pretrained encoders, highlighting how architectural assumptions relate to degradation patterns across modalities and tasks\.
Taken together, our results characterize how performance deteriorates as the temporal gap between training and evaluation widens across domains and model classes\.The features an inductive bias extracts within the training period are what lift in\-distribution performance, yet they are the most tied to it and the first to degrade as the data drifts, so the very features that make a model accurate in distribution are the least robust over time\.Frozen pretrained encoders, relying on coarser and more transferable representations, trade in\-distribution accuracy for steadier degradation\. These findings guide architecture selection in non\-stationary real\-world environments\.
## 2Background and Related Work
### 2\.1Temporal Distribution Shift
A central challenge in real\-world machine learning systems is that the data\-generating process is rarely stationary\. As models are deployed over months, years, or decades, both inputs and label semantics may evolve, causing systematic discrepancies between distributions encountered during training and those observed at inference\. Such temporal evolution, commonly referred to as*temporal distribution shift*, is pervasive across domains\. Understanding the structure of this shift is essential for assessing and improving temporal robustness\. However, prior work has largely focused on static or domain\-level distribution shifts\[[3](https://arxiv.org/html/2607.05908#bib.bib33);[14](https://arxiv.org/html/2607.05908#bib.bib34);[23](https://arxiv.org/html/2607.05908#bib.bib32)\], with limited attention to drift that unfolds sequentially over time, particularly in non\-generated, in\-the\-wild settings\[[39](https://arxiv.org/html/2607.05908#bib.bib13)\]\.
Following established terminology in concept drift research\[[11](https://arxiv.org/html/2607.05908#bib.bib30)\], temporal distribution shift can be characterized along three complementary axes \(Figure[1](https://arxiv.org/html/2607.05908#S2.F1)\):
Covariate shiftoccurs when the input distributionP\(X\)P\(X\)changes while the conditionalP\(Y∣X\)P\(Y\\mid X\)remains fixed\. This type of shift commonly arises in natural data streams where observational setups, sociocultural conventions, or user behavior gradually evolve\.
Label shiftrefers to changes in the marginal label distributionP\(Y\)P\(Y\)\. Long\-term textual or behavioral datasets frequently exhibit such imbalance drift as the prevalence of topics, categories, or rating patterns changes over time\.
Concept driftdenotes changes in the conditional distributionP\(Y∣X\)P\(Y\\mid X\), implying that identical inputs may correspond to different labels at different time points\. This form of drift is particularly pronounced in domains where semantics or visual attributes evolve, such as historical portraits\[[12](https://arxiv.org/html/2607.05908#bib.bib35)\]or online platforms\.
In practice, these forms of drift rarely occur in isolation and often manifest in combination\[[2](https://arxiv.org/html/2607.05908#bib.bib22)\]\. As a result, the temporal robustness of a model depends not only on the magnitude of drift but also on how its architectural assumptions and inductive biases interact with the evolving data distribution\.
### 2\.2Model Robustness to Distribution Shift
OriginalConcept DriftCovariate Drift\(Virtual Drift\)Label Drift
Figure 1:Drift types in a binary classification setting\. Circles and stars indicate the label classes; the curve represents the decision boundary\.Concept driftinduces a change in the decision boundary,Covariate drift\(virtual drift\) changes the input distribution, andLabel driftalters the relative class frequency\[[19](https://arxiv.org/html/2607.05908#bib.bib16)\]\.Understanding how learning algorithms behave under distribution shift has become an important research direction\. Early work approached robustness through domain adaptation\[[3](https://arxiv.org/html/2607.05908#bib.bib33)\]and out\-of\-distribution generalization\[[14](https://arxiv.org/html/2607.05908#bib.bib34)\], formalizing shift as a transition between a small number of discrete source and target domains\. The introduction of large\-scale benchmarks such asWILDS\[[23](https://arxiv.org/html/2607.05908#bib.bib32)\], which focuses on naturally occurring distribution shifts without explicit temporal indexing, and more recentlyWild\-Time\[[39](https://arxiv.org/html/2607.05908#bib.bib13)\], which explicitly models time\-indexed data and temporal distribution shifts, has broadened this perspective by providing evaluation protocols that more closely reflect real\-world deployment scenarios\.
A complementary line of research has examined robustness through model diagnostics and monitoring\. Methods for detecting drift onset by examining changes in latent representations or predictive uncertainty have shown promise in deep learning settings\[[29](https://arxiv.org/html/2607.05908#bib.bib31);[1](https://arxiv.org/html/2607.05908#bib.bib12)\]\. Recent empirical analyses have characterized how concept drift manifests in practice, including its locality and temporal progression across large\-scale data streams\[[2](https://arxiv.org/html/2607.05908#bib.bib22)\]\. At the systems level, work on data\-centric and continual\-learning infrastructures has explored how to maintain model quality over time through cost\-aware retraining and pipeline orchestration\[[26](https://arxiv.org/html/2607.05908#bib.bib18);[27](https://arxiv.org/html/2607.05908#bib.bib19);[34](https://arxiv.org/html/2607.05908#bib.bib20);[4](https://arxiv.org/html/2607.05908#bib.bib7);[19](https://arxiv.org/html/2607.05908#bib.bib16)\]\.
Despite these advances, robustness studies commonly focus either on a single architecture evaluated across multiple datasets or on a single dataset used to compare a narrow set of architectures\. As a result, comparatively little is known about how architectural design choices interact with long\-range temporal drift across heterogeneous modalities and tasks\. This gap is particularly relevant given the diversity of inductive biases exhibited by modern neural models: convolutional networks encode locality and translation equivariance, recurrent networks capture sequential structure\[[18](https://arxiv.org/html/2607.05908#bib.bib27);[5](https://arxiv.org/html/2607.05908#bib.bib28)\], and Transformer\-based encoders rely on self\-attention with minimal structural priors\[[35](https://arxiv.org/html/2607.05908#bib.bib25)\]\. Likewise, large\-scale pretraining in vision and language\[[8](https://arxiv.org/html/2607.05908#bib.bib26);[24](https://arxiv.org/html/2607.05908#bib.bib29)\]introduces representations shaped by broad historical data, yet their behavior under extended temporal drift remains poorly characterized\.
The present study addresses this open question by comparing these architectural families under a shared temporal evaluation protocol and across multiple modalities, enabling a controlled analysis of how model design influences robustness under real\-world temporal distribution shift\.
### 2\.3Robustness Evaluation vs\. Temporal Adaptation
This work evaluates the inherent temporal robustness of neural architectures under distribution shift, measuring how different model families, trained only on cumulative historical data, perform as the temporal gap between training and evaluation widens\. This isolates the robustness arising from architectural design and pretraining alone, prior to any adaptive intervention\.
A complementary line of research instead investigates how models can*adapt*once drift is detected, through scheduled or cost\-aware retraining\[[26](https://arxiv.org/html/2607.05908#bib.bib18);[27](https://arxiv.org/html/2607.05908#bib.bib19)\], continual\-learning pipelines\[[34](https://arxiv.org/html/2607.05908#bib.bib20);[4](https://arxiv.org/html/2607.05908#bib.bib7);[19](https://arxiv.org/html/2607.05908#bib.bib16)\], or accuracy\-aware data maintenance\[[38](https://arxiv.org/html/2607.05908#bib.bib21)\], addressing when to retrain, how much data to incorporate, and how to trade performance against cost\. Our study is orthogonal to that literature and provides a foundation on which such adaptation strategies can be designed, evaluated, and compared\.
## 3Methods and Experimental Approach
Temporal distribution shift manifests differently across modalities, tasks, and time scales, yet existing empirical studies typically vary a single axis at a time, leaving open how*architectural design*,*label structure*, and*drift mechanism*jointly shape temporal robustness \(Section[2](https://arxiv.org/html/2607.05908#S2)\)\. Our setup targets exactly this cross\-cutting comparison: we combine three long\-range, time\-indexed datasets with a diverse suite of neural architectures spanning multiple inductive biases, all evaluated under the unified temporal protocol of Section[4](https://arxiv.org/html/2607.05908#S4)\(temporal drift matrices\), so that architectural differences, rather than differences in splitting or evaluation, drive the observed robustness patterns\.
### 3\.1Datasets
Our empirical analysis spans three qualitatively distinct temporal scenarios, chosen to cover different modalities, tasks, and sources of distribution shift\. Each dataset captures multiple decades of real\-world temporal evolution, making it suitable for cross\-temporal evaluation\.
#### 3\.1\.1Yearbook: A Century of Portraits
TheYearbookdataset\[[13](https://arxiv.org/html/2607.05908#bib.bib1)\]contains 37,921 frontal portraits of American high\-school seniors from 1905 to 2013, which we align and process as 3\-channel32×3232\\times 32tensors\. Although acquisition is largely standardized, stylistic attributes \(e\.g\., hairstyles, clothing, accessories\) vary substantially across decades\. The dataset provides binary sex labels that are approximately balanced over time\. It is a canonical benchmark for temporal shift: Wild\-Time\[[39](https://arxiv.org/html/2607.05908#bib.bib13)\]uses a pre\-/post\-1970 split and reports marked out\-of\-distribution degradation, and Modyn\[[4](https://arxiv.org/html/2607.05908#bib.bib7)\]observes accuracy decay as the train–test time gap grows\. We reproduce this trend \(Fig\.[2](https://arxiv.org/html/2607.05908#S5.F2)\), consistent with covariate and concept drift\.
#### 3\.1\.2Amazon Reviews 2023: E\-commerce
TheAmazon Reviewsdataset\[[20](https://arxiv.org/html/2607.05908#bib.bib15)\]comprises 571\.54 million reviews across 33 product categories, spanning May 1996 to September 2023\. Each review has a timestamp, star rating, and free\-text content\. We cast a review\-level sentiment regression task, predicting the 1–5 rating from the text, which exhibits strong covariate and concept drift due to evolving language, consumer behavior, and platform usage\. We focus on seven categories, restrict to 2014–2023, and draw a stratified sample of 300,000 reviews\.
#### 3\.1\.3arXiv: Scientific Discourse
ThearXivdataset\[[7](https://arxiv.org/html/2607.05908#bib.bib2)\]defines a multi\-label task over 2,866,787 title–abstract records annotated with 176 subject categories\. We concatenate titles and abstracts, keep seven leaf categories \([Appendix˜E](https://arxiv.org/html/2607.05908#A5)\), and use 2000–2025 submissions whose categories fall within them \(papers may carry several\)\. Category mix and terminology shift, inducing label and covariate drift\[[19](https://arxiv.org/html/2607.05908#bib.bib16)\]\. Wild\-Time\[[39](https://arxiv.org/html/2607.05908#bib.bib13)\]reports roughly 20% degradation under temporal splits for a related arXiv task, and the gap is not substantially closed by domain generalization or continual learning methods, motivating our architectural comparison\.
### 3\.2Model Implementation
We evaluate architectures spanning different inductive biases to understand how model design affects temporal robustness\. Rather than covering the full architecture landscape, we select families that span the spectrum of inductive\-bias strength: from simple baselines without structural assumptions that establish lower bounds on performance, to modern architectures that incorporate structural priors, to pre\-trained models that leverage large\-scale external data\. This spread is what lets us attribute robustness differences to the priors themselves\.
#### 3\.2\.1Image Classification
For theYearbookdataset, we evaluate four model families trained on the data, each at three sizes \(small, medium, large\), for 12 architectures, complemented by 9 pretrained vision encoders used as frozen backbones\.
The four families differ in the spatial prior they encode\. At one extreme, the*multilayer perceptron*\(MLP\) flattens the image and applies only fully connected layers, imposing no spatial structure\. The*convolutional network*\(CNN\) builds in locality and translation equivariance through convolutional filters with batch normalization and pooling, and the*residual network*\(ResNet\) extends it with skip connections\[[16](https://arxiv.org/html/2607.05908#bib.bib45)\]that ease optimization at greater depth\. At the other extreme, the*vision Transformer*\(ViT\) drops convolution for self\-attention over patches\[[9](https://arxiv.org/html/2607.05908#bib.bib46)\], a far weaker spatial prior, with a small patch size suited to the32×3232\\times 32inputs\. Across all four, the small, medium, and large variants scale depth and width\.
Finally, we assess*transfer learning*using pretrained vision encoders trained on large\-scale curated or web\-scale datasets\. The underlying hypothesis is that representations learned from diverse data may exhibit stronger temporal robustness than features learned solely from historical portraits\. We consider 9 pretrained models\. The self\-supervisedDINOv2\-S\[[28](https://arxiv.org/html/2607.05908#bib.bib40)\]andDINOv3\-S\[[32](https://arxiv.org/html/2607.05908#bib.bib41)\]are pretrained on large curated image collections;CLIP\-B32\[[30](https://arxiv.org/html/2607.05908#bib.bib38)\]andSigLIP\-B\[[40](https://arxiv.org/html/2607.05908#bib.bib39)\]use contrastive image\-text pretraining, the latter with a sigmoid loss;ConvNeXt\-S\[[25](https://arxiv.org/html/2607.05908#bib.bib44)\],ResNet50\-IN\[[16](https://arxiv.org/html/2607.05908#bib.bib45)\], andViT\-S16\-IN21k\[[9](https://arxiv.org/html/2607.05908#bib.bib46)\]are trained with supervisedImageNetlabels, the last onImageNet\-21k; andMAE\-B\[[15](https://arxiv.org/html/2607.05908#bib.bib42)\]andEVA02\-B\[[10](https://arxiv.org/html/2607.05908#bib.bib43)\]are pretrained with masked image modeling\. In all cases, we freeze the pretrained backbone and train only a linear classification head, isolating the contribution of pretrained representations from the effects of fine\-tuning dynamics\.
#### 3\.2\.2Text Models
For theAmazon ReviewsandarXivdatasets, we evaluate model families that differ in their inductive biases for representing textual structure\. The same architectures serve both datasets, differing only in output layer and loss, withAmazon Reviewsusing regression under a weighted mean squared error and arXiv multi\-label classification under a weighted binary cross\-entropy\.
All four families operate on cachedRoBERTatoken embeddings\. The*feed\-forward baseline*\(FFN\) averages them into a single 768\-dimensional vector and passes it through a small MLP head that discards order entirely, with variants scaling the hidden width from 128 to 2048\. The*convolutional model*\(TextCNN\)\[[21](https://arxiv.org/html/2607.05908#bib.bib24)\]applies one\-dimensional convolutions to capture local n\-gram patterns, scaling the number and width of its filters\. The*recurrent*models read the sequence token by token, as a single\-layer bidirectionalGRU\[[5](https://arxiv.org/html/2607.05908#bib.bib28)\], a single\-layer bidirectionalLSTM\[[18](https://arxiv.org/html/2607.05908#bib.bib27)\], and a two\-layer bidirectionalLSTMwith attention\. The*Transformer*encoders replace recurrence with self\-attention\[[35](https://arxiv.org/html/2607.05908#bib.bib25)\]and learnable positional embeddings, ranging from 1 to 5 layers and 4 to 6 heads\.
Finally, we assess transfer learning from pretrained language encoders\. We consider frozen encoders with a light head trained on the pooled output:BERT\[[8](https://arxiv.org/html/2607.05908#bib.bib26)\],RoBERTa\[[24](https://arxiv.org/html/2607.05908#bib.bib29)\],DeBERTa\-v3\[[17](https://arxiv.org/html/2607.05908#bib.bib47)\],ELECTRA\[[6](https://arxiv.org/html/2607.05908#bib.bib48)\],MPNet\[[33](https://arxiv.org/html/2607.05908#bib.bib49)\],ModernBERT\[[37](https://arxiv.org/html/2607.05908#bib.bib50)\], andDistilBERT\[[31](https://arxiv.org/html/2607.05908#bib.bib52)\]on both text tasks, plusMiniLM\-L6\[[36](https://arxiv.org/html/2607.05908#bib.bib51)\]onarXiv\. As with the vision encoders, freezing isolates pretrained representations from fine\-tuning\.
## 4Evaluation Framework
Standard held\-out evaluation measures performance on data from the training distribution, which under temporal shift can look strong even as the model fails on future data\. We therefore measure cross\-temporal generalization explicitly\.
### 4\.1Temporal Drift Matrices
Let𝒟=\{\(xi,yi,ti\)\}i=1N\\mathcal\{D\}=\\\{\(x\_\{i\},y\_\{i\},t\_\{i\}\)\\\}\_\{i=1\}^\{N\}denote a dataset where each example is associated with a timestamptit\_\{i\}\. We divide the timeline intoKKdisjoint intervals𝒯=\{T1,…,TK\}\\mathcal\{T\}=\\\{T\_\{1\},\\ldots,T\_\{K\}\\\}, whereTk=\[tkstart,tkend\)T\_\{k\}=\[t\_\{k\}^\{\\text\{start\}\},t\_\{k\}^\{\\text\{end\}\}\)\. Let𝒟k=\{\(x,y,t\)∈𝒟:t∈Tk\}\\mathcal\{D\}\_\{k\}=\\\{\(x,y,t\)\\in\\mathcal\{D\}:t\\in T\_\{k\}\\\}denote the subset of data from intervalkk\.
For training, we construct cumulative datasets𝒟≤k=⋃j=1k𝒟j\\mathcal\{D\}\_\{\\leq k\}=\\bigcup\_\{j=1\}^\{k\}\\mathcal\{D\}\_\{j\}containing all data up to and including intervalkk\. A modelfkf\_\{k\}trained on𝒟≤k\\mathcal\{D\}\_\{\\leq k\}has access to historical data over timetkendt\_\{k\}^\{\\text\{end\}\}but no knowledge of future periods\. This cumulative strategy reflects realistic deployment scenarios in which models are periodically retrained on all historical data\.
Thetemporal drift matrixM∈ℝK×KM\\in\\mathbb\{R\}^\{K\\times K\}captures cross\-temporal generalization:
Mij=perf\(fi,𝒟j\)M\_\{ij\}=\\text\{perf\}\(f\_\{i\},\\mathcal\{D\}\_\{j\}\)\(1\)whereperf\(⋅,⋅\)\\text\{perf\}\(\\cdot,\\cdot\)denotes a performance metric \(accuracy, macro AUC, or balanced MSE depending on the task\)\. EntryMijM\_\{ij\}measures how well a model trained on data through periodiiperforms on data from periodjj\.
The structure ofMMreveals aspects of temporal robustness\. Following our plotting convention, rows are the training cutoff \(“trained up to”\) and columns the evaluation period \(“evaluated on”\), with time increasing upward and to the right from a lower\-left origin\. The diagonalMiiM\_\{ii\}is in\-distribution performance on data held out from the training period; the upper\-left \(j<ij<i\) is held\-out performance on earlier in\-training periods; and the lower\-right \(j\>ij\>i\) is forward generalization to data unseen during training\. This lets us read how performance degrades \(or improves\) as the temporal gap between training and evaluation widens\.
### 4\.2Temporal Splitting Strategy
Each slice𝒟k\\mathcal\{D\}\_\{k\}is partitioned once into a stratified training split𝒟ktrain\\mathcal\{D\}\_\{k\}^\{\\text\{train\}\}\(70%\) and a held\-out test split𝒟ktest\\mathcal\{D\}\_\{k\}^\{\\text\{test\}\}\(30%\), shared across all models via a fixed split seed\. For in\-distribution evaluation \(whenj≤ij\\leq i\), we use only the held\-out test split of𝒟j\\mathcal\{D\}\_\{j\}to prevent data leakage:
𝒟eval\(i,j\)=\{𝒟jtestifj≤i\(in\-distribution\)𝒟jtrain∪𝒟jtestifj\>i\(out\-of\-distrib\.\)\\mathcal\{D\}\_\{\\text\{eval\}\}^\{\(i,j\)\}=\\begin\{cases\}\\mathcal\{D\}\_\{j\}^\{\\text\{test\}\}&\\text\{if \}j\\leq i\\text\{ \(in\-distribution\)\}\\\\ \\mathcal\{D\}\_\{j\}^\{\\text\{train\}\}\\cup\\mathcal\{D\}\_\{j\}^\{\\text\{test\}\}&\\text\{if \}j\>i\\text\{ \(out\-of\-distrib\.\)\}\\end\{cases\}
For out\-of\-distribution evaluation on future time slices, we use all available samples from that period to maximize statistical power, since by definition none of this data was seen during training\. Evaluating on periods that overlap the training data \(j≤ij\\leq i\) is deliberate: these held\-out entries verify that a model retains performance across the historical periods it was trained on and provide the in\-distribution reference from which forward decay is measured\.
### 4\.3Training Protocol
The image models are trained with Adam\[[22](https://arxiv.org/html/2607.05908#bib.bib53)\]and the text models with its decoupled\-weight\-decay variant AdamW, a standard choice that behaves robustly across heterogeneous architectures; fixing one optimizer per modality avoids per\-model optimizer tuning as a confound\. Learning rates are task\-specific, with exact hyperparameters pinned in versioned experiment presets\. Training proceeds for a fixed number of epochs, and model selection is based on the final checkpoint\. Every configuration is trained under multiple random seeds, five onYearbookand three on the text tasks \([Appendix˜A](https://arxiv.org/html/2607.05908#A1)\), and all results average over seeds\.
We adopt a cumulative temporal training strategy: for each sliceTkT\_\{k\}we train a modelfkf\_\{k\}on𝒟≤k\\mathcal\{D\}\_\{\\leq k\}, all examples observed up toTkT\_\{k\}, yielding a sequence\{f1,…,fK\}\\\{f\_\{1\},\\dots,f\_\{K\}\\\}that each represent the best model obtainable from the data available at that point in time\. All checkpoints are stored and evaluated on every slice to construct the full temporal drift matrixMM\.
The loss follows the task structure and is tailored to address label imbalance\.
For*binary classification*onYearbook, we minimize standard two\-class cross\-entropy on logits and labels, the maximum\-likelihood objective for a two\-class softmax model\.
For*multi\-label classification*onarXiv, each paper may belong to multiple subject categories\. We keep only seven leaf categories as the label space, retaining papers that carry at least one of them\. The label distribution remains skewed, with negative examples substantially outnumbering positives for each category\. We therefore use a weighted binary cross\-entropy with logits, applied independently to each of theC=7C=7categories:
ℒBCE\\displaystyle\\mathcal\{L\}\_\{\\text\{BCE\}\}=−1n∑i=1n∑c=1C\[wcyiclog\(σ\(zic\)\)\\displaystyle=\-\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\sum\_\{c=1\}^\{C\}\\big\[w\_\{c\}\\,y\_\{ic\}\\log\(\\sigma\(z\_\{ic\}\)\)\(2\)\+\(1−yic\)log\(1−σ\(zic\)\)\],\\displaystyle\\qquad\+\(1\-y\_\{ic\}\)\\log\(1\-\\sigma\(z\_\{ic\}\)\)\\big\],wherezicz\_\{ic\}denotes the logit for classcc,σ\(⋅\)\\sigma\(\\cdot\)is the sigmoid function, andyic∈\{0,1\}y\_\{ic\}\\in\\\{0,1\\\}is the binary label\. The class weightswcw\_\{c\}are defined as
wc=\|\{i:yic=0\}\|\|\{i:yic=1\}\|,w\_\{c\}=\\frac\{\|\\\{i:y\_\{ic\}=0\\\}\|\}\{\|\\\{i:y\_\{ic\}=1\\\}\|\},\(3\)i\.e\., the ratio of negative to positive examples for each category\. This reweighting compensates for label imbalance by amplifying the contribution of rare positive labels, preventing the model from achieving deceptively high accuracy by predicting the all\-zero vector\.
For*regression*onAmazon Reviews, we train models to predict the star rating using a weighted mean squared error:
ℒWMSE=1n∑i=1nwyi\(yi−y^i\)2,\\mathcal\{L\}\_\{\\text\{WMSE\}\}=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}w\_\{y\_\{i\}\}\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\},\(4\)whereyi∈\{1,2,3,4,5\}y\_\{i\}\\in\\\{1,2,3,4,5\\\}is the true rating andy^i\\hat\{y\}\_\{i\}is the prediction\. The rating\-specific weightswrw\_\{r\}are defined as
wr=nnr,w\_\{r\}=\\frac\{n\}\{n\_\{r\}\},\(5\)withnnthe total number of samples andnrn\_\{r\}the number of samples with ratingrr\. Because the empirical rating distribution is heavily skewed toward high scores, an unweightedMSEwould be dominated by the majority class and largely ignore rare low\-rating events\. The inverse\-frequency weighting counteracts this imbalance, ensuring that errors on minority ratings remain visible in the objective and that temporal degradation in performance cannot be explained solely by changes in the prevalence of positive reviews\.
### 4\.4Evaluation Metrics
To populate each entry of the drift matrixMM, we require a scalar performance metric per train\-test time pair\. We match each metric to the task’s label structure: accuracy forYearbook, whose binary labels are approximately balanced \([Section˜B\.1](https://arxiv.org/html/2607.05908#A2.SS1)\); balanced MSE \(MSEbal\\text\{MSE\}\_\{\\text\{bal\}\}\) forAmazon Reviews, which reweights its skewed rating distribution \([Section˜B\.2](https://arxiv.org/html/2607.05908#A2.SS2)\); and macro AUC \(AUC¯\\overline\{\\text\{AUC\}\}\) forarXiv, whose imbalanced multi\-label categories require a threshold\-free, per\-class average \([Section˜B\.3](https://arxiv.org/html/2607.05908#A2.SS3)\)\. The two classification tasks thus receive different metrics because their label structures differ; the drift protocol itself is identical\.
### 4\.5Summary Statistics of the Drift Matrix
In the drift matrixMM, the row indexiiis the training cutoff and the column indexjjis the evaluation period, so the entryMijM\_\{ij\}is the performance of a model trained on all data up to periodiiwhen it is tested on periodjj\. Three numbers summarize the matrix, each averaged only over the cells that are filled, since a train\-test pair with no completed run leaves its cell empty\. The*in\-distribution*score is the average of the diagonal,
ID\(M\)=meaniMii,\\mathrm\{ID\}\(M\)=\\operatorname\{mean\}\_\{i\}M\_\{ii\},\(6\)a model’s performance on the same period it was trained through\. The*future*score averages the cells withj\>ij\>i, where the evaluation period falls after the training cutoff,
Fut\(M\)=meanj\>iMij\.\\mathrm\{Fut\}\(M\)=\\operatorname\{mean\}\_\{j\>i\}M\_\{ij\}\.\(7\)The*decay*is the difference between the two, oriented so a larger value always means worse temporal robustness, since accuracy andAUC¯\\overline\{\\text\{AUC\}\}are better when high whileMSEbal\\text\{MSE\}\_\{\\text\{bal\}\}is better when low,
Dec\(M\)=\{ID\(M\)−Fut\(M\)\(accuracy,AUC¯\),Fut\(M\)−ID\(M\)\(MSEbal\)\.\\mathrm\{Dec\}\(M\)=\\begin\{cases\}\\mathrm\{ID\}\(M\)\-\\mathrm\{Fut\}\(M\)&\(\\text\{accuracy\},\\ \\overline\{\\text\{AUC\}\}\),\\\\ \\mathrm\{Fut\}\(M\)\-\\mathrm\{ID\}\(M\)&\(\\text\{MSE\}\_\{\\text\{bal\}\}\)\.\\end\{cases\}\(8\)so a positive decay is a loss of performance on future periods and a negative one, which is uncommon, a gain\. Together, future score and decay order every model from most to least temporally robust\.
Beyond these whole\-matrix scores, we look at how robustness depends on the training time itself\. Fixing one cutoffii, a single row of the matrix, we pair its diagonal cellMiiM\_\{ii\}with the average over the later periods in that row \(meanj\>iMij\\operatorname\{mean\}\_\{j\>i\}M\_\{ij\}\) and the decay between them, reading these at a few cutoffs spread evenly across the timeline \(the last one is left out, as it has no future\)\. Averaging the same quantities within each architecture family lets us compare the families directly\.
To place each model against the others, we average the matrices over the cohort𝒞\\mathcal\{C\}of models into the*cohort\-mean matrix*
M¯ij=1\|𝒞\|∑m∈𝒞Mij\(m\),\\bar\{M\}\_\{ij\}=\\frac\{1\}\{\|\\mathcal\{C\}\|\}\\sum\_\{m\\in\\mathcal\{C\}\}M^\{\(m\)\}\_\{ij\},\(9\)defined at the cells every model fills; each model’s*deviation*isΔij\(m\)=Mij\(m\)−M¯ij\\Delta^\{\(m\)\}\_\{ij\}=M^\{\(m\)\}\_\{ij\}\-\\bar\{M\}\_\{ij\}\.
## 5Results
Each model is summarised through its drift matrix by the in\-distribution, forward, and decay scores of[Section˜4\.5](https://arxiv.org/html/2607.05908#S4.SS5), namely its performance on the period it was trained through, its mean performance on the later periods held out from training, and the difference between the two\. How far a model degrades between them is governed by two factors that recur across the three domains, the strength of its inductive bias and whether it relies on a frozen pretrained encoder\.
Figure 2:Cohort\-mean drift matrices, one panel per domain\. Each cell\(i,j\)\(i,j\)is the mean performance across all models of that dataset when trained on data through periodii\(row\) and evaluated on periodjj\(column\): accuracy forYearbook, balanced MSE forAmazon Reviews\(lower is better\), and macro AUC forarXiv\. The dashed diagonal marks in\-distribution evaluation; the lower\-right region of each panel is forward generalization to unseen future periods\. Vertical white bands mark periods with insufficient samples\.### 5\.1Image Classification
OnYearbook, the model families separate sharply in distribution, along the diagonal of the cohort\-mean matrix in theYearbookpanel of[Figure˜2](https://arxiv.org/html/2607.05908#S5.F2)\. MLP\-S, which flattens the image into a vector and imposes no spatial structure, sits at about69\.3%69\.3\\%, while the CNNs and ResNets reach near92\.8%92\.8\\%for CNN\-M and CNN\-L \([Table˜5](https://arxiv.org/html/2607.05908#A3.T5)\)\. What sets these apart is their inductive bias toward locality and translation equivariance, by which they build features from small local regions of the image and recognise a pattern wherever it appears, and this lets them exploit the cues that most sharply separate male from female portraits within a given period\. The ViTs and frozen encoders fall between these two ends\.
Temporal robustness reverses this ordering, and we read it from the decay, the accuracy a model loses once the evaluation year moves past its training cutoff \([Table˜5](https://arxiv.org/html/2607.05908#A3.T5)\)\. The CNNs and ResNets, strongest in distribution, decay the most, by13\.713\.7and13\.913\.9points for CNN\-L and CNN\-M, so their accuracy on future years falls to about79\.0%79\.0\\%\. MLP\-S is the steadiest model of all, with only7\.17\.1points of decay, though from its lower starting accuracy of62\.3%62\.3\\%\. The frozen pretrained encoders form a third group, more stable than the CNNs but less accurate to begin with, decaying by7\.97\.9to10\.410\.4points, with the self\-supervised DINOv3\-S steadiest among them and the supervised ImageNet\-21k backbone least\.
Figure 3:Gradient saliency maps onYearbookfor CNN\-L, ResNet\-S, and MLP\-L, each trained through 1950 \(left\) and 1970 \(right\) and evaluated on later years\. Each panel averages gradient saliency over the selected portraits from the indicated evaluation year rather than a single example\.What lifts a model in distribution is also what undermines it over time\. The features a strong inductive bias extracts to separate the classesraise its in\-distribution accuracy, but they arethe most specific to the years it was trained on, and the first to lose their meaningonce hairstyles, clothing, and image quality drift\. The sharper a model’s in\-distribution lead, the faster it erodes as the data ages\. Robustness onYearbookis therefore not a property an architecture optimises on its own, but the other side of fitting a single period too closely, closer to overfitting across time than across samples \([Appendix˜C](https://arxiv.org/html/2607.05908#A3)\)\.
Figure 4:Forgetting curves onYearbook\. Each curve is one model’s mean accuracy as the gap between its training year and the evaluation year grows, averaged over training years and seeds\. A zero gap is in\-distribution, and the slope is the rate of forgetting \([Section˜C\.7](https://arxiv.org/html/2607.05908#A3.SS7)\)\.The forgetting curves in[Figure˜4](https://arxiv.org/html/2607.05908#S5.F4)show this playing out year by year\. Each curve plots a model’s accuracy against the gap between its training cutoff and the evaluation year, so reading from left to right traces how quickly it forgets \([Section˜C\.7](https://arxiv.org/html/2607.05908#A3.SS7)\)\. The strongly biased networks start far above the rest, near90%90\\%at a zero gap, and their curves descend the steepest\. As the gap widens the curves draw together and then cross, so the families that led in distribution lose their edge on the most distant years, and every model settles near two\-class chance\.
#### 5\.1\.1Saliency Maps
The gradient saliency maps in[Figure˜3](https://arxiv.org/html/2607.05908#S5.F3)show where this fragility comes from\. Extending the training window from 1950 to 1970 sharpens where the convolutional models look: the CNN tightens from a diffuse scatter over the cheeks and mouth to a compact, near\-symmetric pair of bright spots at the eyes, the detail that most sharply tells the classes apart, and the ResNet shifts the same way while keeping a broader spread across the central face\. The MLP, without that bias, spreads its attribution across the whole frame at either cutoff, the background included\. The same precision that lets the convolutional models read this discriminative detail binds them to it: those localized features are the most specific to the training period and the most exposed to drift, while the MLP’s blunter reading is less tied to any era\.
### 5\.2Text Models
Each text model is a light head trained on cached, frozenRoBERTaembeddings \([Section˜3](https://arxiv.org/html/2607.05908#S3)\), a shared representation over which the families differ only in inductive bias\.Amazon Reviewsis a rating regression scored by balanced MSE, where lower is better, andarXivis a multi\-label classification scored by macro AUC\.
#### 5\.2\.1Amazon Reviews
OnAmazon Reviewsthe recurrent and Transformer models fit the training reviews most closely, since both read the wording in context, the recurrent networks token by token and the Transformers through self\-attention, and so capture how a review’s words compose into its rating\. That fit shows up as the lowest in\-distribution error, down to0\.6800\.680balanced MSE for BiGRU\-S, with the Transformers alongside them at0\.6860\.686for TX\-S \([Figure˜2](https://arxiv.org/html/2607.05908#S5.F2),[Table˜13](https://arxiv.org/html/2607.05908#A4.T13)\)\. The FFN, which averages the embeddings and discards word order, sits higher at about0\.7610\.761, and the frozen encoders higher still, between0\.8060\.806and1\.0221\.022\.
Temporal robustness runs the other way, and the models that fit the reviews most tightly are the ones whose error grows fastest\. The recurrent networks decay the most, by up to0\.1280\.128balanced MSE, so their error on future reviews climbs to about0\.8240\.824, while the FFN is the steadiest of the trained models, gaining only0\.0750\.075to0\.0890\.089\. The decay is sharpest for models trained on the earliest reviews, where the recurrent family worsens by0\.2420\.242, and it narrows toward the recent cutoffs as fewer years remain ahead \([Appendix˜D](https://arxiv.org/html/2607.05908#A4)\)\. The frozen encoders again sit at higher error but drift less than the trained models, and DeBERTa\-v3 is the most stable model anywhere in the study, at0\.0430\.043of decay\.
What makes these models fit the reviews so well is also what later undermines them\. The way they compose the wording into a rating is the most specific to how reviews were written in the training years, and the first to lose its meaning as the language shifts\.Far enough forward the in\-distribution ranking inverts, and the models that fit the reviews most tightly end among the least accurate, while the order\-free FFN, never sharp, stays the steadiest\.
#### 5\.2\.2arXiv
OnarXiva stronger inductive bias yields no in\-distribution advantage, and so costs no temporal robustness\. Sorting a paper into its subject categories is close to recognising its topic, and the topic is already encoded in the frozenRoBERTaembeddings every model is built on, so no architecture finds structure the others miss\. The feed\-forward, convolutional, recurrent, and Transformer families therefore reach almost the same in\-distribution macro AUC, between about97\.2%97\.2\\%and98\.1%98\.1\\%, within a point of one another \([Figure˜2](https://arxiv.org/html/2607.05908#S5.F2),[Table˜21](https://arxiv.org/html/2607.05908#A5.T21)\)\.
Robustness is just as uniform, and each family loses a similar small2\.72\.7to3\.43\.4points going forward, with the frozen encoders in the same band apart from DeBERTa\-v3 and ELECTRA, which fall further at6\.56\.5and7\.37\.3points \([Appendix˜E](https://arxiv.org/html/2607.05908#A5)\)\.Where the bias gains nothing in distribution it forms no period\-specific features to lose, so no family decays faster than the rest\. The later years stay close to the training distribution: the backbone learned this vocabulary before any model saw it, leaving temporal shift little to take away\.
## 6Conclusion and Future Work
Across image classification, text regression, and multi\-label text classification, in\-distribution accuracy turns out to neither guarantee nor predict temporal robustness\. What governs the rate of degradation is instead the strength of a model’s inductive bias and whether it relies on a frozen pretrained encoder\. A stronger bias extracts the most discriminative, period\-specific features and leads in distribution, yet those same features are the first to lose their meaning as the data drifts, so the architectures that fit the training period most tightly are the ones that degrade fastest\. Frozen pretrained encoders occupy a different regime, conceding in\-distribution accuracy in return for steadier behaviour over time, and onarXiv, where the task is already solved by the pretrained representation and the bias buys no advantage, no family decays faster than the rest\. The practical reading is that the model with the best held\-out score at training time is often the least robust once deployed, so architecture selection should weigh the expected horizon between retraining cycles, not in\-distribution accuracy alone\.
### 6\.1Limitations
Our comparison covers the most common inductive biases, convolutional, recurrent, and attention\-based, against simple baselines and frozen encoders, but many architectures remain untested and the study is best read as a first systematic pass rather than an exhaustive one\. It spans three datasets across two modalities, and broadening it to further domains and other forms of distribution shift would show how widely the pattern holds\. The text tasks are the most constrained, using a narrow label space, a five\-point rating forAmazon Reviewsand seven categories forarXiv, spanning only about a decade and a quarter of a century against a full century forYearbook, and running on a stratified subsample rather than the complete corpora\. Because too few examples fall within each time slice to train a text encoder from scratch, the text models also share a single frozenRoBERTarepresentation, so their absolute scores should be read in that light\. We average over a small fixed seed set rather than conducting formal significance tests, and hyperparameter choices and seed variance may still affect fine\-grained rankings; we therefore emphasize family\-level patterns over individual placements\. The families are also not capacity\-matched\. The small, medium, and large variants within each family expose scale effects, and onYearbookthe smallestMLP,CNN, andResNetvariants decay less than their larger siblings, so inductive bias and capacity remain partially confounded\. The analysis is also descriptive rather than mechanistic\. We measure how robustness varies with inductive bias and pretraining without isolating the precise cues responsible, and we evaluate inherent robustness under cumulative training rather than any adaptive policy, which leaves the question of retraining orchestration to systems such as Modyn\[[4](https://arxiv.org/html/2607.05908#bib.bib7)\]\.
### 6\.2Future Work
These limitations point to several extensions\. The comparison can be widened to a broader range of inductive biases, to capacity\-matched pairs that disentangle bias from scale, and to additional datasets, modalities, and drift mechanisms, and carried to longer temporal spans with the full corpora and richer label spaces, where language drift should be more pronounced and the gap between architectures wider\. The drift matrices can also be paired with scheduled or drift\-triggered retraining under explicit compute budgets, measuring not only how much a model decays but how cheaply that decay can be undone\[[27](https://arxiv.org/html/2607.05908#bib.bib19)\]\. Most of all, moving from description to mechanism, by tying inductive bias and pretraining back to the specific features that drift, would turn the observed regularity into an account of why temporal robustness behaves as it does\.
### 6\.3Availability
Code, experiment definitions, and paper\-generation scripts are available at[github\.com/learning\-mechanisms/drift\-happens](https://github.com/learning-mechanisms/drift-happens/); public run histories and matrix artifacts at[wandb\.ai/drift\-happens/drift\-happens](https://wandb.ai/drift-happens/drift-happens)\. The[extended preprint](https://drift-happens.org/drift-happens.pdf)and[drift\-happens\.org](https://drift-happens.org/)retain the complete per\-model drift\-matrix galleries\. Code is Apache\-2\.0; paper, figures, and documentation are CC BY 4\.0 where we hold the rights, and external datasets and models keep their original licenses\.
## References
- Automatically detecting data drift in machine learning classifiers\.External Links:2111\.05672,[Link](https://arxiv.org/abs/2111.05672)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1)\.
- G\. J\. Aguiar and A\. Cano \(2023\)A comprehensive analysis of concept drift locality in data streams\.External Links:[Link](https://arxiv.org/abs/2311.06396),2311\.06396Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p6.1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1)\.
- S\. Ben\-David, J\. Blitzer, K\. Crammer, A\. Kulesza, F\. Pereira, and J\. W\. Vaughan \(2010\)A theory of learning from different domains\.Machine Learning79\(1\),pp\. 151–175\.Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p1.1)\.
- M\. Böther, T\. Robroek, V\. Gsteiger, R\. Holzinger, X\. Ma, P\. Tözün, and A\. Klimovic \(2025\)Modyn: data\-centric machine learning pipeline orchestration\.Proc\. ACM Manag\. Data3\(1\)\.External Links:[Link](https://doi.org/10.1145/3709705),[Document](https://dx.doi.org/10.1145/3709705)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1),[§3\.1\.1](https://arxiv.org/html/2607.05908#S3.SS1.SSS1.p1.1),[§6\.1](https://arxiv.org/html/2607.05908#S6.SS1.p1.1)\.
- K\. Cho, B\. van Merrienboer, C\. Gulcehre, D\. Bahdanau, F\. Bougares, H\. Schwenk, and Y\. Bengio \(2014\)Learning phrase representations using RNN encoder–decoder for statistical machine translation\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1724–1734\.External Links:[Document](https://dx.doi.org/10.3115/v1/D14-1179)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p3.1),[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p2.1)\.
- K\. Clark, M\. Luong, Q\. V\. Le, and C\. D\. Manning \(2020\)ELECTRA: pre\-training text encoders as discriminators rather than generators\.InInternational Conference on Learning Representations \(ICLR\),External Links:2003\.10555Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- Cornell University \(2024\)ArXiv dataset\.Kaggle\.External Links:[Document](https://dx.doi.org/10.34740/KAGGLE/DSV/7548853),[Link](https://www.kaggle.com/dsv/7548853)Cited by:[§3\.1\.3](https://arxiv.org/html/2607.05908#S3.SS1.SSS3.p1.1)\.
- J\. Devlin, M\. Chang, K\. Lee, and K\. Toutanova \(2019\)BERT: pre\-training of deep bidirectional transformers for language understanding\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(NAACL\-HLT\),pp\. 4171–4186\.External Links:[Document](https://dx.doi.org/10.18653/v1/N19-1423)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p3.1),[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- A\. Dosovitskiy, L\. Beyer, A\. Kolesnikov, D\. Weissenborn, X\. Zhai, T\. Unterthiner, M\. Dehghani, M\. Minderer, G\. Heigold, S\. Gelly, J\. Uszkoreit, and N\. Houlsby \(2021\)An image is worth 16x16 words: transformers for image recognition at scale\.InInternational Conference on Learning Representations \(ICLR\),External Links:2010\.11929Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p2.1),[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- Y\. Fang, Q\. Sun, X\. Wang, T\. Huang, X\. Wang, and Y\. Cao \(2024\)EVA\-02: a visual representation for neon genesis\.Image and Vision Computing149,pp\. 105171\.External Links:2303\.11331Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- J\. Gama, I\. Žliobaitė, A\. Bifet, M\. Pechenizkiy, and A\. Bouchachia \(2014\)A survey on concept drift adaptation\.ACM Computing Surveys46\(4\),pp\. 1–37\.External Links:[Document](https://dx.doi.org/10.1145/2523813)Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p2.1)\.
- S\. Ginosar, K\. Rakelly, S\. Sachs, B\. Yin, and A\. A\. Efros \(2015\)A century of portraits: a visual historical record of american high school yearbooks\.InIEEE International Conference on Computer Vision Workshops,pp\. 1–7\.Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p5.1)\.
- S\. Ginosar, K\. Rakelly, S\. Sachs, B\. Yin, C\. Lee, P\. Krahenbuhl, and A\. A\. Efros \(2019\)A century of portraits: a visual historical record of american high school yearbooks\.External Links:1511\.02575Cited by:[§3\.1\.1](https://arxiv.org/html/2607.05908#S3.SS1.SSS1.p1.1)\.
- I\. Gulrajani and D\. Lopez\-Paz \(2021\)In search of lost domain generalization\.InInternational Conference on Learning Representations,Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p1.1)\.
- K\. He, X\. Chen, S\. Xie, Y\. Li, P\. Dollár, and R\. Girshick \(2022\)Masked autoencoders are scalable vision learners\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2111\.06377Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:1512\.03385Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p2.1),[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- P\. He, J\. Gao, and W\. Chen \(2023\)DeBERTaV3: improving DeBERTa using ELECTRA\-style pre\-training with gradient\-disentangled embedding sharing\.InInternational Conference on Learning Representations \(ICLR\),External Links:2111\.09543Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- S\. Hochreiter and J\. Schmidhuber \(1997\)Long short\-term memory\.Neural Computation9\(8\),pp\. 1735–1780\.External Links:[Document](https://dx.doi.org/10.1162/neco.1997.9.8.1735)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p3.1),[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p2.1)\.
- R\. Holzinger \(2024\)An analysis of drift\- and cost\-aware ml retraining triggering policies in modyn\.Bachelor’s Thesis,Technical University of Munich\.Note:Supervisors: Prof\. Dr\. Viktor Leis\., Jana Vatter, Prof\. Dr\. Ana Klimović, Maximilian BötherCited by:[Figure 1](https://arxiv.org/html/2607.05908#S2.F1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1),[§3\.1\.3](https://arxiv.org/html/2607.05908#S3.SS1.SSS3.p1.1)\.
- Y\. Hou, J\. Li, Z\. He, A\. Yan, X\. Chen, and J\. McAuley \(2024\)Bridging language and items for retrieval and recommendation\.arXiv preprint arXiv:2403\.03952\.Cited by:[§3\.1\.2](https://arxiv.org/html/2607.05908#S3.SS1.SSS2.p1.1)\.
- Y\. Kim \(2014\)Convolutional neural networks for sentence classification\.InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing \(EMNLP\),pp\. 1746–1751\.External Links:[Document](https://dx.doi.org/10.3115/v1/D14-1181)Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p2.1)\.
- D\. P\. Kingma and J\. Ba \(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations,Cited by:[§4\.3](https://arxiv.org/html/2607.05908#S4.SS3.p1.1)\.
- P\. W\. Koh, S\. Sagawa, H\. Marklund, S\. M\. Xie, M\. Zhang, A\. Balsubramani, W\. Hu, M\. Yasunaga, R\. L\. Phillips, I\. Gao,et al\.\(2021\)WILDS: a benchmark of in\-the\-wild distribution shifts\.InInternational Conference on Machine Learning,pp\. 5637–5664\.Cited by:[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p1.1)\.
- Y\. Liu, M\. Ott, N\. Goyal, J\. Du, M\. Joshi, D\. Chen, O\. Levy, M\. Lewis, L\. Zettlemoyer, and V\. Stoyanov \(2019\)RoBERTa: a robustly optimized BERT pretraining approach\.arXiv preprint arXiv:1907\.11692\.Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p3.1),[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- Z\. Liu, H\. Mao, C\. Wu, C\. Feichtenhofer, T\. Darrell, and S\. Xie \(2022\)A ConvNet for the 2020s\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition \(CVPR\),External Links:2201\.03545Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- A\. Mahadevan and M\. Mathioudakis \(2023\)Cost\-effective retraining of machine learning models\.External Links:2310\.04216Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1)\.
- A\. Mahadevan and M\. Mathioudakis \(2024\)Cost\-aware retraining for machine learning\.Knowledge\-Based Systems293,pp\. 111610\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1016/j.knosys.2024.111610),ISSN 0950\-7051,[Link](https://www.sciencedirect.com/science/article/pii/S0950705124002454)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1),[§6\.2](https://arxiv.org/html/2607.05908#S6.SS2.p1.1)\.
- M\. Oquab, T\. Darcet, T\. Moutakanni, H\. Vo, M\. Szafraniec, V\. Khalidov, P\. Fernandez, D\. Haziza, F\. Massa, A\. El\-Nouby, M\. Assran, N\. Ballas, W\. Galuba, R\. Howes, P\. Huang, S\. Li, I\. Misra, M\. Rabbat, V\. Sharma, G\. Synnaeve, H\. Xu, H\. Jégou, J\. Mairal, P\. Labatut, A\. Joulin, and P\. Bojanowski \(2024\)DINOv2: learning robust visual features without supervision\.Transactions on Machine Learning Research \(TMLR\)\.External Links:2304\.07193Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- S\. Rabanser, S\. Günnemann, and Z\. Lipton \(2019\)Failing loudly: an empirical study of methods for detecting dataset shift\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1)\.
- A\. Radford, J\. W\. Kim, C\. Hallacy, A\. Ramesh, G\. Goh, S\. Agarwal, G\. Sastry, A\. Askell, P\. Mishkin, J\. Clark, G\. Krueger, and I\. Sutskever \(2021\)Learning transferable visual models from natural language supervision\.InProceedings of the 38th International Conference on Machine Learning \(ICML\),External Links:2103\.00020Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- V\. Sanh, L\. Debut, J\. Chaumond, and T\. Wolf \(2019\)DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter\.arXiv preprint arXiv:1910\.01108\.External Links:1910\.01108Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- O\. Siméoni, H\. V\. Vo, M\. Seitzer, F\. Baldassarre, M\. Oquab, C\. Jose, V\. Khalidov, M\. Szafraniec, S\. Yi, M\. Ramamonjisoa, F\. Massa, D\. Haziza, L\. Wehrstedt, J\. Wang, T\. Darcet, T\. Moutakanni, L\. Sentana, C\. Roberts, A\. Vedaldi, J\. Tolan, J\. Brandt, C\. Couprie, J\. Mairal, H\. Jégou, P\. Labatut, and P\. Bojanowski \(2025\)DINOv3\.arXiv preprint arXiv:2508\.10104\.External Links:2508\.10104Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
- K\. Song, X\. Tan, T\. Qin, J\. Lu, and T\. Liu \(2020\)MPNet: masked and permuted pre\-training for language understanding\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2004\.09297Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- H\. Tian, M\. Yu, and W\. Wang \(2018\)Continuum: a platform for cost\-aware, low\-latency continual learning\.InProceedings of the ACM Symposium on Cloud Computing,SoCC ’18,New York, NY, USA,pp\. 26–40\.External Links:[Document](https://dx.doi.org/10.1145/3267809.3267817),ISBN 9781450360111,[Link](https://doi.org/10.1145/3267809.3267817)Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p2.1),[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,Vol\.30\.Cited by:[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p3.1),[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p2.1)\.
- W\. Wang, F\. Wei, L\. Dong, H\. Bao, N\. Yang, and M\. Zhou \(2020\)MiniLM: deep self\-attention distillation for task\-agnostic compression of pre\-trained transformers\.InAdvances in Neural Information Processing Systems \(NeurIPS\),External Links:2002\.10957Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- B\. Warner, A\. Chaffin, B\. Clavié, O\. Weller, O\. Hallström, S\. Taghadouini, A\. Gallagher, R\. Biswas, F\. Ladhak, T\. Aarsen, N\. Cooper, G\. Adams, J\. Howard, and I\. Poli \(2024\)Smarter, better, faster, longer: a modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference\.arXiv preprint arXiv:2412\.13663\.External Links:2412\.13663Cited by:[§3\.2\.2](https://arxiv.org/html/2607.05908#S3.SS2.SSS2.p3.1)\.
- S\. Wooders, X\. Mo, A\. Narang, K\. Lin, I\. Stoica, J\. M\. Hellerstein, N\. Crooks, and J\. E\. Gonzalez \(2023\)RALF: accuracy\-aware scheduling for feature store maintenance\.Proc\. VLDB Endow\.17\(3\),pp\. 563–576\.External Links:[Document](https://dx.doi.org/10.14778/3632093.3632116),ISSN 2150\-8097,[Link](https://doi.org/10.14778/3632093.3632116)Cited by:[§2\.3](https://arxiv.org/html/2607.05908#S2.SS3.p2.1)\.
- H\. Yao, C\. Choi, B\. Cao, Y\. Lee, P\. W\. Koh, and C\. Finn \(2022\)Wild\-time: a benchmark of in\-the\-wild distribution shift over time\.InAdvances in Neural Information Processing Systems,Cited by:[2nd item](https://arxiv.org/html/2607.05908#S1.I1.i2.p1.1),[§2\.1](https://arxiv.org/html/2607.05908#S2.SS1.p1.1),[§2\.2](https://arxiv.org/html/2607.05908#S2.SS2.p1.1),[§3\.1\.1](https://arxiv.org/html/2607.05908#S3.SS1.SSS1.p1.1),[§3\.1\.3](https://arxiv.org/html/2607.05908#S3.SS1.SSS3.p1.1)\.
- X\. Zhai, B\. Mustafa, A\. Kolesnikov, and L\. Beyer \(2023\)Sigmoid loss for language image pre\-training\.InProceedings of the IEEE/CVF International Conference on Computer Vision \(ICCV\),External Links:2303\.15343Cited by:[§3\.2\.1](https://arxiv.org/html/2607.05908#S3.SS2.SSS1.p3.1)\.
## Appendix
## Appendix AReproducibility
Experiments are defined as versioned preset snapshots and run through the repository command\-line interface in a Pixi\-pinned environment; run metadata records the git state andpixi\.lockhash, and regenerated paper and website assets are checked against a committed checksum manifest\. Each experiment is a staged run for one dataset, architecture, and seed: the training stage fits all cumulative temporal checkpoints, and the evaluation stage scores them on every time slice to form the drift matrix\. We use seeds0–44forYearbookand0–22forAmazon ReviewsandarXiv, averaging over them\. Public run histories and per\-run matrix artifacts are available in the Weights & Biases project at[https://wandb\.ai/drift\-happens/drift\-happens](https://wandb.ai/drift-happens/drift-happens)\.
Table[1](https://arxiv.org/html/2607.05908#A1.T1)summarizes the measured compute on NVIDIA A100 GPUs, summing the W&B\_runtimeof each finished train and evaluation stage\. GPU\-hours are successful\-stage wall times under one GPU per run; the four\-GPU node\-hour column divides these by four for ideal packed execution, excluding queueing, data preprocessing, failed attempts, and scheduling idle time\.
Table 1:Approximate measured compute for the conference experiment campaign\.
## Appendix BPer\-Dataset Protocol and Metrics
The drift\-matrix protocol of[Section˜4](https://arxiv.org/html/2607.05908#S4)is shared across the three datasets, but each instantiates it at its own time granularity and is scored with the metric matched to its label structure \([Table˜2](https://arxiv.org/html/2607.05908#A2.T2)\)\. For each task we require a scalar performance measure that is both aligned with the task and robust to the class imbalance present in that dataset; a metric dominated by the label distribution would conflate shifts in the data with changes in the model\.
Table 2:Per\-dataset instantiation of the drift\-matrix protocol: time span, slice granularity, number of slicesKK, and primary metric\.### B\.1Yearbook
We slice the 1905–2013 timeline into one\-year intervals, keeping theK=104K=104years with enough samples\. The task is binary classification with roughly balanced classes, so we score it with standard accuracy, the fraction of portraits classified correctly\. Balance is what makes accuracy trustworthy here: when neither class dominates, a model cannot earn a good score by always predicting the same label, so accuracy rises and falls only as the model genuinely classifies more or fewer cases correctly\. Higher values are better\.
### B\.2Amazon Reviews
We slice the 2014–2023 timeline into half\-year intervals, givingK=20K=20slices\. The task is regression: the model predicts a star rating from one to five and is scored by squared error\. The difficulty is that the ratings are strongly skewed toward five\-star reviews, so a single mean squared error taken over all reviews would mainly measure performance on the majority and would barely react to the rarer low ratings\. To keep every rating level visible, we first compute the mean squared error separately within each rating and then average those per\-rating values equally over the rating levels present,
MSEbal=1\|ℛ\|∑r∈ℛMSEr,MSEr=1\|𝒮r\|∑i∈𝒮r\(yi−y^i\)2,\\text\{MSE\}\_\{\\text\{bal\}\}=\\frac\{1\}\{\|\\mathcal\{R\}\|\}\\sum\_\{r\\in\\mathcal\{R\}\}\\text\{MSE\}\_\{r\},\\qquad\\text\{MSE\}\_\{r\}=\\frac\{1\}\{\|\\mathcal\{S\}\_\{r\}\|\}\\sum\_\{i\\in\\mathcal\{S\}\_\{r\}\}\(y\_\{i\}\-\\hat\{y\}\_\{i\}\)^\{2\},where𝒮r=\{i:yi=r\}\\mathcal\{S\}\_\{r\}=\\\{i:y\_\{i\}=r\\\}is the set of reviews whose true rating isrrandℛ=\{r:\|𝒮r\|\>0\}\\mathcal\{R\}=\\\{r:\|\\mathcal\{S\}\_\{r\}\|\>0\\\}is the set of rating levels present in the slice \(at most the five levels one to five\)\. Because every present level contributes equally, a model that began to drift on the scarce one\- and two\-star reviews would reveal it here\. Unlike the other two metrics, lower values are better\.
### B\.3arXiv
We slice the 2000–2025 timeline into one\-year intervals, givingK=26K=26slices\. The task is multi\-label: a paper can belong to several of theC=7C=7subject areas at once, and those areas are very unevenly populated\. This raises two problems\. First, fixing a single decision threshold is arbitrary and tends to favour the frequent subjects, so a threshold\-based score such as accuracy is misleading\. Second, an average that weights papers equally is again dominated by the common subjects\. We address the first by scoring each subject with the Area Under the ROC Curve, which summarises performance over all thresholds at once:AUCc\\text\{AUC\}\_\{c\}is the probability that a randomly chosen paper carrying subjectccis given a higher score for that subject than a randomly chosen paper that does not carry it\. We address the second by averaging the per\-subject values uniformly, so each subject counts the same regardless of how many papers it contains,
AUC¯=1C∑c=1CAUCc,AUCc=∫01TPRc\(t\)𝑑FPRc\(t\),\\overline\{\\text\{AUC\}\}=\\frac\{1\}\{C\}\\sum\_\{c=1\}^\{C\}\\text\{AUC\}\_\{c\},\\qquad\\text\{AUC\}\_\{c\}=\\int\_\{0\}^\{1\}\\text\{TPR\}\_\{c\}\(t\)\\,d\\text\{FPR\}\_\{c\}\(t\),whereTPRc\(t\)\\text\{TPR\}\_\{c\}\(t\)andFPRc\(t\)\\text\{FPR\}\_\{c\}\(t\)denote the true\- and false\-positive rates for subjectccat thresholdtt\. Higher values are better, and a model that ranked every paper correctly within every subject would reach11\.
## Appendix CYearbook – Drift Matrices
The cohort\-mean and per\-model deviation matrices shown here, and the in\-distribution, future, and decay quantities tabulated below, are defined in Section[4\.5](https://arxiv.org/html/2607.05908#S4.SS5)\.
Figure 5:Cohort\-mean Accuracy matrixM¯\\bar\{M\}over the Yearbook models\. Cell\(i,j\)\(i,j\)is the mean across those models of the score from training through sliceiiand evaluating on slicejj\.### C\.1Model Roster
Table 3:Yearbook: models trained from scratch\.
Table 4:Yearbook: frozen pretrained encoders, with trainable head and total parameters\.
### C\.2MLP
\(a\)MLP\-S
\(b\)MLP\-M
\(c\)MLP\-L
Figure 6:MLP models: Accuracy drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### C\.3CNN
\(a\)CNN\-S
\(b\)CNN\-M
\(c\)CNN\-L
Figure 7:CNN models: Accuracy drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### C\.4ResNet
\(a\)ResNet\-S
\(b\)ResNet\-M
\(c\)ResNet\-L
Figure 8:ResNet models: Accuracy drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### C\.5ViT
\(a\)ViT\-S
\(b\)ViT\-M
\(c\)ViT\-L
Figure 9:ViT models: Accuracy drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### C\.6Transfer
\(a\)CLIP\-B32
\(b\)ConvNeXt\-S
\(c\)DINOv2\-S
\(d\)DINOv3\-S
\(e\)EVA02\-B
\(f\)MAE\-B
\(g\)ResNet50\-IN
\(h\)SigLIP\-B
\(i\)ViT\-S16\-IN21k
Figure 10:Transfer models: Accuracy drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### C\.7Forgetting and Rankings
To see how quickly each model forgets, we summarize its drift matrix as a forgetting curve\. The curve plots the Accuracy against the lagℓ=j−i\\ell=j\-i, the number of slices between the training cutoffiiand the evaluation slicejj\. At each lag we average over all training cutoffs,
F\(ℓ\)=meaniMi,i\+ℓ\.F\(\\ell\)=\\operatorname\{mean\}\_\{i\}M\_\{i,\\,i\+\\ell\}\.The result is the Accuracy at a fixed temporal distance, independent of which period a model was trained on\. This separates the effect of temporal distance from the difficulty of any single slice\.
\(a\)Per model
\(b\)Per model family
Figure 11:Forgetting curves: each model \(left\) and averaged within each family \(right\)\.\(a\)By future performance
\(b\)By decay
Figure 12:Models ranked by mean future performance and by temporal decay\.
### C\.8Result Tables
Table 5:Temporal robustness on Yearbook\.Table 6:Yearbook: models trained up to 1905, ordered by future performance\.
Table 7:Yearbook: models trained up to 1944, ordered by future performance\.
Table 8:Yearbook: models trained up to 1978, ordered by future performance\.
Table 9:Yearbook: models trained up to 2012, ordered by future performance\.
Table 10:Yearbook: future performance and decay by model family\.
## Appendix DAmazon Reviews – Drift Matrices
The cohort\-mean and per\-model deviation matrices shown here, and the in\-distribution, future, and decay quantities tabulated below, are defined in Section[4\.5](https://arxiv.org/html/2607.05908#S4.SS5)\.
Figure 13:Cohort\-mean Balanced MSE matrixM¯\\bar\{M\}over the Amazon Reviews models\. Cell\(i,j\)\(i,j\)is the mean across those models of the score from training through sliceiiand evaluating on slicejj\.### D\.1Model Roster
Table 11:Amazon Reviews: models trained from scratch\.
Table 12:Amazon Reviews: frozen pretrained encoders, with trainable head and total parameters\.
### D\.2FFN
\(a\)FFN\-S
\(b\)FFN\-M
\(c\)FFN\-L
Figure 14:FFN models: Balanced MSE drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### D\.3TextCNN
\(a\)TextCNN\-S
\(b\)TextCNN\-M
\(c\)TextCNN\-L
Figure 15:TextCNN models: Balanced MSE drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### D\.4Recurrent
\(a\)BiGRU\-S
\(b\)BiLSTM\-M
\(c\)BiLSTM\-Attn\-L
Figure 16:Recurrent models: Balanced MSE drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### D\.5Transformer
\(a\)TX\-S
\(b\)TX\-M
\(c\)TX\-L
Figure 17:Transformer models: Balanced MSE drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### D\.6Frozen
\(a\)BERT
\(b\)DistilBERT
\(c\)ELECTRA
\(d\)MPNet
\(e\)ModernBERT
\(f\)RoBERTa
\(g\)DeBERTa\-v3
Figure 18:Frozen models: Balanced MSE drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### D\.7Forgetting and Rankings
To see how quickly each model forgets, we summarize its drift matrix as a forgetting curve\. The curve plots the Balanced MSE against the lagℓ=j−i\\ell=j\-i, the number of slices between the training cutoffiiand the evaluation slicejj\. At each lag we average over all training cutoffs,
F\(ℓ\)=meaniMi,i\+ℓ\.F\(\\ell\)=\\operatorname\{mean\}\_\{i\}M\_\{i,\\,i\+\\ell\}\.The result is the Balanced MSE at a fixed temporal distance, independent of which period a model was trained on\. This separates the effect of temporal distance from the difficulty of any single slice\.
\(a\)Per model
\(b\)Per model family
Figure 19:Forgetting curves: each model \(left\) and averaged within each family \(right\)\.\(a\)By future performance
\(b\)By decay
Figure 20:Models ranked by mean future performance and by temporal decay\.
### D\.8Result Tables
Table 13:Temporal robustness on Amazon Reviews\.Table 14:Amazon Reviews: models trained up to 2014\-H1, ordered by future performance\.
Table 15:Amazon Reviews: models trained up to 2017\-H1, ordered by future performance\.
Table 16:Amazon Reviews: models trained up to 2020\-H1, ordered by future performance\.
Table 17:Amazon Reviews: models trained up to 2023\-H1, ordered by future performance\.
Table 18:Amazon Reviews: future performance and decay by model family\.
## Appendix EarXiv – Drift Matrices
The cohort\-mean and per\-model deviation matrices shown here, and the in\-distribution, future, and decay quantities tabulated below, are defined in Section[4\.5](https://arxiv.org/html/2607.05908#S4.SS5)\. The label space comprises the leaf categoriescs\.LG,hep\-ph,cs\.CV,cs\.AI,hep\-th,quant\-ph, andgr\-qc\.
Figure 21:Cohort\-mean Macro AUC matrixM¯\\bar\{M\}over the arXiv models\. Cell\(i,j\)\(i,j\)is the mean across those models of the score from training through sliceiiand evaluating on slicejj\.### E\.1Model Roster
Table 19:arXiv: models trained from scratch\.
Table 20:arXiv: frozen pretrained encoders, with trainable head and total parameters\.
### E\.2FFN
\(a\)FFN\-S
\(b\)FFN\-M
\(c\)FFN\-L
Figure 22:FFN models: Macro AUC drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### E\.3TextCNN
\(a\)TextCNN\-S
\(b\)TextCNN\-M
\(c\)TextCNN\-L
Figure 23:TextCNN models: Macro AUC drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### E\.4Recurrent
\(a\)BiGRU\-S
\(b\)BiLSTM\-M
\(c\)BiLSTM\-Attn\-L
Figure 24:Recurrent models: Macro AUC drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### E\.5Transformer
\(a\)TX\-S
\(b\)TX\-M
\(c\)TX\-L
Figure 25:Transformer models: Macro AUC drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### E\.6Frozen
\(a\)BERT
\(b\)DistilBERT
\(c\)ELECTRA
\(d\)MPNet
\(e\)ModernBERT
\(f\)RoBERTa
\(g\)DeBERTa\-v3
\(h\)MiniLM\-L6
Figure 26:Frozen models: Macro AUC drift matrixM\(m\)M^\{\(m\)\}and deviation from the cohort meanΔ\(m\)=M\(m\)−M¯\\Delta^\{\(m\)\}=M^\{\(m\)\}\-\\bar\{M\}for each model, shown on a sequential and a zero\-centred diverging scale, respectively\.
### E\.7Forgetting and Rankings
To see how quickly each model forgets, we summarize its drift matrix as a forgetting curve\. The curve plots the Macro AUC against the lagℓ=j−i\\ell=j\-i, the number of slices between the training cutoffiiand the evaluation slicejj\. At each lag we average over all training cutoffs,
F\(ℓ\)=meaniMi,i\+ℓ\.F\(\\ell\)=\\operatorname\{mean\}\_\{i\}M\_\{i,\\,i\+\\ell\}\.The result is the Macro AUC at a fixed temporal distance, independent of which period a model was trained on\. This separates the effect of temporal distance from the difficulty of any single slice\.
\(a\)Per model
\(b\)Per model family
Figure 27:Forgetting curves: each model \(left\) and averaged within each family \(right\)\.\(a\)By future performance
\(b\)By decay
Figure 28:Models ranked by mean future performance and by temporal decay\.
### E\.8Result Tables
Table 21:Temporal robustness on arXiv\.Table 22:arXiv: models trained up to 2000, ordered by future performance\.
Table 23:arXiv: models trained up to 2008, ordered by future performance\.
Table 24:arXiv: models trained up to 2016, ordered by future performance\.
Table 25:arXiv: models trained up to 2024, ordered by future performance\.
Table 26:arXiv: future performance and decay by model family\.Similar Articles
Temporal Concept Drift in Legal Judgment Prediction: Neural Baselines Across Three Epochs of Ukrainian Court Decisions
This paper investigates temporal concept drift in legal judgment prediction by fine-tuning transformer models on Ukrainian court decisions from three epochs defined by geopolitical disruptions. Findings show severe forward degradation, asymmetry in backward transfer, and that chronological continual learning effectively mitigates forgetting while domain pretraining reduces degradation magnitude.
Has Anyone Actually Solved Memory Drift?
Discusses the problem of memory drift in AI systems where preferences and facts become outdated but are only appended, leading to conflicting versions and unreliable retrieval.
Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shift
This paper proposes a test-time adaptation approach using semi-supervised learning for AI text detection that adapts to continual distribution shifts from new LLMs, adversarial humanization, and temporal drift, outperforming state-of-the-art supervised detectors.
Neural Variability Enhances Artificial Network Robustness
This paper investigates how correlated noise, inspired by neural variability in the brain, can enhance the robustness of artificial neural networks against adversarial attacks and naturalistic image modifications.
Out-of-distribution Neural Inference in Dynamical Ising Models
This paper investigates out-of-distribution neural inference for reconstructing interaction graphs of dynamical Ising models, finding that Transformer-based and convolutional models exhibit architecture-dependent statistical priors that can produce misleading out-of-distribution robustness.