Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You

# Adapting Without Gradients:Affine Statistics Transport and What Its Certificate Can Tell You
Source: [https://arxiv.org/html/2609.00374](https://arxiv.org/html/2609.00374)
Salim KhazemIbrahim Mohamed SerouisAffiliation:Talan Research CenterAffiliation:Toulouse, FranceEmail:[ibrahim\.mohamed\-serouis@talan\.com](mailto:)

###### Abstract

Test\-time adaptation \(TTA\) typically assumes that model parameters can be updated at inference time\. This assumption is restrictive for inference\-only accelerators, frozen or third\-party models, and memory\-constrained deployments, and standard BatchNorm\-based TTA configurations may also become inactive on architectures without BatchNorm\. We study adaptation when the learned model must remain frozen\. We introduce CASTER, a gradient\-free method that stores source class statistics in a discriminative subspace, estimates a class\-shared affine transformation from target\-batch moments, and analytically transports the source class distributions before classification\. CASTER requires no backward pass, optimizer state, or stored source feature bank\. Across four backbones and seven datasets, it outperformskk\-NN on identical frozen features in 27 of 28 backbone\-dataset settings while retaining a median of18×18\\timesless state\. Affine transport is not always reliable\. On ImageNet\-C, where batches contain only 64 samples for 1000 classes, unconditional transport loses 21\.2 top\-1 points\. We therefore introduce an empirical residual\-to\-margin transportability certificate\. Across 307 evaluation cells, every transport losing more than 10 points has certificate value above 3\.9, although benign and destructive regimes are not perfectly separated\. Gating converts an average−3\.35\-3\.35\-point effect of unconditional transport into a\+1\.69\+1\.69\-point gain, and performance remains within 0\.3 points of the best threshold over a broad threshold range\. Finally, we show that this certificate is mechanism\-specific: when applied to Tent, it accepts only4\.3%4\.3\\%of updates and preserves0\.6%0\.6\\%of Tent’s available gain\. These results position CASTER as a lightweight adaptation mechanism for frozen\-model deployment, together with an explicit account of when its safety signal is informative and when it is not\. Code is available in[https://github\.com/salimkhazem/Caster\.git](https://github.com/salimkhazem/Caster.git)

## 1Introduction

![Refer to caption](https://arxiv.org/html/2609.00374v1/figs/diagram/Caster_diagram.png)Figure 1:CASTER at test time\.Source class moments are transported to the target batch by a class\-shared affine map estimated from batch statistics\. The transported Gaussian discriminant is used only when the transportability certificate passes; otherwise CASTER returns the frozen model\-head prediction\. No gradients are computed and no source feature bank is stored\.Test\-time adaptation \(TTA\) aims to recover accuracy under distribution shift by using unlabeled target data available only at inference time\. Strong methods such as Tent\[[16](https://arxiv.org/html/2609.00374#bib.bib1)\], EATA\[[10](https://arxiv.org/html/2609.00374#bib.bib2)\], CoTTA\[[18](https://arxiv.org/html/2609.00374#bib.bib3)\], and SAR\[[11](https://arxiv.org/html/2609.00374#bib.bib4)\]update model parameters online and can be highly effective when a backward pass and mutable model state are available\. In many deployments, however, those assumptions are restrictive: inference\-only accelerators may not expose backward computation, frozen binaries or external APIs may not expose trainable parameters, and optimizer state can violate latency or memory budgets\. Forward\-only TTA such as FOA\[[9](https://arxiv.org/html/2609.00374#bib.bib15)\]relaxes the backward\-pass requirement through derivative\-free prompt optimization, but still performs a test\-time search\. We study a stricter regime in which the learned network parameters remain frozen and adaptation is performed analytically in feature space\.

A second practical issue is architectural\. In the standard implementations evaluated here, Tent and EATA collect BatchNorm affine parameters\. Consequently, on our four LayerNorm backbones ConvNeXt\-T, ViT\-B/16, DeiT\-B, and Swin\-T their eligible parameter set is empty and their predictions reduce exactly to the frozen source model\. SAR differs because it can update BatchNorm, GroupNorm, or LayerNorm parameters\. This distinction matters when reporting TTA results across modern architectures: an average that mixes active and inert configurations can obscure what a method is actually adapting\.

We proposeCASTER, a gradient\-free TTA method based on*class\-shared affine statistics transport*\. Our hypothesis is that many practically relevant shifts approximately transform a source\-trained feature space by a single affine map shared across classes\. CASTER stores only source class moments in a discriminative subspace\. For each target batch, it estimates a shrinkage\-stabilized map from source and target batch moments, transports the source class Gaussians through that map, and evaluates the resulting Gaussian discriminant\. The feature extractor and learned network parameters are never updated, no source feature bank is retained, and no optimizer state is required\.

Analytic transport is not unconditionally safe\. Its weakest regime is also easy to diagnose: when the number of classes is large relative to the target batch, pseudo\-class coverage is too sparse to support reliable class geometry\. On ImageNet\-C, for example, a batch contains6464samples for10001000classes, and unconditional transport loses21\.221\.2top\-1 points relative to the frozen head\. CASTER therefore includes a*transportability certificate*: after inverse transport, confident pseudo\-class centroids must remain small relative to the minimum source\-class margin, measured in the same Mahalanobis geometry\.

A central result of this paper is a precise characterization of what that certificate does and does not justify\. It reliably identifies the catastrophic transport regime: every admissible evaluation cell that loses more than1010points has certificate value above3\.93\.9\. The induced policy is also insensitive to the exact threshold; across a roughly fourfold interval, performance stays within0\.30\.3points of the best threshold\. However, the raw certificate score does*not*rank adaptation benefit within the non\-degenerate regime: its correlation with the gain from transport is positive, whereas a score used to prefer beneficial updates would need the opposite ordering\. We therefore present the certificate as a safety floor for the transport mechanism, not as a general predictor of how much adaptation will help\.

That distinction becomes sharper when the same gate is applied to a different adaptation mechanism\. On matched CIFAR\-10\-C cells, the certificate admits only4\.3%4\.3\\%of Tent updates and preserves just0\.6%0\.6\\%of Tent’s available gain; a more permissive source\-statistic referee accepts46\.3%46\.3\\%and still keeps only32\.3%32\.3\\%\. At the cell level, the certificate’s acceptance rate decreases as Tent’s ungated gain increases \(ρ=−0\.86\\rho=\-0\.86\), whereas the analogous relationship is positive for CASTER’s own transport\. The failure is structural: a criterion computed from the pre\-update frozen representation cannot, in general, certify the benefit of a mechanism that changes that representation\.

Contributions\.We make three contributions\.\(i\)We introduce CASTER, an analytic, gradient\-free TTA mechanism that stores per\-class moments rather than a source feature bank\. On identical frozen features it outperformskk\-NN in2727of2828backbone\-dataset cells, with an18×18\\timessmaller median retained state, and it remains applicable to architectures on which the BatchNorm\-only Tent/EATA configurations evaluated here are inert\.\(ii\)We derive a scale\-consistent residual\-to\-margin certificate and empirically separate two claims that are often conflated: detecting degenerate transport and ranking adaptation benefit\. The certificate supports the former, not the latter\.\(iii\)We show that this certificate does not transfer to a feature\-updating mechanism, providing both matched experiments and a structural explanation for the failure\. All reported aggregates are regenerated from raw logs by a fail\-closed verification script to reduce silent evaluation errors\.

## 2Related Work

#### Gradient\-based test\-time adaptation\.

Tent minimizes prediction entropy by updating normalization affine parameters and test\-time normalization statistics\[[16](https://arxiv.org/html/2609.00374#bib.bib1)\]\. EATA adds reliability and redundancy filtering together with an anti\-forgetting regularizer\[[10](https://arxiv.org/html/2609.00374#bib.bib2)\]; CoTTA uses teacher\-style averaging, augmentation, and stochastic restoration for non\-stationary streams\[[18](https://arxiv.org/html/2609.00374#bib.bib3)\]; and SAR combines reliable sample selection with sharpness\-aware optimization to improve stability under small batches and dynamic shifts\[[11](https://arxiv.org/html/2609.00374#bib.bib4)\]\. These methods can be strong when backward computation and mutable parameters are available\. In our experiments we report their advantage over CASTER where it exists rather than restricting comparisons to settings favorable to frozen\-feature methods \(Sec\.[4\.5](https://arxiv.org/html/2609.00374#S4.SS5)\)\.

#### Backward\-free and frozen\-model adaptation\.

T3A replaces the linear classifier by pseudo\-prototypes formed from confident target samples while keeping the feature extractor fixed\[[4](https://arxiv.org/html/2609.00374#bib.bib5)\]\. FOA removes backpropagation by optimizing an added input prompt with a derivative\-free evolution strategy and by shifting activations toward source statistics\[[9](https://arxiv.org/html/2609.00374#bib.bib15)\]\. Recent work has also explored parameter\-efficient adaptation of frozen vision backbones, including low\-rank adaptation strategies for Vision Transformers\[[6](https://arxiv.org/html/2609.00374#bib.bib17)\]and topology\-aware LoRA adaptation for segmentation\[[7](https://arxiv.org/html/2609.00374#bib.bib16)\]\. These approaches retain a largely frozen pretrained representation while introducing a small set of trainable parameters\. They nevertheless require parameter updates during adaptation\. CASTER considers a stricter deployment regime in which the learned network remains entirely frozen and adaptation is performed analytically in feature space\. In particular, CASTER estimates a closed\-form affine transport of source class statistics from each target batch, without backpropagation, trainable parameters, or test\-time optimization\. We also compare withkk\-NN over the same frozen features as a deliberately strong nonparametric estimator; unlike CASTER, its retained state grows with the number of source examples\.

#### Selective adaptation and risk monitoring\.

Selective prediction and learning\-to\-defer study whether a model should act when a confidence criterion is satisfied\. Relatedly, recent work monitors predictive risk during TTA and raises an alert when a performance requirement is likely to be violated\[[13](https://arxiv.org/html/2609.00374#bib.bib14)\]\. Our question is narrower and mechanism\-specific: before using an analytic transport, can we detect batches for which that transport is geometrically unsupported? The experiments in Secs\.[4\.3](https://arxiv.org/html/2609.00374#S4.SS3)and[4\.7](https://arxiv.org/html/2609.00374#S4.SS7)show why this scope matters\. A criterion validated for one adaptation mechanism should not be interpreted as a generic certificate for another\.

#### Robustness benchmarks and distribution distances\.

CIFAR\-C and ImageNet\-C evaluate common corruptions\[[3](https://arxiv.org/html/2609.00374#bib.bib6)\]; ImageNet\-V2, ImageNet\-R, and ImageNet\-Sketch test natural or rendition\-based shifts\[[12](https://arxiv.org/html/2609.00374#bib.bib9),[2](https://arxiv.org/html/2609.00374#bib.bib7),[17](https://arxiv.org/html/2609.00374#bib.bib8)\]; and PACS and Office\-Home are standard domain\-shift benchmarks\[[8](https://arxiv.org/html/2609.00374#bib.bib10),[14](https://arxiv.org/html/2609.00374#bib.bib11)\]\. Distribution distances such as maximum mean discrepancy \(MMD\)\[[1](https://arxiv.org/html/2609.00374#bib.bib12)\]and Wasserstein distance\[[15](https://arxiv.org/html/2609.00374#bib.bib13)\]quantify how far two distributions are apart\. Distance from the source, however, is not the same as the expected benefit of a particular adaptation rule\. This distinction is central to our certificate analysis\.

## 3Method

Leth=fθ​\(x\)∈ℝdh=f\_\{\\theta\}\(x\)\\in\\mathbb\{R\}^\{d\}denote the output of a frozen feature extractor and lety∈\{1,…,C\}y\\in\\\{1,\\ldots,C\\\}\. CASTER performs all adaptation outside the learned network\. From labeled source data, we first construct a discriminative linear subspace from the generalized eigenvectors of the between\-class and within\-class scatter matrices\. LetUk0∈ℝd×k0U\_\{k\_\{0\}\}\\in\\mathbb\{R\}^\{d\\times k\_\{0\}\}contain the leading directions, where

k0=min⁡\(kmax,C−1\)\.k\_\{0\}=\\min\(k\_\{\\max\},\\,C\-1\)\.\(1\)TheC−1C\-1bound reflects the maximum rank of between\-class scatter\. At target time, for a batch of sizeBB, we activate the first

k=min⁡\(k0,B−1\)k=\\min\(k\_\{0\},\\,B\-1\)\(2\)directions and usez=Uk⊤​hz=U\_\{k\}^\{\\top\}h\. TheB−1B\-1bound avoids carrying directions that cannot be supported by the empirical target covariance before regularization\. Source statistics are stored once in the largest subspaceUk0U\_\{k\_\{0\}\}and restricted to the firstkkcoordinates as needed\. Thus the retained source state does not depend on the number of source examples\.

For each classcc, CASTER stores the projected source meanμc∈ℝk\\mu\_\{c\}\\in\\mathbb\{R\}^\{k\}, covarianceΣc∈ℝk×k\\Sigma\_\{c\}\\in\\mathbb\{R\}^\{k\\times k\}, and priorπc\\pi\_\{c\}\. Letmsm\_\{s\}andCsC\_\{s\}denote the corresponding global source mean and covariance in the active subspace\.

### 3\.1Affine Statistics Transport

For a target batchℬt=\{zi\}i=1B\\mathcal\{B\}\_\{t\}=\\\{z\_\{i\}\\\}\_\{i=1\}^\{B\}, letm^t\\hat\{m\}\_\{t\}andC^t\\hat\{C\}\_\{t\}be its empirical mean and covariance\. CASTER estimates a shrinkage\-stabilized whiten\-color map

At\\displaystyle A\_\{t\}=\(C^t\+λI\)1/2\(Cs\+λI\)−1/2,\\displaystyle=\(\\hat\{C\}\_\{t\}\+\\lambda I\)^\{1/2\}\(C\_\{s\}\+\\lambda I\)^\{\-1/2\},\(3\)bt\\displaystyle b\_\{t\}=m^t−At​ms,\\displaystyle=\\hat\{m\}\_\{t\}\-A\_\{t\}m\_\{s\},where matrix square roots are the symmetric positive\-semidefinite roots andλ\>0\\lambda\>0controls covariance shrinkage\. WhenB≥2​kB\\geq 2k, we use the full matrix in Eq\. \([3](https://arxiv.org/html/2609.00374#S3.E3)\)\. WhenB<2​kB<2k, we use its diagonal analogue, replacing the two covariance matrices by their diagonal variances\. This rule is a sample\-efficiency safeguard: the dense map has𝒪⁡\(k2\)\\mathcal\{O\}\(k^\{2\}\)degrees of freedom, whereas the diagonal map has𝒪⁡\(k\)\\mathcal\{O\}\(k\)\.

The source class statistics are transported as

μ~c,t=At​μc\+bt,Σ~c,t=At​Σc​At⊤\+β​I,\\tilde\{\\mu\}\_\{c,t\}=A\_\{t\}\\mu\_\{c\}\+b\_\{t\},\\qquad\\tilde\{\\Sigma\}\_\{c,t\}=A\_\{t\}\\Sigma\_\{c\}A\_\{t\}^\{\\top\}\+\\beta I,\(4\)withβ\>0\\beta\>0providing class\-covariance regularization\. The transported Gaussian discriminant score is

gc​\(z\)=\\displaystyle g\_\{c\}\(z\)=−12​\(z−μ~c,t\)⊤​Σ~c,t−1​\(z−μ~c,t\)\\displaystyle\-\\tfrac\{1\}\{2\}\(z\-\\tilde\{\\mu\}\_\{c,t\}\)^\{\\top\}\\tilde\{\\Sigma\}\_\{c,t\}^\{\-1\}\(z\-\\tilde\{\\mu\}\_\{c,t\}\)\(5\)−12logdet\(Σ~c,t\)\+logπc\.\\displaystyle\-\\tfrac\{1\}\{2\}\\log\\det\(\\tilde\{\\Sigma\}\_\{c,t\}\)\+\\log\\pi\_\{c\}\.The proposed target prediction isy^i=arg⁡maxc​gc​\(zi\)\\hat\{y\}\_\{i\}=\\arg\\max\_\{c\}g\_\{c\}\(z\_\{i\}\)\. No network parameter is changed by this operation\.

### 3\.2Transportability Certificate

The affine map is only useful when a target batch contains enough evidence to support the transported class geometry\. CASTER therefore checks the proposed transport before committing to its predictions\.

We retain target samples whose transported\-GDA confidence exceeds a fixed threshold and letIc,tI\_\{c,t\}be the retained samples assigned to classcc\. Each retained feature is inverse\-transported to source coordinates,

z¯i=At−1​\(zi−bt\),\\bar\{z\}\_\{i\}=A\_\{t\}^\{\-1\}\(z\_\{i\}\-b\_\{t\}\),\(6\)and for every occupied pseudo\-classccwe compute the centroid residual

δ^c,t=1\|Ic,t\|​∑i∈Ic,tz¯i−μc\.\\hat\{\\delta\}\_\{c,t\}=\\frac\{1\}\{\|I\_\{c,t\}\|\}\\sum\_\{i\\in I\_\{c,t\}\}\\bar\{z\}\_\{i\}\-\\mu\_\{c\}\.\(7\)LetΣs\\Sigma\_\{s\}be the pooled source within\-class covariance and define the minimum source\-class separation in that same metric as

γs=minc≠c′⁡\[\(μc−μc′\)⊤​Σs−1​\(μc−μc′\)\]1/2\.\\gamma\_\{s\}=\\min\_\{c\\neq c^\{\\prime\}\}\\left\[\(\\mu\_\{c\}\-\\mu\_\{c^\{\\prime\}\}\)^\{\\top\}\\Sigma\_\{s\}^\{\-1\}\(\\mu\_\{c\}\-\\mu\_\{c^\{\\prime\}\}\)\\right\]^\{1/2\}\.\(8\)The certificate is

rt=maxc:\|Ic,t\|\>0\(δ^c,t⊤Σs−1δ^c,t\)1/2γs\+ε\.r\_\{t\}=\\frac\{\\max\_\{c:\\,\|I\_\{c,t\}\|\>0\}\\left\(\\hat\{\\delta\}\_\{c,t\}^\{\\top\}\\Sigma\_\{s\}^\{\-1\}\\hat\{\\delta\}\_\{c,t\}\\right\)^\{1/2\}\}\{\\gamma\_\{s\}\+\\varepsilon\}\.\(9\)CASTER accepts the transported classifier only ifrt≤τr\_\{t\}\\leq\\tauand the retained set satisfies fixed minimum\-sample and minimum\-class\-coverage requirements\. If either coverage requirement fails, or no confident pseudo\-class remains, the batch is rejected by construction and the frozen model head is used instead\. All thresholds are fixed globally rather than tuned per dataset\.

Using the same Mahalanobis metric in the numerator and denominator is essential\. If a Euclidean centroid residual is divided by a Mahalanobis class margin, the ratio changes under an invertible rescaling of feature coordinates and therefore ceases to measure transport quality independently of feature conditioning\. Equation \([9](https://arxiv.org/html/2609.00374#S3.E9)\) is dimensionless and invariant to such linear rescalings\. Empirically, this correction is also necessary: on Flowers\-102, the mixed\-metric version can remain numerically small even when an inverse\-transported centroid approaches a different source class mean, making any useful rejection threshold impossible\. Section[4\.3](https://arxiv.org/html/2609.00374#S4.SS3)evaluates the resulting gate and, importantly, separates catastrophe detection from benefit ranking\.

## 4Experiments

### 4\.1Protocol

#### Frozen\-feature estimator quality\.

We first isolate the quality and storage cost of the classifier used on a fixed representation\. We evaluate four ImageNet\-pretrained backbones \(ViT\-B/16, ResNet\-50, DeiT\-B, and Swin\-T\) on seven datasets \(Oxford\-IIIT Pets, DTD, Flowers\-102, Food\-101, CIFAR\-10, CIFAR\-100, and Tiny\-ImageNet\)\. The backbones remain frozen and receive no task\-specific fine\-tuning\. Every estimator consumes the same extracted features, so accuracy differences are attributable to the estimator rather than representation learning\. We compare CASTER’s source Gaussian discriminant withkk\-NN and report the retained state required at inference: the complete source feature bank forkk\-NN versus per\-class moments for CASTER\.

#### Corruption benchmarks\.

For CIFAR\-10\-C and CIFAR\-100\-C we evaluate all1919corruptions at all55severities over six backbones\. Source checkpoints are trained for the corresponding clean task\. A run is admitted only if \(i\) the checkpoint exists, \(ii\) clean\-source accuracy passes a sanity threshold \(CIFAR\-10≥85%\\geq 85\\%, CIFAR\-100≥55%\\geq 55\\%\), and \(iii\) the source moments were generated from that exact checkpoint\. ResNet\-50 is repeated over three seeds; deterministic or single\-seed backbones are reported without inventing seed replication\. Our verification script fails closed when a required artifact is missing or inconsistent\.

#### Measurement conventions\.

Two conventions are important\. First, harm is always measured against an immutable copy of the unadapted source model\. Comparing an in\-place adaptive model with its own immediately preceding state would make harmful updates appear artificially benign\. Second, when an adaptation baseline has no eligible parameters, its prediction must reproduce source\-only inference exactly, including evaluation mode\. Otherwise dropout or other train\-mode behavior can manufacture an apparent adaptation effect\.

### 4\.2Frozen\-Feature Estimation

Figure[2](https://arxiv.org/html/2609.00374#S4.F2)compares CASTER’s source Gaussian estimator withkk\-NN on identical frozen features\. CASTER is better in2727of2828backbone–dataset cells\. The only exception is ViT\-B/16 on CIFAR\-10, where the two methods differ by less than0\.040\.04top\-1 points\. The largest margins occur on Flowers\-102: CASTER gains approximately1818–2222points on three of the four backbones, a regime with many classes and relatively few labeled examples per class\.

The right panel reports retained state\. Akk\-NN bank grows linearly with the number of source examples and reaches781781MiB on Tiny\-ImageNet\. CASTER stores class moments, with state𝒪⁡\(C​k02\)\\mathcal\{O\}\(Ck\_\{0\}^\{2\}\)and no dependence on the number of source examples\. This advantage is not universal: on Flowers\-102, only10201020training images are stored bykk\-NN whilek0=C−1=101k\_\{0\}=C\-1=101makes CASTER’s covariance state comparatively large\. We report that inversion rather than hiding it\.

Figure 2:Frozen\-feature estimator comparisonover four backbones and seven datasets\. All methods use identical frozen ImageNet\-pretrained features\.Left:CASTER versuskk\-NN top\-1 accuracy; points above the diagonal favor CASTER\.Right:retainedkk\-NN state relative to CASTER, on a log scale\. Cells where the ratio falls below one are reported explicitly in the text\.
### 4\.3What the Certificate Detects

A useful gate can support two very different claims:*catastrophe detection*and*benefit ranking*\. Our certificate supports the first but not the second\.

#### It detects degenerate transport\.

Figure[3](https://arxiv.org/html/2609.00374#S4.F3)pools all admissible evaluation cells from CIFAR\-10\-C, CIFAR\-100\-C, ImageNet\-C, and the additional target datasets used in the transport study\. Every cell that loses more than1010top\-1 points under transport has certificate value above3\.93\.9\. The catastrophic cells are concentrated in low\-coverage regimes in which the class count is large relative to the target batch, most notably ImageNet\-C with10001000classes and batch size6464\.

We do not claim perfect separation\. The safe and unsafe bands overlap: at least one non\-harmful cell has a certificate as high as8\.98\.9\. This matters because an earlier analysis used fine\-grained test streams that were inadvertently class ordered, leaving9797–100%100\\%of many batches dominated by one class\. After shuffling to the intended i\.i\.d\. stream, those cells are benign and the certificate reflects the corrected protocol\. All results below use the i\.i\.d\. streams\.

#### It does not rank benefit\.

Within the non\-degenerate regime, the raw certificate score is positively correlated with the accuracy gain from transport\. A gate intended to rank updates by expected benefit would need the opposite ordering, because lowerrtr\_\{t\}is the acceptance direction\. The explanation is intuitive: larger distribution shifts tend to increase both the residual score and the amount of accuracy the degraded source head can recover\. Consequently, on CIFAR\-10\-C, where catastrophic transport is absent, thresholding the certificate does not beat simply applying transport to every batch\. The certificate should therefore be interpreted as a detector of unsupported transport, not as a utility score\.

#### The threshold is not delicate\.

Across the pooled admissible cells, unconditional transport averages−3\.35\-3\.35points relative to the frozen head\. Gating at the shippedτ=0\.8\\tau=0\.8gives\+1\.69\+1\.69points; anyτ∈\[1\.0,3\.9\]\\tau\\in\[1\.0,3\.9\]gives between\+1\.85\+1\.85and\+1\.92\+1\.92points\. Thus a roughly fourfold threshold interval remains within0\.30\.3points of the best observed value\. We retainτ=0\.8\\tau=0\.8as a conservative operating point: it rejects some transports that would have been harmless rather than accepting transports in the catastrophic regime\.

Figure 3:Certificate score versus realized transport gain, one point per admissible evaluation cell on i\.i\.d\. streams\. Every cell losing more than1010points lies above3\.93\.9, but the safe and unsafe bands are not disjoint\. The symmetric\-log axis makes both the benign and catastrophic regimes visible\.Table 1:Corruption benchmarks, aggregated over1919corruptions and55severities and then grouped by normalization family \(BN: ResNet\-18/50; LN: ConvNeXt\-T, ViT\-B/16, DeiT\-B, Swin\-T\)\. Entries are mean±\\pms\.d\. across backbones within each normalization family after averaging corruption/severity cells \(and seeds where repeated\) within each backbone\. Tent and EATA use their standard BatchNorm\-only parameter collection here and therefore reproduce the frozen head exactly on the four LN backbones\.

### 4\.4What the Gate Buys

We next compare three policies on ResNet\-50 across nine datasets: never transport, always transport, and transport only when the certificate accepts\. No fixed policy dominates across regimes\. Where transport helps, the gate gives up little:0\.10\.1points on Flowers\-102,0\.20\.2on Oxford\-IIIT Pets, and at most2\.22\.2on CIFAR\-10\-C\. Where transport is destructive, it recovers most or all of the loss:\+10\.2\+10\.2points relative to always transporting on CIFAR\-100\-C and\+21\.2\+21\.2on ImageNet\-C, where every batch is rejected and CASTER therefore matches the frozen head\.

Pooled over the admissible ResNet\-50 cells, always transporting scores55\.7055\.70top\-1 and never transporting scores59\.0559\.05; certificate gating reaches60\.0560\.05\. The value of the gate is therefore asymmetric: rejecting a useful transport sacrifices at most the gain that transport could have provided, whereas accepting an unsupported transport can incur a much larger loss\.

### 4\.5Corruption Benchmarks and Normalization

Table[1](https://arxiv.org/html/2609.00374#S4.T1)separates BatchNorm and LayerNorm backbones\. On the two BatchNorm ResNets, gradient\-based adaptation is very strong, and CASTER does not claim a universal advantage\. On the four LayerNorm architectures, the standard Tent and EATA implementations evaluated here collect no eligible BatchNorm parameters and therefore reduce exactly to the frozen source model\. SAR remains active because it can adapt LayerNorm parameters\. CASTER is independent of the network’s normalization type because it operates on frozen extracted features rather than model parameters\.

This comparison is intended to distinguish method capability from implementation artifacts\. We do not claim that entropy minimization is conceptually impossible on LayerNorm networks; rather, the canonical BatchNorm\-parameterized Tent/EATA configurations used in our benchmark are inactive there\. Reporting normalization families separately prevents those inactive cells from being mistaken for evidence about an active adaptation mechanism\.

### 4\.6ImageNet\-C

ImageNet\-C provides the clearest low\-coverage stress test: there are10001000classes but only6464samples per target batch\. The issue is not that a frozen representation is unusable in principle; it is that batch\-level class geometry cannot be reliably inferred when most classes are absent from every batch\.

Ungated feature\-space alternatives fail sharply in this setting\. Relative to the frozen head, Source\-GDA and the non\-transported Gaussian estimator lose19\.019\.0points, unconditional transport loses21\.221\.2, and T3A loses33\.533\.5\. The CASTER certificate rejects every target batch, so the deployed CASTER policy returns the frozen\-head prediction and incurs zero additional harm\. SAR is the only evaluated adaptive method that improves over the frozen head in this experiment \(\+4\.6\+4\.6points\), consistent with its different mechanism of updating normalization parameters rather than estimating target class geometry\.

This result illustrates the intended use of the certificate: when the assumptions required by affine class\-statistics transport are not supported by the observed batch, CASTER abstains from that mechanism instead of forcing an update\.

### 4\.7Certificates Do Not Transfer Across Mechanisms

A mechanism\-specific certificate should not automatically be interpreted as a generic test of whether*any*adaptation is safe\. We test this directly by placing two pre\-update gates in front of Tent and EATA: CASTER’s transportability certificate and an independent source\-statistic referee based on agreement with source\-derived predictions\. The comparison uses5757matched CIFAR\-10\-C cells with ResNet\-50 \(1919corruptions×\\times33severities\), paired with the corresponding ungated runs\.

Table 2:A transportability certificate does not transfer automatically to a different adaptation mechanism\.Results are over5757matched CIFAR\-10\-C cells with ResNet\-50\. For CASTER, gating both improves top\-1 and reduces harm by rejecting destructive transports\. Applied to Tent, the same certificate accepts only4\.3%4\.3\\%of updates and preserves0\.6%0\.6\\%of Tent’s available gain\. “Kept” is omitted for gated CASTER because rejecting harmful transports can make its gain exceed that of the ungated policy\. EATA shows the same qualitative behavior as Tent \(within0\.030\.03top\-1 points\)\.Table[2](https://arxiv.org/html/2609.00374#S4.T2)shows that the transportability certificate leaves Tent essentially indistinguishable from the frozen model\. The source\-statistic referee is more permissive, yet still discards roughly two thirds of Tent’s available improvement\. Neither criterion is a useful safety layer for Tent\.

The ordering explains the failure\. At the cell level, certificate acceptance rate is strongly anti\-correlated with Tent’s ungated gain \(ρ=−0\.86\\rho=\-0\.86\): the gate closes most often on the cells where entropy minimization helps most\. For CASTER’s own transport, the analogous acceptance\-rate relationship is positive \(ρ=\+0\.72\\rho=\+0\.72\)\. These are mechanism\-specific relationships, not evidence that the raw certificate score is a general benefit ranker; Sec\.[4\.3](https://arxiv.org/html/2609.00374#S4.SS3)shows that it is not\.

The failure is structural rather than a threshold\-tuning issue\. CASTER’s certificate evaluates geometry in the*pre\-update frozen representation*\. Tent subsequently changes normalization parameters and therefore changes the representation on which the certificate was computed\. A pre\-update statistic that never evaluates the post\-update representation has no general basis for ordering the benefit of that update\. We therefore treat the certificate as licensing CASTER’s affine transport only, and recommend reporting any adaptation certificate together with the mechanism for which it was derived and validated\.

### 4\.8Cost

Table[3](https://arxiv.org/html/2609.00374#S4.T3)measures throughput and peak memory on the same CIFAR\-10\-C cell \(fog, severity 3, ResNet\-50\), running one method at a time on an otherwise idle device\. CASTER processes10761076images/s, compared with667667for Tent,649649for EATA, and386386for SAR\. Thus CASTER is1\.6×1\.6\\timesfaster than Tent/EATA and2\.8×2\.8\\timesfaster than SAR on this controlled benchmark\. Its12491249MiB peak is22\.7%22\.7\\%of the55105510MiB used by the gradient\-based methods, and it retains90%90\\%of the frozen model’s throughput\.

We report controlled measurements rather than timings extracted from the large sweep, whose jobs shared GPUs\. Under contention, relative timings were substantially distorted\. SAR remains the most expensive evaluated method because its sharpness\-aware update requires two forward–backward evaluations per adaptation step\.

Table 3:Cost of adaptationon CIFAR\-10\-C fog, severity 3, with ResNet\-50\. Each method is measured in isolation on an idle device\. CASTER avoids backward computation and optimizer state, retaining90%90\\%of frozen\-head throughput while using4\.4×4\.4\\timesless peak memory than the gradient\-based methods\.

## 5Discussion and Limitations

#### Gradient\-based TTA remains highly competitive where model updates are available\.

On the BatchNorm backbones, gradient\-based adaptation is strong, but the comparison is not uniformly in its favour: EATA is marginally higher than CASTER on the CIFAR\-10\-C BatchNorm aggregate, whereas CASTER is higher on the CIFAR\-100\-C BatchNorm aggregate\. We therefore do not claim a universal advantage over gradient\-based TTA\. CASTER instead targets deployments in which learned model parameters must remain frozen and backward computation or optimizer state is unavailable, as well as architectures on which the BatchNorm\-only Tent/EATA configurations evaluated here have no eligible parameters\. SAR and other methods that adapt LayerNorm remain applicable in some of these settings, but require a different adaptation mechanism and, in the case of gradient\-based methods, backward computation\. More broadly, efficient visual learning increasingly requires balancing predictive performance with the computational and memory constraints of deployment\[[5](https://arxiv.org/html/2609.00374#bib.bib18)\]\. CASTER follows this deployment\-oriented perspective from a different angle: instead of compressing or modifying the learned representation, it keeps the representation fixed and minimizes the additional state and computation required for test\-time adaptation\.

#### The method inherits its representation\.

CASTER operates on a frozen feature space and cannot repair a representation that does not already separate the target classes sufficiently well\. If the backbone provides poor class separation under a given shift, transporting source statistics cannot recover information that is absent from the representation\. This limitation is shared by other frozen\-feature estimators such askk\-NN, which is why we compare estimators on identical extracted features rather than across independently trained pipelines\.

#### Transport assumes a class\-shared affine shift\.

CASTER assumes that the target feature distribution can be approximated by a single affine transformation shared across classes\. This assumption can fail when the shift is strongly class\-dependent or when the target batch is too small or imbalanced for its empirical moments to represent the target distribution reliably\. In particular, when only a small subset of classes is present in a batch, variation in class composition can be confounded with distribution shift\. The transportability certificate is designed to identify empirically unsupported transports in such regimes, but it is a diagnostic rather than a formal guarantee\. We therefore characterise its behaviour empirically and make no claim of safety over arbitrary target distributions\.

#### The certificate is mechanism\-specific\.

The residual\-to\-margin statistic is derived from the geometry induced by CASTER’s affine transport and should be interpreted only in that context\. Its failure to gate Tent does not imply that feature\-updating methods cannot be monitored; rather, it shows that a criterion computed solely from the pre\-update frozen representation has no general guarantee of ranking the benefit of an update that subsequently changes that representation\. Certificates for other adaptation mechanisms should therefore be derived and validated against the mechanisms they are intended to gate\.

## 6Conclusion

We studied test\-time adaptation under frozen\-model constraints, where learned network parameters cannot be modified and backward computation or optimizer state may be unavailable\. CASTER transports source class statistics to each target batch through a class\-shared affine map estimated from batch moments, requiring neither a backward pass nor a stored source feature bank\. On identical frozen features, it outperformskk\-nearest neighbours in 27 of 28 backbone–dataset settings while achieving a median18×18\\timesreduction in retained state\.

Affine transport is not unconditionally reliable\. Across307307admissible evaluation cells spanning nine datasets, unconditional transport is worse on average than retaining the frozen head\. CASTER therefore uses an empirical residual\-to\-margin transportability certificate to decide whether to apply the transported classifier or fall back to the frozen prediction\. Unconditional transport changes accuracy by an average of−3\.35\-3\.35points relative to the frozen head, whereas gating at the shipped threshold yields\+1\.69\+1\.69points\. Moreover, a roughly fourfold range of thresholds remains within0\.30\.3points of the best observed operating point\. The certificate therefore provides a useful safety floor for affine transport, while our analysis also shows its limitation: it detects empirically degenerate transport but does not rank how much a non\-degenerate adaptation will help\.

Finally, we show that this transportability certificate is mechanism\-specific\. Applied to Tent, it accepts only4\.3%4\.3\\%of updates and preserves0\.6%0\.6\\%of Tent’s available gain, demonstrating that evidence for a certificate on one adaptation mechanism should not be transferred automatically to another\. Where model updates and backward computation are available, gradient\-based TTA remains a strong alternative and can outperform CASTER in some regimes\. CASTER instead provides a lightweight option for frozen\-model deployment, together with an explicit account of when its adaptation mechanism is supported by the observed target batch and when it should abstain\.

## References

- \[1\]A\. Gretton, K\. M\. Borgwardt, M\. J\. Rasch, B\. Schölkopf, and A\. Smola\(2012\)A kernel two\-sample test\.Journal of Machine Learning Research13,pp\. 723–773\.Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[2\]D\. Hendrycks, S\. Basart, N\. Mu, S\. Kadavath, F\. Wang, E\. Dorundo, R\. Desai, T\. Zhu, S\. Parajuli, M\. Guo, D\. Song, J\. Steinhardt, and J\. Gilmer\(2021\)The many faces of robustness: a critical analysis of out\-of\-distribution generalization\.InIEEE/CVF International Conference on Computer Vision,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[3\]D\. Hendrycks and T\. Dietterich\(2019\)Benchmarking neural network robustness to common corruptions and perturbations\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[4\]Y\. Iwasawa and Y\. Matsuo\(2021\)Test\-time classifier adjustment module for model\-agnostic domain generalization\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px2.p1.1)\.
- \[5\]S\. Khazem, J\. Fix, and C\. Pradalier\(2025\)Polygonet: leveraging simplified polygonal representation for effective image classification\.arXiv preprint arXiv:2504\.01214\.Cited by:[§5](https://arxiv.org/html/2609.00374#S5.SS0.SSS0.Px1.p1.1)\.
- \[6\]S\. Khazem\(2026\)AdapterTune: zero\-initialized low\-rank adapters for frozen vision transformers\.arXiv preprint arXiv:2603\.14706\.Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px2.p1.1)\.
- \[7\]S\. Khazem\(2026\)Topolora\-sam: topology\-aware parameter\-efficient adaptation of foundation segmenters for thin\-structure and cross\-domain binary semantic segmentation\.InInternational Conference on Pattern Recognition,pp\. 682–696\.Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px2.p1.1)\.
- \[8\]D\. Li, Y\. Yang, Y\. Song, and T\. Hospedales\(2017\)Deeper, broader and artier domain generalization\.InIEEE International Conference on Computer Vision,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[9\]S\. Niu, C\. Miao, G\. Chen, P\. Wu, and P\. Zhao\(2024\)Test\-Time Model Adaptation with Only Forward Passes\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 38298–38315\.Cited by:[§1](https://arxiv.org/html/2609.00374#S1.p1.1),[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]S\. Niu, J\. Wu, Y\. Zhang, Y\. Chen, S\. Zheng, P\. Zhao, and M\. Tan\(2022\)Efficient test\-time model adaptation without forgetting\.InInternational Conference on Machine Learning,Cited by:[§1](https://arxiv.org/html/2609.00374#S1.p1.1),[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px1.p1.1)\.
- \[11\]S\. Niu, J\. Wu, Y\. Zhang, Z\. Wen, Y\. Chen, P\. Zhao, and M\. Tan\(2023\)Towards stable test\-time adaptation in dynamic wild world\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.00374#S1.p1.1),[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]B\. Recht, R\. Roelofs, L\. Schmidt, and V\. Shankar\(2019\)Do imagenet classifiers generalize to imagenet?\.InInternational Conference on Machine Learning,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[13\]M\. Schirmer, M\. Jazbec, C\. A\. Naesseth, and E\. Nalisnick\(2025\)Monitoring Risks in Test\-Time Adaptation\.InAdvances in Neural Information Processing Systems,Vol\.38\.External Links:[Document](https://dx.doi.org/10.52202/085713-2705)Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px3.p1.1)\.
- \[14\]H\. Venkateswara, J\. Eusebio, S\. Chakraborty, and S\. Panchanathan\(2017\)Deep hashing network for unsupervised domain adaptation\.InIEEE Conference on Computer Vision and Pattern Recognition,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[15\]C\. Villani\(2009\)Optimal transport: old and new\.Springer\.Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[16\]D\. Wang, E\. Shelhamer, S\. Liu, B\. Olshausen, and T\. Darrell\(2021\)Tent: fully test\-time adaptation by entropy minimization\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2609.00374#S1.p1.1),[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]H\. Wang, S\. Ge, Z\. Lipton, and E\. P\. Xing\(2019\)Learning robust global representations by penalizing local predictive power\.InAdvances in Neural Information Processing Systems,Cited by:[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px4.p1.1)\.
- \[18\]Q\. Wang, O\. Fink, L\. Van Gool, and D\. Dai\(2022\)Continual test\-time domain adaptation\.InIEEE/CVF Conference on Computer Vision and Pattern Recognition,Cited by:[§1](https://arxiv.org/html/2609.00374#S1.p1.1),[§2](https://arxiv.org/html/2609.00374#S2.SS0.SSS0.Px1.p1.1)\.