Finite-Horizon Input-Output Dynamics of Minibatch Perturbations in AdamW

# Finite-Horizon Input–Output Dynamics of Minibatch Perturbations in AdamW
Source: [https://arxiv.org/html/2608.19762](https://arxiv.org/html/2608.19762)
Kang LiuAffiliation:School of Future Technology, Xi’an Jiaotong University, Xi’an, China, 710049Correspondence to:[kanyo@foxmail\.com](mailto:[email protected])Suyan LiAffiliation:Department of Electrical and Computer Engineering, National University of Singapore, Singapore, 119077

###### Abstract

A minibatch can influence training beyond the update at which it is observed because AdamW stores past gradient information in its optimizer states\. We study this delayed effect through paired trajectories that differ only in one gradient update and share the same subsequent training sequence\. We formulate AdamW as a finite\-horizon input–state–output \(ISO\) system whose state contains the model parameters and first\- and second\-moment estimates\. Linearizing the joint dynamics yields a signed response operator that maps a localized gradient perturbation to its future loss effects, revealing how optimizer memory shapes their magnitude, timing, and sign\. We further derive an exact multistep error decomposition and establish first\-order finite\-horizon accuracy under local smoothness and controlled activation switching\. Experiments validate the response mechanism and optimizer\-state effects, while repeated\-future analyses reveal substantial prospective structure in delayed influence that can be partially recovered from ISO approximations\. Code is available at[https://github\.com/Kanyooo/Loss\_ISO](https://github.com/Kanyooo/Loss_ISO)\.

###### Keywords:

Optimizer dynamics, Minibatch perturbation, Finite\-Horizon analysis, AdamW

## 1Introduction

Modern neural networks are trained through a sequence of stochastic minibatch updates\. With adaptive optimizers such as AdamW, however, the effect of a minibatch is not confined to the step at which it is observed\. Its gradient changes not only the model parameters, but also the first\- and second\-moment estimates that determine subsequent updates\. As a result, the effect of one minibatch can persist for several training steps and may become visible in the loss only later\. This temporal dependence is particularly relevant to transient training instabilities, including loss spikes\. Sharp local geometry and large\-step training dynamics have been linked to transient loss excursions\([31](https://arxiv.org/html/2608.19762#bib.bib1)\)\. Small\-scale proxies further show that local training statistics can anticipate large\-scale Transformer instabilities\([28](https://arxiv.org/html/2608.19762#bib.bib24)\)\. More recent work attributes another class of spikes to abnormal stochastic gradients and motivates spike\-aware momentum treatment\([12](https://arxiv.org/html/2608.19762#bib.bib4)\)\.

Existing analyses characterize Adam through local geometry, adaptive preconditioning, and optimizer dynamics\([7](https://arxiv.org/html/2608.19762#bib.bib3);[1](https://arxiv.org/html/2608.19762#bib.bib2)\)\. Recent influence studies further show that training\-data effects depend on their position along the optimization trajectory\([27](https://arxiv.org/html/2608.19762#bib.bib5)\), and that AdamW\-aware attribution must propagate perturbations through both parameter and moment states\([8](https://arxiv.org/html/2608.19762#bib.bib6)\)\. Optimizer memory can also make minibatch ordering a first\-order source of finite\-window variation\([25](https://arxiv.org/html/2608.19762#bib.bib7)\)\.These results establish that optimizer state and trajectory matter, but they do not provide a signed, horizon\-resolved account of how a localized minibatch perturbation is stored, propagated, and expressed in future loss\.

We study the delayed effect of minibatch perturbations in AdamW by comparing paired training trajectories that differ only at one update and share the same subsequent training sequence\. This construction isolates how a localized perturbation evolves over a finite future horizon\.

The central idea is to formulate AdamW as a finite\-horizon input–state–output \(ISO\) system, with the model parameters and optimizer moments forming the joint state\. We derive a tangent model that maps an initial gradient perturbation to its future loss response through the intervening AdamW dynamics, revealing how optimizer memory shapes the magnitude, timing, and sign of delayed effects\. We further characterize the finite\-horizon approximation error, including nonlinear dynamics and activation\-pattern changes\. Beyond pathwise analysis, we ask whether delayed influence retains structure before the future training sequence is realized\. Repeated\-future experiments show substantial prospective structure, while its recoverability from present\-time ISO approximations depends on the underlying dynamics\. Figure[1](https://arxiv.org/html/2608.19762#S1.F1)summarizes the ISO mechanism\.

![Refer to caption](https://arxiv.org/html/2608.19762v1/figure1.png)Figure 1:Finite\-horizon ISO view of delayed minibatch influence in AdamW\.Our contributions are threefold:

1. 1\.We formulate localized minibatch influence as a signed, finite\-horizon response under paired AdamW trajectories, isolating the effect of a single gradient perturbation under a shared future realization\.
2. 2\.We derive a joint parameter–moment ISO operator that decomposes the response into perturbation write\-in, optimizer\-state propagation, and loss readout, thereby characterizing how AdamW memory can delay or transiently amplify a minibatch’s effect\.
3. 3\.We establish an exact multistep error decomposition that separates write\-in nonlinearity, smooth propagation error, and activation switching, and prove fixed\-horizon first\-order accuracy\.

## 2Related Work

Training\-data influence has gradually moved from static, endpoint\-based attribution toward trajectory\-aware descriptions of how individual training examples affect learning\. Classical influence functions characterize infinitesimal reweighting around a trained solution\([14](https://arxiv.org/html/2608.19762#bib.bib8)\), while subsequent work has improved their scalability and applicability to modern deep networks and large models\([24](https://arxiv.org/html/2608.19762#bib.bib9);[10](https://arxiv.org/html/2608.19762#bib.bib10);[21](https://arxiv.org/html/2608.19762#bib.bib11);[16](https://arxiv.org/html/2608.19762#bib.bib12);[29](https://arxiv.org/html/2608.19762#bib.bib13)\)\. A parallel line instead follows the optimization path, from retracing SGD updates and propagating hypergradients\([11](https://arxiv.org/html/2608.19762#bib.bib14);[22](https://arxiv.org/html/2608.19762#bib.bib15);[5](https://arxiv.org/html/2608.19762#bib.bib16)\)to approximate unrolling and trajectory\-specific influence\([2](https://arxiv.org/html/2608.19762#bib.bib17);[27](https://arxiv.org/html/2608.19762#bib.bib5)\)\. Recent work further shows that faithful trajectory attribution under AdamW requires accounting for its parameter and moment states\([8](https://arxiv.org/html/2608.19762#bib.bib6)\)\. These developments establish that training order and intermediate optimization states matter\. The object studied here is more local and dynamical: we perturb one realized minibatch gradient and track its signed response over a finite horizon, making the temporal response itself the quantity of interest\.

For AdamW, this distinction is important because the optimizer carries information across updates through its first\- and second\-moment states\. Existing work has characterized Adam and AdamW through convergence, preconditioning, implicit geometry, and dynamical behavior\([13](https://arxiv.org/html/2608.19762#bib.bib18);[23](https://arxiv.org/html/2608.19762#bib.bib19);[18](https://arxiv.org/html/2608.19762#bib.bib20);[19](https://arxiv.org/html/2608.19762#bib.bib21);[1](https://arxiv.org/html/2608.19762#bib.bib2);[17](https://arxiv.org/html/2608.19762#bib.bib22);[30](https://arxiv.org/html/2608.19762#bib.bib23)\)\. Closely related studies of training instability show that adaptive preconditioning, gradient statistics, and optimizer\-state dynamics can shape short\-time behavior and transient loss spikes\([28](https://arxiv.org/html/2608.19762#bib.bib24);[31](https://arxiv.org/html/2608.19762#bib.bib1);[12](https://arxiv.org/html/2608.19762#bib.bib4);[26](https://arxiv.org/html/2608.19762#bib.bib25);[3](https://arxiv.org/html/2608.19762#bib.bib26)\)\. Optimizer memory can also make minibatch ordering a first\-order source of finite\-window variation\([25](https://arxiv.org/html/2608.19762#bib.bib7)\)\. These results motivate treating the optimizer state as part of the perturbation dynamics rather than as an implementation detail\. Our formulation treats parameters and AdamW moments as a joint state, explicitly separating perturbation write\-in, state propagation, and loss readout\. This finite\-horizon ISO view provides a signed, horizon\-resolved account of how a localized minibatch perturbation is stored in optimizer state and later expressed in future loss\.

## 3Finite\-Horizon Minibatch Influence

### 3\.1Overview and Time Convention

We model the effect of a gradient perturbation as a finite\-horizon input–state–output process:

ξt→Btδ​xt\+1→Φt\+h,t\+1δ​xt\+h→ct\+h⊤δ​Ft\+h\.\\xi\_\{t\}\\xrightarrow\{\\;B\_\{t\}\\;\}\\delta x\_\{t\+1\}\\xrightarrow\{\\;\\Phi\_\{t\+h,t\+1\}\\;\}\\delta x\_\{t\+h\}\\xrightarrow\{\\;c\_\{t\+h\}^\{\\top\}\\;\}\\delta F\_\{t\+h\}\.Here,ξt\\xi\_\{t\}denotes the perturbation applied at steptt,BtB\_\{t\}maps it into the joint AdamW state,Φt\+h,t\+1\\Phi\_\{t\+h,t\+1\}propagates the resulting state deviation through subsequent updates, andct\+h⊤c\_\{t\+h\}^\{\\top\}maps the propagated deviation to the probe\-loss response\.

Throughout,xsx\_\{s\}denotes the joint state immediately before minibatchssis processed\. Since the perturbation is applied during updatett, horizonh=1h=1corresponds to the first post\-perturbation statext\+1x\_\{t\+1\}\.

### 3\.2Paired Trajectories and Joint AdamW State

For minibatchℬs\\mathcal\{B\}\_\{s\}, definegs​\(θ\):=∇θℓ​\(θ,ℬs\)∈ℝd\.g\_\{s\}\(\\theta\):=\\nabla\_\{\\theta\}\\ell\(\\theta;\\mathcal\{B\}\_\{s\}\)\\in\\mathbb\{R\}^\{d\}\.Immediately before processing minibatchss, the joint AdamW state isxs:=\[θs,ms−1,vs−1\]⊤∈ℝ3​d,x\_\{s\}:=\[\\theta\_\{s\},m\_\{s\-1\},v\_\{s\-1\}\]^\{\\top\}\\in\\mathbb\{R\}^\{3d\},whereθs\\theta\_\{s\}is the model parameter,ms−1m\_\{s\-1\}is the first\-moment state, andvs−1v\_\{s\-1\}is the second\-moment state\. Under AdamW\([18](https://arxiv.org/html/2608.19762#bib.bib20)\), the first\- and second\-moment estimates evolve as

ms\\displaystyle m\_\{s\}=β1​ms−1\+\(1−β1\)​gs​\(θs\),\\displaystyle=\\beta\_\{1\}m\_\{s\-1\}\+\(1\-\\beta\_\{1\}\)g\_\{s\}\(\\theta\_\{s\}\),vs\\displaystyle v\_\{s\}=β2​vs−1\+\(1−β2\)​gs​\(θs\)⊙2\.\\displaystyle=\\beta\_\{2\}v\_\{s\-1\}\+\(1\-\\beta\_\{2\}\)g\_\{s\}\(\\theta\_\{s\}\)^\{\\odot 2\}\.Letρ1,s:=1−β1s\\rho\_\{1,s\}:=1\-\\beta\_\{1\}^\{s\}andρ2,s:=1−β2s\\rho\_\{2,s\}:=1\-\\beta\_\{2\}^\{s\}, and define the corrected momentsm^s:=ms/ρ1,s\\widehat\{m\}\_\{s\}:=m\_\{s\}/\\rho\_\{1,s\}andv^s:=vs/ρ2,s\\widehat\{v\}\_\{s\}:=v\_\{s\}/\\rho\_\{2,s\}\. The parameter update is written as

θs\+1=Ds​θs\+qs​\(ms,vs\),Ds:=I−ηs​Λs,\\theta\_\{s\+1\}=D\_\{s\}\\theta\_\{s\}\+q\_\{s\}\(m\_\{s\},v\_\{s\}\),~~D\_\{s\}:=I\-\\eta\_\{s\}\\Lambda\_\{s\},whereΛs\\Lambda\_\{s\}denotes decoupled weight decay and

qs\(m,v\):=−ηs\(m/ρ1,s\)⊘\(v/ρ2,s\+ϵ𝟏\)q\_\{s\}\(m,v\):=\-\\eta\_\{s\}\\left\(m/\{\\rho\_\{1,s\}\}\\right\)\\oslash\\left\(\\sqrt\{\{v\}/\{\\rho\_\{2,s\}\}\}\+\\epsilon\\mathbf\{1\}\\right\)is evaluated coordinatewise\.

##### Paired trajectories\.

We compare a control trajectory and a shock trajectory initialized from the same state:xts=xtc=xtx\_\{t\}^\{\\mathrm\{s\}\}=x\_\{t\}^\{\\mathrm\{c\}\}=x\_\{t\}\. At the shock step, their gradients satisfygts=gtc\+ξtg\_\{t\}^\{\\mathrm\{s\}\}=g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}, whereξt\\xi\_\{t\}is the realized minibatch\-gradient perturbation\. From stept\+1t\+1onward, the two trajectories use the same minibatches, stochastic realizations, learning\-rate schedule, and optimizer configuration\.

Define the paired state deviationδ​xs:=xss−xsc\\delta x\_\{s\}:=x\_\{s\}^\{\\mathrm\{s\}\}\-x\_\{s\}^\{\\mathrm\{c\}\}\. Let\{Fs:ℝd→ℝ\}\\\{F\_\{s\}:\\mathbb\{R\}^\{d\}\\rightarrow\\mathbb\{R\}\\\}be a sequence of probe objectives shared by the paired trajectories, and define

δ​Fs:=Fs​\(θss\)−Fs​\(θsc\)\.\\delta F\_\{s\}:=F\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{s\}\}\)\-F\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\)\.A fixed probe corresponds to the special caseFs≡FF\_\{s\}\\equiv F\. Alternatively, taking

Fs​\(θ\):=ℓ⁡\(θ,ℬs\)F\_\{s\}\(\\theta\):=\\ell\(\\theta;\\mathcal\{B\}\_\{s\}\)gives the response of the realized future minibatch loss under the shared future sequence\. Because the probe and future random sequence are shared,δ​Fs\\delta F\_\{s\}isolates the pathwise effect of the gradient difference introduced at steptt\. The response is signed:δ​Fs\>0\\delta F\_\{s\}\>0means that the shock raises the probe loss relative to the control, whereasδ​Fs<0\\delta F\_\{s\}<0means that it lowers the probe loss\.

The control gradientgtcg\_\{t\}^\{\\mathrm\{c\}\}may be the gradient of another realized minibatch, as in the paired experimental protocol\. Alternatively, taking it to be the conditional mean gradient,

g¯t:=𝔼⁡\[gt∣ℱt\],ξt:=gt−g¯t,\\bar\{g\}\_\{t\}:=\\mathbb\{E\}\[g\_\{t\}\\mid\\mathcal\{F\}\_\{t\}\],~~\\xi\_\{t\}:=g\_\{t\}\-\\bar\{g\}\_\{t\},gives a conditional stochastic interpretation of the perturbation, whereℱt\\mathcal\{F\}\_\{t\}denotes the training history available before sampling the minibatch at steptt\.

### 3\.3Minibatch Shock Write\-In

We first derive how the perturbationξt\\xi\_\{t\}enters the three blocks of the joint AdamW state\. Unless stated otherwise, all quantities in this subsection are evaluated along the control trajectory\. Define the sensitivities of the adaptive parameter update to the moment states:

Ms:=∂qs∂ms,Vs:=∂qs∂vs\.M\_\{s\}:=\\frac\{\\partial q\_\{s\}\}\{\\partial m\_\{s\}\},~~V\_\{s\}:=\\frac\{\\partial q\_\{s\}\}\{\\partial v\_\{s\}\}\.Writingds:=v^s\+ϵ​𝟏d\_\{s\}:=\\sqrt\{\\widehat\{v\}\_\{s\}\}\+\\epsilon\\mathbf\{1\}, these sensitivities are diagonal:

Ms\\displaystyle M\_\{s\}=−ηsρ1,s​Diag⁡\(ds−1\),\\displaystyle=\-\\frac\{\\eta\_\{s\}\}\{\\rho\_\{1,s\}\}\\operatorname\{Diag\}\(d\_\{s\}^\{\-1\}\),Vs\\displaystyle V\_\{s\}=ηs​Diag⁡\(m^s2​ρ2,s​v^s⊙ds⊙2\)\.\\displaystyle=\\eta\_\{s\}\\operatorname\{Diag\}\\left\(\\frac\{\\widehat\{m\}\_\{s\}\}\{2\\rho\_\{2,s\}\\sqrt\{\\widehat\{v\}\_\{s\}\}\\odot d\_\{s\}^\{\\odot 2\}\}\\right\)\.Derivatives with respect to the second\-moment state are evaluated at coordinates wherev^s,i\>0\\widehat\{v\}\_\{s,i\}\>0\. The matrixMsM\_\{s\}is negative diagonal, while the sign of each diagonal entry ofVsV\_\{s\}follows the corresponding first\-moment coordinate\. Define the effective gradient\-to\-parameter Jacobian

Γs:=\(1−β1\)​Ms\+2​\(1−β2\)​Vs​Diag⁡\(gsc\)\.\\Gamma\_\{s\}:=\(1\-\\beta\_\{1\}\)M\_\{s\}\+2\(1\-\\beta\_\{2\}\)V\_\{s\}\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\.Its two terms correspond to the pathsg→m→θg\\rightarrow m\\rightarrow\\thetaandg→v→θg\\rightarrow v\\rightarrow\\theta, respectively\.

At the shock step, direct subtraction of the two moment updates gives

δ​mt\\displaystyle\\delta m\_\{t\}=\(1−β1\)​ξt,\\displaystyle=\(1\-\\beta\_\{1\}\)\\xi\_\{t\},δ​vt\\displaystyle\\delta v\_\{t\}=2​\(1−β2\)​Diag⁡\(gtc\)​ξt\+\(1−β2\)​ξt⊙2\.\\displaystyle=2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\\xi\_\{t\}\+\(1\-\\beta\_\{2\}\)\\xi\_\{t\}^\{\\odot 2\}\.Thus, the first\-moment write\-in is exactly linear, whereas the second\-moment write\-in contains both linear and quadratic components\.

Define the composite gradient\-to\-parameter map

ψt​\(g\):=qt​\(β1​mt−1\+\(1−β1\)​g,β2​vt−1\+\(1−β2\)​g⊙2\)\.\\psi\_\{t\}\(g\):=q\_\{t\}\\Bigl\(\\beta\_\{1\}m\_\{t\-1\}\+\(1\-\\beta\_\{1\}\)g,\\;\\beta\_\{2\}v\_\{t\-1\}\+\(1\-\\beta\_\{2\}\)g^\{\\odot 2\}\\Bigr\)\.ThenΓt=D​ψt​\(gtc\)\\Gamma\_\{t\}=D\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\), and

δ​θt\+1=Γt​ξt\+rθ,t​\(ξt\),\\delta\\theta\_\{t\+1\}=\\Gamma\_\{t\}\\xi\_\{t\}\+r\_\{\\theta,t\}\(\\xi\_\{t\}\),where

rθ,t​\(ξt\):=ψt​\(gtc\+ξt\)−ψt​\(gtc\)−Γt​ξt\.r\_\{\\theta,t\}\(\\xi\_\{t\}\):=\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}\)\-\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\-\\Gamma\_\{t\}\\xi\_\{t\}\.IfD​ψtD\\psi\_\{t\}isκθ,t\\kappa\_\{\\theta,t\}\-Lipschitz along the segment joininggtcg\_\{t\}^\{\\mathrm\{c\}\}andgtc\+ξtg\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}, then

‖rθ,t​\(ξt\)‖≤κθ,t2​‖ξt‖2\.\\\|r\_\{\\theta,t\}\(\\xi\_\{t\}\)\\\|\\leq\\frac\{\\kappa\_\{\\theta,t\}\}\{2\}\\\|\\xi\_\{t\}\\\|^\{2\}\.
Stacking the three state blocks yields

δ​xt\+1=Bt​ξt\+rB,t​\(ξt\),\\delta x\_\{t\+1\}=B\_\{t\}\\xi\_\{t\}\+r\_\{B,t\}\(\\xi\_\{t\}\),where

Bt:=\[Γt\(1−β1\)​I2​\(1−β2\)​Diag⁡\(gtc\)\],rB,t​\(ξt\):=\[rθ,t​\(ξt\)0\(1−β2\)​ξt⊙2\]\.B\_\{t\}:=\\begin\{bmatrix\}\\Gamma\_\{t\}\\\\\[2\.84526pt\] \(1\-\\beta\_\{1\}\)I\\\\\[2\.84526pt\] 2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\\end\{bmatrix\},~~r\_\{B,t\}\(\\xi\_\{t\}\):=\\begin\{bmatrix\}r\_\{\\theta,t\}\(\\xi\_\{t\}\)\\\\\[2\.84526pt\] 0\\\\\[2\.84526pt\] \(1\-\\beta\_\{2\}\)\\xi\_\{t\}^\{\\odot 2\}\\end\{bmatrix\}\.\(1\)
The remainder contains two distinct nonlinear effects: the intrinsic quadratic write\-in to the second\-moment state and the nonlinearity of the adaptive parameter update\. The current weight\-decay term does not appear inBtB\_\{t\}because the two trajectories share the same pre\-shock parameterθt\\theta\_\{t\}; weight decay instead enters subsequent transitions throughDs=I−ηs​ΛsD\_\{s\}=I\-\\eta\_\{s\}\\Lambda\_\{s\}\.

### 3\.4Joint\-State Propagation and Loss Readout

Letfsf\_\{s\}denote the exact state transition under future minibatchℬs\\mathcal\{B\}\_\{s\}\. For a smooth network, setf~s:=fs\\widetilde\{f\}\_\{s\}:=f\_\{s\}\. For a piecewise\-smooth network, letf~s\\widetilde\{f\}\_\{s\}denote the smooth extension associated with the activation pattern of the control trajectory atxscx\_\{s\}^\{\\mathrm\{c\}\}\. The tangent transition matrix is

As:=D​f~s​\(xsc\)\.A\_\{s\}:=D\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\.
Although the paired trajectories use the same future minibatch, their gradients generally differ once their parameters diverge\. Letg~s\\widetilde\{g\}\_\{s\}denote the gradient map within the control\-region smooth extension, and define

Hstr:=D​g~s​\(θsc\)\.H\_\{s\}^\{\\mathrm\{tr\}\}:=D\\widetilde\{g\}\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\)\.When the loss is twice differentiable in this region,

Hstr=∇θ2​ℓ​\(θsc,ℬs\)\.H\_\{s\}^\{\\mathrm\{tr\}\}=\\nabla\_\{\\theta\}^\{2\}\\ell\(\\theta\_\{s\}^\{\\mathrm\{c\}\};\\mathcal\{B\}\_\{s\}\)\.The corresponding first\-order gradient variation is

g~s​\(θsc\+δ​θs\)−g~s​\(θsc\)=Hstr​δ​θs\+o⁡\(‖δ​θs‖\)\.\\widetilde\{g\}\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\+\\delta\\theta\_\{s\}\)\-\\widetilde\{g\}\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\)=H\_\{s\}^\{\\mathrm\{tr\}\}\\delta\\theta\_\{s\}\+o\(\\\|\\delta\\theta\_\{s\}\\\|\)\.
Define

Ss:=2​\(1−β2\)​Diag⁡\(gsc\)​Hstr\.S\_\{s\}:=2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)H\_\{s\}^\{\\mathrm\{tr\}\}\.The tangent transition of the joint parameter–moment state is

As=\[Ds\+Γs​Hstrβ1​Msβ2​Vs\(1−β1\)​Hstrβ1​I0Ss0β2​I\]\.A\_\{s\}=\\begin\{bmatrix\}D\_\{s\}\+\\Gamma\_\{s\}H\_\{s\}^\{\\mathrm\{tr\}\}&\\beta\_\{1\}M\_\{s\}&\\beta\_\{2\}V\_\{s\}\\\\\[2\.84526pt\] \(1\-\\beta\_\{1\}\)H\_\{s\}^\{\\mathrm\{tr\}\}&\\beta\_\{1\}I&0\\\\\[2\.84526pt\] S\_\{s\}&0&\\beta\_\{2\}I\\end\{bmatrix\}\.\(2\)The off\-diagonal blocks capture the two feedback paths

θ→g→m→θ,θ→g⊙2→v→θ\.\\theta\\rightarrow g\\rightarrow m\\rightarrow\\theta,~~\\theta\\rightarrow g^\{\\odot 2\}\\rightarrow v\\rightarrow\\theta\.Hence, the persistence parametersβ1\\beta\_\{1\}andβ2\\beta\_\{2\}describe memory within the individual moment states but do not by themselves determine the stability or finite\-horizon gain of the coupled system\.

For indicesa≥ba\\geq b, define the ordered propagator

Φa,b:=\{I,a=b,Aa−1Aa−2⋯Ab,a\>b\.\\Phi\_\{a,b\}:=\\begin\{cases\}I,&a=b,\\\\\[2\.84526pt\] A\_\{a\-1\}A\_\{a\-2\}\\cdots A\_\{b\},&a\>b\.\\end\{cases\}In particular,

Φt\+h,t\+1=At\+h−1⋯At\+1,h≥2,\\Phi\_\{t\+h,t\+1\}=A\_\{t\+h\-1\}\\cdots A\_\{t\+1\},~~h\\geq 2,withΦt\+1,t\+1=I\\Phi\_\{t\+1,t\+1\}=I\. The matricesAsA\_\{s\}vary along the future training trajectory and generally do not commute\.

Define the probe readout vector

cs:=\[∇Fs​\(θsc\)00\]∈ℝ3​d\.c\_\{s\}:=\\begin\{bmatrix\}\\nabla F\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\)\\\\ 0\\\\ 0\\end\{bmatrix\}\\in\\mathbb\{R\}^\{3d\}\.The probe loss depends directly only on the parameter block, so a deviation stored in the moment states affects the probe loss only after it is converted into parameter motion\.

###### Theorem 3\.1\(Finite\-horizon directional response\)\.

Consider the scaled perturbation

gts​\(α\)=gtc\+α​ξt\.g\_\{t\}^\{\\mathrm\{s\}\}\(\\alpha\)=g\_\{t\}^\{\\mathrm\{c\}\}\+\\alpha\\xi\_\{t\}\.Suppose that the control trajectory lies in the interior of the smooth regions used to definef~s\\widetilde\{f\}\_\{s\}, that the corresponding AdamW transitions are differentiable along the trajectory, and thatFsF\_\{s\}is differentiable at the relevant control states\. In particular, the second\-moment coordinates involved in the derivatives satisfyv^s,ic\>0\\widehat\{v\}\_\{s,i\}^\{\\mathrm\{c\}\}\>0\. Letδ​xt\+h​\(α\)\\delta x\_\{t\+h\}\(\\alpha\)andδ​Ft\+h​\(α\)\\delta F\_\{t\+h\}\(\\alpha\)denote the resulting paired state and probe\-loss responses\. Then, for every fixedh≥1h\\geq 1,

dd​α​δ​xt\+h​\(α\)\|α=0\\displaystyle\\left\.\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}\\delta x\_\{t\+h\}\(\\alpha\)\\right\|\_\{\\alpha=0\}=Φt\+h,t\+1​Bt​ξt,\\displaystyle=\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\},dd​α​δ​Ft\+h​\(α\)\|α=0\\displaystyle\\left\.\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}\\delta F\_\{t\+h\}\(\\alpha\)\\right\|\_\{\\alpha=0\}=ct\+h⊤​Φt\+h,t\+1​Bt​ξt\.\\displaystyle=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\.Define the horizon\-hhinput–output operator

𝒢t,h:=ct\+h⊤​Φt\+h,t\+1​Bt\.\\mathcal\{G\}\_\{t,h\}:=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\.Then

δ​Ft\+h​\(α\)=α​𝒢t,h​ξt\+o⁡\(α\),α→0\.\\delta F\_\{t\+h\}\(\\alpha\)=\\alpha\\mathcal\{G\}\_\{t,h\}\\xi\_\{t\}\+o\(\\alpha\),~~\\alpha\\rightarrow 0\.

The complete proof is given in Appendix[B](https://arxiv.org/html/2608.19762#A2)\. Theorem[3\.1](https://arxiv.org/html/2608.19762#S3.Thmtheorem1)gives the central ISO decomposition: a perturbation first enters the joint AdamW state throughBtB\_\{t\}, is propagated by the intervening dynamics throughΦt\+h,t\+1\\Phi\_\{t\+h,t\+1\}, and is finally observed through the loss readoutct\+h⊤c\_\{t\+h\}^\{\\top\}\. The future response therefore depends on the interaction of perturbation direction, optimizer dynamics, and future loss sensitivity rather than on any one of these quantities alone\.

##### Finite\-horizon response summaries\.

For a horizonHH, we summarize the response by its largest magnitude, its timing, and its sign:

Mt,H:=max1≤h≤H⁡\|δ​Ft\+h\|,ht,H⋆:=min⁡arg⁡max1≤h≤H​\|δ​Ft\+h\|,M\_\{t,H\}:=\\max\_\{1\\leq h\\leq H\}\|\\delta F\_\{t\+h\}\|,~~h\_\{t,H\}^\{\\star\}:=\\min\\arg\\max\_\{1\\leq h\\leq H\}\|\\delta F\_\{t\+h\}\|,and

st,H⋆:=sign⁡\(δ​Ft\+ht,H⋆\)\.s\_\{t,H\}^\{\\star\}:=\\operatorname\{sign\}\\left\(\\delta F\_\{t\+h\_\{t,H\}^\{\\star\}\}\\right\)\.Their tangent counterparts are obtained by replacingδ​Ft\+h\\delta F\_\{t\+h\}with𝒢t,h​ξt\\mathcal\{G\}\_\{t,h\}\\xi\_\{t\}\. A delayed extremal response corresponds toht,H⋆\>1h\_\{t,H\}^\{\\star\}\>1\.

When only adverse loss increases are of interest, we additionally use

Pt,H\+:=max1≤h≤H⁡\[δ​Ft\+h\]\+,\[a\]\+:=max⁡\(a,0\)\.P\_\{t,H\}^\{\+\}:=\\max\_\{1\\leq h\\leq H\}\[\\delta F\_\{t\+h\}\]\_\{\+\},~~\[a\]\_\{\+\}:=\\max\(a,0\)\.This distinction is important for loss spikes: a large absolute response need not correspond to an increase in loss\.

##### Relation to loss spikes\.

WhenFt\+hF\_\{t\+h\}is chosen as the realized future minibatch loss, letTt\+hT\_\{t\+h\}denote the corresponding spike threshold and suppose

Ft\+h​\(θt\+hc\)≤Tt\+h\.F\_\{t\+h\}\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\)\\leq T\_\{t\+h\}\.The perturbed trajectory crosses the threshold at horizonhhexactly when

δ​Ft\+h\>Tt\+h−Ft\+h​\(θt\+hc\)\.\\delta F\_\{t\+h\}\>T\_\{t\+h\}\-F\_\{t\+h\}\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\)\.Thus, a positive finite\-horizon response contributes to a loss spike only when it exceeds the remaining margin to the threshold\.

### 3\.5Delayed Expression Through AdamW Memory

To isolate the role of optimizer memory, consider a scalar frozen\-coefficient approximation of the joint AdamW dynamics over a short horizon:

zh\+1=A¯​zh,A¯:=\[abmbvdmβ10dv0β2\],z\_\{h\+1\}=\\overline\{A\}z\_\{h\},~~\\overline\{A\}:=\\begin\{bmatrix\}a&b\_\{m\}&b\_\{v\}\\\\ d\_\{m\}&\\beta\_\{1\}&0\\\\ d\_\{v\}&0&\\beta\_\{2\}\\end\{bmatrix\},where

zh:=\[δ​θh,δ​mh−1,δ​vh−1\]⊤\.z\_\{h\}:=\\begin\{bmatrix\}\\delta\\theta\_\{h\},\\delta m\_\{h\-1\},\\delta v\_\{h\-1\}\\end\{bmatrix\}^\{\\top\}\.The coefficients are the scalar counterparts of the blocks in Equation \([2](https://arxiv.org/html/2608.19762#S3.E2)\):

a:=D\+Γ​Htr,bm:=β1​M,bv:=β2​V,\\displaystyle a:=D\+\\Gamma H^\{\\mathrm\{tr\}\},~~b\_\{m\}:=\\beta\_\{1\}M,~~b\_\{v\}:=\\beta\_\{2\}V,dm:=\(1−β1\)​Htr,dv:=2​\(1−β2\)​g​Htr\.\\displaystyle d\_\{m\}:=\(1\-\\beta\_\{1\}\)H^\{\\mathrm\{tr\}\},~~d\_\{v\}:=2\(1\-\\beta\_\{2\}\)gH^\{\\mathrm\{tr\}\}\.Here,bmb\_\{m\}andbvb\_\{v\}convert moment\-state deviations into parameter motion, whereasdmd\_\{m\}anddvd\_\{v\}feed parameter\-induced gradient changes back into the two moment states\.

To study how a perturbation stored in either memory channel becomes expressed in the parameter state, let

eθ:=\[1,0,0\]⊤,em:=\[0,1,0\]⊤,ev:=\[0,0,1\]⊤,e\_\{\\theta\}:=\[1,0,0\]^\{\\top\},~~e\_\{m\}:=\[0,1,0\]^\{\\top\},~~e\_\{v\}:=\[0,0,1\]^\{\\top\},and define

rk​\(h\):=eθ⊤​A¯h−1​ek,k∈\{m,v\}\.r\_\{k\}\(h\):=e\_\{\\theta\}^\{\\top\}\\overline\{A\}^\{\\,h\-1\}e\_\{k\},~~k\\in\\\{m,v\\\}\.A nonzero scalar probe sensitivity can be applied afterward as an output scaling; the analysis below concerns the magnitude and timing of the memory\-to\-parameter response\.

###### Proposition 3\.3\(Finite\-horizon memory\-channel response\)\.

Fork∈\{m,v\}k\\in\\\{m,v\\\},

rm​\(1\)\\displaystyle r\_\{m\}\(1\)=0,\\displaystyle=0,rm​\(2\)\\displaystyle r\_\{m\}\(2\)=bm,\\displaystyle=b\_\{m\},rm​\(3\)\\displaystyle r\_\{m\}\(3\)=bm​\(a\+β1\),\\displaystyle=b\_\{m\}\(a\+\\beta\_\{1\}\),rv​\(1\)\\displaystyle r\_\{v\}\(1\)=0,\\displaystyle=0,rv​\(2\)\\displaystyle r\_\{v\}\(2\)=bv,\\displaystyle=b\_\{v\},rv​\(3\)\\displaystyle r\_\{v\}\(3\)=bv​\(a\+β2\),\\displaystyle=b\_\{v\}\(a\+\\beta\_\{2\}\),and

rk​\(4\)=bk​\(a2\+a​βk\+βk2\+bm​dm\+bv​dv\)\.r\_\{k\}\(4\)=b\_\{k\}\\Bigl\(a^\{2\}\+a\\beta\_\{k\}\+\\beta\_\{k\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\Bigr\)\.\(3\)
Ifbk≠0b\_\{k\}\\neq 0and

\|a\+βk\|\>1,\|a\+\\beta\_\{k\}\|\>1,\(4\)then

\|rk​\(3\)\|\>\|rk​\(2\)\|\.\|r\_\{k\}\(3\)\|\>\|r\_\{k\}\(2\)\|\.If, in addition,

\|a2\+a​βk\+βk2\+bm​dm\+bv​dv\|\>\|a\+βk\|,\\left\|a^\{2\}\+a\\beta\_\{k\}\+\\beta\_\{k\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\right\|\>\|a\+\\beta\_\{k\}\|,\(5\)then

\|rk​\(4\)\|\>\|rk​\(3\)\|\.\|r\_\{k\}\(4\)\|\>\|r\_\{k\}\(3\)\|\.These finite\-horizon amplification conditions can hold even whenρ⁡\(A¯\)<1\\rho\(\\overline\{A\}\)<1, so asymptotic stability does not preclude transient growth in the response\.

The complete proof and spectral characterization of the frozen system are given in Appendix[C](https://arxiv.org/html/2608.19762#A3)\.

Proposition[3\.3](https://arxiv.org/html/2608.19762#S3.Thmtheorem3)makes the delay mechanism explicit\. A perturbation stored entirely inmmorvvis initially absent from the parameter output and becomes visible only after the corresponding memory state feeds back into the parameter update\. Its subsequent magnitude depends on both memory persistence and the return termsbm​dm\+bv​dvb\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\. In particular, the second\-moment feedback depends on the current momentum, gradient, and local curvature, so it can reinforce or oppose the evolving parameter deviation\.

The frozen model therefore illustrates how delayed and transiently amplified responses can arise from AdamW memory even when the local dynamics are asymptotically stable\. The full ISO operator

ct\+h⊤​Φt\+h,t\+1​Btc\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}extends this mechanism to the high\-dimensional, anisotropic, and time\-varying dynamics of an actual training trajectory\.

### 3\.6Approximation Error in Smooth and Piecewise\-Smooth Networks

The finite\-horizon tangent model linearizes a nonlinear, time\-varying training trajectory\. Its approximation error has two sources: the smooth Taylor remainder within the local control region and, for piecewise\-smooth networks, the defect caused by activation\-pattern changes\.

Using the control\-region extensionf~s\\widetilde\{f\}\_\{s\}, define

rssm\\displaystyle r\_\{s\}^\{\\mathrm\{sm\}\}:=f~s​\(xsc\+δ​xs\)−f~s​\(xsc\)−As​δ​xs,\\displaystyle:=\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\)\-\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\-A\_\{s\}\\delta x\_\{s\},rssw\\displaystyle r\_\{s\}^\{\\mathrm\{sw\}\}:=fs​\(xsc\+δ​xs\)−f~s​\(xsc\+δ​xs\)\.\\displaystyle:=f\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\)\-\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\)\.Sincefsf\_\{s\}andf~s\\widetilde\{f\}\_\{s\}agree at the control state, the exact perturbation recursion is

δ​xs\+1=As​δ​xs\+rssm\+rssw\.\\delta x\_\{s\+1\}=A\_\{s\}\\delta x\_\{s\}\+r\_\{s\}^\{\\mathrm\{sm\}\}\+r\_\{s\}^\{\\mathrm\{sw\}\}\.\(6\)For smooth networks,rssw=0r\_\{s\}^\{\\mathrm\{sw\}\}=0\. More generally, it also vanishes whenever the paired states remain in the same activation region\.

Let the tangent prediction satisfy

δ​x^t\+1:=Bt​ξt,δ​x^s\+1:=As​δ​x^s\.\\widehat\{\\delta x\}\_\{t\+1\}:=B\_\{t\}\\xi\_\{t\},~~\\widehat\{\\delta x\}\_\{s\+1\}:=A\_\{s\}\\widehat\{\\delta x\}\_\{s\}\.\(7\)
###### Theorem 3\.4\(Finite\-horizon error decomposition\)\.

Letes:=δ​xs−δ​x^s\.e\_\{s\}:=\\delta x\_\{s\}\-\\widehat\{\\delta x\}\_\{s\}\.Then, for everyh≥1h\\geq 1,

et\+h=\\displaystyle e\_\{t\+h\}=\{\}Φt\+h,t\+1​rB,t​\(ξt\)\\displaystyle\\Phi\_\{t\+h,t\+1\}r\_\{B,t\}\(\\xi\_\{t\}\)\(8\)\+∑j=t\+1t\+h−1Φt\+h,j\+1\(rjsm\+rjsw\)\.\\displaystyle\+\\sum\_\{j=t\+1\}^\{t\+h\-1\}\\Phi\_\{t\+h,j\+1\}\\left\(r\_\{j\}^\{\\mathrm\{sm\}\}\+r\_\{j\}^\{\\mathrm\{sw\}\}\\right\)\.
IfD​f~jD\\widetilde\{f\}\_\{j\}isLjL\_\{j\}\-Lipschitz along the segment joiningxjcx\_\{j\}^\{\\mathrm\{c\}\}andxjc\+δ​xjx\_\{j\}^\{\\mathrm\{c\}\}\+\\delta x\_\{j\}, then

‖et\+h‖≤‖Φt\+h,t\+1‖​‖rB,t​\(ξt\)‖\\displaystyle\\\|e\_\{t\+h\}\\\|\\leq\{\}\\\|\\Phi\_\{t\+h,t\+1\}\\\|\\\|r\_\{B,t\}\(\\xi\_\{t\}\)\\\|\(9\)\+∑j=t\+1t\+h−1∥Φt\+h,j\+1∥\(Lj2∥δxj∥2\+∥rjsw∥\)\.\\displaystyle\+\\sum\_\{j=t\+1\}^\{t\+h\-1\}\\\|\\Phi\_\{t\+h,j\+1\}\\\|\\left\(\\frac\{L\_\{j\}\}\{2\}\\\|\\delta x\_\{j\}\\\|^\{2\}\+\\\|r\_\{j\}^\{\\mathrm\{sw\}\}\\\|\\right\)\.
If, in addition,Ft\+hF\_\{t\+h\}has anLF,t\+hL\_\{F,t\+h\}\-Lipschitz gradient along the segment joiningθt\+hc\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}andθt\+hs\\theta\_\{t\+h\}^\{\\mathrm\{s\}\}, then

\|δ​Ft\+h−ct\+h⊤​Φt\+h,t\+1​Bt​ξt\|\\displaystyle\\bigl\|\\delta F\_\{t\+h\}\-c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\\bigr\|≤‖ct\+h‖​‖et\+h‖\\displaystyle\\leq\{\}\\\|c\_\{t\+h\}\\\|\\,\\\|e\_\{t\+h\}\\\|\(10\)\+LF,t\+h2​‖δ​θt\+h‖2\.\\displaystyle\+\\frac\{L\_\{F,t\+h\}\}\{2\}\\\|\\delta\\theta\_\{t\+h\}\\\|^\{2\}\.

Theorem[3\.4](https://arxiv.org/html/2608.19762#S3.Thmtheorem4)shows that the same propagators that carry the first\-order perturbation also propagate the approximation defects introduced at each step\. Large finite\-horizon gain can therefore amplify both the response of interest and the error of its tangent approximation\.

###### Corollary 3\.5\(Fixed\-horizon first\-order accuracy\)\.

Consider the scaled perturbationα​ξt\\alpha\\xi\_\{t\}and a fixed horizonHH\. Suppose that, in a neighborhood of the control trajectory,D​ψtD\\psi\_\{t\}andD​f~sD\\widetilde\{f\}\_\{s\}are locally Lipschitz, the finite\-horizon propagators are uniformly bounded, the probe objectives have locally Lipschitz gradients, and the switching defects satisfy

‖rssw‖≤Cssw​‖δ​xs‖2,1≤s−t<H\.\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq C\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\delta x\_\{s\}\\\|^\{2\},~~1\\leq s\-t<H\.Then, for every1≤h≤H1\\leq h\\leq H,

δ​Ft\+h​\(α\)=α​ct\+h⊤​Φt\+h,t\+1​Bt​ξt\+O⁡\(α2\),α→0\.\\delta F\_\{t\+h\}\(\\alpha\)=\\alpha c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\+O\(\\alpha^\{2\}\),~~\\alpha\\rightarrow 0\.TheO⁡\(α2\)O\(\\alpha^\{2\}\)constant may depend on the fixed horizon and control trajectory but not onα\\alpha\.

For smooth networks the switching condition holds withCssw=0C\_\{s\}^\{\\mathrm\{sw\}\}=0\. For piecewise\-smooth networks, a sufficient activation\-margin condition under which the aggregate switching defect is quadratic is given in Appendix[D](https://arxiv.org/html/2608.19762#A4)\. The appendix also provides the complete proofs and a recursive error envelope\.

## 4Experiments

Our experiments address two questions\. First, does the proposed joint\-state ISO model capture the finite\-horizon response mechanism across increasingly realistic training systems? Second, although the pathwise ISO conditions on a realized future training sequence, does delayed influence retain structure that is already identifiable before that future unfolds?

For mechanism validation, a control and a shock trajectory start from the same AdamW state, differ only in the gradient applied at steptt, and then process the same future minibatches\. With a fixed probe objectiveFF, we write

di,h=F⁡\(θi,t\+hs\)−F⁡\(θt\+hc\),d^i,h=ct\+h⊤​Φt\+h,t\+1​Bt​ξi,d\_\{i,h\}=F\(\\theta^\{\\mathrm\{s\}\}\_\{i,t\+h\}\)\-F\(\\theta^\{\\mathrm\{c\}\}\_\{t\+h\}\),~~\\widehat\{d\}\_\{i,h\}=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{i\},\(11\)and summarize magnitude byMi=max1≤h≤H⁡\|di,h\|M\_\{i\}=\\max\_\{1\\leq h\\leq H\}\|d\_\{i,h\}\|\. We evaluate trajectory fidelity, sign agreement, and within\-system Spearman correlation withMiM\_\{i\}\. Candidates and horizons are nested observations: the controlled and neural\-network studies aggregate within independent training systems, whereas the language\-model study is reported descriptively over fixed model–dataset conditions\. The second experiment holds the present state–shock pair fixed and instead resamples unseen future continuations\. Complete protocols, estimators, and additional results are provided in Appendix[E](https://arxiv.org/html/2608.19762#A5)\.

### 4\.1Experiment 1: Finite\-Horizon Mechanism Validation

##### Controlled quadratic systems\.

We begin with quadratic minibatch losses

ℓs​\(θ\)=12​θ⊤​Ds​θ\+12​r​‖Us⊤​θ‖22\+qs⊤​θ,\\ell\_\{s\}\(\\theta\)=\\frac\{1\}\{2\}\\theta^\{\\top\}D\_\{s\}\\theta\+\\frac\{1\}\{2r\}\\\|U\_\{s\}^\{\\top\}\\theta\\\|\_\{2\}^\{2\}\+q\_\{s\}^\{\\top\}\\theta,\(12\)withd=512d=512, rankr=16r=16, temporally correlated minibatches, and three curvature regimes\. Four seeds with eight independently generated systems each give 32 systems\. After 40 burn\-in updates, each system uses four reference minibatches, 16 candidate shocks, a common future of lengthH=32H=32, and both a standard and an anisotropic probe\. A separate rotating\-readout probe is used only in the exact\-one\-step\-matched stress test\.

We varyα∈\{1/32,1/16,1/8,1/4,1/2,1\}\\alpha\\in\\\{1/32,1/16,1/8,1/4,1/2,1\\\}\. Table[1](https://arxiv.org/html/2608.19762#S4.T1)shows that the tangent trajectory remains accurate over the local range\. Atα=1/8\\alpha=1/8, median NRMSE is0\.04830\.0483for the standard probe and0\.07190\.0719for the anisotropic probe, with perfect median sign agreement\. Fitting\|dh​\(α\)−α​d^h\|∝αph\|d\_\{h\}\(\\alpha\)\-\\alpha\\widehat\{d\}\_\{h\}\|\\propto\\alpha^\{p\_\{h\}\}overα≤1/4\\alpha\\leq 1/4givesph≈2p\_\{h\}\\approx 2throughout the horizon, with medianR2\>0\.99998R^\{2\}\>0\.99998, matching the quadratic local remainder predicted by the theory\.

Table 1:Representative signed\-trajectory fidelity in the controlled stage of Experiment 1\. Entries are medians over 32 independent systems after candidate aggregation\.To separate future propagation from the immediate response, we construct shocks whose exact\|di,1\|\|d\_\{i,1\}\|values are matched without using any response ath\>1h\>1\. The resulting within\-system CV of\|d1\|\|d\_\{1\}\|is1\.10×10−121\.10\\times 10^\{\-12\}for the standard probe and6\.70×10−126\.70\\times 10^\{\-12\}for the rotating\-readout probe, while the CV of futureMMremains0\.0920\.092and0\.4590\.459\. Full ISO recovers this future ordering with median Spearman correlations0\.9930\.993and1\.0001\.000\(Table[2](https://arxiv.org/html/2608.19762#S4.T2)\)\. Exact state interventions further separate the parameter, first\-moment, and second\-moment time scales, whose isolated responses peak near horizons66,1616, and2626, respectively\. Matched\-first\-displacement sweeps move the extremum later asβ1\\beta\_\{1\}orβ2\\beta\_\{2\}increases\. These controls isolate delayed state propagation from the immediate parameter write\.

##### Nonconvex neural networks\.

We next apply the same paired\-trajectory protocol to CIFAR\-10\([15](https://arxiv.org/html/2608.19762#bib.bib27)\)using a94,53894\{,\}538\-parameter CNN–ReLU and an855,050855\{,\}050\-parameter MLP–GELU\. Each architecture contributes 16 independently trained systems, with 12 candidate shocks per system andH=12H=12after 100 burn\-in updates\. Atα=0\.0625\\alpha=0\.0625, median trajectory NRMSE is0\.04580\.0458for CNN–ReLU and0\.03770\.0377for MLP–GELU; atα=0\.25\\alpha=0\.25it is0\.10970\.1097and0\.15300\.1530\. The MLP error exponent remains2\.0012\.001–2\.0052\.005across horizons, whereas the CNN exponent decreases from2\.0172\.017ath=1h=1to1\.1201\.120ath=12h=12as activation\-pattern differences increase, consistent with the switching term in the finite\-horizon error decomposition\.

At full scale, Full ISO ranks future magnitude with correlations0\.8880\.888and0\.7620\.762, compared with0\.5450\.545and0\.5660\.566for the exact one\-step response\. State interventions again show an early parameter response and a later first\-moment response; increasingβ1\\beta\_\{1\}from0\.50\.5to0\.990\.99multiplies accumulated response by14\.1714\.17and11\.9611\.96in the two architectures\. Among the 96 exact intervention trajectories used for the signed\-response diagnostic, 57 extrema are positive and 39 are negative\.

Table 2:Within\-system Spearman correlation with future magnitudeMMduring mechanism validation\. The controlled rows use exact\-one\-step\-matched candidates; the neural rows use natural candidates\. The one\-step score is tied in the matched controlled stress test\.
##### Pretrained language models\.

Finally, we evaluate Pythia\-410M, Pythia\-1B, and Pythia\-1\.4B\([4](https://arxiv.org/html/2608.19762#bib.bib28)\)on WikiText\-103\([20](https://arxiv.org/html/2608.19762#bib.bib29)\), OpenWebText\([9](https://arxiv.org/html/2608.19762#bib.bib30)\), and CodeParrot\([6](https://arxiv.org/html/2608.19762#bib.bib31)\)\. Each of the nine model–dataset systems is continued for 500 AdamW updates before measurement, producing nontrivial first\- and second\-moment states\. Each condition then uses two reference minibatches, eight candidate shocks, seven common\-future minibatches, andH=8H=8\.

At this scale we estimate the end\-to\-end ISO directional response numerically using centered finite differences of the probe logits, followed by the exact cross\-entropy differential; we denote this quantity byISO Tangent \(FD\)\. All 72 candidates pass the adjacent\-scale consistency test, with median consistency NRMSE0\.003780\.00378\. Trajectory NRMSE is0\.03870\.0387,0\.04770\.0477, and0\.09460\.0946atα=0\.0625\\alpha=0\.0625,0\.1250\.125, and0\.250\.25, respectively, with median trajectory cosine above0\.99980\.9998and perfect sign accuracy over this local range\.

Across model sizes, local NRMSE is0\.10900\.1090,0\.05190\.0519, and0\.05670\.0567\(Table[3](https://arxiv.org/html/2608.19762#S4.T3)\), while trajectory cosine remains above0\.9990\.999\.ISO Tangent \(FD\)has positive rank correlation in all nine model–dataset conditions, with median correlations0\.7140\.714,0\.8330\.833, and0\.7620\.762across the three model sizes\. It also recovers the full\-scale extremum sign for 65 of 72 candidates\. These results show that the signed finite\-horizon tangent response remains locally accurate and informative across model scale and data domain\.

Table 3:Language\-model scaling results in Experiment 1\. Local metrics pool the three datasets andα≤0\.25\\alpha\\leq 0\.25; rank correlations are medians over the three fixed data\-domain conditions at each model size\.

### 4\.2Experiment 2: Prospective Structure Under Unknown Futures

The pathwise operatorct\+h⊤​Φt\+h,t\+1​Btc\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}depends on the realized future training sequence and is therefore not, by itself, a present\-time predictor\. We ask a more basic question: if the current AdamW state and initiating shock are held fixed, does their finite\-horizon effect remain candidate\-specific when the unseen future minibatches are resampled?

For each fixed system and candidateii, we drawK=32K=32independent future continuationsωk\\omega\_\{k\}, while sharing each continuation between its control and shock trajectory\. Let

Mi,k:=max1≤h≤H⁡\|di,h​\(ωk\)\|,μi:=1K​∑k=1KMi,k\.M\_\{i,k\}:=\\max\_\{1\\leq h\\leq H\}\|d\_\{i,h\}\(\\omega\_\{k\}\)\|,~~\\mu\_\{i\}:=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}M\_\{i,k\}\.\(13\)Within each system we summarize repeated\-future structure by

ΠH:=Vari⁡\(μi\)Vari⁡\(μi\)\+𝔼i​\[Vark⁡\(Mi,k\)\]\.\\Pi\_\{H\}:=\\frac\{\\operatorname\{Var\}\_\{i\}\(\\mu\_\{i\}\)\}\{\\operatorname\{Var\}\_\{i\}\(\\mu\_\{i\}\)\+\\mathbb\{E\}\_\{i\}\[\\operatorname\{Var\}\_\{k\}\(M\_\{i,k\}\)\]\}\.\(14\)This is a protocol\-specific variance ratio, not an information\-theoretic fraction of predictable risk\. We also report the median Spearman correlation between each branch ranking and the conditional\-mean ranking\. To test whether the structure can be extracted without observing any sampled future branch, we compare the exact one\-step response, gradient norm, parameter\-write norm, a*present\-frozen ISO*that repeatedly applies one current reference\-derived Jacobian with theh=1h=1probe readout frozen, and a*reference\-surrogate ISO*that deterministically rolls out only the current reference minibatches\. Neither prospective ISO score uses future minibatches\.

Table 4:Prospective structure under 32 independently resampled future continuations\.ΠH\\Pi\_\{H\}is defined in Eq\. \([14](https://arxiv.org/html/2608.19762#S4.E14)\); branchρ\\rhois the median single\-future Spearman correlation with the conditional\-mean candidate ranking\. Present\-score correlations targetμi\\mu\_\{i\}\. All entries are medians over independent systems\.##### Controlled repeated futures\.

In the standard quadratic regime, natural candidates giveΠ32=0\.801\\Pi\_\{32\}=0\.801, but their exact one\-step response already correlates0\.8010\.801withμi\\mu\_\{i\}\. We therefore repeat the analysis after exact one\-step matching\. The matched\|d1\|\|d\_\{1\}\|has median within\-system CV1\.07×10−121\.07\\times 10^\{\-12\}, yetΠ32\\Pi\_\{32\}remains0\.7300\.730and the median single\-branch ranking correlation withμi\\mu\_\{i\}is0\.9090\.909\. Thus the delayed response retains substantial candidate\-specific structure after immediate magnitude is removed\. However, this structure is not recovered by the simplest present\-time compressions in the controlled system: correlations withμi\\mu\_\{i\}are0\.0490\.049for gradient norm,−0\.057\-0\.057for parameter\-write norm, and−0\.066\-0\.066for present\-frozen ISO\. The anisotropic\-probe results show the same qualitative separation and are reported in Appendix[E\.3\.1](https://arxiv.org/html/2608.19762#A5.SS3.SSS1)\.

##### Neural\-network repeated futures\.

The same construction is applied to the fixed post\-burn\-in CIFAR\-10 systems\. Exact one\-step matching succeeds for every candidate, with median CV3\.57×10−133\.57\\times 10^\{\-13\}for MLP–GELU and1\.36×10−121\.36\\times 10^\{\-12\}for CNN–ReLU\. After matching, prospective structure remains strong:Π12=0\.921\\Pi\_\{12\}=0\.921for MLP–GELU and0\.7000\.700for CNN–ReLU\. Unlike the controlled setting, present\-time ISO scores now recover much of the conditional\-mean ordering\. Present\-frozen ISO reaches median correlations0\.9410\.941and0\.7550\.755, compared with gradient\-norm correlations0\.0660\.066and0\.5000\.500for MLP–GELU and CNN–ReLU, respectively\. The reference\-surrogate ISO reaches0\.7520\.752and0\.8110\.811\. The stronger frozen\-ISO agreement in the smooth MLP is consistent with the greater local tangent coherence observed in Experiment 1\.

Taken together, Experiment 1 establishes the pathwise finite\-horizon mechanism, whereas Experiment 2 shows that its delayed effects are not created entirely by the subsequently realized minibatches: substantial candidate\-specific structure can already be present at the perturbation time\. Whether that structure admits an accurate present\-time representation is regime\-dependent, as illustrated by the contrast between the controlled, MLP–GELU, and CNN–ReLU results\. Characterizing the conditions for such prospective identifiability is distinct from the pathwise mechanism studied here\.

## 5Conclusion and Discussion

We studied delayed minibatch influence in AdamW through a finite\-horizon input–state–output formulation that tracks how perturbations enter optimizer state, propagate through future updates, and appear in later losses\. The resulting tangent model is supported across controlled systems, neural networks, and pretrained language models, while repeated\-future experiments show that delayed influence can also contain prospective structure\. The main limitations are that the exact ISO is pathwise and its present\-time approximation may degrade under nonlinear dynamics, activation switching, and future dynamical drift\. Future work should characterize when such prospective influence is identifiable from the current optimizer state, ideally through necessary and sufficient conditions, and determine whether this can support reliable training\-time warning or control\.

## Acknowledgments

We gratefully acknowledge Hongqian Huang for providing the computational resources used in this work\.

## References

- K\. Ahn, Z\. Zhang, Y\. Kook, and Y\. DaiUnderstanding Adam optimizer via online learning of updates: Adam is FTRL in disguise\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 619–640\.External Links:[Link](https://proceedings.mlr.press/v235/ahn24b.html)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p2.1),[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Baeet al\.\(2024\)J\. Bae, W\. Lin, J\. Lorraine, and R\. B\. GrosseTraining data attribution via approximate unrolling\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 66647–66686\.External Links:[Link](https://openreview.net/forum?id=3NaqGg92KZ)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Baiet al\.\(2025\)Z\. Bai, Z\. Zhou, J\. Zhao, X\. Li, Z\. Li, F\. Xiong, H\. Yang, Y\. Zhang, and Z\. J\. XuAdaptive preconditioners trigger loss spikes in adam\.arXiv preprint arXiv:2506\.04805\.External Links:[Link](https://arxiv.org/abs/2506.04805)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Bidermanet al\.\(2023\)S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. Van Der WalPythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,Vol\.202,pp\. 2397–2430\.External Links:[Link](https://proceedings.mlr.press/v202/biderman23a.html)Cited by:[§4\.1](https://arxiv.org/html/2608.19762#S4.SS1.SSS0.Px3.p1.1)\.
- Chenet al\.\(2021\)Y\. Chen, B\. Li, H\. Yu, P\. Wu, and C\. MiaoHyDRA: hypergradient data relevance analysis for interpreting deep neural networks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.35,pp\. 7081–7089\.External Links:[Link](https://ojs.aaai.org/index.php/AAAI/article/view/16871)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- CodeParrot \(2022\)CodeParrotCodeParrot clean\.Note:[https://huggingface\.co/datasets/codeparrot/codeparrot\-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean)Hugging Face datasetCited by:[§4\.1](https://arxiv.org/html/2608.19762#S4.SS1.SSS0.Px3.p1.1)\.
- Daset al\.\(2024\)R\. Das, N\. Agarwal, S\. Sanghavi, and I\. S\. DhillonTowards quantifying the preconditioning effect of Adam\.arXiv preprint arXiv:2402\.07114\.External Links:[Link](https://arxiv.org/abs/2402.07114)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p2.1)\.
- Denget al\.\(2026\)J\. Deng, P\. Hu, S\. Jin, H\. Lu, J\. T\. Wang, S\. Zhang, and J\. W\. MaHow faithful is trajectory\-based data attribution? error sources, remedies, and practical guidelines\.arXiv preprint arXiv:2605\.18814\.External Links:[Link](https://arxiv.org/abs/2605.18814)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p2.1),[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Gokaslan and Cohen \(2019\)A\. Gokaslan and V\. CohenOpenWebText corpus\.Note:[https://skylion007\.github\.io/OpenWebTextCorpus/](https://skylion007.github.io/OpenWebTextCorpus/)Cited by:[§4\.1](https://arxiv.org/html/2608.19762#S4.SS1.SSS0.Px3.p1.1)\.
- Grosseet al\.\(2023\)R\. Grosse, J\. Bae, C\. Anil, N\. Elhage, A\. Tamkin, A\. Tajdini, B\. Steiner, D\. Li, E\. Durmus, E\. Perez, E\. Hubinger, K\. Lukošiūtė, K\. Nguyen, N\. Joseph, S\. McCandlish, J\. Kaplan, and S\. R\. BowmanStudying large language model generalization with influence functions\.arXiv preprint arXiv:2308\.03296\.External Links:[Link](https://arxiv.org/abs/2308.03296)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Haraet al\.\(2019\)S\. Hara, A\. Nitanda, and T\. MaeharaData cleansing for models trained with SGD\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper/2019/hash/5f14615696649541a025d3d0f8e0447f-Abstract.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Huanget al\.\(2025\)T\. Huang, Z\. Zhu, G\. Jin, L\. Liu, Z\. Wang, and S\. LiuSPAM: spike\-aware Adam with momentum reset for stable LLM training\.arXiv preprint arXiv:2501\.06842\.External Links:[Link](https://arxiv.org/abs/2501.06842)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p1.1),[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1412.6980)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. LiangUnderstanding black\-box predictions via influence functions\.InProceedings of the 34th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.70,pp\. 1885–1894\.External Links:[Link](https://proceedings.mlr.press/v70/koh17a.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Krizhevsky \(2009\)A\. KrizhevskyLearning multiple layers of features from tiny images\.Technical reportUniversity of Toronto\.External Links:[Link](https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf)Cited by:[§4\.1](https://arxiv.org/html/2608.19762#S4.SS1.SSS0.Px2.p1.1)\.
- Kwonet al\.\(2024\)Y\. Kwon, E\. Wu, K\. Wu, and J\. ZouDataInf: efficiently estimating data influence in LoRA\-tuned LLMs and diffusion models\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=9m02ib92Wz)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Linet al\.\(2024\)W\. Lin, F\. Dangel, R\. Eschenhagen, J\. Bae, R\. E\. Turner, and A\. MakhzaniCan we remove the square\-root in adaptive gradient methods? a second\-order perspective\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 29949–29973\.External Links:[Link](https://proceedings.mlr.press/v235/lin24e.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Loshchilov and Hutter \(2019\)I\. Loshchilov and F\. HutterDecoupled weight decay regularization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=Bkg6RiCqY7)Cited by:[§A\.1](https://arxiv.org/html/2608.19762#A1.SS1.p1.2),[§2](https://arxiv.org/html/2608.19762#S2.p2.1),[§3\.2](https://arxiv.org/html/2608.19762#S3.SS2.p1.1)\.
- Maet al\.\(2022\)C\. Ma, L\. Wu, and W\. EA qualitative study of the dynamic behavior for adaptive gradient algorithms\.InProceedings of the 2nd Mathematical and Scientific Machine Learning Conference,Proceedings of Machine Learning Research, Vol\.145,pp\. 671–692\.External Links:[Link](https://proceedings.mlr.press/v145/ma22a.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Merityet al\.\(2016\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. SocherPointer sentinel mixture models\.arXiv preprint arXiv:1609\.07843\.External Links:[Link](https://arxiv.org/abs/1609.07843)Cited by:[§4\.1](https://arxiv.org/html/2608.19762#S4.SS1.SSS0.Px3.p1.1)\.
- Parket al\.\(2023\)S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. MadryTRAK: attributing model behavior at scale\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 27074–27113\.External Links:[Link](https://proceedings.mlr.press/v202/park23c.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Pruthiet al\.\(2020\)G\. Pruthi, F\. Liu, S\. Kale, and M\. SundararajanEstimating training data influence by tracing gradient descent\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 19920–19930\.External Links:[Link](https://proceedings.neurips.cc/paper/2020/hash/e6385d39ec9394f2f3a354d9d2b88eec-Abstract.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Reddiet al\.\(2018\)S\. J\. Reddi, S\. Kale, and S\. KumarOn the convergence of adam and beyond\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=ryQu7f-RZ)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Schioppaet al\.\(2022\)A\. Schioppa, P\. Zablotskaia, D\. Vilar, and A\. SokolovScaling up influence functions\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 8179–8186\.External Links:[Link](https://arxiv.org/abs/2112.03052)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Sweeney \(2026\)J\. SweeneyOptimizer memory makes shuffle order a first\-order source of fine\-tuning noise\.arXiv preprint arXiv:2606\.29554\.External Links:[Link](https://arxiv.org/abs/2606.29554)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p2.1),[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Wanget al\.\(2025a\)G\. Wang, S\. Li, C\. Chen, J\. Zeng, J\. Yang, T\. Sun, Y\. Ma, D\. Yu, and L\. ShenAdaGC: improving training stability for large language model pretraining\.arXiv preprint arXiv:2502\.11034\.External Links:[Link](https://arxiv.org/abs/2502.11034)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Wanget al\.\(2025b\)J\. T\. Wang, D\. Song, J\. Zou, P\. Mittal, and R\. JiaCapturing the temporal dependence of training data influence\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=uHLgDEgiS5)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p2.1),[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Wortsmanet al\.\(2024\)M\. Wortsman, P\. J\. Liu, L\. Xiao, K\. E\. Everett, A\. A\. Alemi, B\. Adlam, J\. D\. Co\-Reyes, I\. Gur, A\. Kumar, R\. Novak, J\. Pennington, J\. Sohl\-Dickstein, K\. Xu, J\. Lee, J\. Gilmer, and S\. KornblithSmall\-scale proxies for large\-scale transformer training instabilities\.InThe Twelfth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=d8w0pmvXbZ)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p1.1),[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Xiaet al\.\(2024\)M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. ChenLESS: selecting influential data for targeted instruction tuning\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 54104–54132\.External Links:[Link](https://proceedings.mlr.press/v235/xia24c.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p1.1)\.
- Xie and Li \(2024\)S\. Xie and Z\. LiImplicit bias of AdamW:ℓ∞\\ell\_\{\\infty\}\-norm constrained optimization\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 54488–54510\.External Links:[Link](https://proceedings.mlr.press/v235/xie24e.html)Cited by:[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.
- Zhuet al\.\(2024\)L\. Zhu, C\. Liu, A\. Radhakrishnan, and M\. BelkinCatapults in SGD: spikes in the training loss and their impact on generalization through feature learning\.InProceedings of the 41st International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.235,pp\. 62476–62509\.External Links:[Link](https://proceedings.mlr.press/v235/zhu24h.html)Cited by:[§1](https://arxiv.org/html/2608.19762#S1.p1.1),[§2](https://arxiv.org/html/2608.19762#S2.p2.1)\.

## Appendix ADetailed Derivation of the Minibatch Shock Write\-In

This appendix derives the input operatorBtB\_\{t\}, its nonlinear remainder, and the second\-order gradient\-to\-parameter map used in the one\-step output expansion\.

### A\.1AdamW Update as a State Transition

For a fixed stepss, define

ρ1,s:=1−β1s,ρ2,s:=1−β2s\.\\rho\_\{1,s\}:=1\-\\beta\_\{1\}^\{s\},~~\\rho\_\{2,s\}:=1\-\\beta\_\{2\}^\{s\}\.Following AdamW\([18](https://arxiv.org/html/2608.19762#bib.bib20)\), given the pre\-update state

xs=\[θsms−1vs−1\],x\_\{s\}=\\begin\{bmatrix\}\\theta\_\{s\}\\\\ m\_\{s\-1\}\\\\ v\_\{s\-1\}\\end\{bmatrix\},and an input gradientg∈ℝdg\\in\\mathbb\{R\}^\{d\}, the moment states after the update are

ms​\(g\)\\displaystyle m\_\{s\}\(g\):=β1​ms−1\+\(1−β1\)​g,\\displaystyle:=\\beta\_\{1\}m\_\{s\-1\}\+\(1\-\\beta\_\{1\}\)g,\(15\)vs​\(g\)\\displaystyle v\_\{s\}\(g\):=β2​vs−1\+\(1−β2\)​g⊙2\.\\displaystyle:=\\beta\_\{2\}v\_\{s\-1\}\+\(1\-\\beta\_\{2\}\)g^\{\\odot 2\}\.\(16\)The bias\-corrected moments are

m^s​\(g\):=ms​\(g\)ρ1,s,v^s​\(g\):=vs​\(g\)ρ2,s,\\widehat\{m\}\_\{s\}\(g\):=\\frac\{m\_\{s\}\(g\)\}\{\\rho\_\{1,s\}\},~~\\widehat\{v\}\_\{s\}\(g\):=\\frac\{v\_\{s\}\(g\)\}\{\\rho\_\{2,s\}\},and the adaptive parameter displacement is

qs\(m,v\):=−ηs\(mρ1,s\)⊘\(vρ2,s\+ϵ𝟏\)\.q\_\{s\}\(m,v\):=\-\\eta\_\{s\}\\left\(\\frac\{m\}\{\\rho\_\{1,s\}\}\\right\)\\oslash\\left\(\\sqrt\{\\frac\{v\}\{\\rho\_\{2,s\}\}\}\+\\epsilon\\mathbf\{1\}\\right\)\.Define the composite gradient\-to\-parameter map

ψs​\(g\):=qs​\(ms​\(g\),vs​\(g\)\)\.\\psi\_\{s\}\(g\):=q\_\{s\}\\bigl\(m\_\{s\}\(g\),v\_\{s\}\(g\)\\bigr\)\.At the shock step,

θt\+1​\(g\)=Dt​θt\+ψt​\(g\)\.\\theta\_\{t\+1\}\(g\)=D\_\{t\}\\theta\_\{t\}\+\\psi\_\{t\}\(g\)\.Because the control and shock trajectories share the same pre\-update parameterθt\\theta\_\{t\}, the termDt​θtD\_\{t\}\\theta\_\{t\}cancels from their difference\.

All derivatives below are evaluated along the control trajectory\. Whenever a derivative with respect to the raw second\-moment state is used, we assume

v^s,ic\>0\\widehat\{v\}\_\{s,i\}^\{\\mathrm\{c\}\}\>0for the corresponding coordinates\.

### A\.2Moment\-State Sensitivities

Write

ds:=v^sc\+ϵ​𝟏\.d\_\{s\}:=\\sqrt\{\\widehat\{v\}\_\{s\}^\{\\mathrm\{c\}\}\}\+\\epsilon\\mathbf\{1\}\.Since the adaptive map is coordinate\-separable, its derivatives with respect tommandvvare diagonal\.

For coordinateii,

qs,i​\(mi,vi\)=−ηs​mi/ρ1,svi/ρ2,s\+ϵ\.q\_\{s,i\}\(m\_\{i\},v\_\{i\}\)=\-\\eta\_\{s\}\\frac\{m\_\{i\}/\\rho\_\{1,s\}\}\{\\sqrt\{v\_\{i\}/\\rho\_\{2,s\}\}\+\\epsilon\}\.Differentiating with respect tomim\_\{i\}gives

∂qs,i∂mi=−ηsρ1,s​ds,i\.\\frac\{\\partial q\_\{s,i\}\}\{\\partial m\_\{i\}\}=\-\\frac\{\\eta\_\{s\}\}\{\\rho\_\{1,s\}d\_\{s,i\}\}\.Hence

Ms=∂qs∂ms=−ηsρ1,s​Diag⁡\(ds−1\)\.M\_\{s\}=\\frac\{\\partial q\_\{s\}\}\{\\partial m\_\{s\}\}=\-\\frac\{\\eta\_\{s\}\}\{\\rho\_\{1,s\}\}\\operatorname\{Diag\}\(d\_\{s\}^\{\-1\}\)\.\(17\)
For the derivative with respect toviv\_\{i\}, define

rs,i:=v^s,ic=vs,icρ2,s\.r\_\{s,i\}:=\\widehat\{v\}\_\{s,i\}^\{\\mathrm\{c\}\}=\\frac\{v\_\{s,i\}^\{\\mathrm\{c\}\}\}\{\\rho\_\{2,s\}\}\.Using

∂vi/ρ2,s∂vi=12​ρ2,s​rs,i,\\frac\{\\partial\\sqrt\{v\_\{i\}/\\rho\_\{2,s\}\}\}\{\\partial v\_\{i\}\}=\\frac\{1\}\{2\\rho\_\{2,s\}\\sqrt\{r\_\{s,i\}\}\},we obtain

∂qs,i∂vi\\displaystyle\\frac\{\\partial q\_\{s,i\}\}\{\\partial v\_\{i\}\}=ηs​ms,ic/ρ1,s2​ρ2,s​rs,i​\(rs,i\+ϵ\)2\\displaystyle=\\eta\_\{s\}\\frac\{m\_\{s,i\}^\{\\mathrm\{c\}\}/\\rho\_\{1,s\}\}\{2\\rho\_\{2,s\}\\sqrt\{r\_\{s,i\}\}\\left\(\\sqrt\{r\_\{s,i\}\}\+\\epsilon\\right\)^\{2\}\}=ηs​m^s,ic2​ρ2,s​v^s,ic​ds,i2\.\\displaystyle=\\eta\_\{s\}\\frac\{\\widehat\{m\}\_\{s,i\}^\{\\mathrm\{c\}\}\}\{2\\rho\_\{2,s\}\\sqrt\{\\widehat\{v\}\_\{s,i\}^\{\\mathrm\{c\}\}\}d\_\{s,i\}^\{2\}\}\.Therefore

Vs=∂qs∂vs=ηs​Diag⁡\(m^sc2​ρ2,s​v^sc⊙ds⊙2\)\.V\_\{s\}=\\frac\{\\partial q\_\{s\}\}\{\\partial v\_\{s\}\}=\\eta\_\{s\}\\operatorname\{Diag\}\\left\(\\frac\{\\widehat\{m\}\_\{s\}^\{\\mathrm\{c\}\}\}\{2\\rho\_\{2,s\}\\sqrt\{\\widehat\{v\}\_\{s\}^\{\\mathrm\{c\}\}\}\\odot d\_\{s\}^\{\\odot 2\}\}\\right\)\.\(18\)
Equation \([17](https://arxiv.org/html/2608.19762#A1.E17)\) shows thatMsM\_\{s\}is negative diagonal\. The sign of theiith diagonal entry ofVsV\_\{s\}is the sign ofm^s,ic\\widehat\{m\}\_\{s,i\}^\{\\mathrm\{c\}\}\.

### A\.3Gradient\-to\-Parameter Jacobian

The derivatives of the moment maps in Equations \([15](https://arxiv.org/html/2608.19762#A1.E15)\)–\([16](https://arxiv.org/html/2608.19762#A1.E16)\) are

D​ms​\(g\)​\[u\]\\displaystyle Dm\_\{s\}\(g\)\[u\]=\(1−β1\)​u,\\displaystyle=\(1\-\\beta\_\{1\}\)u,D​vs​\(g\)​\[u\]\\displaystyle Dv\_\{s\}\(g\)\[u\]=2​\(1−β2\)​Diag⁡\(g\)​u\.\\displaystyle=2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\)u\.Applying the chain rule toψs​\(g\)=qs​\(ms​\(g\),vs​\(g\)\)\\psi\_\{s\}\(g\)=q\_\{s\}\(m\_\{s\}\(g\),v\_\{s\}\(g\)\)gives

D​ψs​\(gsc\)​\[u\]\\displaystyle D\\psi\_\{s\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\[u\]=Ms​D​ms​\(gsc\)​\[u\]\+Vs​D​vs​\(gsc\)​\[u\]\\displaystyle=M\_\{s\}Dm\_\{s\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\[u\]\+V\_\{s\}Dv\_\{s\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\[u\]=\[\(1−β1\)​Ms\+2​\(1−β2\)​Vs​Diag⁡\(gsc\)\]​u\.\\displaystyle=\\Bigl\[\(1\-\\beta\_\{1\}\)M\_\{s\}\+2\(1\-\\beta\_\{2\}\)V\_\{s\}\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\\Bigr\]u\.Thus

Γs:=D​ψs​\(gsc\)=\(1−β1\)​Ms\+2​\(1−β2\)​Vs​Diag⁡\(gsc\)\.\\Gamma\_\{s\}:=D\\psi\_\{s\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)=\(1\-\\beta\_\{1\}\)M\_\{s\}\+2\(1\-\\beta\_\{2\}\)V\_\{s\}\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)\.\(19\)The two terms correspond to the differential paths

g→m→θ,g→v→θ\.g\\rightarrow m\\rightarrow\\theta,~~g\\rightarrow v\\rightarrow\\theta\.

### A\.4Exact Shock\-Step State Difference

At the shock step,

gts=gtc\+ξt\.g\_\{t\}^\{\\mathrm\{s\}\}=g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}\.Subtracting the first\-moment updates gives

δ​mt\\displaystyle\\delta m\_\{t\}=mts−mtc\\displaystyle=m\_\{t\}^\{\\mathrm\{s\}\}\-m\_\{t\}^\{\\mathrm\{c\}\}\(20\)=\(1−β1\)​\(gts−gtc\)\\displaystyle=\(1\-\\beta\_\{1\}\)\\left\(g\_\{t\}^\{\\mathrm\{s\}\}\-g\_\{t\}^\{\\mathrm\{c\}\}\\right\)=\(1−β1\)​ξt\.\\displaystyle=\(1\-\\beta\_\{1\}\)\\xi\_\{t\}\.
For the second moment,

δ​vt\\displaystyle\\delta v\_\{t\}=\(1−β2\)​\[\(gtc\+ξt\)⊙2−\(gtc\)⊙2\]\\displaystyle=\(1\-\\beta\_\{2\}\)\\left\[\(g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}\)^\{\\odot 2\}\-\(g\_\{t\}^\{\\mathrm\{c\}\}\)^\{\\odot 2\}\\right\]\(21\)=2​\(1−β2\)​gtc⊙ξt\+\(1−β2\)​ξt⊙2\\displaystyle=2\(1\-\\beta\_\{2\}\)g\_\{t\}^\{\\mathrm\{c\}\}\\odot\\xi\_\{t\}\+\(1\-\\beta\_\{2\}\)\\xi\_\{t\}^\{\\odot 2\}=2​\(1−β2\)​Diag⁡\(gtc\)​ξt\+\(1−β2\)​ξt⊙2\.\\displaystyle=2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\\xi\_\{t\}\+\(1\-\\beta\_\{2\}\)\\xi\_\{t\}^\{\\odot 2\}\.
The parameter difference is

δ​θt\+1\\displaystyle\\delta\\theta\_\{t\+1\}=ψt​\(gtc\+ξt\)−ψt​\(gtc\)\\displaystyle=\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\_\{t\}\)\-\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\(22\)=Γt​ξt\+rθ,t​\(ξt\),\\displaystyle=\\Gamma\_\{t\}\\xi\_\{t\}\+r\_\{\\theta,t\}\(\\xi\_\{t\}\),where

rθ,t​\(ξ\):=ψt​\(gtc\+ξ\)−ψt​\(gtc\)−Γt​ξ\.r\_\{\\theta,t\}\(\\xi\):=\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\+\\xi\)\-\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\-\\Gamma\_\{t\}\\xi\.
Stacking Equations \([20](https://arxiv.org/html/2608.19762#A1.E20)\), \([21](https://arxiv.org/html/2608.19762#A1.E21)\), and \([22](https://arxiv.org/html/2608.19762#A1.E22)\) gives

δ​xt\+1=Bt​ξt\+rB,t​\(ξt\),\\delta x\_\{t\+1\}=B\_\{t\}\\xi\_\{t\}\+r\_\{B,t\}\(\\xi\_\{t\}\),where

Bt:=\[Γt\(1−β1\)​I2​\(1−β2\)​Diag⁡\(gtc\)\]B\_\{t\}:=\\begin\{bmatrix\}\\Gamma\_\{t\}\\\\\[2\.84526pt\] \(1\-\\beta\_\{1\}\)I\\\\\[2\.84526pt\] 2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{t\}^\{\\mathrm\{c\}\}\)\\end\{bmatrix\}and

rB,t​\(ξ\):=\[rθ,t​\(ξ\)0\(1−β2\)​ξ⊙2\]\.r\_\{B,t\}\(\\xi\):=\\begin\{bmatrix\}r\_\{\\theta,t\}\(\\xi\)\\\\\[2\.84526pt\] 0\\\\\[2\.84526pt\] \(1\-\\beta\_\{2\}\)\\xi^\{\\odot 2\}\\end\{bmatrix\}\.
IfD​ψtD\\psi\_\{t\}isκθ,t\\kappa\_\{\\theta,t\}\-Lipschitz in a neighborhood containing the segment

\{gtc\+τ​ξt:0≤τ≤1\},\\left\\\{g\_\{t\}^\{\\mathrm\{c\}\}\+\\tau\\xi\_\{t\}:0\\leq\\tau\\leq 1\\right\\\},Taylor’s theorem gives

‖rθ,t​\(ξt\)‖≤κθ,t2​‖ξt‖2\.\\\|r\_\{\\theta,t\}\(\\xi\_\{t\}\)\\\|\\leq\\frac\{\\kappa\_\{\\theta,t\}\}\{2\}\\\|\\xi\_\{t\}\\\|^\{2\}\.Moreover,

‖ξ⊙2‖2=\(∑iξi4\)1/2≤∑iξi2=‖ξ‖22\.\\\|\\xi^\{\\odot 2\}\\\|\_\{2\}=\\left\(\\sum\_\{i\}\\xi\_\{i\}^\{4\}\\right\)^\{1/2\}\\leq\\sum\_\{i\}\\xi\_\{i\}^\{2\}=\\\|\\xi\\\|\_\{2\}^\{2\}\.Consequently,

‖rB,t​\(ξt\)‖2≤\(κθ,t2\+1−β2\)​‖ξt‖22\.\\\|r\_\{B,t\}\(\\xi\_\{t\}\)\\\|\_\{2\}\\leq\\left\(\\frac\{\\kappa\_\{\\theta,t\}\}\{2\}\+1\-\\beta\_\{2\}\\right\)\\\|\\xi\_\{t\}\\\|\_\{2\}^\{2\}\.\(23\)For a scaled perturbationα​ξt\\alpha\\xi\_\{t\}, Equation \([23](https://arxiv.org/html/2608.19762#A1.E23)\) implies

‖rB,t​\(α​ξt\)‖=O⁡\(α2\)\.\\\|r\_\{B,t\}\(\\alpha\\xi\_\{t\}\)\\\|=O\(\\alpha^\{2\}\)\.

### A\.5Second Derivative of the Gradient\-to\-Parameter Map

For completeness, we derive

𝒬t:=D2​ψt​\(gtc\),\\mathcal\{Q\}\_\{t\}:=D^\{2\}\\psi\_\{t\}\(g\_\{t\}^\{\\mathrm\{c\}\}\),which is used in the one\-step second\-order output expansion\.

Let

a1:=1−β1,a2:=1−β2\.a\_\{1\}:=1\-\\beta\_\{1\},~~a\_\{2\}:=1\-\\beta\_\{2\}\.The moment maps satisfy

D​mt​\(g\)​\[u\]\\displaystyle Dm\_\{t\}\(g\)\[u\]=a1​u,\\displaystyle=a\_\{1\}u,D2​mt​\(g\)​\[u,w\]\\displaystyle D^\{2\}m\_\{t\}\(g\)\[u,w\]=0,\\displaystyle=0,D​vt​\(g\)​\[u\]\\displaystyle Dv\_\{t\}\(g\)\[u\]=2​a2​Diag⁡\(g\)​u,\\displaystyle=2a\_\{2\}\\operatorname\{Diag\}\(g\)u,D2​vt​\(g\)​\[u,w\]\\displaystyle D^\{2\}v\_\{t\}\(g\)\[u,w\]=2​a2​\(u⊙w\)\.\\displaystyle=2a\_\{2\}\(u\\odot w\)\.
Becauseqtq\_\{t\}is coordinate\-separable,𝒬t\\mathcal\{Q\}\_\{t\}is also coordinate\-separable\. Define

st,i:=v^t,ic,dt,i:=st,i\+ϵ\.s\_\{t,i\}:=\\sqrt\{\\widehat\{v\}\_\{t,i\}^\{\\mathrm\{c\}\}\},~~d\_\{t,i\}:=s\_\{t,i\}\+\\epsilon\.The nonzero second derivatives ofqt,iq\_\{t,i\}are

qm​v,t,i\\displaystyle q\_\{mv,t,i\}:=∂2qt,i∂mi​∂vi=ηt2​ρ1,t​ρ2,t​st,i​dt,i2,\\displaystyle:=\\frac\{\\partial^\{2\}q\_\{t,i\}\}\{\\partial m\_\{i\}\\partial v\_\{i\}\}=\\frac\{\\eta\_\{t\}\}\{2\\rho\_\{1,t\}\\rho\_\{2,t\}s\_\{t,i\}d\_\{t,i\}^\{2\}\},qv​v,t,i\\displaystyle q\_\{vv,t,i\}:=∂2qt,i∂vi2=−ηt​m^t,ic​\(3​st,i\+ϵ\)4​ρ2,t2​st,i3​dt,i3\.\\displaystyle:=\\frac\{\\partial^\{2\}q\_\{t,i\}\}\{\\partial v\_\{i\}^\{2\}\}=\-\\frac\{\\eta\_\{t\}\\widehat\{m\}\_\{t,i\}^\{\\mathrm\{c\}\}\(3s\_\{t,i\}\+\\epsilon\)\}\{4\\rho\_\{2,t\}^\{2\}s\_\{t,i\}^\{3\}d\_\{t,i\}^\{3\}\}\.Also,

qv,t,i=\[Vt\]i​i\.q\_\{v,t,i\}=\[V\_\{t\}\]\_\{ii\}\.
The second\-order chain rule gives

𝒬t​\[u,w\]=\\displaystyle\\mathcal\{Q\}\_\{t\}\[u,w\]=\{\}Dm​v2​qt​\[D​mt​\[u\],D​vt​\[w\]\]\\displaystyle D\_\{mv\}^\{2\}q\_\{t\}\\left\[Dm\_\{t\}\[u\],Dv\_\{t\}\[w\]\\right\]\+Dv​m2​qt​\[D​vt​\[u\],D​mt​\[w\]\]\\displaystyle\+D\_\{vm\}^\{2\}q\_\{t\}\\left\[Dv\_\{t\}\[u\],Dm\_\{t\}\[w\]\\right\]\+Dv​v2​qt​\[D​vt​\[u\],D​vt​\[w\]\]\\displaystyle\+D\_\{vv\}^\{2\}q\_\{t\}\\left\[Dv\_\{t\}\[u\],Dv\_\{t\}\[w\]\\right\]\+Vt​D2​vt​\[u,w\]\.\\displaystyle\+V\_\{t\}D^\{2\}v\_\{t\}\[u,w\]\.Coordinatewise,

\[𝒬t​\[u,w\]\]i=χt,i​ui​wi,\[\\mathcal\{Q\}\_\{t\}\[u,w\]\]\_\{i\}=\\chi\_\{t,i\}u\_\{i\}w\_\{i\},where

χt,i:=\\displaystyle\\chi\_\{t,i\}:=\{\}4​a1​a2​gt,ic​qm​v,t,i\\displaystyle 4a\_\{1\}a\_\{2\}g\_\{t,i\}^\{\\mathrm\{c\}\}q\_\{mv,t,i\}\+4​a22​\(gt,ic\)2​qv​v,t,i\+2​a2​\[Vt\]i​i\.\\displaystyle\+4a\_\{2\}^\{2\}\(g\_\{t,i\}^\{\\mathrm\{c\}\}\)^\{2\}q\_\{vv,t,i\}\+2a\_\{2\}\[V\_\{t\}\]\_\{ii\}\.Therefore, for a scaled perturbationα​ξt\\alpha\\xi\_\{t\},

δ​θt\+1​\(α\)=α​Γt​ξt\+α22​𝒬t​\[ξt,ξt\]\+o⁡\(α2\)\.\\delta\\theta\_\{t\+1\}\(\\alpha\)=\\alpha\\Gamma\_\{t\}\\xi\_\{t\}\+\\frac\{\\alpha^\{2\}\}\{2\}\\mathcal\{Q\}\_\{t\}\[\\xi\_\{t\},\\xi\_\{t\}\]\+o\(\\alpha^\{2\}\)\.\(24\)

## Appendix BProof of the Finite\-Horizon Directional Response

This appendix derives the joint AdamW transition Jacobian, proves Theorem[3\.1](https://arxiv.org/html/2608.19762#S3.Thmtheorem1), and records several properties of the resulting input–output operator\.

### B\.1Control\-Region State Transition

Letfsf\_\{s\}denote the exact AdamW transition under future minibatchℬs\\mathcal\{B\}\_\{s\}:

xs\+1=fs​\(xs\)\.x\_\{s\+1\}=f\_\{s\}\(x\_\{s\}\)\.For a smooth network, definef~s:=fs\\widetilde\{f\}\_\{s\}:=f\_\{s\}\. For a piecewise\-smooth network, letf~s\\widetilde\{f\}\_\{s\}denote the smooth extension associated with the activation pattern of the control trajectory atxscx\_\{s\}^\{\\mathrm\{c\}\}\.

The control\-region tangent matrix is

As:=D​f~s​\(xsc\)\.A\_\{s\}:=D\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\.All quantities in the following block derivation are evaluated at the control state and its corresponding future minibatch\.

Letg~s​\(θ\)\\widetilde\{g\}\_\{s\}\(\\theta\)denote the gradient map induced by the same control\-region smooth extension, and define

Hstr:=D​g~s​\(θsc\)\.H\_\{s\}^\{\\mathrm\{tr\}\}:=D\\widetilde\{g\}\_\{s\}\(\\theta\_\{s\}^\{\\mathrm\{c\}\}\)\.When the loss is twice differentiable in the control region,

Hstr=∇θ2​ℓ​\(θsc,ℬs\)\.H\_\{s\}^\{\\mathrm\{tr\}\}=\\nabla\_\{\\theta\}^\{2\}\\ell\(\\theta\_\{s\}^\{\\mathrm\{c\}\};\\mathcal\{B\}\_\{s\}\)\.

### B\.2Blockwise Derivation of the Joint Jacobian

For a generic state

x=\[θ,m−,v−\]⊤,x=\[\\theta,m\_\{\-\},v\_\{\-\}\]^\{\\top\},the control\-region transition has components

m\+\\displaystyle m^\{\+\}=β1​m−\+\(1−β1\)​g~s​\(θ\),\\displaystyle=\\beta\_\{1\}m\_\{\-\}\+\(1\-\\beta\_\{1\}\)\\widetilde\{g\}\_\{s\}\(\\theta\),v\+\\displaystyle v^\{\+\}=β2​v−\+\(1−β2\)​g~s​\(θ\)⊙2,\\displaystyle=\\beta\_\{2\}v\_\{\-\}\+\(1\-\\beta\_\{2\}\)\\widetilde\{g\}\_\{s\}\(\\theta\)^\{\\odot 2\},θ\+\\displaystyle\\theta^\{\+\}=Ds​θ\+qs​\(m\+,v\+\)\.\\displaystyle=D\_\{s\}\\theta\+q\_\{s\}\(m^\{\+\},v^\{\+\}\)\.
The derivatives of the first\-moment update are

∂m\+∂θ\\displaystyle\\frac\{\\partial m^\{\+\}\}\{\\partial\\theta\}=\(1−β1\)​Hstr,\\displaystyle=\(1\-\\beta\_\{1\}\)H\_\{s\}^\{\\mathrm\{tr\}\},∂m\+∂m−\\displaystyle\\frac\{\\partial m^\{\+\}\}\{\\partial m\_\{\-\}\}=β1​I,\\displaystyle=\\beta\_\{1\}I,∂m\+∂v−\\displaystyle\\frac\{\\partial m^\{\+\}\}\{\\partial v\_\{\-\}\}=0\.\\displaystyle=0\.
For the second moment,

D⁡\[g~s​\(θ\)⊙2\]=2​Diag⁡\(g~s​\(θ\)\)​D​g~s​\(θ\),D\\left\[\\widetilde\{g\}\_\{s\}\(\\theta\)^\{\\odot 2\}\\right\]=2\\operatorname\{Diag\}\\left\(\\widetilde\{g\}\_\{s\}\(\\theta\)\\right\)D\\widetilde\{g\}\_\{s\}\(\\theta\),so

∂v\+∂θ\\displaystyle\\frac\{\\partial v^\{\+\}\}\{\\partial\\theta\}=2​\(1−β2\)​Diag⁡\(gsc\)​Hstr:=Ss,\\displaystyle=2\(1\-\\beta\_\{2\}\)\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)H\_\{s\}^\{\\mathrm\{tr\}\}:=S\_\{s\},∂v\+∂m−\\displaystyle\\frac\{\\partial v^\{\+\}\}\{\\partial m\_\{\-\}\}=0,\\displaystyle=0,∂v\+∂v−\\displaystyle\\frac\{\\partial v^\{\+\}\}\{\\partial v\_\{\-\}\}=β2​I\.\\displaystyle=\\beta\_\{2\}I\.
For the parameter update,

∂θ\+∂θ=\\displaystyle\\frac\{\\partial\\theta^\{\+\}\}\{\\partial\\theta\}=\{\}Ds\+Ms​∂m\+∂θ\+Vs​∂v\+∂θ\\displaystyle D\_\{s\}\+M\_\{s\}\\frac\{\\partial m^\{\+\}\}\{\\partial\\theta\}\+V\_\{s\}\\frac\{\\partial v^\{\+\}\}\{\\partial\\theta\}=\\displaystyle=\{\}Ds\+\(1−β1\)​Ms​Hstr\\displaystyle D\_\{s\}\+\(1\-\\beta\_\{1\}\)M\_\{s\}H\_\{s\}^\{\\mathrm\{tr\}\}\+2​\(1−β2\)​Vs​Diag⁡\(gsc\)​Hstr\\displaystyle\+2\(1\-\\beta\_\{2\}\)V\_\{s\}\\operatorname\{Diag\}\(g\_\{s\}^\{\\mathrm\{c\}\}\)H\_\{s\}^\{\\mathrm\{tr\}\}=\\displaystyle=\{\}Ds\+Γs​Hstr\.\\displaystyle D\_\{s\}\+\\Gamma\_\{s\}H\_\{s\}^\{\\mathrm\{tr\}\}\.Similarly,

∂θ\+∂m−\\displaystyle\\frac\{\\partial\\theta^\{\+\}\}\{\\partial m\_\{\-\}\}=β1​Ms,\\displaystyle=\\beta\_\{1\}M\_\{s\},∂θ\+∂v−\\displaystyle\\frac\{\\partial\\theta^\{\+\}\}\{\\partial v\_\{\-\}\}=β2​Vs\.\\displaystyle=\\beta\_\{2\}V\_\{s\}\.
Combining the nine blocks gives

As=\[Ds\+Γs​Hstrβ1​Msβ2​Vs\(1−β1\)​Hstrβ1​I0Ss0β2​I\]\.A\_\{s\}=\\begin\{bmatrix\}D\_\{s\}\+\\Gamma\_\{s\}H\_\{s\}^\{\\mathrm\{tr\}\}&\\beta\_\{1\}M\_\{s\}&\\beta\_\{2\}V\_\{s\}\\\\\[2\.84526pt\] \(1\-\\beta\_\{1\}\)H\_\{s\}^\{\\mathrm\{tr\}\}&\\beta\_\{1\}I&0\\\\\[2\.84526pt\] S\_\{s\}&0&\\beta\_\{2\}I\\end\{bmatrix\}\.\(25\)
For later use, recall the general propagator convention

Φa,b:=\{I,a=b,Aa−1Aa−2⋯Ab,a\>b\.\\Phi\_\{a,b\}:=\\begin\{cases\}I,&a=b,\\\\\[2\.84526pt\] A\_\{a\-1\}A\_\{a\-2\}\\cdots A\_\{b\},&a\>b\.\\end\{cases\}

### B\.3Proof of Theorem[3\.1](https://arxiv.org/html/2608.19762#S3.Thmtheorem1)

Consider

gts​\(α\)=gtc\+α​ξt\.g\_\{t\}^\{\\mathrm\{s\}\}\(\\alpha\)=g\_\{t\}^\{\\mathrm\{c\}\}\+\\alpha\\xi\_\{t\}\.Atα=0\\alpha=0, the shock and control trajectories coincide\.

By assumption, the control states lie in the interior of the smooth regions used to define the transitionsf~t\+1,…,f~t\+h−1\\widetilde\{f\}\_\{t\+1\},\\ldots,\\widetilde\{f\}\_\{t\+h\-1\}\. For a fixed horizon, continuity of the trajectory implies that there existsα0\>0\\alpha\_\{0\}\>0such that, for sufficiently small\|α\|<α0\|\\alpha\|<\\alpha\_\{0\}, the perturbed trajectory follows the same sequence of local smooth extensions\. The differentiability assumptions on the AdamW transition, including the required positivity of the second\-moment coordinates, ensure that the Jacobians used below are well defined\.

Define

x˙s:=dd​α​δ​xs​\(α\)\|α=0\.\\dot\{x\}\_\{s\}:=\\left\.\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}\\delta x\_\{s\}\(\\alpha\)\\right\|\_\{\\alpha=0\}\.By Appendix[A](https://arxiv.org/html/2608.19762#A1),

x˙t\+1=Bt​ξt\.\\dot\{x\}\_\{t\+1\}=B\_\{t\}\\xi\_\{t\}\.
For every future steps≥t\+1s\\geq t\+1,

xs\+1s​\(α\)=f~s​\(xss​\(α\)\),x\_\{s\+1\}^\{\\mathrm\{s\}\}\(\\alpha\)=\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{s\}\}\(\\alpha\)\\right\),while

xs\+1c=f~s​\(xsc\)\.x\_\{s\+1\}^\{\\mathrm\{c\}\}=\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\\right\)\.Differentiating atα=0\\alpha=0yields

x˙s\+1=D​f~s​\(xsc\)​x˙s=As​x˙s\.\\dot\{x\}\_\{s\+1\}=D\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\\dot\{x\}\_\{s\}=A\_\{s\}\\dot\{x\}\_\{s\}\.\(26\)Repeated application gives

x˙t\+h\\displaystyle\\dot\{x\}\_\{t\+h\}=At\+h−1At\+h−2⋯At\+1Btξt\\displaystyle=A\_\{t\+h\-1\}A\_\{t\+h\-2\}\\cdots A\_\{t\+1\}B\_\{t\}\\xi\_\{t\}=Φt\+h,t\+1​Bt​ξt\.\\displaystyle=\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\.Forh=1h=1,Φt\+1,t\+1=I\\Phi\_\{t\+1,t\+1\}=I\.

The probe function at horizont\+ht\+his shared by the paired trajectories, so

δ​Ft\+h​\(α\)=Ft\+h​\(θt\+hs​\(α\)\)−Ft\+h​\(θt\+hc\)\.\\delta F\_\{t\+h\}\(\\alpha\)=F\_\{t\+h\}\\left\(\\theta\_\{t\+h\}^\{\\mathrm\{s\}\}\(\\alpha\)\\right\)\-F\_\{t\+h\}\\left\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\\right\)\.Differentiating atα=0\\alpha=0gives

dd​α​δ​Ft\+h​\(α\)\|α=0\\displaystyle\\left\.\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\alpha\}\\delta F\_\{t\+h\}\(\\alpha\)\\right\|\_\{\\alpha=0\}=∇Ft\+h\(θt\+hc\)⊤θ˙t\+h\\displaystyle=\\nabla F\_\{t\+h\}\\left\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\\right\)^\{\\top\}\\dot\{\\theta\}\_\{t\+h\}=ct\+h⊤​x˙t\+h\\displaystyle=c\_\{t\+h\}^\{\\top\}\\dot\{x\}\_\{t\+h\}=ct\+h⊤​Φt\+h,t\+1​Bt​ξt\.\\displaystyle=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\.This proves both directional identities in Theorem[3\.1](https://arxiv.org/html/2608.19762#S3.Thmtheorem1)\. The first\-order expansion

δ​Ft\+h​\(α\)=α​ct\+h⊤​Φt\+h,t\+1​Bt​ξt\+o⁡\(α\)\\delta F\_\{t\+h\}\(\\alpha\)=\\alpha c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\+o\(\\alpha\)follows directly from differentiability atα=0\\alpha=0\.

### B\.4Complete One\-Step Second\-Order Output Expansion

We now recover the one\-step second\-order geometry that is omitted from the main text\.

From Equation \([24](https://arxiv.org/html/2608.19762#A1.E24)\),

δ​θt\+1​\(α\)=α​pt\+α22​ut\+o⁡\(α2\),\\delta\\theta\_\{t\+1\}\(\\alpha\)=\\alpha p\_\{t\}\+\\frac\{\\alpha^\{2\}\}\{2\}u\_\{t\}\+o\(\\alpha^\{2\}\),where

pt:=Γt​ξt,ut:=𝒬t​\[ξt,ξt\]\.p\_\{t\}:=\\Gamma\_\{t\}\\xi\_\{t\},~~u\_\{t\}:=\\mathcal\{Q\}\_\{t\}\[\\xi\_\{t\},\\xi\_\{t\}\]\.
Assume thatFt\+1F\_\{t\+1\}is twice differentiable in the relevant local region and define

Ht\+1F:=∇2Ft\+1​\(θt\+1c\)\.H\_\{t\+1\}^\{F\}:=\\nabla^\{2\}F\_\{t\+1\}\\left\(\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}\\right\)\.Taylor expansion aroundθt\+1c\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}gives

δ​Ft\+1​\(α\)=\\displaystyle\\delta F\_\{t\+1\}\(\\alpha\)=\{\}∇Ft\+1\(θt\+1c\)⊤δθt\+1\(α\)\\displaystyle\\nabla F\_\{t\+1\}\\left\(\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}\\right\)^\{\\top\}\\delta\\theta\_\{t\+1\}\(\\alpha\)\+12​δ​θt\+1​\(α\)⊤​Ht\+1F​δ​θt\+1​\(α\)\\displaystyle\+\\frac\{1\}\{2\}\\delta\\theta\_\{t\+1\}\(\\alpha\)^\{\\top\}H\_\{t\+1\}^\{F\}\\delta\\theta\_\{t\+1\}\(\\alpha\)\+o⁡\(‖δ​θt\+1​\(α\)‖2\)\.\\displaystyle\+o\\left\(\\\|\\delta\\theta\_\{t\+1\}\(\\alpha\)\\\|^\{2\}\\right\)\.The linear output contribution is

∇Ft\+1⊤δθt\+1\(α\)=\\displaystyle\\nabla F\_\{t\+1\}^\{\\top\}\\delta\\theta\_\{t\+1\}\(\\alpha\)=\{\}α∇Ft\+1⊤pt\\displaystyle\\alpha\\nabla F\_\{t\+1\}^\{\\top\}p\_\{t\}\+α22∇Ft\+1⊤ut\+o\(α2\),\\displaystyle\+\\frac\{\\alpha^\{2\}\}\{2\}\\nabla F\_\{t\+1\}^\{\\top\}u\_\{t\}\+o\(\\alpha^\{2\}\),where the gradients are evaluated atθt\+1c\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}\. The quadratic output contribution satisfies

12​δ​θt\+1​\(α\)⊤​Ht\+1F​δ​θt\+1​\(α\)=α22​pt⊤​Ht\+1F​pt\+o⁡\(α2\)\.\\frac\{1\}\{2\}\\delta\\theta\_\{t\+1\}\(\\alpha\)^\{\\top\}H\_\{t\+1\}^\{F\}\\delta\\theta\_\{t\+1\}\(\\alpha\)=\\frac\{\\alpha^\{2\}\}\{2\}p\_\{t\}^\{\\top\}H\_\{t\+1\}^\{F\}p\_\{t\}\+o\(\\alpha^\{2\}\)\.Therefore

δ​Ft\+1​\(α\)=\\displaystyle\\delta F\_\{t\+1\}\(\\alpha\)=\{\}α∇Ft\+1\(θt\+1c\)⊤Γtξt\\displaystyle\\alpha\\nabla F\_\{t\+1\}\\left\(\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}\\right\)^\{\\top\}\\Gamma\_\{t\}\\xi\_\{t\}\(27\)\+α22\[\(Γtξt\)⊤Ht\+1F\(Γtξt\)\\displaystyle\+\\frac\{\\alpha^\{2\}\}\{2\}\\Bigl\[\(\\Gamma\_\{t\}\\xi\_\{t\}\)^\{\\top\}H\_\{t\+1\}^\{F\}\(\\Gamma\_\{t\}\\xi\_\{t\}\)\+∇Ft\+1\(θt\+1c\)⊤𝒬t\[ξt,ξt\]\]\\displaystyle\+\\nabla F\_\{t\+1\}\\left\(\\theta\_\{t\+1\}^\{\\mathrm\{c\}\}\\right\)^\{\\top\}\\mathcal\{Q\}\_\{t\}\[\\xi\_\{t\},\\xi\_\{t\}\]\\Bigr\]\+o⁡\(α2\)\.\\displaystyle\+o\(\\alpha^\{2\}\)\.
The first second\-order term in Equation \([27](https://arxiv.org/html/2608.19762#A2.E27)\) is the probe\-curvature contribution induced by the first\-order parameter displacement\. Equivalently, it is generated by the effective curvature operator

Γt⊤​Ht\+1F​Γt\.\\Gamma\_\{t\}^\{\\top\}H\_\{t\+1\}^\{F\}\\Gamma\_\{t\}\.The second term is the output effect of the nonlinear AdamW write\-in itself\. The training\-batch HessianHstrH\_\{s\}^\{\\mathrm\{tr\}\}governs the subsequent state propagation, whereasHsFH\_\{s\}^\{F\}describes curvature of the probe output\. Neither curvature term by itself determines the sign of the response\.

### B\.5Coordinate Invariance of the Input–Output Operator

Let

x~s:=Ts​xs\\widetilde\{x\}\_\{s\}:=T\_\{s\}x\_\{s\}for invertible matricesTsT\_\{s\}\. The transformed state transition is

A~s:=Ts\+1​As​Ts−1\.\\widetilde\{A\}\_\{s\}:=T\_\{s\+1\}A\_\{s\}T\_\{s\}^\{\-1\}\.Hence

Φ~t\+h,t\+1\\displaystyle\\widetilde\{\\Phi\}\_\{t\+h,t\+1\}=A~t\+h−1⋯A~t\+1\\displaystyle=\\widetilde\{A\}\_\{t\+h\-1\}\\cdots\\widetilde\{A\}\_\{t\+1\}=Tt\+hAt\+h−1Tt\+h−1−1⋯Tt\+2At\+1Tt\+1−1\\displaystyle=T\_\{t\+h\}A\_\{t\+h\-1\}T\_\{t\+h\-1\}^\{\-1\}\\cdots T\_\{t\+2\}A\_\{t\+1\}T\_\{t\+1\}^\{\-1\}=Tt\+h​Φt\+h,t\+1​Tt\+1−1\.\\displaystyle=T\_\{t\+h\}\\Phi\_\{t\+h,t\+1\}T\_\{t\+1\}^\{\-1\}\.The input and output maps transform as

B~t=Tt\+1​Bt,c~t\+h⊤=ct\+h⊤​Tt\+h−1\.\\widetilde\{B\}\_\{t\}=T\_\{t\+1\}B\_\{t\},~~\\widetilde\{c\}\_\{t\+h\}^\{\\top\}=c\_\{t\+h\}^\{\\top\}T\_\{t\+h\}^\{\-1\}\.Therefore

c~t\+h⊤​Φ~t\+h,t\+1​B~t\\displaystyle\\widetilde\{c\}\_\{t\+h\}^\{\\top\}\\widetilde\{\\Phi\}\_\{t\+h,t\+1\}\\widetilde\{B\}\_\{t\}=ct\+h⊤​Tt\+h−1​Tt\+h​Φt\+h,t\+1​Tt\+1−1​Tt\+1​Bt\\displaystyle=c\_\{t\+h\}^\{\\top\}T\_\{t\+h\}^\{\-1\}T\_\{t\+h\}\\Phi\_\{t\+h,t\+1\}T\_\{t\+1\}^\{\-1\}T\_\{t\+1\}B\_\{t\}=ct\+h⊤​Φt\+h,t\+1​Bt\.\\displaystyle=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\.Thus the finite\-horizon input–output operator is invariant under invertible state reparameterization\. Internal state\-gain quantities can depend on the relative scaling chosen for the parameter and moment blocks, whereas the signed scalar input–output response does not\.

### B\.6Additional Finite\-Horizon Response Summaries

The main text uses the maximum response magnitude, its timing, its sign, and the largest positive excursion as the primary finite\-horizon summaries\. We record additional cumulative and direction\-specific quantities here\.

The accumulated absolute response is

AREt,H:=∑h=1H\|δ​Ft\+h\|\.\\operatorname\{ARE\}\_\{t,H\}:=\\sum\_\{h=1\}^\{H\}\|\\delta F\_\{t\+h\}\|\.WhileMt,HM\_\{t,H\}measures the largest deviation over the horizon,AREt,H\\operatorname\{ARE\}\_\{t,H\}measures the total magnitude accumulated along the response trajectory\.

The largest negative excursion is

Pt,H−:=max1≤h≤H⁡\[−δ​Ft\+h\]\+\.P\_\{t,H\}^\{\-\}:=\\max\_\{1\\leq h\\leq H\}\[\-\\delta F\_\{t\+h\}\]\_\{\+\}\.Together,Pt,H\+P\_\{t,H\}^\{\+\}andPt,H−P\_\{t,H\}^\{\-\}distinguish the largest positive and negative deviations from the control trajectory\.

Their accumulated counterparts are

AELt,H\+:=∑h=1H\[δ​Ft\+h\]\+,AELt,H−:=∑h=1H\[−δ​Ft\+h\]\+\.\\operatorname\{AEL\}\_\{t,H\}^\{\+\}:=\\sum\_\{h=1\}^\{H\}\[\\delta F\_\{t\+h\}\]\_\{\+\},~~\\operatorname\{AEL\}\_\{t,H\}^\{\-\}:=\\sum\_\{h=1\}^\{H\}\[\-\\delta F\_\{t\+h\}\]\_\{\+\}\.The corresponding tangent quantities are obtained by replacingδ​Ft\+h\\delta F\_\{t\+h\}with𝒢t,h​ξt\\mathcal\{G\}\_\{t,h\}\\xi\_\{t\}\. For example,

ARE^t,H:=∑h=1H\|𝒢t,h​ξt\|,\\widehat\{\\operatorname\{ARE\}\}\_\{t,H\}:=\\sum\_\{h=1\}^\{H\}\|\\mathcal\{G\}\_\{t,h\}\\xi\_\{t\}\|,and

P^t,H−:=max1≤h≤H⁡\[−𝒢t,h​ξt\]\+\.\\widehat\{P\}\_\{t,H\}^\{\-\}:=\\max\_\{1\\leq h\\leq H\}\[\-\\mathcal\{G\}\_\{t,h\}\\xi\_\{t\}\]\_\{\+\}\.These quantities are secondary summaries of the same signed finite\-horizon response rather than separate dynamical objects\.

## Appendix CFrozen AdamW Memory\-Channel Analysis

This appendix proves Proposition[3\.3](https://arxiv.org/html/2608.19762#S3.Thmtheorem3)and gives spectral, transfer\-function, and feedback\-loop characterizations of the frozen three\-state model\.

### C\.1Short\-Horizon Responses

Consider

A¯=\[abmbvdmβ10dv0β2\],\\overline\{A\}=\\begin\{bmatrix\}a&b\_\{m\}&b\_\{v\}\\\\ d\_\{m\}&\\beta\_\{1\}&0\\\\ d\_\{v\}&0&\\beta\_\{2\}\\end\{bmatrix\},with normalized parameter readout

eθ:=\[100\],e\_\{\\theta\}:=\\begin\{bmatrix\}1\\\\ 0\\\\ 0\\end\{bmatrix\},and memory\-channel basis vectors

em:=\[010\],ev:=\[001\]\.e\_\{m\}:=\\begin\{bmatrix\}0\\\\ 1\\\\ 0\\end\{bmatrix\},~~e\_\{v\}:=\\begin\{bmatrix\}0\\\\ 0\\\\ 1\\end\{bmatrix\}\.Fork∈\{m,v\}k\\in\\\{m,v\\\}, define

rk​\(h\):=eθ⊤​A¯h−1​ek\.r\_\{k\}\(h\):=e\_\{\\theta\}^\{\\top\}\\overline\{A\}^\{h\-1\}e\_\{k\}\.
If the scalar probe sensitivity at the frozen operating point isγF≠0\\gamma\_\{F\}\\neq 0, then the corresponding first\-order probe\-loss response isγF​rk​\(h\)\\gamma\_\{F\}r\_\{k\}\(h\)\. Thus,rk​\(h\)r\_\{k\}\(h\)isolates the timing and amplification produced by the memory\-to\-parameter dynamics, while the probe readout supplies the final output scaling and sign\.

Ath=1h=1,

rm​\(1\)=rv​\(1\)=0\.r\_\{m\}\(1\)=r\_\{v\}\(1\)=0\.Ath=2h=2,

A¯​em=\[bmβ10\],A¯​ev=\[bv0β2\],\\overline\{A\}e\_\{m\}=\\begin\{bmatrix\}b\_\{m\}\\\\ \\beta\_\{1\}\\\\ 0\\end\{bmatrix\},~~\\overline\{A\}e\_\{v\}=\\begin\{bmatrix\}b\_\{v\}\\\\ 0\\\\ \\beta\_\{2\}\\end\{bmatrix\},so

rm​\(2\)=bm,rv​\(2\)=bv\.r\_\{m\}\(2\)=b\_\{m\},~~r\_\{v\}\(2\)=b\_\{v\}\.
ApplyingA¯\\overline\{A\}again,

A¯2​em=\[bm​\(a\+β1\)bm​dm\+β12bm​dv\],\\overline\{A\}^\{2\}e\_\{m\}=\\begin\{bmatrix\}b\_\{m\}\(a\+\\beta\_\{1\}\)\\\\ b\_\{m\}d\_\{m\}\+\\beta\_\{1\}^\{2\}\\\\ b\_\{m\}d\_\{v\}\\end\{bmatrix\},and

A¯2​ev=\[bv​\(a\+β2\)bv​dmbv​dv\+β22\]\.\\overline\{A\}^\{2\}e\_\{v\}=\\begin\{bmatrix\}b\_\{v\}\(a\+\\beta\_\{2\}\)\\\\ b\_\{v\}d\_\{m\}\\\\ b\_\{v\}d\_\{v\}\+\\beta\_\{2\}^\{2\}\\end\{bmatrix\}\.Hence

rm​\(3\)=bm​\(a\+β1\),rv​\(3\)=bv​\(a\+β2\)\.r\_\{m\}\(3\)=b\_\{m\}\(a\+\\beta\_\{1\}\),~~r\_\{v\}\(3\)=b\_\{v\}\(a\+\\beta\_\{2\}\)\.
A third multiplication gives

rm​\(4\)\\displaystyle r\_\{m\}\(4\)=a​bm​\(a\+β1\)\+bm​\(bm​dm\+β12\)\+bv​bm​dv\\displaystyle=a\\,b\_\{m\}\(a\+\\beta\_\{1\}\)\+b\_\{m\}\(b\_\{m\}d\_\{m\}\+\\beta\_\{1\}^\{2\}\)\+b\_\{v\}b\_\{m\}d\_\{v\}=bm​\(a2\+a​β1\+β12\+bm​dm\+bv​dv\),\\displaystyle=b\_\{m\}\\left\(a^\{2\}\+a\\beta\_\{1\}\+\\beta\_\{1\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\right\),and

rv​\(4\)\\displaystyle r\_\{v\}\(4\)=a​bv​\(a\+β2\)\+bm​bv​dm\+bv​\(bv​dv\+β22\)\\displaystyle=a\\,b\_\{v\}\(a\+\\beta\_\{2\}\)\+b\_\{m\}b\_\{v\}d\_\{m\}\+b\_\{v\}\(b\_\{v\}d\_\{v\}\+\\beta\_\{2\}^\{2\}\)=bv​\(a2\+a​β2\+β22\+bm​dm\+bv​dv\)\.\\displaystyle=b\_\{v\}\\left\(a^\{2\}\+a\\beta\_\{2\}\+\\beta\_\{2\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\right\)\.Therefore,

rk​\(4\)=bk​\(a2\+a​βk\+βk2\+bm​dm\+bv​dv\)\.r\_\{k\}\(4\)=b\_\{k\}\\left\(a^\{2\}\+a\\beta\_\{k\}\+\\beta\_\{k\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\right\)\.
Ifbk≠0b\_\{k\}\\neq 0, then

\|rk​\(3\)\|\>\|rk​\(2\)\|\|r\_\{k\}\(3\)\|\>\|r\_\{k\}\(2\)\|is equivalent to

\|a\+βk\|\>1\.\|a\+\\beta\_\{k\}\|\>1\.Likewise,

\|rk​\(4\)\|\>\|rk​\(3\)\|\|r\_\{k\}\(4\)\|\>\|r\_\{k\}\(3\)\|holds whenever

\|a2\+a​βk\+βk2\+bm​dm\+bv​dv\|\>\|a\+βk\|\.\\left\|a^\{2\}\+a\\beta\_\{k\}\+\\beta\_\{k\}^\{2\}\+b\_\{m\}d\_\{m\}\+b\_\{v\}d\_\{v\}\\right\|\>\|a\+\\beta\_\{k\}\|\.
These are finite\-horizon algebraic conditions and do not require asymptotic stability\. When they hold together with

ρ⁡\(A¯\)<1,\\rho\(\\overline\{A\}\)<1,the response grows over the corresponding short horizon even though

A¯h→0​as​h→∞\.\\overline\{A\}^\{h\}\\rightarrow 0~~\\text\{as \}h\\rightarrow\\infty\.This is the transient amplification regime described in the main text\.

### C\.2Characteristic Polynomial

The characteristic polynomial is

p⁡\(λ\)\\displaystyle p\(\\lambda\):=det\(λ​I−A¯\)\\displaystyle:=\\det\(\\lambda I\-\\overline\{A\}\)\(28\)=\(λ−a\)​\(λ−β1\)​\(λ−β2\)\\displaystyle=\(\\lambda\-a\)\(\\lambda\-\\beta\_\{1\}\)\(\\lambda\-\\beta\_\{2\}\)−bm​dm​\(λ−β2\)−bv​dv​\(λ−β1\)\.\\displaystyle\-b\_\{m\}d\_\{m\}\(\\lambda\-\\beta\_\{2\}\)\-b\_\{v\}d\_\{v\}\(\\lambda\-\\beta\_\{1\}\)\.The two loop gainsbm​dmb\_\{m\}d\_\{m\}andbv​dvb\_\{v\}d\_\{v\}shift the poles of the joint system away from the uncoupled valuesa,β1,β2a,\\beta\_\{1\},\\beta\_\{2\}\. The frozen system is asymptotically stable when every root of Equation \([28](https://arxiv.org/html/2608.19762#A3.E28)\) lies strictly inside the unit disk\.

### C\.3Memory\-to\-Parameter Transfer Functions

For a complex variablezzoutside the spectrum ofA¯\\overline\{A\}, define

ℋm​\(z\):=eθ⊤​\(z​I−A¯\)−1​em,\\mathscr\{H\}\_\{m\}\(z\):=e\_\{\\theta\}^\{\\top\}\(zI\-\\overline\{A\}\)^\{\-1\}e\_\{m\},and

ℋv​\(z\):=eθ⊤​\(z​I−A¯\)−1​ev\.\\mathscr\{H\}\_\{v\}\(z\):=e\_\{\\theta\}^\{\\top\}\(zI\-\\overline\{A\}\)^\{\-1\}e\_\{v\}\.Using the corresponding cofactors,

ℋm​\(z\)\\displaystyle\\mathscr\{H\}\_\{m\}\(z\)=bm​\(z−β2\)p⁡\(z\),\\displaystyle=\\frac\{b\_\{m\}\(z\-\\beta\_\{2\}\)\}\{p\(z\)\},ℋv​\(z\)\\displaystyle\\mathscr\{H\}\_\{v\}\(z\)=bv​\(z−β1\)p⁡\(z\)\.\\displaystyle=\\frac\{b\_\{v\}\(z\-\\beta\_\{1\}\)\}\{p\(z\)\}\.Both channels share the poles of the complete joint system\. Their numerators differ because the momentum input bypasses thevvstate, whereas the second\-moment input bypasses themmstate\.

### C\.4Modal Decomposition and Nonnormal Residues

SupposeA¯\\overline\{A\}is diagonalizable overℂ\\mathbb\{C\}:

A¯=V​Λ​V−1,\\overline\{A\}=V\\Lambda V^\{\-1\},where

V=\[v1v2v3\],V−1=\[w1∗w2∗w3∗\],V=\\begin\{bmatrix\}v\_\{1\}&v\_\{2\}&v\_\{3\}\\end\{bmatrix\},~~V^\{\-1\}=\\begin\{bmatrix\}w\_\{1\}^\{\*\}\\\\ w\_\{2\}^\{\*\}\\\\ w\_\{3\}^\{\*\}\\end\{bmatrix\},and

wi∗​vj=δi​j\.w\_\{i\}^\{\*\}v\_\{j\}=\\delta\_\{ij\}\.Then

A¯h−1=∑i=13λih−1​vi​wi∗\.\\overline\{A\}^\{h\-1\}=\\sum\_\{i=1\}^\{3\}\\lambda\_\{i\}^\{h\-1\}v\_\{i\}w\_\{i\}^\{\*\}\.Therefore

rk​\(h\)\\displaystyle r\_\{k\}\(h\)=eθ⊤​A¯h−1​ek\\displaystyle=e\_\{\\theta\}^\{\\top\}\\overline\{A\}^\{h\-1\}e\_\{k\}\(29\)=∑i=13\(eθ⊤​vi\)​\(wi∗​ek\)​λih−1\.\\displaystyle=\\sum\_\{i=1\}^\{3\}\\left\(e\_\{\\theta\}^\{\\top\}v\_\{i\}\\right\)\\left\(w\_\{i\}^\{\*\}e\_\{k\}\\right\)\\lambda\_\{i\}^\{h\-1\}\.For a real matrix, complex eigenvalues and residues occur in conjugate pairs, so Equation \([29](https://arxiv.org/html/2608.19762#A3.E29)\) remains real\.

The modal expansion implies

\|rk​\(h\)\|≤∑i=13\|eθ⊤​vi\|​\|wi∗​ek\|​\|λi\|h−1\.\|r\_\{k\}\(h\)\|\\leq\\sum\_\{i=1\}^\{3\}\\left\|e\_\{\\theta\}^\{\\top\}v\_\{i\}\\right\|\\left\|w\_\{i\}^\{\*\}e\_\{k\}\\right\|\|\\lambda\_\{i\}\|^\{h\-1\}\.A coarser matrix\-norm bound is

\|rk​\(h\)\|\\displaystyle\|r\_\{k\}\(h\)\|≤‖eθ⊤​V‖2​‖Λh−1‖2​‖V−1​ek‖2\\displaystyle\\leq\\\|e\_\{\\theta\}^\{\\top\}V\\\|\_\{2\}\\\|\\Lambda^\{h\-1\}\\\|\_\{2\}\\\|V^\{\-1\}e\_\{k\}\\\|\_\{2\}≤κ2​\(V\)​ρ​\(A¯\)h−1,\\displaystyle\\leq\\kappa\_\{2\}\(V\)\\rho\(\\overline\{A\}\)^\{h\-1\},where

κ2​\(V\):=‖V‖2​‖V−1‖2\.\\kappa\_\{2\}\(V\):=\\\|V\\\|\_\{2\}\\\|V^\{\-1\}\\\|\_\{2\}\.
For a normal matrix,VVcan be chosen unitary andκ2​\(V\)=1\\kappa\_\{2\}\(V\)=1\. For a nonnormal matrix, the eigenvector condition number and individual input–output residues can be much larger\. Nonnormality can therefore enlarge the finite\-horizon memory\-to\-parameter response even when all eigenmodes are asymptotically decaying\.

The finite\-horizon peak

hk⋆:=min⁡arg⁡maxh≥1​\|rk​\(h\)\|h\_\{k\}^\{\\star\}:=\\min\\arg\\max\_\{h\\geq 1\}\|r\_\{k\}\(h\)\|depends jointly on the modal decay rates, oscillatory phases, and input–output residues\. Multiple decaying modes can interfere constructively at intermediate horizons, placing the largest response after the initial memory\-to\-parameter conversion\. The sign of the corresponding probe\-loss response additionally depends on the scalar probe readout\.

### C\.5Signs of the Two Feedback Loops

In the scalar restriction,

bm=β1​M,dm=\(1−β1\)​Htr\.b\_\{m\}=\\beta\_\{1\}M,~~d\_\{m\}=\(1\-\\beta\_\{1\}\)H^\{\\mathrm\{tr\}\}\.SinceM<0M<0,

sign⁡\(bm​dm\)=−sign⁡\(Htr\)\.\\operatorname\{sign\}\(b\_\{m\}d\_\{m\}\)=\-\\operatorname\{sign\}\(H^\{\\mathrm\{tr\}\}\)\.Thus, positive local curvature gives a negative momentum\-loop return gain, which can contribute to oscillatory or sign\-changing parameter responses\.

For the second\-moment loop,

bv=β2​V,dv=2​\(1−β2\)​g​Htr\.b\_\{v\}=\\beta\_\{2\}V,~~d\_\{v\}=2\(1\-\\beta\_\{2\}\)gH^\{\\mathrm\{tr\}\}\.Since the sign ofVVfollows the sign of the current first\-moment state,

sign⁡\(bv​dv\)=sign⁡\(m​g​Htr\)\.\\operatorname\{sign\}\(b\_\{v\}d\_\{v\}\)=\\operatorname\{sign\}\\left\(m\\,g\\,H^\{\\mathrm\{tr\}\}\\right\)\.The second\-moment feedback can therefore reinforce or oppose the evolving parameter response depending on the local operating point\.

### C\.6Weight Decay and Bias Correction

For scalar weight decayλwd\\lambda\_\{\\mathrm\{wd\}\},

a=1−η​λwd\+Γ​Htr\.a=1\-\\eta\\lambda\_\{\\mathrm\{wd\}\}\+\\Gamma H^\{\\mathrm\{tr\}\}\.Weight decay therefore modifies the direct parameter\-retention term and, throughaa, changes the short\-horizon response coefficients and their interaction with the two moment\-memory channels\.

Bias correction enters through

ρ1,s=1−β1s,ρ2,s=1−β2s,\\rho\_\{1,s\}=1\-\\beta\_\{1\}^\{s\},~~\\rho\_\{2,s\}=1\-\\beta\_\{2\}^\{s\},and hence throughMsM\_\{s\},VsV\_\{s\}, andΓs\\Gamma\_\{s\}\. In the full AdamW dynamics these quantities are time dependent\. The frozen model treats their values at the selected operating point as fixed over the local analysis window\. Bias correction therefore changes the numerical coefficients of the frozen system without changing its parameter–moment coupling structure\.

## Appendix DFinite\-Horizon Approximation Error

This appendix proves Theorem[3\.4](https://arxiv.org/html/2608.19762#S3.Thmtheorem4)and Corollary[3\.5](https://arxiv.org/html/2608.19762#S3.Thmtheorem5)\. It also gives a sufficient activation\-margin condition for quadratic switching error, a recursive error envelope, and the pathwise interpretation under future training randomness\.

### D\.1Exact Smooth–Switching Decomposition

Let

xss:=xsc\+δ​xs\.x\_\{s\}^\{\\mathrm\{s\}\}:=x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\.The exact paired state difference after one future update is

δ​xs\+1\\displaystyle\\delta x\_\{s\+1\}=fs​\(xsc\+δ​xs\)−fs​\(xsc\)\.\\displaystyle=f\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\-f\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\\right\)\.By construction,

fs​\(xsc\)=f~s​\(xsc\)\.f\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)=\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\.Adding and subtractingf~s​\(xsc\+δ​xs\)\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\)gives

δ​xs\+1=\\displaystyle\\delta x\_\{s\+1\}=\{\}f~s​\(xsc\+δ​xs\)−f~s​\(xsc\)\\displaystyle\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\-\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\\right\)\+fs​\(xsc\+δ​xs\)−f~s​\(xsc\+δ​xs\)\.\\displaystyle\+f\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\-\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\.Using

rssm\\displaystyle r\_\{s\}^\{\\mathrm\{sm\}\}:=f~s​\(xsc\+δ​xs\)−f~s​\(xsc\)−As​δ​xs,\\displaystyle:=\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\-\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\\right\)\-A\_\{s\}\\delta x\_\{s\},rssw\\displaystyle r\_\{s\}^\{\\mathrm\{sw\}\}:=fs​\(xsc\+δ​xs\)−f~s​\(xsc\+δ​xs\),\\displaystyle:=f\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\-\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\),we obtain

δ​xs\+1=As​δ​xs\+rssm\+rssw\.\\delta x\_\{s\+1\}=A\_\{s\}\\delta x\_\{s\}\+r\_\{s\}^\{\\mathrm\{sm\}\}\+r\_\{s\}^\{\\mathrm\{sw\}\}\.\(30\)
IfD​f~sD\\widetilde\{f\}\_\{s\}isLsL\_\{s\}\-Lipschitz along

ℒs=\{xsc\+τ​δ​xs:0≤τ≤1\},\\mathcal\{L\}\_\{s\}=\\left\\\{x\_\{s\}^\{\\mathrm\{c\}\}\+\\tau\\delta x\_\{s\}:0\\leq\\tau\\leq 1\\right\\\},the integral remainder formula gives

rssm=∫01\[D​f~s​\(xsc\+τ​δ​xs\)−D​f~s​\(xsc\)\]​δ​xs​𝑑τ\.r\_\{s\}^\{\\mathrm\{sm\}\}=\\int\_\{0\}^\{1\}\\left\[D\\widetilde\{f\}\_\{s\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\tau\\delta x\_\{s\}\\right\)\-D\\widetilde\{f\}\_\{s\}\(x\_\{s\}^\{\\mathrm\{c\}\}\)\\right\]\\delta x\_\{s\}\\,\\mathrm\{d\}\\tau\.Therefore

‖rssm‖\\displaystyle\\\|r\_\{s\}^\{\\mathrm\{sm\}\}\\\|≤∫01Ls​τ​‖δ​xs‖2​𝑑τ\\displaystyle\\leq\\int\_\{0\}^\{1\}L\_\{s\}\\tau\\\|\\delta x\_\{s\}\\\|^\{2\}\\,\\mathrm\{d\}\\tau\(31\)=Ls2​‖δ​xs‖2\.\\displaystyle=\\frac\{L\_\{s\}\}\{2\}\\\|\\delta x\_\{s\}\\\|^\{2\}\.
For a smooth network,rssw=0r\_\{s\}^\{\\mathrm\{sw\}\}=0\. The same holds in a piecewise\-smooth network whenever the paired states remain in the same activation region\.

### D\.2Proof of the Multistep State\-Error Identity

The tangent approximation is initialized by

δ​x^t\+1:=Bt​ξt\\widehat\{\\delta x\}\_\{t\+1\}:=B\_\{t\}\\xi\_\{t\}and propagated according to

δ​x^s\+1:=As​δ​x^s\.\\widehat\{\\delta x\}\_\{s\+1\}:=A\_\{s\}\\widehat\{\\delta x\}\_\{s\}\.Define

es:=δ​xs−δ​x^s\.e\_\{s\}:=\\delta x\_\{s\}\-\\widehat\{\\delta x\}\_\{s\}\.At the first post\-shock state,

et\+1\\displaystyle e\_\{t\+1\}=δ​xt\+1−δ​x^t\+1\\displaystyle=\\delta x\_\{t\+1\}\-\\widehat\{\\delta x\}\_\{t\+1\}\(32\)=Bt​ξt\+rB,t​\(ξt\)−Bt​ξt\\displaystyle=B\_\{t\}\\xi\_\{t\}\+r\_\{B,t\}\(\\xi\_\{t\}\)\-B\_\{t\}\\xi\_\{t\}=rB,t​\(ξt\)\.\\displaystyle=r\_\{B,t\}\(\\xi\_\{t\}\)\.
For a future step, subtracting the tangent recursion from Equation \([30](https://arxiv.org/html/2608.19762#A4.E30)\) gives

es\+1\\displaystyle e\_\{s\+1\}=As​δ​xs\+rssm\+rssw−As​δ​x^s\\displaystyle=A\_\{s\}\\delta x\_\{s\}\+r\_\{s\}^\{\\mathrm\{sm\}\}\+r\_\{s\}^\{\\mathrm\{sw\}\}\-A\_\{s\}\\widehat\{\\delta x\}\_\{s\}=As​es\+rssm\+rssw\.\\displaystyle=A\_\{s\}e\_\{s\}\+r\_\{s\}^\{\\mathrm\{sm\}\}\+r\_\{s\}^\{\\mathrm\{sw\}\}\.Repeated substitution yields

et\+h=\\displaystyle e\_\{t\+h\}=\{\}Φt\+h,t\+1​rB,t​\(ξt\)\\displaystyle\\Phi\_\{t\+h,t\+1\}r\_\{B,t\}\(\\xi\_\{t\}\)\(33\)\+∑j=t\+1t\+h−1Φt\+h,j\+1\(rjsm\+rjsw\)\.\\displaystyle\+\\sum\_\{j=t\+1\}^\{t\+h\-1\}\\Phi\_\{t\+h,j\+1\}\\left\(r\_\{j\}^\{\\mathrm\{sm\}\}\+r\_\{j\}^\{\\mathrm\{sw\}\}\\right\)\.Forh=1h=1, the sum is empty and Equation \([33](https://arxiv.org/html/2608.19762#A4.E33)\) reduces to Equation \([32](https://arxiv.org/html/2608.19762#A4.E32)\)\.

Taking norms gives

‖et\+h‖≤\\displaystyle\\\|e\_\{t\+h\}\\\|\\leq\{\}‖Φt\+h,t\+1‖​‖rB,t​\(ξt\)‖\\displaystyle\\\|\\Phi\_\{t\+h,t\+1\}\\\|\\\|r\_\{B,t\}\(\\xi\_\{t\}\)\\\|\+∑j=t\+1t\+h−1∥Φt\+h,j\+1∥\(∥rjsm∥\+∥rjsw∥\)\.\\displaystyle\+\\sum\_\{j=t\+1\}^\{t\+h\-1\}\\\|\\Phi\_\{t\+h,j\+1\}\\\|\\left\(\\\|r\_\{j\}^\{\\mathrm\{sm\}\}\\\|\+\\\|r\_\{j\}^\{\\mathrm\{sw\}\}\\\|\\right\)\.Using Equation \([31](https://arxiv.org/html/2608.19762#A4.E31)\) proves the state\-error bound in Theorem[3\.4](https://arxiv.org/html/2608.19762#S3.Thmtheorem4)\.

### D\.3Probe\-Output Error

Let

Πθ:=\[I00\]\\Pi\_\{\\theta\}:=\\begin\{bmatrix\}I&0&0\\end\{bmatrix\}denote projection onto the parameter block\. Then

δ​θt\+h=Πθ​δ​xt\+h\.\\delta\\theta\_\{t\+h\}=\\Pi\_\{\\theta\}\\delta x\_\{t\+h\}\.The exact paired probe response is

δ​Ft\+h=Ft\+h​\(θt\+hc\+δ​θt\+h\)−Ft\+h​\(θt\+hc\)\.\\displaystyle\\delta F\_\{t\+h\}=F\_\{t\+h\}\\left\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\+\\delta\\theta\_\{t\+h\}\\right\)\-F\_\{t\+h\}\\left\(\\theta\_\{t\+h\}^\{\\mathrm\{c\}\}\\right\)\.If∇Ft\+h\\nabla F\_\{t\+h\}isLF,t\+hL\_\{F,t\+h\}\-Lipschitz along the connecting segment, then

δ​Ft\+h=ct\+h⊤​δ​xt\+h\+rF,t\+h,\\delta F\_\{t\+h\}=c\_\{t\+h\}^\{\\top\}\\delta x\_\{t\+h\}\+r\_\{F,t\+h\},where

\|rF,t\+h\|≤LF,t\+h2​‖δ​θt\+h‖2\.\|r\_\{F,t\+h\}\|\\leq\\frac\{L\_\{F,t\+h\}\}\{2\}\\\|\\delta\\theta\_\{t\+h\}\\\|^\{2\}\.\(34\)
The tangent output prediction is

δ​F^t\+h\\displaystyle\\widehat\{\\delta F\}\_\{t\+h\}=ct\+h⊤​δ​x^t\+h\\displaystyle=c\_\{t\+h\}^\{\\top\}\\widehat\{\\delta x\}\_\{t\+h\}=ct\+h⊤​Φt\+h,t\+1​Bt​ξt\.\\displaystyle=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\.Therefore

δ​Ft\+h−δ​F^t\+h\\displaystyle\\delta F\_\{t\+h\}\-\\widehat\{\\delta F\}\_\{t\+h\}=ct\+h⊤​\(δ​xt\+h−δ​x^t\+h\)\+rF,t\+h\\displaystyle=c\_\{t\+h\}^\{\\top\}\\left\(\\delta x\_\{t\+h\}\-\\widehat\{\\delta x\}\_\{t\+h\}\\right\)\+r\_\{F,t\+h\}=ct\+h⊤​et\+h\+rF,t\+h\.\\displaystyle=c\_\{t\+h\}^\{\\top\}e\_\{t\+h\}\+r\_\{F,t\+h\}\.Taking absolute values yields

\|δ​Ft\+h−ct\+h⊤​Φt\+h,t\+1​Bt​ξt\|≤\\displaystyle\\left\|\\delta F\_\{t\+h\}\-c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\\right\|\\leq\{\}‖ct\+h‖​‖et\+h‖\\displaystyle\\\|c\_\{t\+h\}\\\|\\\|e\_\{t\+h\}\\\|\+LF,t\+h2​‖δ​θt\+h‖2,\\displaystyle\+\\frac\{L\_\{F,t\+h\}\}\{2\}\\\|\\delta\\theta\_\{t\+h\}\\\|^\{2\},which proves the output\-error bound in Theorem[3\.4](https://arxiv.org/html/2608.19762#S3.Thmtheorem4)\.

### D\.4A Sufficient Activation\-Margin Condition

We now give a sufficient condition under which the aggregate switching defect satisfies the quadratic bound required by Corollary[3\.5](https://arxiv.org/html/2608.19762#S3.Thmtheorem5)\.

Let𝒮s\\mathcal\{S\}\_\{s\}denote the set of activation gates whose states differ between the paired trajectories during transitionss\. For multiple simultaneous switches, the total switching defect can be decomposed by a telescoping construction\.

Choose an arbitrary ordering

𝒮s=\{j1,…,jq\}\.\\mathcal\{S\}\_\{s\}=\\\{j\_\{1\},\\ldots,j\_\{q\}\\\}\.Forr=0,…,qr=0,\\ldots,q, letfs\(r\)f\_\{s\}^\{\(r\)\}denote the local transition map that uses the shock activation state for gatesj1,…,jrj\_\{1\},\\ldots,j\_\{r\}and the control activation state for the remaining switched gates\. At the perturbed state,

fs\(0\)=f~s,fs\(q\)=fs\.f\_\{s\}^\{\(0\)\}=\\widetilde\{f\}\_\{s\},~~f\_\{s\}^\{\(q\)\}=f\_\{s\}\.Define

rs,jrsw:=\\displaystyle r\_\{s,j\_\{r\}\}^\{\\mathrm\{sw\}\}:=\{\}fs\(r\)​\(xsc\+δ​xs\)\\displaystyle f\_\{s\}^\{\(r\)\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)−fs\(r−1\)​\(xsc\+δ​xs\)\.\\displaystyle\-f\_\{s\}^\{\(r\-1\)\}\\left\(x\_\{s\}^\{\\mathrm\{c\}\}\+\\delta x\_\{s\}\\right\)\.The total switching defect telescopes:

rssw=∑r=1qrs,jrsw\.r\_\{s\}^\{\\mathrm\{sw\}\}=\\sum\_\{r=1\}^\{q\}r\_\{s,j\_\{r\}\}^\{\\mathrm\{sw\}\}\.
Assume that each local switch satisfies

‖rs,jsw‖≤Ks,j​‖δ​xs‖\.\\\|r\_\{s,j\}^\{\\mathrm\{sw\}\}\\\|\\leq K\_\{s,j\}\\\|\\delta x\_\{s\}\\\|\.\(35\)Letas,jca\_\{s,j\}^\{\\mathrm\{c\}\}denote the control pre\-activation of gatejj, and letδ​as,j\\delta a\_\{s,j\}denote its change between the paired states\. A gate can switch only if

\|as,jc\|≤\|δ​as,j\|\.\|a\_\{s,j\}^\{\\mathrm\{c\}\}\|\\leq\|\\delta a\_\{s,j\}\|\.Assume

\|δ​as,j\|≤La,s,j​‖δ​xs‖,La,s,j≤L¯a,s\.\|\\delta a\_\{s,j\}\|\\leq L\_\{a,s,j\}\\\|\\delta x\_\{s\}\\\|,~~L\_\{a,s,j\}\\leq\\overline\{L\}\_\{a,s\}\.Every switched unit therefore satisfies

\|as,jc\|≤L¯a,s​‖δ​xs‖,\|a\_\{s,j\}^\{\\mathrm\{c\}\}\|\\leq\\overline\{L\}\_\{a,s\}\\\|\\delta x\_\{s\}\\\|,and hence

𝒮s⊆\{j:\|as,jc\|≤L¯a,s​‖δ​xs‖\}\.\\mathcal\{S\}\_\{s\}\\subseteq\\left\\\{j:\|a\_\{s,j\}^\{\\mathrm\{c\}\}\|\\leq\\overline\{L\}\_\{a,s\}\\\|\\delta x\_\{s\}\\\|\\right\\\}\.\(36\)
Suppose further that, for all sufficiently smallu\>0u\>0, the weighted mass of units near the activation boundary satisfies

∑j:\|as,jc\|≤uKs,j≤ρsu\.\\sum\_\{j:\|a\_\{s,j\}^\{\\mathrm\{c\}\}\|\\leq u\}K\_\{s,j\}\\leq\\rho\_\{s\}u\.\(37\)Using Equations \([35](https://arxiv.org/html/2608.19762#A4.E35)\) and \([36](https://arxiv.org/html/2608.19762#A4.E36)\),

‖rssw‖\\displaystyle\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|≤∑j∈𝒮s‖rs,jsw‖\\displaystyle\\leq\\sum\_\{j\\in\\mathcal\{S\}\_\{s\}\}\\\|r\_\{s,j\}^\{\\mathrm\{sw\}\}\\\|≤‖δ​xs‖​∑j∈𝒮sKs,j\\displaystyle\\leq\\\|\\delta x\_\{s\}\\\|\\sum\_\{j\\in\\mathcal\{S\}\_\{s\}\}K\_\{s,j\}≤∥δxs∥∑j:\|as,jc\|≤L¯a,s​‖δ​xs‖Ks,j\.\\displaystyle\\leq\\\|\\delta x\_\{s\}\\\|\\sum\_\{j:\|a\_\{s,j\}^\{\\mathrm\{c\}\}\|\\leq\\overline\{L\}\_\{a,s\}\\\|\\delta x\_\{s\}\\\|\}K\_\{s,j\}\.Applying Equation \([37](https://arxiv.org/html/2608.19762#A4.E37)\) with

u=L¯a,s​‖δ​xs‖u=\\overline\{L\}\_\{a,s\}\\\|\\delta x\_\{s\}\\\|gives

‖rssw‖≤ρs​L¯a,s​‖δ​xs‖2\.\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq\\rho\_\{s\}\\overline\{L\}\_\{a,s\}\\\|\\delta x\_\{s\}\\\|^\{2\}\.\(38\)Thus the activation\-margin condition provides the quadratic switching bound used by the fixed\-horizon accuracy result\.

### D\.5Proof of Fixed\-Horizon First\-Order Accuracy

Consider the scaled perturbationα​ξt\\alpha\\xi\_\{t\}and a fixed horizonHH\.

Local Lipschitz continuity ofD​ψtD\\psi\_\{t\}gives, from Equation \([23](https://arxiv.org/html/2608.19762#A1.E23)\),

‖rB,t​\(α​ξt\)‖≤CB,t​α2​‖ξt‖2\\\|r\_\{B,t\}\(\\alpha\\xi\_\{t\}\)\\\|\\leq C\_\{B,t\}\\alpha^\{2\}\\\|\\xi\_\{t\}\\\|^\{2\}for sufficiently smallα\\alpha\.

Local Lipschitz continuity ofD​f~sD\\widetilde\{f\}\_\{s\}gives

‖rssm‖≤Ls2​‖δ​xs‖2\.\\\|r\_\{s\}^\{\\mathrm\{sm\}\}\\\|\\leq\\frac\{L\_\{s\}\}\{2\}\\\|\\delta x\_\{s\}\\\|^\{2\}\.Under the switching assumption in Corollary[3\.5](https://arxiv.org/html/2608.19762#S3.Thmtheorem5),

‖rssw‖≤Cssw​‖δ​xs‖2\.\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq C\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\delta x\_\{s\}\\\|^\{2\}\.Hence

‖rssm‖\+‖rssw‖≤Cs​‖δ​xs‖2,Cs:=Ls2\+Cssw\.\\\|r\_\{s\}^\{\\mathrm\{sm\}\}\\\|\+\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq C\_\{s\}\\\|\\delta x\_\{s\}\\\|^\{2\},~~C\_\{s\}:=\\frac\{L\_\{s\}\}\{2\}\+C\_\{s\}^\{\\mathrm\{sw\}\}\.\(39\)
At the first post\-shock state,

‖δ​xt\+1​\(α\)‖\\displaystyle\\\|\\delta x\_\{t\+1\}\(\\alpha\)\\\|≤\|α\|​‖Bt​ξt‖\+CB,t​α2​‖ξt‖2\\displaystyle\\leq\|\\alpha\|\\\|B\_\{t\}\\xi\_\{t\}\\\|\+C\_\{B,t\}\\alpha^\{2\}\\\|\\xi\_\{t\}\\\|^\{2\}=O⁡\(\|α\|\)\.\\displaystyle=O\(\|\\alpha\|\)\.Suppose inductively that

‖δ​xs​\(α\)‖=O⁡\(\|α\|\)\\\|\\delta x\_\{s\}\(\\alpha\)\\\|=O\(\|\\alpha\|\)for somet\+1≤s<t\+Ht\+1\\leq s<t\+H\. The local derivatives are bounded on the relevant neighborhood, so‖As‖\\\|A\_\{s\}\\\|is bounded\. Using Equation \([30](https://arxiv.org/html/2608.19762#A4.E30)\),

‖δ​xs\+1​\(α\)‖\\displaystyle\\\|\\delta x\_\{s\+1\}\(\\alpha\)\\\|≤‖As‖​‖δ​xs​\(α\)‖\+Cs​‖δ​xs​\(α\)‖2\\displaystyle\\leq\\\|A\_\{s\}\\\|\\\|\\delta x\_\{s\}\(\\alpha\)\\\|\+C\_\{s\}\\\|\\delta x\_\{s\}\(\\alpha\)\\\|^\{2\}=O⁡\(\|α\|\)\+O⁡\(α2\)\\displaystyle=O\(\|\\alpha\|\)\+O\(\\alpha^\{2\}\)=O⁡\(\|α\|\)\.\\displaystyle=O\(\|\\alpha\|\)\.By induction,

‖δ​xt\+h​\(α\)‖=O⁡\(\|α\|\)\\\|\\delta x\_\{t\+h\}\(\\alpha\)\\\|=O\(\|\\alpha\|\)for every fixed1≤h≤H1\\leq h\\leq H\.

The exact multistep error identity now gives

‖et\+h​\(α\)‖≤\\displaystyle\\\|e\_\{t\+h\}\(\\alpha\)\\\|\\leq\{\}‖Φt\+h,t\+1‖​O​\(α2\)\\displaystyle\\\|\\Phi\_\{t\+h,t\+1\}\\\|O\(\\alpha^\{2\}\)\+∑j=t\+1t\+h−1∥Φt\+h,j\+1∥O\(α2\)\.\\displaystyle\+\\sum\_\{j=t\+1\}^\{t\+h\-1\}\\\|\\Phi\_\{t\+h,j\+1\}\\\|O\(\\alpha^\{2\}\)\.SinceHHis fixed and the finite\-horizon propagators are uniformly bounded,

‖et\+h​\(α\)‖=O⁡\(α2\),1≤h≤H\.\\\|e\_\{t\+h\}\(\\alpha\)\\\|=O\(\\alpha^\{2\}\),~~1\\leq h\\leq H\.
The probe objectives have locally Lipschitz gradients, and‖δ​θt\+h​\(α\)‖=O⁡\(\|α\|\)\\\|\\delta\\theta\_\{t\+h\}\(\\alpha\)\\\|=O\(\|\\alpha\|\), so Equation \([34](https://arxiv.org/html/2608.19762#A4.E34)\) gives

\|rF,t\+h​\(α\)\|=O⁡\(α2\)\.\|r\_\{F,t\+h\}\(\\alpha\)\|=O\(\\alpha^\{2\}\)\.Therefore

δ​Ft\+h​\(α\)\\displaystyle\\delta F\_\{t\+h\}\(\\alpha\)=ct\+h⊤​δ​x^t\+h​\(α\)\+O⁡\(α2\)\\displaystyle=c\_\{t\+h\}^\{\\top\}\\widehat\{\\delta x\}\_\{t\+h\}\(\\alpha\)\+O\(\\alpha^\{2\}\)=α​ct\+h⊤​Φt\+h,t\+1​Bt​ξt\+O⁡\(α2\),\\displaystyle=\\alpha c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{t\}\+O\(\\alpha^\{2\}\),for every1≤h≤H1\\leq h\\leq H\. The constants may depend on the fixed control trajectory and horizonHHbut not onα\\alpha\. This proves Corollary[3\.5](https://arxiv.org/html/2608.19762#S3.Thmtheorem5)\.

When the sufficient activation\-margin condition of Appendix[D\.4](https://arxiv.org/html/2608.19762#A4.SS4)holds, Equation \([39](https://arxiv.org/html/2608.19762#A4.E39)\) can be instantiated with

Cssw=ρs​L¯a,s\.C\_\{s\}^\{\\mathrm\{sw\}\}=\\rho\_\{s\}\\overline\{L\}\_\{a,s\}\.

### D\.6Recursive Error Envelope

The structural error bound contains the true deviationsδ​xs\\delta x\_\{s\}\. A recursive envelope can instead be expressed in terms of the tangent trajectory and previously accumulated error bounds\.

Suppose

‖es‖≤εs\.\\\|e\_\{s\}\\\|\\leq\\varepsilon\_\{s\}\.Since

δ​xs=δ​x^s\+es,\\delta x\_\{s\}=\\widehat\{\\delta x\}\_\{s\}\+e\_\{s\},we have

‖δ​xs‖≤‖δ​x^s‖\+εs\.\\\|\\delta x\_\{s\}\\\|\\leq\\\|\\widehat\{\\delta x\}\_\{s\}\\\|\+\\varepsilon\_\{s\}\.Using the error recursion,

‖es\+1‖\\displaystyle\\\|e\_\{s\+1\}\\\|≤‖As‖​‖es‖\+‖rssm‖\+‖rssw‖\\displaystyle\\leq\\\|A\_\{s\}\\\|\\\|e\_\{s\}\\\|\+\\\|r\_\{s\}^\{\\mathrm\{sm\}\}\\\|\+\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|≤‖As‖​εs\+Ls2​\(‖δ​x^s‖\+εs\)2\+‖rssw‖\.\\displaystyle\\leq\\\|A\_\{s\}\\\|\\varepsilon\_\{s\}\+\\frac\{L\_\{s\}\}\{2\}\\left\(\\\|\\widehat\{\\delta x\}\_\{s\}\\\|\+\\varepsilon\_\{s\}\\right\)^\{2\}\+\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\.Thus the recursion

εs\+1:=\\displaystyle\\varepsilon\_\{s\+1\}:=\{\}‖As‖​εs\\displaystyle\\\|A\_\{s\}\\\|\\varepsilon\_\{s\}\(40\)\+Ls2​\(‖δ​x^s‖\+εs\)2\+‖rssw‖\\displaystyle\+\\frac\{L\_\{s\}\}\{2\}\\left\(\\\|\\widehat\{\\delta x\}\_\{s\}\\\|\+\\varepsilon\_\{s\}\\right\)^\{2\}\+\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|preserves

‖es\+1‖≤εs\+1\.\\\|e\_\{s\+1\}\\\|\\leq\\varepsilon\_\{s\+1\}\.Initializing with

εt\+1:=‖rB,t​\(ξt\)‖\\varepsilon\_\{t\+1\}:=\\\|r\_\{B,t\}\(\\xi\_\{t\}\)\\\|therefore gives an envelope for all subsequent states\.

If a quadratic switching bound

‖rssw‖≤Cssw​‖δ​xs‖2\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq C\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\delta x\_\{s\}\\\|^\{2\}is available, then

‖rssw‖≤Cssw​\(‖δ​x^s‖\+εs\)2\.\\\|r\_\{s\}^\{\\mathrm\{sw\}\}\\\|\\leq C\_\{s\}^\{\\mathrm\{sw\}\}\\left\(\\\|\\widehat\{\\delta x\}\_\{s\}\\\|\+\\varepsilon\_\{s\}\\right\)^\{2\}\.The envelope becomes

εs\+1:=\\displaystyle\\varepsilon\_\{s\+1\}:=\{\}‖As‖​εs\\displaystyle\\\|A\_\{s\}\\\|\\varepsilon\_\{s\}\(41\)\+\(Ls2\+Cssw\)​\(‖δ​x^s‖\+εs\)2\.\\displaystyle\+\\left\(\\frac\{L\_\{s\}\}\{2\}\+C\_\{s\}^\{\\mathrm\{sw\}\}\\right\)\\left\(\\\|\\widehat\{\\delta x\}\_\{s\}\\\|\+\\varepsilon\_\{s\}\\right\)^\{2\}\.Under the activation\-margin sufficient condition,

Cssw=ρs​L¯a,s\.C\_\{s\}^\{\\mathrm\{sw\}\}=\\rho\_\{s\}\\overline\{L\}\_\{a,s\}\.
The same transition dynamics therefore govern both the desired first\-order response and the accumulation of approximation error: large finite\-horizon gain amplifies the propagated perturbation as well as nonlinear defects introduced along the trajectory\.

### D\.7Pathwise Conditioning and Distributional Extension

Letω\\omegadenote a realized future random sequence, including future minibatches, dropout masks, data augmentation, and other stochastic training operations\. Conditional onω\\omegaand the pre\-shock history, the future operators are deterministic:

As​\(ω\),Φt\+h,t\+1​\(ω\),ct\+h​\(ω\)\.A\_\{s\}\(\\omega\),~~\\Phi\_\{t\+h,t\+1\}\(\\omega\),~~c\_\{t\+h\}\(\\omega\)\.When the probe sequence is itself defined from the future minibatches, such as

Fs​\(θ\)=ℓ⁡\(θ,ℬs\),F\_\{s\}\(\\theta\)=\\ell\(\\theta;\\mathcal\{B\}\_\{s\}\),the probe functions are also fixed after conditioning onω\\omega\.

The corresponding pathwise input–output operator is

𝒢t,h​\(ω\)=ct\+h​\(ω\)⊤​Φt\+h,t\+1​\(ω\)​Bt\.\\mathcal\{G\}\_\{t,h\}\(\\omega\)=c\_\{t\+h\}\(\\omega\)^\{\\top\}\\Phi\_\{t\+h,t\+1\}\(\\omega\)B\_\{t\}\.All directional\-response identities and error decompositions above apply separately to each realized common\-future sequence\. Averaging a pathwise quantity over independently sampled futures yields its corresponding distributional version\. For example,

𝔼ω​\[δ​Ft\+h​\(ω\)\]\\mathbb\{E\}\_\{\\omega\}\\left\[\\delta F\_\{t\+h\}\(\\omega\)\\right\]describes the mean paired response at horizonhh, while

𝔼ω​\[𝒢t,h​\(ω\)​ξt\]\\mathbb\{E\}\_\{\\omega\}\\left\[\\mathcal\{G\}\_\{t,h\}\(\\omega\)\\xi\_\{t\}\\right\]gives the corresponding first\-order mean response whenever the expectation and local expansion may be interchanged\.

The common\-future construction retains the same future realization for the control and perturbed trajectories within each pair\. It therefore isolates the pathwise propagation associated with the perturbation at steptt, while repetition across different future realizations characterizes variability of that response\.

## Appendix EDetailed Experimental Protocols and Results

This appendix gives the complete protocols and the additional numerical results supporting Section[4](https://arxiv.org/html/2608.19762#S4)\. The experiments are organized by scientific question rather than by model class\. Experiment 1 validates the finite\-horizon mechanism across controlled quadratic systems, nonconvex neural networks, and pretrained language models\. Experiment 2 then asks whether the resulting delayed response retains candidate\-specific structure when the future training sequence is unknown and independently resampled\.

For the controlled and neural\-network mechanism\-validation stages, horizonh=1h=1denotes the first post\-shock state and

di,h​\(α\)\\displaystyle d\_\{i,h\}\(\\alpha\):=F⁡\(θi,t\+hs​\(α\)\)−F⁡\(θt\+hc\),\\displaystyle:=F\(\\theta^\{\\mathrm\{s\}\}\_\{i,t\+h\}\(\\alpha\)\)\-F\(\\theta^\{\\mathrm\{c\}\}\_\{t\+h\}\),\(42\)d^i,h\\displaystyle\\widehat\{d\}\_\{i,h\}:=ct\+h⊤​Φt\+h,t\+1​Bt​ξi,\\displaystyle:=c\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{i\},\(43\)d^i,h​\(α\)\\displaystyle\\widehat\{d\}\_\{i,h\}\(\\alpha\):=α​d^i,h\.\\displaystyle:=\\alpha\\widehat\{d\}\_\{i,h\}\.\(44\)We use

Mi​\(α\)\\displaystyle M\_\{i\}\(\\alpha\)=maxh⁡\|di,h​\(α\)\|,\\displaystyle=\\max\_\{h\}\|d\_\{i,h\}\(\\alpha\)\|,AREi⁡\(α\)\\displaystyle\\operatorname\{ARE\}\_\{i\}\(\\alpha\)=∑h\|di,h​\(α\)\|,\\displaystyle=\\sum\_\{h\}\|d\_\{i,h\}\(\\alpha\)\|,\(45\)hi⋆\\displaystyle h\_\{i\}^\{\\star\}=min⁡arg⁡maxh​\|di,h\|,\\displaystyle=\\min\\arg\\max\_\{h\}\|d\_\{i,h\}\|,si⋆\\displaystyle s\_\{i\}^\{\\star\}=sign⁡\(di,hi⋆\)\.\\displaystyle=\\operatorname\{sign\}\(d\_\{i,h\_\{i\}^\{\\star\}\}\)\.\(46\)The primary trajectory error is

NRMSEi,α=\(∑h\(di,h​\(α\)−α​d^i,h\)2∑hdi,h​\(α\)2\)1/2\.\\operatorname\{NRMSE\}\_\{i,\\alpha\}=\\left\(\\frac\{\\sum\_\{h\}\(d\_\{i,h\}\(\\alpha\)\-\\alpha\\widehat\{d\}\_\{i,h\}\)^\{2\}\}\{\\sum\_\{h\}d\_\{i,h\}\(\\alpha\)^\{2\}\}\\right\)^\{1/2\}\.\(47\)For the analytic controlled and neural\-network tangents, the horizon\-wise symmetric relative error is

ei,h​\(α\)=\|di,h​\(α\)−α​d^i,h\|\|di,h​\(α\)\|\+\|α​d^i,h\|\+10−30\.e\_\{i,h\}\(\\alpha\)=\\frac\{\|d\_\{i,h\}\(\\alpha\)\-\\alpha\\widehat\{d\}\_\{i,h\}\|\}\{\|d\_\{i,h\}\(\\alpha\)\|\+\|\\alpha\\widehat\{d\}\_\{i,h\}\|\+10^\{\-30\}\}\.\(48\)The empirical validity radius is defined on the tested scale grid by

αivalid=max⁡\{α:median1≤h≤H⁡ei,h​\(α\)≤0\.2\},\\alpha\_\{i\}^\{\\mathrm\{valid\}\}=\\max\\left\\\{\\alpha:\\operatorname\{median\}\_\{1\\leq h\\leq H\}e\_\{i,h\}\(\\alpha\)\\leq 0\.2\\right\\\},\(49\)with value zero when the set is empty\. Candidate radii are aggregated within each system before system\-level summaries are formed\. The language\-model stage reports the corresponding valid\-candidate fraction at every tested scale\.

Candidates, scales, horizons, and future branches are always treated as nested observations\. The controlled and neural\-network studies first aggregate candidate\-level quantities within independently generated training systems\. The Pythia study is a single\-seed scaling analysis: its nine model–dataset systems are fixed conditions rather than independent random replications, and its statistics are descriptive\. Experiment 2 preserves these system units while introducing 32 independently resampled future continuations per fixed state–shock pair\.

### E\.1Common Baselines and Tangent Ablations

The controlled and neural\-network stages of Experiment 1 explicitly construct the initial AdamW tangent

δ​xi,1:=Bt​ξi=\[δ​θi,t\+1δ​mi,tδ​vi,t\]\\delta x\_\{i,1\}:=B\_\{t\}\\xi\_\{i\}=\\begin\{bmatrix\}\\delta\\theta\_\{i,t\+1\}\\\\ \\delta m\_\{i,t\}\\\\ \\delta v\_\{i,t\}\\end\{bmatrix\}\(50\)and recursively propagateδ​xi,h\+1=At\+h​δ​xi,h\\delta x\_\{i,h\+1\}=A\_\{t\+h\}\\delta x\_\{i,h\}\. LetPθ​δ​x=δ​θP\_\{\\theta\}\\delta x=\\delta\\thetaand use the local shorthandδ​θi,h:=Pθ​δ​xi,h\\delta\\theta\_\{i,h\}:=P\_\{\\theta\}\\delta x\_\{i,h\}\. Define the parameter\-space probe gradient

c¯h:=∇F​\(θt\+hc\),ct\+h=Pθ⊤​c¯h\.\\bar\{c\}\_\{h\}:=\\nabla F\(\\theta^\{\\mathrm\{c\}\}\_\{t\+h\}\),\\qquad c\_\{t\+h\}=P\_\{\\theta\}^\{\\top\}\\bar\{c\}\_\{h\}\.\(51\)Every trajectory score below is the maximum absolute predicted response over1≤h≤H1\\leq h\\leq H\.

##### Full ISO\.

The complete analytic/JVP trajectory and its score are

ri,hISO=c¯h⊤​Pθ​Φt\+h,t\+1​Bt​ξi,SiISO=maxh⁡\|ri,hISO\|\.r\_\{i,h\}^\{\\mathrm\{ISO\}\}=\\bar\{c\}\_\{h\}^\{\\top\}P\_\{\\theta\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{i\},\\qquad S\_\{i\}^\{\\mathrm\{ISO\}\}=\\max\_\{h\}\|r\_\{i,h\}^\{\\mathrm\{ISO\}\}\|\.\(52\)

##### Temporal ablations\.

The implementations used in both mechanism\-validation stages are

ri,hno​\-​prop\\displaystyle r\_\{i,h\}^\{\\mathrm\{no\\text\{\-\}prop\}\}=c¯h⊤​δ​θi,1,\\displaystyle=\\bar\{c\}\_\{h\}^\{\\top\}\\delta\\theta\_\{i,1\},\(53\)ri,hinit​\-​θ\\displaystyle r\_\{i,h\}^\{\\mathrm\{init\\text\{\-\}\\theta\}\}=c¯h⊤​Pθ​Φt\+h,t\+1​Pθ⊤​δ​θi,1,\\displaystyle=\\bar\{c\}\_\{h\}^\{\\top\}P\_\{\\theta\}\\Phi\_\{t\+h,t\+1\}P\_\{\\theta\}^\{\\top\}\\delta\\theta\_\{i,1\},\(54\)ri,hclamp​\-​θ\\displaystyle r\_\{i,h\}^\{\\mathrm\{clamp\\text\{\-\}\\theta\}\}=c¯h⊤​\(∏s=t\+1t\+h−1\[As\]θ​θ\)​δ​θi,1,\\displaystyle=\\bar\{c\}\_\{h\}^\{\\top\}\\left\(\\prod\_\{s=t\+1\}^\{t\+h\-1\}\[A\_\{s\}\]\_\{\\theta\\theta\}\\right\)\\delta\\theta\_\{i,1\},\(55\)ri,hfrozen​\-​dyn\\displaystyle r\_\{i,h\}^\{\\mathrm\{frozen\\text\{\-\}dyn\}\}=c¯h⊤​Pθ​At\+1h−1​δ​xi,1,\\displaystyle=\\bar\{c\}\_\{h\}^\{\\top\}P\_\{\\theta\}A\_\{t\+1\}^\{h\-1\}\\delta x\_\{i,1\},\(56\)ri,hfrozen​\-​readout\\displaystyle r\_\{i,h\}^\{\\mathrm\{frozen\\text\{\-\}readout\}\}=c¯1⊤​δ​θi,h\.\\displaystyle=\\bar\{c\}\_\{1\}^\{\\top\}\\delta\\theta\_\{i,h\}\.\(57\)Here,No Propagationreuses the initial parameter write at every horizon\.Initial Parameter Onlyremoves the two moment components ofδ​xi,1\\delta x\_\{i,1\}once and then applies the complete time\-varying dynamics\.Clamped Parameter Statesets the moment tangent blocks to zero before and after every transition, retaining only\[As\]θ​θ\[A\_\{s\}\]\_\{\\theta\\theta\}\.Frozen Dynamicsrepeatedly applies the first future Jacobian, andFrozen Readoutappliesc¯1\\bar\{c\}\_\{1\}to the correctly propagated parameter tangent at every horizon\.

##### Scalar baselines\.

The exact immediate\-response oracle and the static scores are

Si1​s​t​e​p\\displaystyle S\_\{i\}^\{\\mathrm\{1step\}\}=\|di,1​\(1\)\|,\\displaystyle=\|d\_\{i,1\}\(1\)\|,Sigrad\\displaystyle S\_\{i\}^\{\\mathrm\{grad\}\}=‖ξi‖2,\\displaystyle=\\\|\\xi\_\{i\}\\\|\_\{2\},\(58\)Siwrite\\displaystyle S\_\{i\}^\{\\mathrm\{write\}\}=‖δ​θi,1‖2,\\displaystyle=\\\|\\delta\\theta\_\{i,1\}\\\|\_\{2\},Sicurv\\displaystyle S\_\{i\}^\{\\mathrm\{curv\}\}=\|ξi⊤​Href​ξi\|,\\displaystyle=\|\\xi\_\{i\}^\{\\top\}H\_\{\\mathrm\{ref\}\}\\xi\_\{i\}\|,\(59\)Sinorm\\displaystyle S\_\{i\}^\{\\mathrm\{norm\}\}=maxh⁡‖c¯h‖2​‖δ​θi,h‖2,\\displaystyle=\\max\_\{h\}\\\|\\bar\{c\}\_\{h\}\\\|\_\{2\}\\\|\\delta\\theta\_\{i,h\}\\\|\_\{2\},\(60\)Href\\displaystyle H\_\{\\mathrm\{ref\}\}=1R​∑r=1R∇2ℓ​\(θt,ℬrref\)\.\\displaystyle=\\frac\{1\}\{R\}\\sum\_\{r=1\}^\{R\}\\nabla^\{2\}\\ell\(\\theta\_\{t\};\\mathcal\{B\}\_\{r\}^\{\\mathrm\{ref\}\}\)\.\(61\)The curvature score therefore uses the shock direction in the average reference\-minibatch Hessian\. The norm\-product baseline removes directional alignment while retaining the horizon\-wise state and readout norms\. For the readout\-order null, one shared permutationπ\\piis applied to all candidates in a system and

Si,πshuffle=maxh⁡\|c¯π⁡\(h\)⊤​δ​θi,h\|\.S\_\{i,\\pi\}^\{\\mathrm\{shuffle\}\}=\\max\_\{h\}\|\\bar\{c\}\_\{\\pi\(h\)\}^\{\\top\}\\delta\\theta\_\{i,h\}\|\.\(62\)We use 100 independently drawn permutations per system\.

### E\.2Experiment 1: Finite\-Horizon Mechanism Validation

#### E\.2\.1Controlled Quadratic Systems

##### Systems and protocol\.

Each minibatch loss is

ℓs​\(θ\)=12​θ⊤​Ds​θ\+12​r​‖Us⊤​θ‖22\+qs⊤​θ,\\ell\_\{s\}\(\\theta\)=\\frac\{1\}\{2\}\\theta^\{\\top\}D\_\{s\}\\theta\+\\frac\{1\}\{2r\}\\\|U\_\{s\}^\{\\top\}\\theta\\\|\_\{2\}^\{2\}\+q\_\{s\}^\{\\top\}\\theta,\(63\)whereDsD\_\{s\}is positive diagonal,Us∈ℝd×rU\_\{s\}\\in\\mathbb\{R\}^\{d\\times r\},d=512d=512, andr=16r=16\. For condition multiplierκ∈\{1,4,16\}\\kappa\\in\\\{1,4,16\\\}, independent innovations are sampled as

D~s,j\\displaystyle\\widetilde\{D\}\_\{s,j\}∼Unif⁡\(0\.05,0\.25​κ\),\\displaystyle\\sim\\operatorname\{Unif\}\(0\.05,0\.25\\kappa\),\(64\)U~s,j​k\\displaystyle\\widetilde\{U\}\_\{s,jk\}∼𝒩⁡\(0,1/d\),\\displaystyle\\sim\\mathcal\{N\}\(0,1/d\),\(65\)q~s,j\\displaystyle\\widetilde\{q\}\_\{s,j\}∼𝒩⁡\(0,0\.052\)\.\\displaystyle\\sim\\mathcal\{N\}\(0,0\.05^\{2\}\)\.\(66\)Writingρ=0\.85\\rho=0\.85, the temporally correlated sequence is generated by

Ds\\displaystyle D\_\{s\}=ρ​Ds−1\+\(1−ρ\)​D~s,\\displaystyle=\\rho D\_\{s\-1\}\+\(1\-\\rho\)\\widetilde\{D\}\_\{s\},\(67\)Us\\displaystyle U\_\{s\}=ρ​Us−1\+1−ρ2​U~s,\\displaystyle=\\rho U\_\{s\-1\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,\\widetilde\{U\}\_\{s\},\(68\)qs\\displaystyle q\_\{s\}=ρ​qs−1\+1−ρ2​q~s,\\displaystyle=\\rho q\_\{s\-1\}\+\\sqrt\{1\-\\rho^\{2\}\}\\,\\widetilde\{q\}\_\{s\},\(69\)with the first batch equal to its innovation\. The condition multiplier cycles through\{1,4,16\}\\\{1,4,16\\\}with the system index, andθ0∼𝒩⁡\(0,0\.12​I\)\\theta\_\{0\}\\sim\\mathcal\{N\}\(0,0\.1^\{2\}I\)\. AdamW uses

η=2×10−3,β1=0\.9,β2=0\.999,ϵ=10−8,λ=0\.01\.\\begin\{gathered\}\\eta=2\\times 10^\{\-3\},\\quad\\beta\_\{1\}=0\.9,\\quad\\beta\_\{2\}=0\.999,\\\\ \\epsilon=10^\{\-8\},\\quad\\lambda=0\.01\.\\end\{gathered\}\(70\)All calculations use double precision\. Each independently initialized system is advanced for 40 burn\-in steps\. Four reference minibatches define the control gradient, 16 candidate minibatches define shocks, and a shared future sequence is followed forH=32H=32\. Four seeds and eight systems per seed give 32 independent base systems\. Standard and anisotropic probes are paired views of each base system and are not counted as separate replications\.

##### Probe constructions\.

The main controlled experiment uses two paired probes\. The*standard probe*is sampled independently from the same quadratic family, with linear standard deviation0\.020\.02\. The*anisotropic probe*starts from that draw, sets its linear term to zero, and replaces its diagonal by

DFaniso\\displaystyle D\_\{F\}^\{\\mathrm\{aniso\}\}=DF​Diag⁡\(w\),\\displaystyle=D\_\{F\}\\operatorname\{Diag\}\(w\),\(71\)\{wj\}j=1d\\displaystyle\\\{w\_\{j\}\\\}\_\{j=1\}^\{d\}=perm⁡\(\{32\(j−1\)/\(d−1\)\}j=1d\),\\displaystyle=\\operatorname\{perm\}\\\!\\left\(\\left\\\{32^\{\(j\-1\)/\(d\-1\)\}\\right\\\}\_\{j=1\}^\{d\}\\right\),\(72\)while retaining the sampled low\-rank factor\. This is thereadout\_activeregime in the code\.

The exact\-one\-step\-matched supplement uses the standard probe and a separate*rotating\-readout probe*\. Letθhmc\\theta^\{\\mathrm\{c\}\}\_\{h\_\{m\}\}andθhm\+1c\\theta^\{\\mathrm\{c\}\}\_\{h\_\{m\}\+1\}be the two central states of theHH\-step common control trajectory and set

μ\\displaystyle\\mu=12​\(θhmc\+θhm\+1c\),\\displaystyle=\\frac\{1\}\{2\}\\left\(\\theta^\{\\mathrm\{c\}\}\_\{h\_\{m\}\}\+\\theta^\{\\mathrm\{c\}\}\_\{h\_\{m\}\+1\}\\right\),\(73\)W\\displaystyle W=Diag⁡\(perm⁡\(\{32\(j−1\)/\(d−1\)\}j=1d\)\)\.\\displaystyle=\\operatorname\{Diag\}\\\!\\left\(\\operatorname\{perm\}\\\!\\left\(\\left\\\{32^\{\(j\-1\)/\(d\-1\)\}\\right\\\}\_\{j=1\}^\{d\}\\right\)\\right\)\.\(74\)The fixed probe is

Frot​\(θ\)=12​\(θ−μ\)⊤​W​\(θ−μ\),c¯h=W⁡\(θhc−μ\)\.F\_\{\\mathrm\{rot\}\}\(\\theta\)=\\frac\{1\}\{2\}\(\\theta\-\\mu\)^\{\\top\}W\(\\theta\-\\mu\),\\qquad\\bar\{c\}\_\{h\}=W\(\\theta\_\{h\}^\{\\mathrm\{c\}\}\-\\mu\)\.\(75\)It depends only on the common control trajectory\. Centering it between the two middle states makes the readout direction change as the control trajectory passes the probe center\. The standard, anisotropic, and rotating\-readout probes are therefore three distinct constructions\.

##### Local fidelity and finite\-scale error\.

We useα∈\{1/32,1/16,1/8,1/4,1/2,1\}\\alpha\\in\\\{1/32,1/16,1/8,1/4,1/2,1\\\}\. Table[5](https://arxiv.org/html/2608.19762#A5.T5)reports system\-level medians\. At the smallest scale, NRMSE is0\.01170\.0117under the standard probe and0\.01730\.0173under the anisotropic probe\. The median local trajectory NRMSE pooled across the prespecified local range is0\.04500\.0450, and median horizon\-wise sign agreement is one\. The median empirical validity radius is0\.50\.5\. Fitting

log⁡\|di,h​\(α\)−α​d^i,h\|=ai,h\+pi,h​log⁡α\\log\|d\_\{i,h\}\(\\alpha\)\-\\alpha\\widehat\{d\}\_\{i,h\}\|=a\_\{i,h\}\+p\_\{i,h\}\\log\\alpha\(76\)over the local scales yields an approximately quadratic remainder, with system\-levelR2R^\{2\}values essentially one\. The independently evaluated recursive state\-error identity has maximum residual3\.11×10−173\.11\\times 10^\{\-17\}, providing an implementation audit of the recursive decomposition\.

Table 5:Controlled\-stage signed\-trajectory fidelity in Experiment 1\. Entries are medians over 32 independent systems after candidate\-level aggregation\.
##### Natural candidate ranking\.

For unmodified candidate minibatches, Full ISO reaches median Spearman correlation0\.8350\.835for the standard probe and0\.7620\.762for the anisotropic probe\. The corresponding exact one\-step correlations are0\.7740\.774and0\.7340\.734\. Their difference is modest because natural immediate and future responses can be strongly correlated\. Static gradient, parameter\-write, and curvature summaries attain correlations in the approximate range0\.330\.33–0\.480\.48\.

##### Exact\-one\-step matching\.

Letaabe the one\-step tangent functional satisfyinga⊤​ξ=c¯1⊤​Pθ​Bt​ξa^\{\\top\}\\xi=\\bar\{c\}\_\{1\}^\{\\top\}P\_\{\\theta\}B\_\{t\}\\xi\. In the implementation, the diagonal write\-in map is recovered by applyingBtB\_\{t\}to the all\-ones direction, soa=c¯1⊙Pθ​Bt​𝟏a=\\bar\{c\}\_\{1\}\\odot P\_\{\\theta\}B\_\{t\}\\mathbf\{1\}\. For natural residualri=gi−g¯r\_\{i\}=g\_\{i\}\-\\bar\{g\}, define

zi\\displaystyle z\_\{i\}=ri−a​a⊤​ri‖a‖22,\\displaystyle=r\_\{i\}\-a\\frac\{a^\{\\top\}r\_\{i\}\}\{\\\|a\\\|\_\{2\}^\{2\}\},\(77\)bi\\displaystyle b\_\{i\}=si​μ1​a‖a‖22,si=\(−1\)i,\\displaystyle=s\_\{i\}\\mu\_\{1\}\\frac\{a\}\{\\\|a\\\|\_\{2\}^\{2\}\},\\qquad s\_\{i\}=\(\-1\)^\{i\},\(78\)z~i\\displaystyle\\widetilde\{z\}\_\{i\}=\{4​‖bi‖2​zi/‖zi‖2,‖zi‖2\>0,0,otherwise,\\displaystyle=\\begin\{cases\}4\\\|b\_\{i\}\\\|\_\{2\}z\_\{i\}/\\\|z\_\{i\}\\\|\_\{2\},&\\\|z\_\{i\}\\\|\_\{2\}\>0,\\\\ 0,&\\text\{otherwise\},\\end\{cases\}\(79\)ξ~i\\displaystyle\\widetilde\{\\xi\}\_\{i\}=bi\+z~i\.\\displaystyle=b\_\{i\}\+\\widetilde\{z\}\_\{i\}\.\(80\)The common target is

μ1=0\.25​medianj​\|a⊤​rj\|\.\\mu\_\{1\}=0\.25\\operatorname\{median\}\_\{j\}\|a^\{\\top\}r\_\{j\}\|\.\(81\)Thusa⊤​ξ~i=si​μ1a^\{\\top\}\\widetilde\{\\xi\}\_\{i\}=s\_\{i\}\\mu\_\{1\}, while the candidate\-specific component lies in the nullspace of the immediate output functional and has four times the norm of the common component\.

We then calibrate one scalarγi≥0\\gamma\_\{i\}\\geq 0per candidate using the exact nonlinear AdamW update: LetΘ\+​\(x,g\)\\Theta^\{\+\}\(x,g\)denote the parameter component after one exact AdamW update from statexxwith gradientgg\. The calibration equation is

F⁡\(Θ\+​\(xt,g¯\+γi​ξ~i\)\)−F⁡\(Θ\+​\(xt,g¯\)\)=si​μ1\.F\\\!\\left\(\\Theta^\{\+\}\(x\_\{t\},\\bar\{g\}\+\\gamma\_\{i\}\\widetilde\{\\xi\}\_\{i\}\)\\right\)\-F\\\!\\left\(\\Theta^\{\+\}\(x\_\{t\},\\bar\{g\}\)\\right\)=s\_\{i\}\\mu\_\{1\}\.\(82\)Starting with\[0,1\]\[0,1\], the upper endpoint is doubled until the target is bracketed, up toγ=64\\gamma=64; 64 bisection iterations are then applied\. A match is accepted when the signed residual is at most5×10−8​μ15\\times 10^\{\-8\}\\mu\_\{1\}\. All candidates across the 64 system–probe instances \(32 systems evaluated under two probe constructions\) are successfully calibrated\. This procedure uses the control state, the one\-step functional, and natural residual directions, but no response ath\>1h\>1\.

Table[6](https://arxiv.org/html/2608.19762#A5.T6)shows that the matched immediate response is numerically constant whereas future magnitudes remain heterogeneous\. The median ratioMi/\|di,1\|M\_\{i\}/\|d\_\{i,1\}\|is6\.216\.21under the standard probe and4\.544\.54under the rotating\-readout probe; corresponding parameter transient gains are approximately7\.77\.7\.

Table 6:Controlled\-stage exact\-one\-step\-matched stress test in Experiment 1\. CV is computed over 16 candidates within a system; entries are medians across 32 independent systems, with IQRs shown for the matching diagnostics\.Table 7:Median Spearman correlation under exact\-one\-step matching\. The one\-step score is tied by construction\.
##### Readout controls\.

The standard readout remains close to its initial direction, with median minimum cosine0\.9140\.914\. The rotating\-readout probe produces substantial horizon\-dependent rotation while keeping a fixed scalar probe objective\. We additionally apply 100 common permutations to the readout sequence while preserving the propagated states\. Correct temporal alignment places Full ISO at the maximum observed percentile of the shuffled\-readout null in both regimes\. This control distinguishes directional state–readout alignment from a product of state and readout norms\.

##### State\-channel dependence and persistence\.

Let the exact post\-shock joint\-state deviation be

Δ​xt\+1=\(Δ​θt\+1,Δ​mt,Δ​vt\)\.\\Delta x\_\{t\+1\}=\(\\Delta\\theta\_\{t\+1\},\\Delta m\_\{t\},\\Delta v\_\{t\}\)\.\(83\)For\(a,b,c\)∈\{0,1\}3\(a,b,c\)\\in\\\{0,1\\\}^\{3\}, we form

xt\+1a​b​c=xt\+1c\+\(a​Δ​θt\+1,b​Δ​mt,c​Δ​vt\)x\_\{t\+1\}^\{abc\}=x\_\{t\+1\}^\{\\mathrm\{c\}\}\+\(a\\Delta\\theta\_\{t\+1\},b\\Delta m\_\{t\},c\\Delta v\_\{t\}\)\(84\)and follow the common future exactly\. Parameter\-only interventions peak early, first\-moment interventions peak later and usually carry the largest isolated response, and second\-moment deviations strongly modulate the response when coupled withmm\. These hybrid counterfactuals can lie away from states reached by ordinary AdamW; we use them to measure channel dependence and interaction\.

Table 8:Complete post\-shock state\-channel intervention in the controlled stage of Experiment 1\. Entries are medians after first aggregating selected candidates within each of the 32 independent systems\. Channel order is\(θ,m,v\)\(\\theta,m,v\)\.In matched\-first\-displacement sweeps, the first future parameter displacement is held fixed whileβ1\\beta\_\{1\}orβ2\\beta\_\{2\}is varied\. Increasingβ1\\beta\_\{1\}from0\.50\.5to0\.990\.99moves the momentum\-channel extremum from approximately horizon 5 toward the end of the measured interval and substantially increases ARE\. A horizon\-128128extension confirms the ordering of time scales: parameter deviations act earliest, momentum dominates intermediate delays, and second\-moment effects persist longest\. This rules out a purely larger first\-update explanation for the memory effect\.

Table 9:Controlled\-stage matched\-first\-displacement persistence sweep in Experiment 1\. The first future parameter displacement is held fixed within each channel\.

#### E\.2\.2Nonconvex Neural Networks

##### Models, data, and replication\.

We use CIFAR\-10 with a two\-hidden\-layer width\-256 MLP with GELU activations \(855,050855\{,\}050parameters\) and a three\-layer width\-32 CNN with ReLU activations \(94,53894\{,\}538parameters\)\. Training batches contain 128 examples and fixed test probes contain 256 examples\. Images use standard CIFAR\-10 channel normalization\. AdamW uses learning rate2×10−42\\times 10^\{\-4\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, and weight decay0\.010\.01; state and tangent calculations use double precision\.

For each architecture, four top\-level seeds \(2026–2029\) and four independent initializations per seed give 16 independent systems, or 32 total\. Each system receives 100 burn\-in updates, four reference batches, 12 candidate batches, 11 common\-future batches, and one fixed probe, givingH=12H=12and 384 candidate shocks\. We useα∈\{0\.0625,0\.125,0\.25,0\.5,1\}\\alpha\\in\\\{0\.0625,0\.125,0\.25,0\.5,1\\\}\.

##### Trajectory fidelity\.

Table[10](https://arxiv.org/html/2608.19762#A5.T10)gives the complete finite\-scale curve for the neural\-network stage\. The pooled median local NRMSE is0\.06780\.0678and median local sign agreement is one\. The median empirical validity radius is0\.750\.75for CNN–ReLU \(IQR\[0\.5,1\]\[0\.5,1\]\) and0\.50\.5for MLP–GELU \(IQR\[0\.375,0\.5\]\[0\.375,0\.5\]\)\. The recursive state\-error identity has maximum residual9\.33×10−179\.33\\times 10^\{\-17\}\.

Table 10:Neural\-network finite\-horizon tangent fidelity in Experiment 1\. Entries are medians over independent systems after aggregating candidates within systems\.
##### Finite\-scale error behavior\.

The local absolute\-error exponent is fitted overα∈\{0\.0625,0\.125,0\.25\}\\alpha\\in\\\{0\.0625,0\.125,0\.25\\\}\. For MLP–GELU, its median remains in\[2\.001,2\.005\]\[2\.001,2\.005\]at all horizons, withR2≈1R^\{2\}\\approx 1\. For CNN–ReLU, the median exponent is2\.0172\.017,1\.9161\.916,1\.5731\.573, and1\.1201\.120at horizons11,44,88, and1212\. ReLU activation\-sign differences are nonzero and increase with scale \(ath=8h=8, their median rises from1\.79×10−41\.79\\times 10^\{\-4\}atα=\.0625\\alpha=\.0625to2\.82×10−32\.82\\times 10^\{\-3\}atα=1\\alpha=1\)\. The degradation coincides with increasing activation\-pattern changes, consistent with a growing switching defect at later horizons\.

##### Ranking future absolute influence\.

Within each system, we rank candidates byMi=maxh⁡\|di,h​\(1\)\|M\_\{i\}=\\max\_\{h\}\|d\_\{i,h\}\(1\)\|\. Table[11](https://arxiv.org/html/2608.19762#A5.T11)reports the main ablations for the neural\-network stage\. Full ISO obtains pooled median Spearman0\.8320\.832\. It exceeds the exact one\-step oracle and static norms in both architectures, whereas frozen dynamics preserves much of the ordering over the shortH=12H=12horizon\. Paired Wilcoxon tests use the independent system as the unit and Holm correction within each architecture\. For CNN–ReLU, Full ISO significantly exceeds all alternatives except frozen dynamics\. For MLP–GELU, its advantages over exact one\-step, gradient norm, parameter write, curvature, norm product, and frozen readout survive correction; its differences from the remaining propagation ablations are positive in median but not significant after correction\.

Table 11:Median within\-system Spearman correlation with future magnitude in the neural\-network stage of Experiment 1\.
##### Readout order and signed extrema\.

Under 100 shared permutations of future readout order, CNN–ReLU Full ISO decreases from0\.8880\.888to median0\.7760\.776\(p=0\.0019p=0\.0019\), with the true score at the median 99th percentile of the system\-specific null\. For MLP–GELU, the corresponding values are0\.7620\.762and0\.7130\.713\(p=0\.191p=0\.191\); its nearby readouts are sufficiently similar that permutation preserves much of the ordering\. Across the 96 exact intervention trajectories included in this diagnostic, 57 extrema are positive and 39 are negative\. A one\-sided positive\-response analysis would therefore omit40\.6%40\.6\\%of these extremal events\.

##### State\-channel dependence\.

Table[12](https://arxiv.org/html/2608.19762#A5.T12)reports exact hybrid\-state interventions in the neural\-network stage\. The parameter\-only response peaks early, while the first\-moment\-only response peaks later and has substantially greater accumulated magnitude\. The second\-moment\-only response is small, but addingvvto\(θ,m\)\(\\theta,m\)slightly reduces the median response in both architectures\. The conditionalvvcontrast is negative in 15 of 16 CNN systems and 13 of 16 MLP systems\. This is evidence thatvvmodulates a coupled parameter–moment state, not that it has a uniquely defined negative additive contribution\.

Table 12:Neural\-network exact state\-channel interventions in Experiment 1\. Values are median ARE relative to the full111111intervention; bit order is\(θ,m,v\)\(\\theta,m,v\)\.
##### Matched\-displacement persistence\.

Holding the first future parameter displacement fixed, increasingβ1\\beta\_\{1\}from0\.50\.5to0\.990\.99multipliesMMby8\.478\.47\(CNN\) and10\.8010\.80\(MLP\), and multiplies ARE by14\.1714\.17and11\.9611\.96\. The median extremum moves from approximatelyh=3\.5h=3\.5–44toh=12h=12\. Increasingβ2\\beta\_\{2\}from0\.90\.9to0\.99990\.9999multiplies ARE by4\.244\.24and2\.752\.75, respectively\. Since the first displacement is matched, these differences isolate persistence from immediate update magnitude\.

#### E\.2\.3Pretrained Language Models

##### Models, domains, and warm optimizer state\.

We use Pythia\-410M, Pythia\-1B, and Pythia\-1\.4B with WikiText\-103, OpenWebText, and CodeParrot, giving nine model–dataset systems\. From each domain we prepare 2,048 token sequences of length 129\. The training batch size is one and the fixed probe contains two sequences\. AdamW uses learning rate10−510^\{\-5\},β1=0\.9\\beta\_\{1\}=0\.9,β2=0\.999\\beta\_\{2\}=0\.999,ϵ=10−8\\epsilon=10^\{\-8\}, and weight decay0\.010\.01\. Starting from pretrained weights, every system is first continued for 500 updates\. This creates a nontrivial trained AdamW moment state and prevents a zero\-moment cold start from dominating the derivative\. The intervention uses two reference batches, eight candidate shocks, seven common\-future batches, andH=8H=8\. We evaluateα∈\{0\.0625,0\.125,0\.25,0\.5,1\}\\alpha\\in\\\{0\.0625,0\.125,0\.25,0\.5,1\\\}under seed 2026\.

##### Numerical ISO directional response\.

Directly differencing nearly equal scalar cross\-entropies is inaccurate at this scale\. For candidateii, horizonhh, and probe examplenn, we therefore compute the centered logit derivative

δ​zi,h,n​\(ε\)=zi,h,n\+​\(ε\)−zi,h,n−​\(ε\)2​ε\\delta z\_\{i,h,n\}\(\\varepsilon\)=\\frac\{z^\{\+\}\_\{i,h,n\}\(\\varepsilon\)\-z^\{\-\}\_\{i,h,n\}\(\\varepsilon\)\}\{2\\varepsilon\}\(85\)and apply the exact control cross\-entropy differential

d^i,h​\(ε\)=1Ntok​∑n⟨softmax⁡\(zh,n0\)−eyn,δ​zi,h,n​\(ε\)⟩\.\\widehat\{d\}\_\{i,h\}\(\\varepsilon\)=\\frac\{1\}\{N\_\{\\mathrm\{tok\}\}\}\\sum\_\{n\}\\left\\langle\\operatorname\{softmax\}\(z^\{0\}\_\{h,n\}\)\-e\_\{y\_\{n\}\},\\delta z\_\{i,h,n\}\(\\varepsilon\)\\right\\rangle\.\(86\)The resulting trajectory estimates the directional response

dd​α​F​\(θi,t\+h​\(α\)\)\|α=0,\\left\.\\frac\{d\}\{d\\alpha\}F\(\\theta\_\{i,t\+h\}\(\\alpha\)\)\\right\|\_\{\\alpha=0\},\(87\)which equalsct\+h⊤​Φt\+h,t\+1​Bt​ξic\_\{t\+h\}^\{\\top\}\\Phi\_\{t\+h,t\+1\}B\_\{t\}\\xi\_\{i\}under the theorem’s local conditions\. We denote this numerical quantity byISO Tangent \(FD\)\. Unlike the controlled and neural\-network stages above, this stage estimates the end\-to\-end tangent response without separately materializingBtB\_\{t\}, everyAsA\_\{s\}, and the readoutct\+hc\_\{t\+h\}\. The exact finite\-scale target remains the FP64\-reduced difference between the shock and control probe losses\. We testε∈\{\.5,\.25,\.125,\.0625,\.03125,\.015625\}\\varepsilon\\in\\\{\.5,\.25,\.125,\.0625,\.03125,\.015625\\\}\. Adjacent derivative trajectories are compared with the symmetric NRMSE

EFD​\(ε,ε/2\)=‖d^​\(ε\)−d^​\(ε/2\)‖212​\(‖d^​\(ε\)‖2\+‖d^​\(ε/2\)‖2\)\+10−30\.E\_\{\\mathrm\{FD\}\}\(\\varepsilon,\\varepsilon/2\)=\\frac\{\\\|\\widehat\{d\}\(\\varepsilon\)\-\\widehat\{d\}\(\\varepsilon/2\)\\\|\_\{2\}\}\{\\tfrac\{1\}\{2\}\(\\\|\\widehat\{d\}\(\\varepsilon\)\\\|\_\{2\}\+\\\|\\widehat\{d\}\(\\varepsilon/2\)\\\|\_\{2\}\)\+10^\{\-30\}\}\.\(88\)The smaller scale in the most consistent adjacent pair is selected, and a candidate is identifiable when the selected discrepancy is at most0\.250\.25\. All 72 candidates pass\. Median selected consistency is0\.003780\.00378and the maximum accepted value is0\.227650\.22765\. Selected smaller scales \.25, \.125, \.0625, \.03125, and \.015625 occur for 35, 15, 14, 4, and 4 candidates\.

##### Finite\-scale fidelity and validity\.

Table[13](https://arxiv.org/html/2608.19762#A5.T13)reports all shock scales\. The empirical validity criterion is the median horizon\-wise symmetric relative error at most0\.20\.2\. Valid fractions are93\.1%93\.1\\%,97\.2%97\.2\\%,87\.5%87\.5\\%,73\.6%73\.6\\%, and55\.6%55\.6\\%from the smallest to the largest scale\. The slight first\-pair nonmonotonicity reflects numerical variation around small responses\. The result characterizes the finite\-scale range over which the local mechanism provides accurate pointwise predictions\.

Table 13:Language\-model finite\-scale trajectory fidelity in Experiment 1 over 72 candidates\.
##### Model\- and domain\-scale behavior\.

Table[14](https://arxiv.org/html/2608.19762#A5.T14)shows no systematic local degradation from 410M to 1\.4B\. Pooling scales, local NRMSE is0\.04260\.0426on CodeParrot,0\.10940\.1094on OpenWebText, and0\.11610\.1161on WikiText\-103\. Atα=1\\alpha=1,87\.5%87\.5\\%,37\.5%37\.5\\%, and41\.7%41\.7\\%of the candidates in these domains satisfy the validity criterion, demonstrating that finite\-scale range varies more by domain than by model size in these conditions\. The overall median fitted local error exponent is1\.9211\.921\. Per\-system exponents are\(1\.878,1\.089,0\.658\)\(1\.878,1\.089,0\.658\)at 410M,\(1\.988,1\.916,1\.936\)\(1\.988,1\.916,1\.936\)at 1B, and\(1\.992,1\.951,2\.025\)\(1\.992,1\.951,2\.025\)at 1\.4B for CodeParrot, OpenWebText, and WikiText\-103, respectively\. Thus, the overall local trend is near quadratic, with the main deviations concentrated in the 410M OpenWebText and WikiText\-103 conditions\.

Table 14:Language\-model scaling summary in Experiment 1\. Ranking correlations are medians over the three data\-domain conditions at each scale\.
##### Ranking at full shock scale\.

The exact one\-step baseline is

Si1​s​t​e​p=\|di,1​\(1\)\|,S\_\{i\}^\{\\mathrm\{1step\}\}=\|d\_\{i,1\}\(1\)\|,\(89\)which executes the complete nonlinear shock and control updates and evaluates both on the fixed probe, but sees no subsequent common\-future propagation\. It is therefore a strong counterfactual oracle that directly observes the immediate fixed\-probe effect\. Table[15](https://arxiv.org/html/2608.19762#A5.T15)reports the system\-wise results\.ISO Tangent \(FD\)is positive in all nine conditions and exceeds gradient and parameter\-write norms in eight\. It exceeds exact one\-step in five conditions, while exact one\-step is higher in four; their medians are equal\. The two statistics capture complementary information: one\-step observes the immediate output, whereasISO Tangent \(FD\)represents the propagated signed trajectory\.

Table 15:Language\-model within\-system Spearman correlation in Experiment 1 withMi=maxh⁡\|di,h​\(1\)\|M\_\{i\}=\\max\_\{h\}\|d\_\{i,h\}\(1\)\|\. Each condition contains eight candidates\.As a descriptive secondary analysis, ranks normalized within each condition give pooled correlations0\.7910\.791for one\-step and0\.7330\.733for ISO\-FD\. After linearly controlling one\-step rank, the partial ISO\-FD–target correlation is0\.3100\.310; adding ISO\-FD rank raises descriptiveR2R^\{2\}from0\.6260\.626to0\.6620\.662\. Because candidates are nested within nine fixed conditions, these values are descriptive and are not treated as 72 independent replications\.

The exact one\-step response is a strong baseline in the pretrained\-language\-model setting and exceeds ISO\-FD in several conditions\. This does not contradict the finite\-horizon mechanism: the one\-step\-matched controlled experiments show that immediate response is not sufficient in general, while the present scaling experiment tests whether the ISO tangent remains faithful at larger model scale\.

##### Signed extrema\.

At full scale, 51 of 72 candidate extrema are positive and 21 are negative\. ISO\-FD recovers the extremum sign for 65 candidates \(90\.3%90\.3\\%\)\. Counts by domain are\(24,0\)\(24,0\)for CodeParrot,\(13,11\)\(13,11\)for OpenWebText, and\(14,10\)\(14,10\)for WikiText\-103, where each pair is \(positive, negative\)\. This domain dependence reinforces the signed formulation: positive loss excursions are one subclass of the broader finite\-horizon response\.

##### State\-channel interactions\.

For the single candidate selected per system in this diagnostic, we evaluate the same eight post\-shock hybrid states used in the smaller experiments\. The most striking behavior occurs on WikiText\-103: injectingmmwithout its matchedvvproduces maximum absolute responses50\.32250\.322,8\.57488\.5748, and26\.33926\.339at 410M, 1B, and 1\.4B, whereas injecting the coupled\(m,v\)\(m,v\)deviations gives0\.010840\.01084,0\.008860\.00886, and0\.008470\.00847\. Full\-state responses are similarly of order10−210^\{\-2\}\. These extreme isolated\-momentum hybrids demonstrate strong cross\-channel coupling and off\-trajectory sensitivity\. We therefore interpret the intervention family jointly as an interaction probe\.

##### Computational cost and scope\.

Median complete runtime per system, including 500 continuation updates, eight shocks, the finite\-difference grid, exact trajectories, and channel interventions, is approximately 464 seconds at 410M, 645 seconds at 1B, and 941 seconds at 1\.4B\. The nine measured runtimes sum to approximately 5,849 seconds and were parallelized over four GPUs\. With one seed, this experiment provides a descriptive scaling comparison over three sizes of the Pythia family and three data domains\. Multi\-seed replication of the same mechanism is provided by the controlled and neural\-network stages of Experiment 1\.

### E\.3Experiment 2: Prospective Structure Under Unknown Futures

##### Repeated\-future construction\.

Experiment 1 conditions on one realized common future\. Experiment 2 instead holds the pre\-shock history, post\-burn\-in AdamW state, reference minibatches, candidate shock, fixed probe, optimizer configuration, and horizon fixed, and resamples only the unseen future minibatches\. For each candidateiiand future branchk∈\{1,…,K\}k\\in\\\{1,\\ldots,K\\\}, the control and shock trajectories share the same branch, so that

di,h\(k\)=F⁡\(θi,t\+hs,\(k\)\)−F⁡\(θt\+hc,\(k\)\)d\_\{i,h\}^\{\(k\)\}=F\(\\theta\_\{i,t\+h\}^\{\\mathrm\{s\},\(k\)\}\)\-F\(\\theta\_\{t\+h\}^\{\\mathrm\{c\},\(k\)\}\)\(90\)continues to isolate the response to the initiating shock within that branch\. We useK=32K=32throughout\.

In the controlled systems, the original burn\-in state, four reference minibatches, 16 candidates, probes, andH=32H=32are unchanged\. The latent state of the correlated quadratic process immediately before the future is held fixed and each branch resamples only subsequent innovations\. One branch is shared across all candidates and both probe views in a base system\. Thestandardandreadout\_activeresults below correspond to the standard and anisotropic probes from Experiment 1; the rotating\-readout probe is not used in this repeated\-future experiment\.

In the neural systems, the original CIFAR\-10 architecture, initialization, 100\-step burn\-in state, four reference minibatches, 12 candidates, fixed test probe, AdamW configuration, andH=12H=12are unchanged\. Each future branch independently resamples the subsequent CIFAR\-10 training minibatches and is shared across candidates and candidate modes within a system\.

##### Candidate families and response summaries\.

Both settings evaluate the original*natural*gradient residuals and an*exact\-one\-step\-matched*family\. The matched family reuses the construction in Eq\. \([82](https://arxiv.org/html/2608.19762#A5.E82)\): the common tangent target is0\.250\.25times the median natural one\-step magnitude, the candidate\-specific component is placed in the nullspace of the immediate output functional with four times the norm of the common component, and one nonnegative scalar per candidate is calibrated through the exact nonlinear AdamW update\. No response ath\>1h\>1and no resampled future branch is used in this construction\. All candidates calibrate successfully\. Table[18](https://arxiv.org/html/2608.19762#A5.T18)shows that the resulting exact\|d1\|\|d\_\{1\}\|values are constant to numerical precision\.

For each candidate–branch pair we record

Mi,k\\displaystyle M\_\{i,k\}:=maxh⁡\|di,h\(k\)\|,\\displaystyle:=\\max\_\{h\}\|d\_\{i,h\}^\{\(k\)\}\|,\(91\)Pi,k\+\\displaystyle P^\{\+\}\_\{i,k\}:=maxh⁡\[di,h\(k\)\]\+,\\displaystyle:=\\max\_\{h\}\[d\_\{i,h\}^\{\(k\)\}\]\_\{\+\},\(92\)Ai,k\\displaystyle A\_\{i,k\}:=∑h\|di,h\(k\)\|\.\\displaystyle:=\\sum\_\{h\}\|d\_\{i,h\}^\{\(k\)\}\|\.\(93\)MagnitudeMMis the primary response used in the main text;P\+P^\{\+\}andAAare secondary checks\. For a generic summaryRi,kR\_\{i,k\}, define

μiR:=1K​∑k=1KRi,k,\\mu\_\{i\}^\{R\}:=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}R\_\{i,k\},\(94\)and the within\-system prospective\-structure ratio

ΠH​\(R\)=Vari⁡\(μiR\)Vari⁡\(μiR\)\+𝔼i​\[Vark⁡\(Ri,k\)\]\.\\Pi\_\{H\}\(R\)=\\frac\{\\operatorname\{Var\}\_\{i\}\(\\mu\_\{i\}^\{R\}\)\}\{\\operatorname\{Var\}\_\{i\}\(\\mu\_\{i\}^\{R\}\)\+\\mathbb\{E\}\_\{i\}\[\\operatorname\{Var\}\_\{k\}\(R\_\{i,k\}\)\]\}\.\(95\)The implementation uses population normalization within the finite candidate and branch grids\.ΠH\\Pi\_\{H\}is therefore a protocol\-specific variance decomposition: it compares candidate\-specific variation in the repeated\-future mean with variation induced by resampling the future, and is not an information\-theoretic percentage or a universal fraction of predictable risk\. We additionally compute

ρbranch:=mediank⁡Spearman⁡\(R⋅,k,μR\),\\rho\_\{\\mathrm\{branch\}\}:=\\operatorname\{median\}\_\{k\}\\operatorname\{Spearman\}\(R\_\{\\cdot,k\},\\mu^\{R\}\),\(96\)which measures the stability of candidate ordering across individual future branches\.

##### Present\-time scores\.

The ranking target for a present\-time score is the repeated\-future conditional meanμiM\\mu\_\{i\}^\{M\}\. The scalar baselines are the exact one\-step absolute response, gradient norm, and initial parameter\-write norm from Eq\. \([58](https://arxiv.org/html/2608.19762#A5.E58)\)\. We add two ISO\-based scores that are computed without accessing any sampled future branch\. The*present\-frozen ISO*evaluates one local transition JacobianA¯t\\overline\{A\}\_\{t\}at the first post\-control state using only the current reference minibatches, freezes theh=1h=1probe readoutc¯1\\bar\{c\}\_\{1\}, and uses

SiPF=max1≤h≤H⁡\|c¯1⊤​Pθ​A¯th−1​Bt​ξi\|\.S\_\{i\}^\{\\mathrm\{PF\}\}=\\max\_\{1\\leq h\\leq H\}\\left\|\\bar\{c\}\_\{1\}^\{\\top\}P\_\{\\theta\}\\overline\{A\}\_\{t\}^\{\\,h\-1\}B\_\{t\}\\xi\_\{i\}\\right\|\.\(97\)The*reference\-surrogate ISO*instead constructs a deterministic future using only the current reference\-minibatch mean \(controlled systems\) or the current reference minibatches \(neural systems\), updates the resulting surrogate control state, readout, and tangent along that deterministic rollout, and takes the maximum absolute predicted response\. The distinction from the pathwise Full ISO in Experiment 1 is essential: neither prospective score uses a realized or resampled future minibatch\.

#### E\.3\.1Controlled Repeated Futures

Table[16](https://arxiv.org/html/2608.19762#A5.T16)reports the primary magnitude results\. Under the standard probe, natural candidates have medianΠ32​\(M\)=0\.801\\Pi\_\{32\}\(M\)=0\.801, but the exact one\-step response already correlates0\.8010\.801withμiM\\mu\_\{i\}^\{M\}\. After exact one\-step matching,Π32​\(M\)=0\.730\\Pi\_\{32\}\(M\)=0\.730and the median branch\-ranking correlation remains0\.9090\.909, despite the one\-step magnitude being tied to numerical precision\. The anisotropic probe gives the same qualitative conclusion, with matchedΠ32​\(M\)=0\.603\\Pi\_\{32\}\(M\)=0\.603and branch correlation0\.8950\.895\. In contrast, the simple present\-time scores do not recover the matched conditional\-mean ordering in this controlled setting: present\-frozen ISO correlations are−0\.066\-0\.066and0\.0310\.031for the standard and anisotropic probes\.

#### E\.3\.2Neural\-Network Repeated Futures

The neural results preserve the same protocol while replacing the controlled quadratic process by independently resampled CIFAR\-10 future minibatches\. Matched MLP–GELU yieldsΠ12​\(M\)=0\.921\\Pi\_\{12\}\(M\)=0\.921with branch\-ranking correlation0\.9550\.955; matched CNN–ReLU yields0\.7000\.700and0\.8720\.872\. Present\-time ISO scores are substantially more informative here than in the controlled systems\. Present\-frozen ISO correlates0\.9410\.941withμiM\\mu\_\{i\}^\{M\}for MLP–GELU and0\.7550\.755for CNN–ReLU, while reference\-surrogate ISO reaches0\.7520\.752and0\.8110\.811\. These results establish prospective relevance under the measured protocols, but they do not by themselves identify the conditions that make a present\-time representation accurate\.

Table 16:Primary repeated\-future magnitude results in Experiment 2\. Each cell reports the median across independent systems; brackets give the system\-level IQR\. Branchρ\\rhois the within\-system median correlation between an individual future\-branch ranking and the repeated\-future mean ranking\.Table 17:Present\-time Spearman correlation with the repeated\-future mean magnitudeμiM\\mu\_\{i\}^\{M\}in Experiment 2\. The exact one\-step score is tied in the matched family and its correlation is therefore undefined\.Table 18:Exact\-one\-step matching audit for Experiment 2\. CV is computed within each candidate family; entries are medians with IQRs\. Calibration success is 1\.000 in every condition\.Table 19:Absolute variance components for the matched primary magnitude\. Values are medians of the system\-specific quantities in Eq\. \([95](https://arxiv.org/html/2608.19762#A5.E95)\); the ratio of the two medians need not equal the median of the system\-specific ratios\.Table 20:Prospective\-structure ratio for the primary magnitude and two secondary response summaries\.P\+P^\{\+\}is sign\-sensitive; because the matched construction enforces alternating signed one\-step targets, the positive\-peak column is reported only as a descriptive robustness check and is not used to support the main prospective\-identifiability claim\.The combined result is deliberately narrower than an online\-prediction claim\. Repeated futures show that candidate identity can remain a strong source of finite\-horizon variation even after the immediate output magnitude is matched, while the success of a particular present\-time ISO representation varies substantially across systems\. Experiment 2 therefore motivates prospective identifiability as a separate question without assuming that the pathwise ISO itself is deployable before the future trajectory is observed\.