When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

# When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?
Source: [https://arxiv.org/html/2610.41038](https://arxiv.org/html/2610.41038)
Kağan Akman kagan\.akman@bilkent\.edu\.trAffiliation:Department of MathematicsAffiliation:Bilkent UniversityAffiliation:Ankara, 06800, TurkeyJake Feldman Starosta jake\.feldmanstarosta@queensu\.caAffiliation:Department of Mathematics and StatisticsAffiliation:Queen’s UniversityAffiliation:Kingston K7L 3N6, ON, CanadaNaci Saldi naci\.saldi@bilkent\.edu\.trAffiliation:Department of MathematicsAffiliation:Bilkent UniversityAffiliation:Ankara, 06800, TurkeySerdar Yüksel yuksel@queensu\.caAffiliation:Department of Mathematics and StatisticsAffiliation:Queen’s UniversityAffiliation:Kingston K7L 3N6, ON, Canada

###### Abstract

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non\-Markovian data\. To this end, this paper studies next\-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non\-Markovian sources or those which are hidden Markov models of some finite order \(possibly non\-stationary\)\. In either case, we first obtain error bounds via finite models with finite window approximations\. For stationary ergodic sources, the resulting error is bounded by an entropy gap; forNNth order Markovian sources, it decays at an exponential rateα⌊M/N⌋\\alpha^\{\\lfloor M/N\\rfloor\}, whereMMis the window length andα\\alphais determined by filter stability \(and the Dobrushin coefficients of the transition and measurement kernels\)\. We then construct a Transformer of depthM\+1M\+1, one head per layer, realizing the truncated filter, and we optimally train it with stochastic control developed in earlier work, which rigorously returns a near optimal Transformer\. In particular, we show that for a non\-Markovian source, either under stationarity and ergodicity for general models or under filter stability for hidden Markov models, an optimally trained Transformer has arbitrarily small performance error with increasing context length, data, and approximation refinement\. Numerical studies accompany the theory\.

###### keywords

Transformer training, next token prediction, finite memory approximation, non\-Markovian models, optimal control, expressivity\.

## 1Introduction

Transformer architectures are empirically strong predictors of the next token from a finite context, but how long must that context be? Practice has an answer by means of experiments, and asserts that longer context is often better\.

In this paper, we answer this question in terms of an intrinsic property of the general possibly non\-Markovian source by investigating how much such a non\-Markovian source can be approximated by anNNth order finite Markov model, in a suitable sense, together with optimal training loss due to approximation and sample complexity errors\.

A specific application will involve partially observed Markov models \(known also as hidden Markov models\) for which stronger bounds will be presented in terms of filter stability\.

The theory of partially observed Markov process \(POMP\) tells us that if the source is a latent Markov process, i\.e\. an unseen Markov process coupled with an observation process obtained via a noisy channel conditioned on the source, then the optimal predictor is a filter\. Going one step further, we predict the next observation variable based on the history of previous observation variables, corresponding to the language modeling task of next\-token prediction\. Using the full history as context is often challenging in practice\. Therefore, we truncate the input, that is, initialize it from a \(possibly wrong\) prior and take the finite window as the context\. Then, the predictive performance on this truncated context is a question of filter stability[Atar and Zeitouni \(1997\)](https://arxiv.org/html/2610.41038#bib.bib5);[Le Gland and Oudjane \(2004\)](https://arxiv.org/html/2610.41038#bib.bib30);[Chigansky and Liptser \(2004\)](https://arxiv.org/html/2610.41038#bib.bib10);[Chigansky et al\. \(2009\)](https://arxiv.org/html/2610.41038#bib.bib11)\.

On the other hand, practice demonstrates that Transformer architectures are efficient for processing sequential data such as sentences for large language models\.[Yun et al\. \(2020a\)](https://arxiv.org/html/2610.41038#bib.bib51)reveal that Transformers are universal approximators of continuous, sequence\-to\-sequence functions on compact domains\. Thus, a natural question arises:Can Transformers approximate finite\-window filter processes arising from POMPs?In this paper, we show that the answer is affirmative and we find quantitative bounds for the approximation error while specifying the class of Transformers achieving such task by explicit width and depth constructions\.

On anNthN^\{\\rm th\}order Markovian model, that is, the law of the next state is determined by the previousNNstates, we define noisy observation channels in order to make it latent\. Then we define the associated prediction process: full history, finite\-window and finite\-window with quantized observations\. There are three major unknowns: \(i\) Are finitely many quantized observations sufficient statistics? \(ii\) Can Transformers realize finite\-window predictive distribution with quantized observations? \(iii\) How can we learn the optimal weight trajectory for such Transformer? We answer \(i\) and \(ii\) in this paper, whereas \(iii\) is studied by[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3), where an optimal control theoretic formulation of Transformers is studied with rigorously justified finite model approximations and dynamic programming\.

Transformer training amortizes the output cost: paying upfront at training to reduce inference costs compared to an empirical lookup table\. This amortization is done by compressing the information from the empirical distribution lookup table to a representation that lives in the model weights, built on the structure of the Transformer architecture\. While this compression requires an initial computational cost in the form of training, it saves the user compute and storage by replacing a large input table search with a deterministic equation that approximates the target distribution\. This representation has other useful properties as well, such as continuity and generalization, and thus the ability to interpolate sequences absent from the training data set\.

We begin by defining a generally non\-Markovian process that we segment to create the training data set\. In the context of LLMs, this can be thought of as a sequence of tokens on the internet\. We then formulate the Transformer dynamics as a control theoretic problem: selecting the optimal model weights \(controls\), to approximate the empirical next\-token distribution \(target state\) given the input sequence \(current state\)\. The sequence flows through the Transformer one layer at a time, each comprised of a feed\-forward layer and self\-attention block\. We employ dynamic programming to train a quantized Transformer model to minimize the cross\-entropy between the actual predicted and target empirical distributions\. We show that the empirical prediction gets arbitrarily close to the truncated\-window quantized prediction\. Then, we prove the existence of a Transformer that approximates this truncated\-window quantized prediction, which we approximate experimentally with quantized Transformer training\.

### 1\.1Related Works

#### Control–theoretic approaches to neural networks and, in particular, to Transformers

There is a developing literature focusing on the control–theoretic formulations and properties of Transformers\.[E \(2017\)](https://arxiv.org/html/2610.41038#bib.bib15)suggests to see machine learning problems \-in particular, residual neural networks \(ResNets\)\- as optimal control of continuous time dynamical systems governed by ordinary differential equations where the time index corresponds to the layers of a neural network\. Around the same time,[Haber and Ruthotto \(2018\)](https://arxiv.org/html/2610.41038#bib.bib20)study a similar ODE based analysis of ResNets by proving stability and well\-posedness results\. They also study discretization of such systems for numerical analysis and implementability\.[Ruthotto and Haber \(2020\)](https://arxiv.org/html/2610.41038#bib.bib44)categorize convolutional neural networks \(CNNs\) into PDE inspired classes and authors discuss the regularity of CNNs via the theory of parabolic and hyperbolic PDEs\. An alternative approach by seeing neural networks as solutions of suitable dynamical systems, called*neural ordinary differential equations*, is studied by[Chen et al\. \(2018\)](https://arxiv.org/html/2610.41038#bib.bib9);[Lu et al\. \(2018\)](https://arxiv.org/html/2610.41038#bib.bib32), where the neural networks structure is replaced by the solution of the governing ODEs, obtained by the Runge\-Kutta methods\([Butcher, 2016](https://arxiv.org/html/2610.41038#bib.bib7), Chapter 3\)\.[Ruiz\-Balet and Zuazua \(2023\)](https://arxiv.org/html/2610.41038#bib.bib42)study controllability of neural ordinary differential equations while[Ruiz\-Balet and Zuazua \(2024\)](https://arxiv.org/html/2610.41038#bib.bib43)study the controllability of*neural transport equations*, which are continuity equations governed by a neural network’s velocity field\.

[Geshkovski et al\. \(2025\)](https://arxiv.org/html/2610.41038#bib.bib19)study Transformers in a mathematically rigorous way while focusing on clustering properties, building on their earlier work\([Geshkovski et al\., 2023](https://arxiv.org/html/2610.41038#bib.bib17)\)\. The training scheme used in this paper is by[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3)where Transformers are represented as measure\-state Markov decision processes lifted from discrete\-time McKean–Vlasov dynamics\([McKean Jr, 1966](https://arxiv.org/html/2610.41038#bib.bib35)\)exhibited at the token level, and then, the training is reformulated as a dynamic programming problem\. For computational tractability,[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3)derive a finite model approximation to the measure\-state MDP using the algorithm of[Reznik \(2011\)](https://arxiv.org/html/2610.41038#bib.bib41)and\([Saldi et al\., 2018](https://arxiv.org/html/2610.41038#bib.bib45), Chapter 3\)\. The generalization properties of the quantized training method are studied by[Akman et al\. \(2026a\)](https://arxiv.org/html/2610.41038#bib.bib2)from two angles: norm\-based excess risk bounds and Wasserstein distributional robustness\.

A different use of control theory appears in[Chang and Mehta \(2025\)](https://arxiv.org/html/2610.41038#bib.bib8), who construct a Transformer\-like architecture for prediction in hidden Markov models through a duality between nonlinear filtering and optimal control\.[Kudre et al\. \(2026\)](https://arxiv.org/html/2610.41038#bib.bib29)extend this to linear\-Gaussian models and compare the optimal control weights with the attention weights of a trained Transformer\. Our work differs by the usage of control classes; in[Chang and Mehta \(2025\)](https://arxiv.org/html/2610.41038#bib.bib8), control describes inference over the full history \(and is thus closed\-loop or context dependent\), whereas we use control for training at the distributional level leading to open\-loop \(and thus context\-independent\) parameters consistent with standard Transformer architecture, and quantify the cost of a finite context window via filter stability\.

#### What Transformers can represent

Transformers are universal approximators of continuous sequence\-to\-sequence functions on compact domains with fixed width and arbitrary depth\([Yun et al\., 2020a](https://arxiv.org/html/2610.41038#bib.bib51)\)while[Yun et al\. \(2020b\)](https://arxiv.org/html/2610.41038#bib.bib52)show that sparse Transformers can preserve such universal approximation capabilities\. In contrast, our target, in this paper, is a measure\-valued map on a finite quantized domain for which we give explicit depth and width with one head per layer\. As another refinement direction,[Kajitsuka and Sato \(2024\)](https://arxiv.org/html/2610.41038#bib.bib24)prove Transformers with one\-headed single self\-attention having low\-rank weight matrices and double feed\-forward layers can approximate continuous and permutation\-equivariant maps on compact domains\. There is a line of work treating the context as a probability measure over tokens;[Furuya et al\. \(2025\)](https://arxiv.org/html/2610.41038#bib.bib16)prove universal approximation for this setup, while[Geshkovski et al\. \(2024\)](https://arxiv.org/html/2610.41038#bib.bib18)focuses on Transformer’s interpolation ability from measures to measures\.[Agrachev and Letrouit \(2025\)](https://arxiv.org/html/2610.41038#bib.bib1)show controllability of general equivariant systems and treat self\-attention mechanism as a special case\. There are several studies providing explicit approximation rates:[Jiang and Li \(2024\)](https://arxiv.org/html/2610.41038#bib.bib23)prove Jackson\-type rates for sequence\-to\-sequence matching scenario, while[Wang et al\. \(2024\)](https://arxiv.org/html/2610.41038#bib.bib49)gives approximation rates for Transformers with long but sparse memory, and they characterize how various hyperparameters such as depth, width, and the number of heads contribute\.

Forkk\-th order fully observable Markov sources,[Rajaraman et al\. \(2024\)](https://arxiv.org/html/2610.41038#bib.bib40)show that three layers with one head suffice to represent the conditionalkk\-gram with an attention\-only Transformer\. Their constructed network represents a statistic of the prompt, namely, the empirical distribution of successors to the current token with the sequence itself, which is unbiased for a fully observed source and is kernel\-agnostic; whereas we represent the predictive distribution of a partially observed source in the weights\.

A complementary line bounds what Transformers cannot do\. Transformers with hard attention fail to recognize even relatively simple formal languages\([Hahn, 2020](https://arxiv.org/html/2610.41038#bib.bib21)\)\.[Merrill and Sabharwal \(2023\)](https://arxiv.org/html/2610.41038#bib.bib36)show that a fixed\-depth Transformer with limited numerical precision cannot carry out a computation that requires many sequential steps\.[Strobl et al\. \(2024\)](https://arxiv.org/html/2610.41038#bib.bib46)survey the expressibility of Transformers\.

#### Filter stability and memory in prediction

The filter stability problem refers to the correction of an incorrectly initialized nonlinear filter for a partially observed stochastic dynamical system \(controlled or control\-free\) with increasing measurements[Chigansky and Liptser \(2004\)](https://arxiv.org/html/2610.41038#bib.bib10);[Chigansky et al\. \(2009\)](https://arxiv.org/html/2610.41038#bib.bib11);[Atar and Zeitouni \(1997\)](https://arxiv.org/html/2610.41038#bib.bib5);[Le Gland and Oudjane \(2004\)](https://arxiv.org/html/2610.41038#bib.bib30)\.[McDonald and Yüksel \(2020\)](https://arxiv.org/html/2610.41038#bib.bib33)give an explicit geometric rate for filter stability through Dobrushin coefficient\([Dobrushin, 1956](https://arxiv.org/html/2610.41038#bib.bib14)\)of the transition kernel and the observation channel of the latent Markov source\. Truncating a predictor to a finite window is exactly this situation since discarding the data amounts to replacing the true conditional law of the hidden state by a fixed prior\.[Kara and Yuksel \(2022\)](https://arxiv.org/html/2610.41038#bib.bib26)use a similar observation to establish near optimality of finite\-memory feedback policies in partially observed MDPs, converting the belief\-state error into an error of discrepancy in the values;[Kara et al\. \(2025\)](https://arxiv.org/html/2610.41038#bib.bib28)extend this to continuous spaces\. In this paper, we use belief error and propagate it through the one\-step prediction kernel instead\. Alternative approaches to filter stability also establish exponential filter stability through a different route:[Atar and Zeitouni \(1997\)](https://arxiv.org/html/2610.41038#bib.bib5)and[Le Gland and Oudjane \(2004\)](https://arxiv.org/html/2610.41038#bib.bib30)prove exponential stability rates for the Hilbert projective metric via Birkhoff’s contraction coefficient\. Alternatively, asymptotic filter stability can also be utilized if one only wishes to have asymptotic performance bounds without explicit rates of convergence[Chigansky et al\. \(2009\)](https://arxiv.org/html/2610.41038#bib.bib11);[McDonald and Yüksel \(2024\)](https://arxiv.org/html/2610.41038#bib.bib34)\. An infinite product space approach, where finite memory truncation is viewed as quantization, is presented in[Yüksel \(2025\)](https://arxiv.org/html/2610.41038#bib.bib50)\.

### 1\.2Contributions

Our main results can be summarized as follows\. Given a general non\-Markovian model\{Yr:r∈ℤ\+\}\\\{Y\_\{r\}:r\\in\\mathbb\{Z\}\_\{\+\}\\\}taking values from a compact metric space𝕐\\mathbb\{Y\}, consider its finite approximationqs​\(Yr\),r∈ℤ\+q^\{s\}\(Y\_\{r\}\),r\\in\\mathbb\{Z\}\_\{\+\}where\|q⁡\(𝕐\)\|=:R⁡\(s\)<∞\|q\(\\mathbb\{Y\}\)\|=:R\(s\)<\\inftyandqqis a quantizer\. We show that Transformer performance is fundamentally related to finite model with finite memory approximability\.

Such a finite model with finite memory approximability is a highly qualified expression and requires care with regard to its precise presentation\. For a general stationary and ergodic model, this term is given by

supr≥M𝔼Y0:r\[∥Pr\(Yr\+1∈⋅∣Y0:r\)−Pr\(Yr\+1∈⋅∣qs\(Yr\),qs\(Yr−1\),…,qs\(Yr−M\+1\)\)∥TV\]:=F⁡\(M,qs\)\\begin\{split\}\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\bigg\[\\\|\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\\mid Y\_\{0:r\}\)\-\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\\mid q^\{s\}\(Y\_\{r\}\),q^\{s\}\(Y\_\{r\-1\}\),\\ldots,&q^\{s\}\(Y\_\{r\-M\+1\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\bigg\]\\\\ &:=F\(M,q^\{s\}\)\\end\{split\}for the case with stationary and ergodic data\.

We show that asF⁡\(M,qs\)→0F\(M,q^\{s\}\)\\to 0withM→∞,R⁡\(s\)→∞M\\to\\infty,R\(s\)\\to\\infty, Transformers, trained optimally via stochastic control, perform asymptotically well, with explicit convergence bounds\. Information theoretic or ergodic theoretic bounds onF⁡\(M,qs\)F\(M,q^\{s\}\)can be utilized in the quantitative expressions\.

To make our results stronger and more explicit in a particular context but one with significant applied relevance, we will also consider the special case of hidden Markov models, where there is an underlying Markov processXr,r∈ℤ\+X\_\{r\},r\\in\\mathbb\{Z\}\_\{\+\}which generatesYrY\_\{r\}, the observable measurement process, throughYr=H⁡\(Xr,Vr\)Y\_\{r\}=H\(X\_\{r\},V\_\{r\}\)for some measurableHHand i\.i\.d\. noiseVrV\_\{r\}\(whose representation is without loss for such models\)\. In this case, the above approximation bounds can be obtained more explicitly in terms of filter stability; stationarity can be relaxed away and more explicit conditions can be obtained\. Without imposing ergodicity, define

supr≥M𝔼Y0:r∥Pr\(Yr\+1∈⋅\|Y0:r\)−Pr\(Yr\+1∈⋅\|qs\(Yr\),qs\(Yr−1\),⋯,qs\(Yr−M\+1,π^\)∥TV:=F~​\(M,qs\),\\begin\{split\}\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\|Y\_\{0:r\}\)\-\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\|q^\{s\}\(Y\_\{r\}\),q^\{s\}\(Y\_\{r\-1\}\),\\cdots,&q^\{s\}\(Y\_\{r\-M\+1\},\\hat\{\\pi\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &:=\\tilde\{F\}\(M,q^\{s\}\),\\end\{split\}whereπ^\\hat\{\\pi\}denotes a fixed probability measure serving as an artificial and incorrect prior at time00, to facilitate Bayesian updates\.

We show that asF~​\(M,qs\)→0\\tilde\{F\}\(M,q^\{s\}\)\\to 0withM→∞,R⁡\(s\)→∞M\\to\\infty,R\(s\)\\to\\infty, Transformers, trained via optimal stochastic control\([Akman et al\., 2026b](https://arxiv.org/html/2610.41038#bib.bib3)\), can rigorously be shown to perform prediction and filtering with near lossless performance under filter stability and regularity conditions\. We thus address a fundamental question on Transformer architectures and obtain quantified error bounds due to finite model with finite memory approximation for general non\-Markovian models\.

Figure[1](https://arxiv.org/html/2610.41038#S1.F1)summarizes our technical results\. Extensive simulations corroborate the mathematical results\. We note that our ingredients in the analysis are: \(i\) approximation via finite memory models \(in a suitable, expectation or uniform over priors, sense\), \(ii\) approximation of finite memory models with finite Markov models, \(iii\) optimal Transformer learning for finite Markov models and their performance, and \(iv\) a sample complexity analysis\.

Figure 1:Summary of the results of this paper\.Our analysis also highlights what Transformers can and cannot do: Since our analysis builds on rigorously justified near optimal training via control theory, a corollary of our analysis is that when sequence to sequence approximations is realizable with no error usingsomeTransformer, an optimally trained scheme can construct a Transformer which attains an arbitrarily small error performance\. That is, analogous to standard control theoretic paradigm \(see e\.g\.[Anderson and Moore \(2007\)](https://arxiv.org/html/2610.41038#bib.bib4)\), controllability/reachability certifies that under a suitably formulated cost, optimality arrives at arbitrarily small performance error whenever mild regularity conditions hold\.

## 2Sufficiency of Finite Memory in Prediction

LetSSbe a finite alphabet and\(𝕐,ρ\)\(\\mathbb\{Y\},\\rho\)be a compact metric space endowed with a reference probability measureμ\\mudefined on the Borelσ\\sigma\-field generated byρ\\rho\. Let\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}be a stochastic process taking values inSSandQ∈𝒫⁡\(𝕐∣S\)Q\\in\\mathcal\{P\}\(\\mathbb\{Y\}\\mid S\)be a stochastic kernel admitting a Radon\-Nikodym derivative

ψ⁡\(y∣z\):=dQ\(⋅∣z\)d​μ​\(y\),y∈𝕐,z∈S\.\\psi\(y\\mid z\):=\\frac\{\\mathrm\{d\}Q\(\\,\\cdot\\mid z\)\}\{\\mathrm\{d\}\\mu\}\(y\),\\quad y\\in\\mathbb\{Y\},\\,z\\in S\.SinceSSis finite, suchμ\\mualways exists; we simply takeμ\(⋅\)=∑z∈S1\|S\|Q\(⋅∣z\)\\mu\(\\cdot\)=\\sum\_\{z\\in S\}\\frac\{1\}\{\|S\|\}Q\(\\,\\cdot\\mid z\)\. We take observation variablesYr∼Q\(⋅∣Zr\)Y\_\{r\}\\sim Q\(\\,\\cdot\\mid Z\_\{r\}\)from this channel for allr∈ℤ\+r\\in\\mathbb\{Z\}\_\{\+\}and we assume thatYrY\_\{r\}is independent ofZtZ\_\{t\}andYtY\_\{t\}wheneverr≠tr\\neq t, conditioned onZrZ\_\{r\}\.

Z0Z\_\{0\}Z1Z\_\{1\}⋯\\cdotsZrZ\_\{r\}Zr\+1Z\_\{r\+1\}⋯\\cdotsY0Y\_\{0\}Y1Y\_\{1\}YrY\_\{r\}Yr\+1Y\_\{r\+1\}Q\(⋅∣Z0\)Q\(\\,\\cdot\\mid Z\_\{0\}\)Q\(⋅∣Z1\)Q\(\\,\\cdot\\mid Z\_\{1\}\)Q\(⋅∣Zr\)Q\(\\,\\cdot\\mid Z\_\{r\}\)Q\(⋅∣Zr\+1\)Q\(\\,\\cdot\\mid Z\_\{r\+1\}\)Figure 2:Flows of the latent process and observations\.As𝕐\\mathbb\{Y\}is a compact space, by total boundedness, for anys∈ℕs\\in\\mathbb\{N\}, there exists a1/s1/s\-netΛs:=\{ys,i:i=1,…,R\(s\)\}\\Lambda\_\{s\}:=\\\{y\_\{s,i\}:i=1,\\ldots,R\(s\)\\\}, whereR⁡\(s\)∈ℕR\(s\)\\in\\mathbb\{N\}, that is

mini=1,…,R⁡\(s\)⁡ρ⁡\(y,ys,i\)≤1s\\min\_\{i=1,\\ldots,R\(s\)\}\\rho\(y,y\_\{s,i\}\)\\leq\\frac\{1\}\{s\}\(1\)for ally∈𝕐y\\in\\mathbb\{Y\}\.

###### Definition 1\.

The quantizerqs:𝕐→Λsq^\{s\}:\\mathbb\{Y\}\\to\\Lambda\_\{s\}is defined as

qs​\(⋅\):=arg⁡mini=1,…,R⁡\(s\)⁡ρ⁡\(⋅,ys,i\),q^\{s\}\(\\cdot\):=\\arg\\min\_\{i=1,\\ldots,R\(s\)\}\\rho\(\\cdot,y\_\{s,i\}\),where the ties are broken in a measurable way\.

We separate𝕐\\mathbb\{Y\}intoR⁡\(s\)R\(s\)\-cells represented by the elements ofΛs\\Lambda\_\{s\}\. SayC⁡\(s,i\)C\(s,i\)be the cell associated toys,iy\_\{s,i\}, that is,C⁡\(s,i\):=\(qs\)−1​\(ys,i\)C\(s,i\):=\(q^\{s\}\)^\{\-1\}\(y\_\{s,i\}\)\. Then\{C⁡\(s,i\)\}i=1R⁡\(s\)\\\{C\(s,i\)\\\}\_\{i=1\}^\{R\(s\)\}form a disjoint partition of𝕐\\mathbb\{Y\}, i\.e\.⨆i=1R⁡\(s\)C⁡\(s,i\)=𝕐\\bigsqcup\_\{i=1\}^\{R\(s\)\}C\(s,i\)=\\mathbb\{Y\}\. Note thatdiam⁡\(C⁡\(s,i\)\)≤2/s\\operatorname\{diam\}\(C\(s,i\)\)\\leq 2/sfor alls∈ℕs\\in\\mathbb\{N\}andi=1,…,R⁡\(s\)i=1,\\ldots,R\(s\)\.

We define four objects: First, we have the true predictive distribution which makes a prediction on the next token based on the full history\. This is an object with the highest informative value provided that the underlying structure is well\-behaved\. However, as we progress further into the time horizon, computing it may require vast computation power and memory requirements\. To this end, we also introduce theMM\-window predictive distribution which uses only theMM\-many past observations prior to the next time step\. After that we define the quantized version of theMM\-window predictive distribution, which uses theMM\-window data from the1s\\frac\{1\}\{s\}\-net approximation of𝕐\\mathbb\{Y\}\. Finally, since Transformers realize maps via finite set of logits, we quantize the target as well\. Eventually, one part of our objective is to reveal that, depending on the window sizeMMand quantization levelss,MM\-windowss\-quantized data are near\-sufficient statistics with explicit error rates\.

###### Definition 2\.

For anyr∈ℕr\\in\\mathbb\{N\}, we define

1. 1\.The true predictive distribution Pr\(⋅\):=Pr\(Yr\+1∈⋅∣Y0:r\)\.P\_\{r\}\(\\cdot\):=\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\\mid Y\_\{0:r\}\)\.
2. 2\.TheMM\-window predictive distribution PrM\(⋅\):=Pr\(Yr\+1∈⋅∣Yr−M\+1,…,Yr\),P\_\{r\}^\{M\}\(\\cdot\):=\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\\mid Y\_\{r\-M\+1\},\\ldots,Y\_\{r\}\),
3. 3\.TheMM\-window quantized\-data predictive distribution PrM,s\(⋅\):=Pr\(Yr\+1∈⋅∣qs\(Yr−M\+1\),…,qs\(Yr\)\),P\_\{r\}^\{M,s\}\(\\cdot\):=\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in\\cdot\\mid q^\{s\}\(Y\_\{r\-M\+1\}\),\\ldots,q^\{s\}\(Y\_\{r\}\)\),
4. 4\.TheMM\-window fully quantized predictive distribution P^rM,s\(⋅\):=q\#sPrM,s\(⋅∣qs\(Yr−M\+1\),…,qs\(Yr\)\)\.\\hat\{P\}\_\{r\}^\{M,s\}\(\\cdot\):=q^\{s\}\_\{\\\#\}P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid q^\{s\}\(Y\_\{r\-M\+1\}\),\\ldots,q^\{s\}\(Y\_\{r\}\)\)\.

In 2\-4, we take the initial distribution asLaw⁡\(Zr−M\)\\mathrm\{Law\}\(Z\_\{r\-M\}\)\.

In the theory of partially observed Markov decision processes \(POMDPs\), one desires to compute the*filter process*πr\(⋅\)=Pr\(Zr∈⋅∣Y0:r\)\\pi\_\{r\}\(\\cdot\)=\\mathrm\{Pr\}\(Z\_\{r\}\\in\\cdot\\mid Y\_\{0:r\}\)\([Kara and Yüksel, 2024](https://arxiv.org/html/2610.41038#bib.bib27)\)\. Hence, the problem we study in this paper is slightly different from the usual nonlinear filtering problem\. A filter is an estimator of hidden states, based on the information given by the observation variables\. In this paper, we are trying to estimate the next observation given the observations seen so far\. For Markovian processes, these two problems are equivalent since one can arrive atPrP\_\{r\}by composing the filterπr\\pi\_\{r\}with the transition and observation kernels of the system \(which are non\-expansive in the total variation distance\), in that order\. However, this is not true in general for an ergodic source\.

The analysis of this problem requires several assumptions, which we regard as mild as they balance the level of difficulty imposed by the generality\.

###### Assumption 3\.

The following hold throughout:

- \(A1\)The densityψ\\psisatisfies the minorization conditionψ⁡\(y∣z\)≥qmin\>0\\psi\(y\\mid z\)\\geq q\_\{\\rm min\}\>0for ally∈𝕐y\\in\\mathbb\{Y\}and allz∈Sz\\in S\.
- \(A2\)There existsLψ≥0L\_\{\\psi\}\\geq 0such that for allz∈Sz\\in Sand ally,y′∈𝕐y,y^\{\\prime\}\\in\\mathbb\{Y\},\|ψ⁡\(y∣z\)−ψ⁡\(y′∣z\)\|≤Lψ​ρ​\(y,y′\)\|\\psi\(y\\mid z\)\-\\psi\(y^\{\\prime\}\\mid z\)\|\\leq L\_\{\\psi\}\\rho\(y,y^\{\\prime\}\)\.
- \(A3\)For alls∈ℕs\\in\\mathbb\{N\}, there isμmins\>0\\mu\_\{\\rm min\}^\{s\}\>0such that, for alli=1,…,R⁡\(s\)i=1,\\ldots,R\(s\),μ⁡\(C⁡\(s,i\)\)≥μmins\\mu\(C\(s,i\)\)\\geq\\mu\_\{\\rm min\}^\{s\}holds\.

Once we obtain the necessary results, we need to be able to come back from the quantized space to the original space\. In particular, in §[4](https://arxiv.org/html/2610.41038#S4), we build a Transformer to realizeP^rM,s\\hat\{P\}\_\{r\}^\{M,s\}in Definition[2](https://arxiv.org/html/2610.41038#Thmtheorem2), using which we obtain the final error bounds in §[5](https://arxiv.org/html/2610.41038#S5)\. Since the Transformer gives a probability distribution overΛs\\Lambda\_\{s\}, we have to revert back to a distribution on𝕐\\mathbb\{Y\}, namelyPrM,sP\_\{r\}^\{M,s\}\.

###### Definition 4\.

For anys∈ℕs\\in\\mathbb\{N\}, defineΞs:𝒫⁡\(Λs\)→𝒫⁡\(𝕐\)\\Xi\_\{s\}:\\mathcal\{P\}\(\\Lambda\_\{s\}\)\\to\\mathcal\{P\}\(\\mathbb\{Y\}\)by

Ξs​\(ν\)​\(A\):=∑i=1R⁡\(s\)ν⁡\(ys,i\)​μ⁡\(A∩C⁡\(s,i\)\)μ⁡\(C⁡\(s,i\)\),ν∈𝒫⁡\(Λs\)\.\\Xi\_\{s\}\(\\nu\)\(A\):=\\sum\_\{i=1\}^\{R\(s\)\}\\nu\(y\_\{s,i\}\)\\frac\{\\mu\(A\\cap C\(s,i\)\)\}\{\\mu\(C\(s,i\)\)\},\\qquad\\nu\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)\.

The functionΞs\\Xi\_\{s\}is well\-defined if Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A3\) holds: Then the denominatorμ⁡\(C⁡\(s,i\)\)\\mu\(C\(s,i\)\)is positive for alls∈ℕs\\in\\mathbb\{N\}andi=1,…,R⁡\(s\)i=1,\\ldots,R\(s\)\.

We study this problem under two scenarios: \(i\) We add the assumption that the latent source\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is stationary ergodic, in addition to Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\. \(ii\)\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}isNthN^\{\\rm th\}order Markov, which we initially treat the first order case, then generalize to theNthN^\{\\rm th\}order case by reducing to the first order case back\. This setting allows us to derive explicit rates showing exponentially fast convergence using Dobrushin coefficient\-based analyses, following\([McDonald and Yüksel, 2020](https://arxiv.org/html/2610.41038#bib.bib33)\)\.

###### Proposition 5\.

Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\-2\), for allr≥Mr\\geq Mand alls∈ℕs\\in\\mathbb\{N\},

sup𝐲∈𝕐M∥PrM\(⋅∣𝐲\)−PrM,s\(⋅∣𝐪M,s\(𝐲\)\)∥TV≤2​M​Lψqmin​s\\sup\_\{\\mathbf\{y\}\\in\\mathbb\{Y\}^\{M\}\}\\\|P\_\{r\}^\{M\}\(\\,\\cdot\\mid\\mathbf\{y\}\)\-P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid\\mathbf\{q\}^\{M,s\}\(\\mathbf\{y\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{2ML\_\{\\psi\}\}\{q\_\{\\rm min\}s\}whereqM,sq^\{M,s\}is the application ofqsq^\{s\}to sequences of lengthMM\.

### 2\.1The case where\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is a general stationary and ergodic process

In this part, we cover the case where the latent source is stationary and ergodic\. This setting allows us to derive vanishing error bounds using information–theoretic tools\. However, the bounds are of qualitative nature\. We commence with stating our base assumption for this subsection, and then we investigate how the regularity of the latent source affects the regularity of the observed variables\. Finally, we show that the finite memory approximation yields a entropy gap based bound decreasing to00\.

###### Assumption 6\.

\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is ergodic with probability one, i\.e\.

limT→\+∞1T​∑r=0T−1f⁡\(Zr\)​=a\.s\.​∑z∈SπZ​\(z\)​f​\(z\)\\lim\_\{T\\to\+\\infty\}\\frac\{1\}\{T\}\\sum\_\{r=0\}^\{T\-1\}f\(Z\_\{r\}\)\\overset\{a\.s\.\}\{=\}\\sum\_\{z\\in S\}\\pi\_\{Z\}\(z\)\\,f\(z\)for anyf:S→ℝf:S\\to\\mathbb\{R\}\. We further assume that\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is stationary\.

Let𝒜\\mathcal\{A\}andℬ\\mathcal\{B\}be measurable spaces,AAandBBbe two continuous random variables with a joint probability density functionf:𝒜×ℬ→\[0,1\]f:\\mathcal\{A\}\\times\\mathcal\{B\}\\to\[0,1\], whereAAis𝒜\\mathcal\{A\}\-valued andBBisℬ\\mathcal\{B\}\-valued\. Then the*conditional differential entropy is defined as*

H\(A∣B\):=−∫𝒜×ℬf\(a,b\)logf\(a∣b\)d\(a,b\)H\(A\\mid B\):=\-\\int\_\{\\mathcal\{A\}\\times\\mathcal\{B\}\}f\(a,b\)\\log\{f\(a\\mid b\)\}\\,\\mathrm\{d\}\(a,b\)\(2\)whenever the integral above exists, wheref⁡\(a∣b\)f\(a\\mid b\)is the conditional probability ofAAgivenBB\. The*memory gap*is then defined as

ΔM,r:=H\(Yr\+1∣Yr−M\+1:r\)−H\(Yr\+1∣Y0:r\)\\Delta\_\{M,r\}:=H\(Y\_\{r\+1\}\\mid Y\_\{r\-M\+1:r\}\)\-H\(Y\_\{r\+1\}\\mid Y\_\{0:r\}\)\(3\)Note that for fixedrr,ΔM,r→0\\Delta\_\{M,r\}\\to 0monotonically asMMincreases torrsince the conditioningσ\\sigma\-field grows monotonically to the full historyσ\\sigma\-field generated byY0:rY\_\{0:r\}\. In addition, for the proof of next lemma, we require the notationHj:=H\(Yj∣Y0:j−1\)H\_\{j\}:=H\(Y\_\{j\}\\mid Y\_\{0:j\-1\}\)for allj∈ℕj\\in\\mathbb\{N\}and defineH∞:=limj→∞HjH\_\{\\infty\}:=\\lim\_\{j\\to\\infty\}H\_\{j\}\. Note that Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\) and \(A3\) ensure that the conditional entropies in \([3](https://arxiv.org/html/2610.41038#S2.E3)\) andHjH\_\{j\}’s exist sinceψ\(⋅∣z\)\\psi\(\\,\\cdot\\mid z\)is \(Lipschitz\) continuous on a compact space𝕐\\mathbb\{Y\}, therefore, it is bounded\. Hence,Hj∈\[−log⁡‖ψ‖∞,−log⁡qmin\]H\_\{j\}\\in\[\-\\log\{\\\|\\psi\\\|\_\{\\infty\}\},\-\\log\{q\_\{\\rm min\}\}\]\. Note that\{Hj\}j∈ℕ\\\{H\_\{j\}\\\}\_\{j\\in\\mathbb\{N\}\}forms a non\-increasing sequence:

Hj\+1:=H\(Yj\+1∣Y0:j\)≤H\(Yj\+1∣Y1:j\)=H\(Yj∣Y0:j−1\)=:Hj,j∈ℕH\_\{j\+1\}:=H\(Y\_\{j\+1\}\\mid Y\_\{0:j\}\)\\leq H\(Y\_\{j\+1\}\\mid Y\_\{1:j\}\)=H\(Y\_\{j\}\\mid Y\_\{0:j\-1\}\)=:H\_\{j\},\\quad j\\in\\mathbb\{N\}The inequalityH\(Yj\+1∣Y0:j\)≤H\(Yj\+1∣Y1:j\)H\(Y\_\{j\+1\}\\mid Y\_\{0:j\}\)\\leq H\(Y\_\{j\+1\}\\mid Y\_\{1:j\}\)holds for allj∈ℕj\\in\\mathbb\{N\}since conditioning on a coarserσ\\sigma\-field results in higher entropy\. The equalityH\(Yj\+1∣Y1:j\)=H\(Yj∣Y0:j−1\)H\(Y\_\{j\+1\}\\mid Y\_\{1:j\}\)=H\(Y\_\{j\}\\mid Y\_\{0:j\-1\}\)is due to the stationarity of\{Yr\}r∈ℤ\+\\\{Y\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}, established in Lemma[30](https://arxiv.org/html/2610.41038#Thmtheorem30)\. Therefore, by monotone convergence theorem for real\-valued sequences,H∞H\_\{\\infty\}exists\.

We define the*Kullback\-Leibler divergence*between two probability measuresP1P\_\{1\}andP2P\_\{2\}on a measurable space\(Ω,ℱ\)\(\\Omega,\\mathcal\{F\}\)as follows:

DKL\(P1∥P2\):=\{∫Ωd​P1​log⁡d​P1d​P2,P1≪P2;\+∞,otherwise\.D\_\{\\rm KL\}\(P\_\{1\}\\parallel P\_\{2\}\):=\\begin\{cases\}\\int\_\{\\Omega\}\\mathrm\{d\}P\_\{1\}\\log\{\\frac\{\\mathrm\{d\}P\_\{1\}\}\{\\mathrm\{d\}P\_\{2\}\}\},&P\_\{1\}\\ll P\_\{2\};\\\\ \+\\infty,&\\text\{otherwise\}\.\\end\{cases\}\(4\)
In addition, recall that the*mutual information*of two random variablesAAandBB, defined on the same probability space, is defined as

I\(A;B\):=DKL\(Law\(A,B\)∥Law\(A\)⊗Law\(B\)\)\.I\(A;B\):=D\_\{\\rm KL\}\(\\mathrm\{Law\}\(A,B\)\\parallel\\mathrm\{Law\}\(A\)\\otimes\\mathrm\{Law\}\(B\)\)\.\(5\)An additional third random variableCCallows us to define the*conditional mutual information*:

I\(A;B∣C\):=𝔼CDKL\(Law\(A,B∣C\)∥Law\(A∣C\)⊗Law\(B∣C\)\)\.I\(A;B\\mid C\):=\\mathbb\{E\}\_\{C\}D\_\{\\rm KL\}\(\\mathrm\{Law\}\(A,B\\mid C\)\\parallel\\mathrm\{Law\}\(A\\mid C\)\\otimes\\mathrm\{Law\}\(B\\mid C\)\)\.\(6\)It can be shown that the conditional mutual information satisfies

I⁡\(A;B∣C\)=H⁡\(A∣C\)−H⁡\(A∣B,C\)\.I\(A;B\\mid C\)=H\(A\\mid C\)\-H\(A\\mid B,C\)\.\(7\)Letζ1\\zeta\_\{1\}andζ2\\zeta\_\{2\}be probability measures on a measurable space\(Ω,ℱ\)\(\\Omega,\\mathcal\{F\}\)\. Thetotal variation distanceis defined as‖ζ1−ζ2‖TV:=supA∈ℱ\|ζ1​\(A\)−ζ2​\(A\)\|\\\|\\zeta\_\{1\}\-\\zeta\_\{2\}\\\|\_\{\\rm\\scriptscriptstyle TV\}:=\\sup\_\{A\\in\\mathcal\{F\}\}\|\\zeta\_\{1\}\(A\)\-\\zeta\_\{2\}\(A\)\|\.

###### Proposition 7\.

Under Assumption[6](https://arxiv.org/html/2610.41038#Thmtheorem6),ΔM:=supr≥MΔM,r=HM−H∞\\Delta\_\{M\}:=\\sup\_\{r\\geq M\}\\Delta\_\{M,r\}=H\_\{M\}\-H\_\{\\infty\}, andΔM↓0\\Delta\_\{M\}\\downarrow 0asM→∞M\\to\\infty\. Consequently,supr≥M𝔼Y0:r∥Pr−PrM∥TV≤ΔM/2↓0\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sqrt\{\\Delta\_\{M\}/2\}\\downarrow 0\.

### 2\.2The hidden Markov model setup:\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is Markov but not necessarily stationary

#### 2\.2\.1The case where\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is first order Markov

We assume that\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is a time\-homogeneous Markov chain, such that

Pr\(Zr\+1∈⋅∣Z0:r=z0:r\)=Pr\(Zr\+1∈⋅∣Zr=zr\)=:κ\(⋅∣zr\)\\mathrm\{Pr\}\(Z\_\{r\+1\}\\in\\cdot\\mid Z\_\{0:r\}=\\ z\_\{0:r\}\)=\\mathrm\{Pr\}\(Z\_\{r\+1\}\\in\\cdot\\mid Z\_\{r\}=z\_\{r\}\)=:\\kappa\(\\,\\cdot\\mid z\_\{r\}\)\(8\)for allr∈ℤ\+r\\in\\mathbb\{Z\}\_\{\+\}andz0:r∈Sr\+1z\_\{0:r\}\\in S^\{r\+1\}\. We have the observation channelQQand sequence\{Yr\}r∈ℤ\+\\\{Y\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}as before\.

The filter stability bounds derived by[McDonald and Yüksel \(2020\)](https://arxiv.org/html/2610.41038#bib.bib33)uses Dobrushin’s coefficient\([Dobrushin, 1956](https://arxiv.org/html/2610.41038#bib.bib14)\), thereby, it is a centerpiece to our analysis as well\. In the following, we define Dobrushin coefficient of a stochastic kernel between general measurable spaces in the*minorization form*\.

###### Definition 8\.

Dobrushin coefficient of a stochastic kernelKKon a measurable space𝖡\\mathsf\{B\}given another measurable space𝖠\\mathsf\{A\}is defined as

δ⁡\(K\):=inf∑i=1nmin⁡\{K⁡\(Bi∣a\),K⁡\(Bi∣a′\)\}\\delta\(K\):=\\inf\\sum\_\{i=1\}^\{n\}\\min\\\{K\(B\_\{i\}\\mid a\),K\(B\_\{i\}\\mid a^\{\\prime\}\)\\\}where the infimum is taken over alla,a′∈𝖠a,a^\{\\prime\}\\in\\mathsf\{A\}and all finite measurable partitions\{Bi\}i=1n\\\{B\_\{i\}\\\}\_\{i=1\}^\{n\}of𝖡\\mathsf\{B\}\.

Informally, the Dobrushin coeffient of a kernelKKcan be interpreted as the worst\-case one\-step probability of forgetting where you started\. It takes values in\[0,1\]\[0,1\]where11is often interpreted as total forgetting in one\-step and00tells us that there is no forced forgetting\. For any two probability measuresζ1,ζ2∈𝒫⁡\(𝖡\)\\zeta\_\{1\},\\zeta\_\{2\}\\in\\mathcal\{P\}\(\\mathsf\{B\}\), one has the Lipschitz property‖ζ1​K−ζ2​K‖TV≤\(1−δ⁡\(K\)\)​‖ζ1−ζ2‖TV\\\|\\zeta\_\{1\}K\-\\zeta\_\{2\}K\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\(1\-\\delta\(K\)\)\\\|\\zeta\_\{1\}\-\\zeta\_\{2\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\.

In the following proposition, we prove the exponential filter stability bounds for the next token predictive conditional distributions with full and truncated data, using\([McDonald and Yüksel, 2020](https://arxiv.org/html/2610.41038#bib.bib33), Corollary 3\.7\)\. We show that the difference is bounded by an exponential rate depending on the Dobrushin coefficients of the transition kernelκ\\kappaand the observation channelQQ\.

###### Proposition 9\.

Suppose Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\) holds\. Then, wheneverr≥M≥1r\\geq M\\geq 1, we have

𝔼Y0:r∥Pr−PrM∥TV≤αM\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\alpha^\{M\}withα=\(1−δ⁡\(κ\)\)​\(2−δ⁡\(Q\)\)\\alpha=\(1\-\\delta\(\\kappa\)\)\(2\-\\delta\(Q\)\)\.

Under suitable conditions, for which an example would beδ⁡\(κ\)\>1/2\\delta\(\\kappa\)\>1/2, it can be shown that the rate in Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9)gives an exponential decay\. In general, the faster convergence requires bothδ⁡\(κ\)\\delta\(\\kappa\)andδ⁡\(Q\)\\delta\(Q\)to be near11\. This is in parallel with the intuitive meaning of Dobrushin’s coefficient: Whenever it is close to11, the system forgets where it started, hence, the truncated data filter converges to the full history filter\.

#### 2\.2\.2The case where\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}isNthN^\{\\rm th\}order Markov

Now, we assume that\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is anNthN^\{\\rm th\}order Markov chain, that is

Pr\(Zr\+1∈⋅∣Z0:r=z0:r\)=Pr\(Zr\+1∈⋅∣Zr−N\+1:r=zr−N\+1:r\)\\mathrm\{Pr\}\(Z\_\{r\+1\}\\in\\cdot\\mid Z\_\{0:r\}=z\_\{0:r\}\)=\\mathrm\{Pr\}\(Z\_\{r\+1\}\\in\\cdot\\mid Z\_\{r\-N\+1:r\}=z\_\{r\-N\+1:r\}\)\(9\)for allr∈ℤ\+r\\in\\mathbb\{Z\}\_\{\+\}and allz0:r∈Sr\+1z\_\{0:r\}\\in S^\{r\+1\}\. Then we define the quantity on the right hand side in \([9](https://arxiv.org/html/2610.41038#S2.E9)\),𝜿\(⋅∣zr−N\+1:r\)\\bm\{\\kappa\}\(\\,\\cdot\\mid z\_\{r\-N\+1:r\}\)which defines a conditional probability distribution onSSgivenSNS^\{N\}\. We call𝜿\\bm\{\\kappa\}the*transition kernel*of\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}\. As before,\{Yr\}r∈ℤ\+\\\{Y\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is the induced observation variables emitted through the channelQQ\. The objects in Definition[2](https://arxiv.org/html/2610.41038#Thmtheorem2)transfer directly to here as well\. In this setting,\{\(Zr,Yr\)\}r∈ℤ\+\\\{\(Z\_\{r\},Y\_\{r\}\)\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}defines a hidden \(partially observed\)NthN^\{\\rm th\}order Markov model\.

We use a block hidden Markov model that we derive from the originalNthN^\{\\rm th\}\-order partially observed Markovian process as an apparatus for enabling us to use the machinery provided by[McDonald and Yüksel \(2020\)](https://arxiv.org/html/2610.41038#bib.bib33)\. To this end, we fix a time indexr∈ℕr\\in\\mathbb\{N\}and a window lengthMMthat is larger thanNNthroughout\. Moreover, we definem:=⌊M/N⌋m:=\\lfloor M/N\\rfloorandres:=r−m​N\\mathrm\{res\}:=r\-mN, where the former serves as the time\-index for the block process and the latter is the residual time\-index due to block lifting\.

For eachi=1,…,mi=1,\\ldots,m, we define two processes:

𝐗i:=\(Zres\+\(i−1\)​N\+1,…,Zres\+i​N\)∈SN\\mathbf\{X\}\_\{i\}:=\(Z\_\{\\mathrm\{res\}\+\(i\-1\)N\+1\},\\ldots,Z\_\{\\mathrm\{res\}\+iN\}\)\\in S^\{N\}\(10\)and

𝐘i:=\(Yres\+\(i−1\)​N\+1,…,Yres\+i​N\)∈𝕐N\\mathbf\{Y\}\_\{i\}:=\(Y\_\{\\mathrm\{res\}\+\(i\-1\)N\+1\},\\ldots,Y\_\{\\mathrm\{res\}\+iN\}\)\\in\\mathbb\{Y\}^\{N\}\(11\)with𝐗0:=\(Z1,…,Zres\)\\mathbf\{X\}\_\{0\}:=\(Z\_\{1\},\\ldots,Z\_\{\\mathrm\{res\}\}\)\. The indices are chosen in a way that𝐗m=\(Zr−N\+1,…,Zr\)\\mathbf\{X\}\_\{m\}=\(Z\_\{r\-N\+1\},\\ldots,Z\_\{r\}\)and𝐘1:m\\mathbf\{Y\}\_\{1:m\}lies inside\{Yr−M\+1,…,Yr\}\\\{Y\_\{r\-M\+1\},\\ldots,Y\_\{r\}\\\}\. In addition, we haveM−m​N<NM\-mN<Nobservations as leftover\. For these processes, we define three kernels: For any𝐳=\(z1,…,zN\),𝐳′=\(z1′,…,zN′\)∈SN\\mathbf\{z\}=\(z\_\{1\},\\ldots,z\_\{N\}\),\\mathbf\{z\}^\{\\prime\}=\(z\_\{1\}^\{\\prime\},\\ldots,z\_\{N\}^\{\\prime\}\)\\in S^\{N\}and𝐲=\(y1,…,yN\)∈𝕐N\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{N\}\)\\in\\mathbb\{Y\}^\{N\}

- \(K1\)Theblock transition kernel 𝐓⁡\(𝐳′∣𝐳\):=∏j=1N𝜿⁡\(zj′∣zj,…,zN,z1′,…,zj−1′\)∈𝒫⁡\(SN∣SN\)\.\\mathbf\{T\}\(\\mathbf\{z\}^\{\\prime\}\\mid\\mathbf\{z\}\):=\\prod\_\{j=1\}^\{N\}\\bm\{\\kappa\}\(z\_\{j\}^\{\\prime\}\\mid z\_\{j\},\\ldots,z\_\{N\},z\_\{1\}^\{\\prime\},\\ldots,z\_\{j\-1\}^\{\\prime\}\)\\in\\mathcal\{P\}\(S^\{N\}\\mid S^\{N\}\)\.
- \(K2\)Theblock observation kernel 𝐐⁡\(𝐲∣𝐳\):=∏j=1NQ⁡\(yj∣zj\)∈𝒫⁡\(𝕐N∣SN\)\.\\mathbf\{Q\}\(\\mathbf\{y\}\\mid\\mathbf\{z\}\):=\\prod\_\{j=1\}^\{N\}Q\(y\_\{j\}\\mid z\_\{j\}\)\\in\\mathcal\{P\}\(\\mathbb\{Y\}^\{N\}\\mid S^\{N\}\)\.
- \(K3\)Theone\-step prediction kernel 𝐏𝐫𝐞𝐝⁡\(y∣𝐳\)=∑z∈Sψ⁡\(y∣z\)​𝜿​\(z∣𝐳\)∈𝒫⁡\(𝕐∣SN\),y∈𝕐\.\\mathbf\{Pred\}\(y\\mid\\mathbf\{z\}\)=\\sum\_\{z\\in S\}\\psi\(y\\mid z\)\\,\\bm\{\\kappa\}\(z\\mid\\mathbf\{z\}\)\\in\\mathcal\{P\}\(\\mathbb\{Y\}\\mid S^\{N\}\),\\quad y\\in\\mathbb\{Y\}\.

Our proof strategy is to reduce theNthN^\{\\rm th\}order Markov chain to a first order one with the block lifting, which is now a known territory to us\.

Sliding lift𝐗r=\(Zr−N\+1,…,Zr\)\\mathbf\{X\}\_\{r\}=\(Z\_\{r\-N\+1\},\\dots,Z\_\{r\}\)Z1Z\_\{1\}Z2Z\_\{2\}⋯\\cdotsZNZ\_\{N\}ZN\+1Z\_\{N\+1\}N−1N\-1shared coordinates𝐗1\\mathbf\{X\}\_\{1\}𝐗2\\mathbf\{X\}\_\{2\}Block lift𝐗i=\(Zres\+\(i−1\)​N\+1,…,Zres\+i​N\)\\mathbf\{X\}\_\{i\}=\(Z\_\{\\mathrm\{res\}\+\(i\-1\)N\+1\},\\dots,Z\_\{\\mathrm\{res\}\+iN\}\)Z1Z\_\{1\}⋯\\cdotsZNZ\_\{N\}ZN\+1Z\_\{N\+1\}⋯\\cdotsZ2​NZ\_\{2N\}⋯\\cdots𝐗1\\mathbf\{X\}\_\{1\}𝐗2\\mathbf\{X\}\_\{2\}𝐓\\mathbf\{T\}Figure 3:Two ways of lifting anNthN^\{\\rm th\}order Markov chain to a first order one\. Sliding lift makes the process share coordinates as it evolves, therefore, resulting in zero Dobrushin coefficient\. The block lifting solves this issue\.The following proof is a reiteration of the proof of Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9)for the block Markov reduction of theNthN^\{\\rm th\}order Markov model with additional worked out details\.

###### Proposition 10\.

Suppose Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1, A2\) hold\. Then, wheneverr≥M≥N≥1r\\geq M\\geq N\\geq 1, we have

𝔼Y0:r∥Pr−PrM∥TV≤𝜶⌊M/N⌋\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}with𝛂:=\(1−δ⁡\(𝐓\)\)​\(2−δ⁡\(𝐐\)\)\\bm\{\\alpha\}:=\(1\-\\delta\(\\mathbf\{T\}\)\)\(2\-\\delta\(\\mathbf\{Q\}\)\)\.

#### 2\.2\.3Filter stability and sufficiency of finite memory

We finally merge the findings of §[2\.2\.1](https://arxiv.org/html/2610.41038#S2.SS2.SSS1)and §[2\.2\.2](https://arxiv.org/html/2610.41038#S2.SS2.SSS2)to fully characterize the effect of memory on next token prediction\.

###### Theorem 12\.

Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1, A2\), for allM≥NM\\geq Nand alls∈ℕs\\in\\mathbb\{N\}, we have

- \(i\)withα=\(1−δ⁡\(κ\)\)​\(2−δ⁡\(Q\)\)\\alpha=\(1\-\\delta\(\\kappa\)\)\(2\-\\delta\(Q\)\), supr≥M𝔼Y0:r∥Pr−PrM,s∥TV≤αM\+2​M​Lψqmin​s\.\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\alpha^\{M\}\+\\frac\{2ML\_\{\\psi\}\}\{q\_\{\\rm min\}s\}\.
- \(ii\)with𝜶=\(1−δ⁡\(𝐓\)\)​\(2−δ⁡\(𝐐\)\)\\bm\{\\alpha\}=\(1\-\\delta\(\\mathbf\{T\}\)\)\(2\-\\delta\(\\mathbf\{Q\}\)\), supr≥M𝔼Y0:r∥Pr−PrM,s∥TV≤𝜶⌊M/N⌋\+2​M​Lψqmin​s\.\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}\+\\frac\{2ML\_\{\\psi\}\}\{q\_\{\\rm min\}s\}\.

Theorem[12](https://arxiv.org/html/2610.41038#Thmtheorem12)’s findings raise two naturally arising questions: \(i\) How much memory is sufficient for next token estimation? \(ii\) How fine must the quantization be?

###### Corollary 13\.

Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1, A2\), for any precisionε\>0\\varepsilon\>0ifs≥4​M​Lψ/\(qmin​ε\)s\\geq 4ML\_\{\\psi\}/\(q\_\{\\rm min\}\\varepsilon\)and, in addition

- \(i\)α<1\\alpha<1holds, and whenever M≥log⁡\(2/ε\)log⁡\(1/α\),M\\geq\\frac\{\\log\{\(2/\\varepsilon\)\}\}\{\\log\(1/\\alpha\)\},holds, whereα\\alphaas in Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9),
- \(ii\)𝜶<1\\bm\{\\alpha\}<1holds, and whenever M≥N​⌈log⁡\(2/ε\)log⁡\(1/𝜶\)⌉,M\\geq N\\bigg\\lceil\\frac\{\\log\{\(2/\\varepsilon\)\}\}\{\\log\(1/\\bm\{\\alpha\}\)\}\\bigg\\rceil,holds, where𝜶\\bm\{\\alpha\}as in Proposition[10](https://arxiv.org/html/2610.41038#Thmtheorem10),

we get

supr≥M𝔼Y0:r∥Pr−PrM,s∥TV≤ε\.\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\varepsilon\.

## 3The Optimal Control Formulation of Transformers

We follow[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3)’s formulation; however, with a divergence: we do not consider positional encoding separately in the first lift, but rather encode it in the token embedding itself\. This depicts how position is used in early Transformers \(now it is done at the attention matrix level, see\([Barbero et al\., 2024](https://arxiv.org/html/2610.41038#bib.bib6)\), which requires generating a token from a static ’vocabulary’ and then embedding it with positional information\.

### 3\.1Transformers as Discrete\-Time Dynamical Systems with Open\-Loop Control

We recall the definitions and conventions of[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3)for a self\-contained discussion\. Let𝕏\\mathbb\{X\}be the token embedding space and𝕌\\mathbb\{U\}be the space of network weights in one Transformer layer\. We define the particle flow functionf:𝕏×𝒫⁡\(𝕏\)×𝕌→𝕏f:\\mathbb\{X\}\\times\\mathcal\{P\}\(\\mathbb\{X\}\)\\times\\mathbb\{U\}\\to\\mathbb\{X\}for allx∈𝕏x\\in\\mathbb\{X\},μ∈𝒫⁡\(𝕏\)\\mu\\in\\mathcal\{P\}\(\\mathbb\{X\}\), andU=\(WW,WA,Wb,WQ,WK,WV,EE,EU\)∈𝕌U=\(W\_\{W\},W\_\{A\},W\_\{b\},W\_\{Q\},W\_\{K\},W\_\{V\},E\_\{E\},E\_\{U\}\)\\in\\mathbb\{U\}by

f⁡\(x,μ,U\):=WW​σ​\(WA​x\+Wb\)\+∫𝕏μ⁡\(𝑑z\)​eβ⁡⟨WQ​x,WK​z⟩​WV​z∫𝕏μ⁡\(𝑑z\)​eβ⁡⟨WQ​x,WK​z⟩f\(x,\\mu,U\):=W\_\{W\}\\sigma\(W\_\{A\}x\+W\_\{b\}\)\+\\dfrac\{\\displaystyle\\int\_\{\\mathbb\{X\}\}\\mu\(\\mathrm\{d\}z\)\\,e^\{\\beta\\langle W\_\{Q\}x,W\_\{K\}z\\rangle\}\\,W\_\{V\}z\}\{\\displaystyle\\int\_\{\\mathbb\{X\}\}\\mu\(\\mathrm\{d\}z\)\\,e^\{\\beta\\langle W\_\{Q\}x,W\_\{K\}z\\rangle\}\}for a sharpening parameterβ≥0\\beta\\geq 0\. The feed\-forward block is the left term, and the attention block is the right term\. The number of layers is denoted byT∈ℕT\\in\\mathbb\{N\}where𝒯:=\{0,…,T−1\}\\mathcal\{T\}:=\\\{0,\\ldots,T\-1\\\}and𝒯¯:=𝒯∪\{T\}\\overline\{\\mathcal\{T\}\}:=\\mathcal\{T\}\\cup\\\{T\\\}\.

We encode the positionp∈PELp\\in\\mathrm\{PE\}\_\{L\}intos∈Ss\\in SviaEE:S×PEL→𝕏E\_\{E\}:S\\times\\mathrm\{PE\}\_\{L\}\\to\\mathbb\{X\}to producex∈𝕏x\\in\\mathbb\{X\}whereL∈ℕL\\in\\mathbb\{N\}is the input sequence length\. Henceforth, we assume that positional information is embedded and retrievable inxix^\{i\}viaEEE\_\{E\}fori∈ℒ:=\{1,…,L\}i\\in\\mathcal\{L\}:=\\\{1,\\ldots,L\\\}including under transitions\. That is, we assume the following:

###### Assumption 15\.

There exists an unembedding functionEU:𝕏→ℝ\|Λs\|×PELE\_\{U\}:\\mathbb\{X\}\\to\\mathbb\{R\}^\{\|\\Lambda\_\{s\}\|\}\\times\\mathrm\{PE\}\_\{L\}such thatEU∘EE=idS×PELE\_\{U\}\\circ E\_\{E\}=\{\\text\{id\}\_\{S\\times\\mathrm\{PE\}\_\{L\}\}\}andEU\(f\(x,μ,U\)\)=\(\.,p\)⇔EU\(x\)=\(\.,p\)E\_\{U\}\(f\(x,\\mu,U\)\)=\(\.,p\)\\iff E\_\{U\}\(x\)=\(\.,p\)\.

In practice, this may correspond to a positional encoder and decoder where the particle flow function sufficiently preserves proximity to the original positional encoding such that projecting it onto the image of the embedded vocabularyEE​\(S×PEL\)E\_\{E\}\(S\\times\\mathrm\{PE\}\_\{L\}\)in𝕏\\mathbb\{X\}retrieves the decoded position\.

For our realization, for anyt∈𝒯¯t\\in\\overline\{\\mathcal\{T\}\}, we takexti∈𝕏x^\{i\}\_\{t\}\\in\\mathbb\{X\},Ut∈𝕌U\_\{t\}\\in\\mathbb\{U\}, and defineμt:=1L​∑i=1Lδxti∈𝒫⁡\(𝕏\)\\mu\_\{t\}:=\\frac\{1\}\{L\}\\sum\_\{i=1\}^\{L\}\\delta\_\{x\_\{t\}^\{i\}\}\\in\\mathcal\{P\}\(\\mathbb\{X\}\)fori∈ℒi\\in\\mathcal\{L\}andt∈𝒯t\\in\\mathcal\{T\}\. The weights ofUtU\_\{t\}are specified asWW,t∈ℝd×d1,WA,t∈ℝd1×d,Wb,t∈ℝd1,WQ,t,WK,t∈ℝd2×d,WV,t∈ℝd×dW\_\{W,t\}\\in\\mathbb\{R\}^\{d\\times d\_\{1\}\},W\_\{A,t\}\\in\\mathbb\{R\}^\{d\_\{1\}\\times d\},W\_\{b,t\}\\in\\mathbb\{R\}^\{d\_\{1\}\},W\_\{Q,t\},W\_\{K,t\}\\in\\mathbb\{R\}^\{d\_\{2\}\\times d\},W\_\{V,t\}\\in\\mathbb\{R\}^\{d\\times d\}ford1,d2∈ℕd\_\{1\},d\_\{2\}\\in\\mathbb\{N\}\. The first layerU0U\_\{0\}additionally includes the embedding matrixEE∈ℝd×\|Λs\|E\_\{E\}\\in\\mathbb\{R\}^\{d\\times\|\\Lambda\_\{s\}\|\}and the last layerUTU\_\{T\}includes the unembedding matrixEU∈ℝ\|Λs\|×dE\_\{U\}\\in\\mathbb\{R\}^\{\|\\Lambda\_\{s\}\|\\times d\}\. The particle flow function represents the evolution of a single token embedding, such that

f⁡\(xti,μt,Ut\)\\displaystyle f\(x\_\{t\}^\{i\},\\mu\_\{t\},U\_\{t\}\)=Wt​σ​\(WA,t​xti\+Wb,t\)\+∫𝕏μt​\(𝑑z\)​eβ⁡⟨WQ,t​xti,WK,t​z⟩​WV,t​z∫𝕏μt​\(𝑑z\)​eβ⁡⟨WQ,t​xti,WK,t​z⟩\\displaystyle=W\_\{t\}\\sigma\(W\_\{A,t\}x\_\{t\}^\{i\}\+W\_\{b,t\}\)\+\\frac\{\\int\_\{\\mathbb\{X\}\}\\mu\_\{t\}\(\\mathrm\{d\}z\)e^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}z\\rangle\}W\_\{V,t\}z\}\{\\int\_\{\\mathbb\{X\}\}\\mu\_\{t\}\(\\mathrm\{d\}z\)e^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}z\\rangle\}\}=Wt​σ​\(WA,t​xti\+Wb,t\)\+∑j=1Leβ⁡⟨WQ,t​xti,WK,t​xtj,k⟩​WV,t​xtj,k∑l=1Leβ⁡⟨WQ,t​xti,WK,t​xtl,k⟩=xt\+1i\.\\displaystyle=W\_\{t\}\\sigma\(W\_\{A,t\}x\_\{t\}^\{i\}\+W\_\{b,t\}\)\+\\frac\{\\sum\_\{j=1\}^\{L\}e^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x^\{j,k\}\_\{t\}\\rangle\}W\_\{V,t\}x^\{j,k\}\_\{t\}\}\{\\sum\_\{l=1\}^\{L\}e^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x^\{l,k\}\_\{t\}\\rangle\}\}=x^\{i\}\_\{t\+1\}\.
We define the*measure flow map*Φ:𝒫⁡\(𝕏\)×𝕌→𝒫⁡\(𝕏\)\\Phi:\\mathcal\{P\}\(\\mathbb\{X\}\)\\times\\mathbb\{U\}\\to\\mathcal\{P\}\(\\mathbb\{X\}\)such that

Φ⁡\(μ,U\):=μ∘Tμ,U−1\\Phi\(\\mu,U\):=\\mu\\circ T^\{\-1\}\_\{\\mu,U\}\(12\)whereTμ,U:𝕏→𝕏T\_\{\\mu,U\}:\\mathbb\{X\}\\to\\mathbb\{X\}is given byTμ,U​\(x\)=f⁡\(x,μ,U\)T\_\{\\mu,U\}\(x\)=f\(x,\\mu,U\)for allx∈𝕏x\\in\\mathbb\{X\}\. A change\-of\-measures argument yields thatΦ⁡\(μt,U\)=μt\+1\\Phi\(\\mu\_\{t\},U\)=\\mu\_\{t\+1\}for allt∈𝒯t\\in\\mathcal\{T\}andk∈𝒦k\\in\\mathcal\{K\}\. This shows thatΦ\\Phiis indeed the right tool to describe measure\-level Transformer dynamics\. Under compactness assumptions that can be realizable via practical tools,Φ\\Phiis jointly continuous on its domain\. Moreover,\{μt\}t∈𝒯¯\\\{\\mu\_\{t\}\\\}\_\{t\\in\\overline\{\\mathcal\{T\}\}\}defines a weak Feller Markov decision process, that is, the process\{\(μt,Ut\):t∈𝒯\}∪\{μT\}\\\{\(\\mu\_\{t\},U\_\{t\}\):t\\in\\mathcal\{T\}\\\}\\cup\\\{\\mu\_\{T\}\\\}is controlled Markov chain with state space𝒫⁡\(𝕏\)\\mathcal\{P\}\(\\mathbb\{X\}\), action space𝕌\\mathbb\{U\}, and with transition kernel

η\(⋅∣μ,U\):=δΦ⁡\(μ,U\)\(⋅\)\\eta\(\\,\\cdot\\mid\\mu,U\):=\\delta\_\{\\Phi\(\\mu,U\)\}\(\\cdot\)which satisfies: For any continuous and boundedc:𝒫⁡\(𝕏\)→ℝc:\\mathcal\{P\}\(\\mathbb\{X\}\)\\to\\mathbb\{R\}, the function

𝒫⁡\(𝕏\)×𝕌∋\(μ,U\)↦∫𝒫⁡\(𝕏\)η⁡\(d​μ′∣μ,U\)​c​\(μ′\)\\mathcal\{P\}\(\\mathbb\{X\}\)\\times\\mathbb\{U\}\\ni\(\\mu,U\)\\mapsto\\int\_\{\\mathcal\{P\}\(\\mathbb\{X\}\)\}\\eta\(\\mathrm\{d\}\\mu^\{\\prime\}\\mid\\mu,U\)\\,c\(\\mu^\{\\prime\}\)is jointly continuous on𝒫⁡\(𝕏\)×𝕌\\mathcal\{P\}\(\\mathbb\{X\}\)\\times\\mathbb\{U\}, where𝒫⁡\(𝕏\)\\mathcal\{P\}\(\\mathbb\{X\}\)is given the weak∗\-topology, metrized by the22\-Wasserstein loss

W2​\(P,Q\)=infπ∈Π⁡\(P,Q\)∫S×S‖x−y‖22​π​\(𝑑x,𝑑y\)W\_\{2\}\(P,Q\)=\\inf\_\{\\pi\\in\\Pi\(P,Q\)\}\\int\_\{S\\times S\}\\\|x\-y\\\|\_\{2\}^\{2\}\\,\\pi\(\\mathrm\{d\}x,\\mathrm\{d\}y\)\(13\)whereΠ\(P,Q\):=\{ζ∈𝒫\(S×S\):ζ\(S×⋅\)=P\(⋅\)andζ\(⋅×S\)=Q\(⋅\)\}\\Pi\(P,Q\):=\\\{\\zeta\\in\\mathcal\{P\}\(S\\times S\):\\zeta\(S\\times\\cdot\)=P\(\\cdot\)\\text\{ and \}\\zeta\(\\cdot\\times S\)=Q\(\\cdot\)\\\}\.

Note that the Transformer dynamics define a deterministic Markov chain\. The noise in Transformer training, or in general, neural network training is due to mini\-batching process\.

### 3\.2Data Set Construction

We generateK~\\tilde\{K\}sequences of lengthL\+1L\+1from the quantized process\{qs​\(Yr\)\}r∈ℤ\+\\\{q^\{s\}\(Y\_\{r\}\)\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}\. We split each sequence into a training pair by taking𝒙k=\(x1,k,…,xL,k\)\\bm\{x\}^\{k\}=\(x^\{1,k\},\\dots,x^\{L,k\}\)as the firstLLelements of the sequence andy~k\\tilde\{y\}^\{k\}as the last element, fork∈𝒦~=\{1,…,K~\}k\\in\\tilde\{\\mathcal\{K\}\}=\\\{1,\\dots,\\tilde\{K\}\\\}\. We aggregate these prompts to an initial single state\-labeled data set, given by

D~=\{\(𝒙k,y~k\):k∈𝒦~\}∈\(ΛsL×Λs\)K~\\tilde\{D\}=\\\{\(\\bm\{x\}^\{k\},\\tilde\{y\}^\{k\}\):k\\in\\tilde\{\\mathcal\{K\}\}\\\}\\in\(\\Lambda\_\{s\}^\{L\}\\times\\Lambda\_\{s\}\)^\{\\tilde\{K\}\}where𝒙k\\bm\{x\}^\{k\}is the feature vector andy~k∈Λs\\tilde\{y\}^\{k\}\\in\\Lambda\_\{s\}is the target token embedding\. The initial data setD~\\tilde\{D\}has training pairs indexed by𝒦~\\tilde\{\\mathcal\{K\}\}, with possibly some duplicate feature vectors, i\.e\.,𝒙j=𝒙l\\bm\{x\}^\{j\}=\\bm\{x\}^\{l\}forj≠lj\\neq l, andj,l∈𝒦~j,l\\in\\tilde\{\\mathcal\{K\}\}\.

We construct a new data setDDfromD~\\tilde\{D\}with unique feature vectors and labels as probability measures instead of individual token embeddings\. We index the unique feature vectors with𝒦=\{1,…,K\}\\mathcal\{K\}=\\\{1,\\dots,K\\\}, picking one representative index from𝒦~\\tilde\{\\mathcal\{K\}\}for each unique feature vector\. Fork∈𝒦k\\in\\mathcal\{K\}, we define the empirical measureyk∈𝒫⁡\(Λs\)y^\{k\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)conditioned on𝒙k\\bm\{x\}^\{k\}as

yk=∑l=1K~δy~l𝟏\{𝒙l=𝒙k\}∑l=1K~𝟏\{𝒙l=𝒙k\}y^\{k\}=\\frac\{\\sum\_\{l=1\}^\{\\tilde\{K\}\}\\delta\_\{\\tilde\{y\}^\{l\}\}\\mathbf\{1\}\_\{\\\{\\bm\{x\}^\{l\}=\\bm\{x\}^\{k\}\\\}\}\}\{\\sum\_\{l=1\}^\{\\tilde\{K\}\}\\mathbf\{1\}\_\{\\\{\\bm\{x\}^\{l\}=\\bm\{x\}^\{k\}\\\}\}\}
ThenD=\{\(𝒙k,yk\):k∈𝒦\}∈\(ΛsL×𝒫⁡\(Λs\)\)KD=\\\{\(\\bm\{x\}^\{k\},y^\{k\}\):k\\in\\mathcal\{K\}\\\}\\in\(\\Lambda\_\{s\}^\{L\}\\times\\mathcal\{P\}\(\\Lambda\_\{s\}\)\)^\{K\}has unique training samples𝒙k\\bm\{x\}^\{k\}, with labelsyky^\{k\}as empirical probability measures\. Note that the denominator is non\-zero since it is the count of samples inD^\\hat\{D\}with feature vector𝒙k\\bm\{x\}^\{k\}\.

A distinction between this data set construction and that of most large language models \(LLMs\) is that here we exclude elements of generated process appearing after the target label before training at the data set level, whereas most current pre\-trained LLMs use the technique of masking of[Devlin et al\. \(2019\)](https://arxiv.org/html/2610.41038#bib.bib13)during training, see\([Zhao et al\., 2024](https://arxiv.org/html/2610.41038#bib.bib53)\)\.

We then constructμ0k:=1L​∑i=1Lδxi,k\\mu\_\{0\}^\{k\}:=\\frac\{1\}\{L\}\\sum\_\{i=1\}^\{L\}\\delta\_\{x^\{i,k\}\}and push it through the time horizon via lifted Transformer dynamicsΦ\\Phifor allk∈𝒦:=\{1,…,K\}k\\in\\mathcal\{K\}:=\\\{1,\\ldots,K\\\}\. We write\{μtk\}𝒯¯\\\{\\mu\_\{t\}^\{k\}\\\}\_\{\\overline\{\\mathcal\{T\}\}\}for the resulting process\. If we concatenate all of these processes over the data set, then we obtain another MDP with state process\{𝝁t\}t∈𝒯¯=\{\(μt1,…,μtK\)\}t∈𝒯¯\\\{\\bm\{\\mu\}\_\{t\}\\\}\_\{t\\in\\overline\{\\mathcal\{T\}\}\}=\\\{\(\\mu\_\{t\}^\{1\},\\ldots,\\mu\_\{t\}^\{K\}\)\\\}\_\{t\\in\\overline\{\\mathcal\{T\}\}\}for which the state space is𝒫​\(𝕏\)K\\mathcal\{P\}\(\\mathbb\{X\}\)^\{K\}, the action space is𝕌\\mathbb\{U\}and the transition kernel is

𝜼K\(⋅∣𝝁t,U\):=δ𝚽⁡\(𝝁t,U\)\(⋅\),t∈𝒯\.\\bm\{\\eta\}^\{K\}\(\\,\\cdot\\mid\\bm\{\\mu\}\_\{t\},U\):=\\delta\_\{\\bm\{\\Phi\}\(\\bm\{\\mu\}\_\{t\},U\)\}\(\\cdot\),\\quad t\\in\\mathcal\{T\}\.\(14\)This can be regarded as an ensemble control problem\([Li and Khaneja, 2009](https://arxiv.org/html/2610.41038#bib.bib31)\)\.

### 3\.3The Loss Structure

Having defined the particle dynamics and the data setDD, we now specify how to train the policyγ\\gamma\. The Transformer outputs an unembedding score vectorEU​xT,γk,L∈ℝ\|Λs\|E\_\{U\}x^\{k,L\}\_\{T,\\gamma\}\\in\\mathbb\{R\}^\{\|\\Lambda\_\{s\}\|\}for each prompt𝒙k\\bm\{x\}^\{k\}\. We convert this into a probability measure and penalize its distance from the empirical labelyk∈𝒫⁡\(Λs\)y^\{k\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)\.

We employ the softmax function to obtain probability distributions from our predicted unembedding scores\. For a vectorz∈ℝ\|Λs\|z\\in\\mathbb\{R\}^\{\|\\Lambda\_\{s\}\|\},softmax:ℝ\|Λs\|→𝒫⁡\(Λs\)\\text\{softmax\}:\\mathbb\{R\}^\{\|\\Lambda\_\{s\}\|\}\\to\\mathcal\{P\}\(\\Lambda\_\{s\}\)mapszzonto the probability simplex as

softmax⁡\(z\)=\(ez1∑j=1\|Λs\|ezj,…,ez\|Λs\|∑j=1\|Λs\|ez\|Λs\|\)\.\\mathrm\{softmax\}\(z\)=\\left\(\\frac\{e^\{z\_\{1\}\}\}\{\\sum\_\{j=1\}^\{\|\\Lambda\_\{s\}\|\}e^\{z\_\{j\}\}\},\\dots,\\frac\{e^\{z\_\{\|\\Lambda\_\{s\}\|\}\}\}\{\\sum\_\{j=1\}^\{\|\\Lambda\_\{s\}\|\}e^\{z\_\{\|\\Lambda\_\{s\}\|\}\}\}\\right\)\.This grants ussoftmax​\(z\)j∈\(0,1\)\\text\{softmax\}\(z\)\_\{j\}\\in\(0,1\)and∑j=1\|Λs\|softmax​\(z\)j=1\\sum\_\{j=1\}^\{\|\\Lambda\_\{s\}\|\}\\text\{softmax\}\(z\)\_\{j\}=1, providing us a valid probability measure in𝒫⁡\(S\)\\mathcal\{P\}\(S\)\. A useful property of softmax is shifting invariance, wheresoftmax​\(z\)=softmax​\(z\+c​𝟏\)\\text\{softmax\}\(z\)=\\text\{softmax\}\(z\+c\\mathbf\{1\}\), forc∈ℝc\\in\\mathbb\{R\}applied component\-wise tozz\.

We defineνk:=softmax​\(EU​xT,γk,L\)∈𝒫⁡\(S\)\\nu^\{k\}:=\\text\{softmax\}\(E\_\{U\}\{x\_\{T,\\gamma\}^\{k,L\}\}\)\\in\\mathcal\{P\}\(S\), wherexT,γk,L∈𝕏x^\{k,L\}\_\{T,\\gamma\}\\in\\mathbb\{X\}is initialized atx0,γk,L∈𝕏x^\{k,L\}\_\{0,\\gamma\}\\in\\mathbb\{X\}with its empirical measureμ0,γk\\mu^\{k\}\_\{0,\\gamma\}evolving according to the pushforwardΦ\\Phiwith actions determined by a policyγ:𝒫​\(𝕏\)K→𝕌\\gamma:\\mathcal\{P\}\(\\mathbb\{X\}\)^\{K\}\\to\\mathbb\{U\}\. We aim to find a policyγ∈Γ\\gamma\\in\\Gammathat minimizes the following loss structure\.

Cross\-Entropy loss is commonly used for machine learning classification tasks and autoregressive generative models\([Ciampiconi et al\., 2026](https://arxiv.org/html/2610.41038#bib.bib12)\)\. It is well\-suited for next\-token prediction, which can be seen as classification over the entire vocabulary\. For a true probability measureP1P\_\{1\}and a predicted measureP2P\_\{2\}in𝒫⁡\(Λs\)\\mathcal\{P\}\(\\Lambda\_\{s\}\), withP2P\_\{2\}being absolutely continuous with respect to some reference measureλo\\lambda\_\{o\}, the cross entropy is given by

CE\(P1,P2\)=𝔼P1\[−logp2\(x\)\]=−∫ΛsP1\(dv\)logp2\(v\)\\mathrm\{CE\}\(P\_\{1\},P\_\{2\}\)=\\mathbb\{E\}\_\{P\_\{1\}\}\[\-\\log p\_\{2\}\(x\)\]=\-\\int\_\{\\Lambda\_\{s\}\}P\_\{1\}\(\\mathrm\{d\}v\)\\,\\log p\_\{2\}\(v\)where the density ofP2P\_\{2\},p2=d​P2d​λop\_\{2\}=\\frac\{\\mathrm\{d\}P\_\{2\}\}\{\\mathrm\{d\}\\lambda\_\{o\}\}, is given by the Radon\-Nikodym derivative\. The continuous case has undesirable properties, such as potentially negative entropy, so we use the following discrete case, which is applicable for next\-token prediction over a finite vocabulary\. For discreteP1,P2∈𝒫⁡\(Λs\)P\_\{1\},P\_\{2\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\), the cross entropy is given by

CE\(P1,P2\)=−∑v∈ΛsP1\(v\)logP2\(v\)\.\\mathrm\{CE\}\(P\_\{1\},P\_\{2\}\)=\-\\sum\_\{v\\in\\Lambda\_\{s\}\}P\_\{1\}\(v\)\\log P\_\{2\}\(v\)\.For our formulation, we sum the prompt\-wise losses over the entire data set to get our final cross entropy loss functionalVCE:\(ΛsL×𝒫⁡\(Λs\)\)K×Γ→ℝV\_\{\\rm CE\}:\(\\Lambda\_\{s\}^\{L\}\\times\\mathcal\{P\}\(\\Lambda\_\{s\}\)\)^\{K\}\\times\\Gamma\\to\\mathbb\{R\}given by

VCE\(D,γ\)=∑k=1KnkK~CE\(yk,νk\)=−∑k=1KnkK~∑v∈Λsyk\(v\)logνk\(v\)V\_\{\\rm CE\}\(D,\\gamma\)=\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}\\mathrm\{CE\}\(y^\{k\},\\nu^\{k\}\)=\-\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}\\sum\_\{v\\in\\Lambda\_\{s\}\}y^\{k\}\(v\)\\log\\nu^\{k\}\(v\)\(15\)wherenk=∑l=1K~χ𝐱k​\(𝐱l\)n\_\{k\}=\\sum\_\{l=1\}^\{\\tilde\{K\}\}\\chi\_\{\\mathbf\{x\}^\{k\}\}\(\\mathbf\{x\}^\{l\}\)for allk=1,…,Kk=1,\\ldots,K\.

\([15](https://arxiv.org/html/2610.41038#S3.E15)\) is equivalent to the empirical risk of the original sample:

1K~∑l=1K~−logν\(y~l∣𝐱l\)=∑k=1KnkK~\(1nk∑l=1𝐱l=𝐱kK~−logν\(y~l∣𝐱l\)\)=∑k=1KnkK~CE\(yk,νk\)\.\\frac\{1\}\{\\tilde\{K\}\}\\sum\_\{l=1\}^\{\\tilde\{K\}\}\-\\log\\nu\(\\tilde\{y\}^\{l\}\\mid\\mathbf\{x\}^\{l\}\)=\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}\\bigg\(\\frac\{1\}\{n\_\{k\}\}\\sum\_\{\\begin\{subarray\}\{c\}l=1\\\\ \\mathbf\{x\}^\{l\}=\\mathbf\{x\}^\{k\}\\end\{subarray\}\}^\{\\tilde\{K\}\}\-\\log\{\\nu\(\\tilde\{y\}^\{l\}\\mid\\mathbf\{x\}^\{l\}\)\}\\bigg\)=\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}\\mathrm\{CE\}\(y^\{k\},\\nu^\{k\}\)\.
The cross entropy loss has no notion of the underlying geometry of the state space\. It depends only on whereppassigns the same probability mass asqq, not the proximity of non\-overlapping masses forp,q∈𝒫⁡\(𝕏\)p,q\\in\\mathcal\{P\}\(\\mathbb\{X\}\)\. This property is useful for vocabularies where the token geometry doesn’t reflect semantic information; adjacent tokens can have unrelated meanings\.

We only compare the probability measure of the last position \(pL=L∈PELp\_\{L\}=L\\in\\mathrm\{PE\}\_\{L\}\) in the output vectors to our label\. This is also a trick used in practice in Transformer fine\-tuning, where features and labels are separated in this way, see\([Radford et al\., 2018](https://arxiv.org/html/2610.41038#bib.bib39), Equation 3\)\.

### 3\.4Finite Model Approximations to Transformers

We select parametersl,m,n∈ℤ\+l,m,n\\in\\mathbb\{Z\}\_\{\+\}to quantize the above MDP into MDP\(l,m,n\)as follows: we quantize the values of the probability measure on the state space withll, the state space itself withnn, and the action space withmm\. We quantize the token embedding space𝕏\\mathbb\{X\}as𝕏n\\mathbb\{X\}\_\{n\}\. By the compactness of𝕏\\mathbb\{X\}, there exist someM𝕏∈ℝ\+M\_\{\\mathbb\{X\}\}\\in\\mathbb\{R\}\_\{\+\}that tightly bounds all coordinates of pointsx=\(x1,x2,…,xd\)x=\(x\_\{1\},x\_\{2\},\\dots,x\_\{d\}\)in the embedding space𝕏⊂ℝd\\mathbb\{X\}\\subset\\mathbb\{R\}^\{d\}, i\.e\.

M𝕏:=maxx∈𝕏,j∈\{0,…,d\}⁡‖xj‖\.M\_\{\\mathbb\{X\}\}:=\\max\_\{x\\in\\mathbb\{X\},\{j\\in\\\{0,\\dots,d\\\}\}\}\|\|x\_\{j\}\|\|\.So𝕏⊂\[−M𝕏,M𝕏\]d\\mathbb\{X\}\\subset\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]^\{d\}\. We place a grid𝕏n\\mathbb\{X\}\_\{n\}ofndn^\{d\}many points uniformly over\[−M𝕏,M𝕏\]d\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]^\{d\}with spacing2​M𝕏n−1\\frac\{2M\_\{\\mathbb\{X\}\}\}\{n\-1\}, which is finite forn\>1n\>1\. Then, givenx∈𝕏x\\in\\mathbb\{X\}, we can find somex^∈𝕏n\\hat\{x\}\\in\\mathbb\{X\}\_\{n\}that satisfies

‖x−x^‖≤12​2​M𝕏n−1​d=M𝕏​dn−1\\\|x\-\\hat\{x\}\\\|\\leq\\frac\{1\}\{2\}\\frac\{2M\_\{\\mathbb\{X\}\}\}\{n\-1\}\\sqrt\{d\}=\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}whered\\sqrt\{d\}comes from taking the L2 norm indddimensions\. We can arrive at such a grid arrangement by taking

𝕏n=\{\(−M𝕏\+2​M𝕏​k^1n−1,…,−M𝕏\+2​M𝕏​k^dn−1\):k^j∈\{0,…,n−1\}\}\.\\mathbb\{X\}\_\{n\}=\\\{\(\-M\_\{\\mathbb\{X\}\}\+\\frac\{2M\_\{\\mathbb\{X\}\}\\hat\{k\}\_\{1\}\}\{n\-1\},\\dots,\-M\_\{\\mathbb\{X\}\}\+\\frac\{2M\_\{\\mathbb\{X\}\}\\hat\{k\}\_\{d\}\}\{n\-1\}\):\\hat\{k\}\_\{j\}\\in\\\{0,\\dots,n\-1\\\}\\\}\.Quantizing𝕏\\mathbb\{X\}in this way ensures that the number of points is alwaysndn^\{d\}which lends itself nicely for implementation\.

We quantize the action space, imposing a grid on the entries of the parameter matrices\. Due to the compactness of𝕌\\mathbb\{U\}, there exists someM𝕌<∞M\_\{\\mathbb\{U\}\}<\\inftythat is the tightest bound for all elements of the weights\(W,A,b,Q,K,V,EE,EU\)\(W,A,b,Q,K,V,E\_\{E\},E\_\{U\}\)ofU∈𝕌U\\in\\mathbb\{U\}, i\.e\.

M𝕌:=maxU∈𝕌,θ∈U;i,j⁡\|θi,j\|M\_\{\\mathbb\{U\}\}:=\\max\_\{U\\in\\mathbb\{U\},\\theta\\in U;i,j\}\|\\theta\_\{i,j\}\|whereθi,j\\theta\_\{i,j\}are matrix entries ofθ\\theta\. Thus,−M𝕌≤θi,j≤M𝕌\-M\_\{\\mathbb\{U\}\}\\leq\\theta\_\{i,j\}\\leq M\_\{\\mathbb\{U\}\}, and we placemmpoints uniformly over each parameter’s matrix entries with spacing2​M𝕌m−1\\frac\{2M\_\{\\mathbb\{U\}\}\}\{m\-1\}, which is finite form\>1m\>1\.

Then, givenU=\(WW,WA,Wb,WQ,WK,WV,EE,EU\)∈𝕌U=\(W\_\{W\},W\_\{A\},W\_\{b\},W\_\{Q\},W\_\{K\},W\_\{V\},E\_\{E\},E\_\{U\}\)\\in\\mathbb\{U\}, we can find someU^=\(WW^,WA^,Wb^CLOSE,\\hat\{U\}=\(\\hat\{W\_\{W\}\},\\hat\{W\_\{A\}\},\\hat\{W\_\{b\}\},OPENWQ^,WK^,WV^,E^E,E^U\)\\hat\{W\_\{Q\}\},\\hat\{W\_\{K\}\},\\hat\{W\_\{V\}\},\\hat\{E\}\_\{E\},\\hat\{E\}\_\{U\}\)such that

‖U−U^‖F\\displaystyle\|\|U\-\\hat\{U\}\|\|\_\{F\}≤12​2​M𝕌m−1​\(d⋅d1\+d1⋅d\+d1\+d2⋅d\+d2⋅dCLOSE\\displaystyle\\leq\\frac\{1\}\{2\}\\frac\{2M\_\{\\mathbb\{U\}\}\}\{m\-1\}\\bigg\(\\sqrt\{d\\cdot d\_\{1\}\}\+\\sqrt\{d\_\{1\}\\cdot d\}\+\\sqrt\{d\_\{1\}\}\+\\sqrt\{d\_\{2\}\\cdot d\}\+\\sqrt\{d\_\{2\}\\cdot d\}OPEN\+d⋅d\+\|Λs\|⋅d\+d⋅\|Λs\|\)\\displaystyle\\hskip 120\.00018pt\+\\sqrt\{d\\cdot d\}\+\\sqrt\{\|\\Lambda\_\{s\}\|\\cdot d\}\+\\sqrt\{d\\cdot\|\\Lambda\_\{s\}\|\}\\bigg\)=M𝕌m−1​\(2​d⋅d1\+d1\+2​d2⋅d\+d\+2​\|Λs\|⋅d\)\\displaystyle=\\frac\{M\_\{\\mathbb\{U\}\}\}\{m\-1\}\\bigg\(2\\sqrt\{d\\cdot d\_\{1\}\}\+\\sqrt\{d\_\{1\}\}\+2\\sqrt\{d\_\{2\}\\cdot d\}\+d\+2\\sqrt\{\|\\Lambda\_\{s\}\|\\cdot d\}\\bigg\)where the coefficient is determined from the dimensions of the parameter matrices and\|\|\.\|\|F\|\|\.\|\|\_\{F\}is the product\-wise Frobenius norm \(as squared\):

‖U‖F2=Tr⁡\(W​WT\)\+Tr⁡\(A​AT\)\+Tr⁡\(b​bT\)\+Tr⁡\(Q​QT\)\+Tr⁡\(K​KT\)\+Tr⁡\(V​VT\)\+Tr⁡\(EE​EET\)\+Tr⁡\(EU​EUT\)\.\\begin\{split\}\\\|U\\\|\_\{F\}^\{2\}&=\\mathrm\{Tr\}\(WW^\{T\}\)\+\\mathrm\{Tr\}\(AA^\{T\}\)\+\\mathrm\{Tr\}\(bb^\{T\}\)\\\\ &\\qquad\+\\mathrm\{Tr\}\(QQ^\{T\}\)\+\\mathrm\{Tr\}\(KK^\{T\}\)\+\\mathrm\{Tr\}\(VV^\{T\}\)\+\\mathrm\{Tr\}\(E\_\{E\}E\_\{E\}^\{T\}\)\+\\mathrm\{Tr\}\(E\_\{U\}E\_\{U\}^\{T\}\)\.\\end\{split\}Note that Transformers withT\>1T\>1layers do not have actions containing bothEEE\_\{E\}andEUE\_\{U\}; however, we keep them here to provide an upper bound\. A quantization that adheres to the above conditions can be obtained by taking

𝕌m=\{\(WW^,WA^,Wb^,WQ^,WK^,WV^,E^E,E^U\):\\displaystyle\\mathbb\{U\}\_\{m\}=\\big\\\{\(\\hat\{W\_\{W\}\},\\hat\{W\_\{A\}\},\\hat\{W\_\{b\}\},\\hat\{W\_\{Q\}\},\\hat\{W\_\{K\}\},\\hat\{W\_\{V\}\},\\hat\{E\}\_\{E\},\\hat\{E\}\_\{U\}\):θi,j=−M𝕌\+2​M𝕌​k^mm−1,\\displaystyle\\,\\theta\_\{i,j\}=\-M\_\{\\mathbb\{U\}\}\+\\frac\{2M\_\{\\mathbb\{U\}\}\\hat\{k\}\_\{m\}\}\{m\-1\},θ∈\{WW^,WA^,Wb^,WQ^,WK^,WV^,E^E,E^U\},k^m∈\{0,…,m−1\}\}\.\\displaystyle\\hskip\-120\.00018pt\\theta\\in\\\{\\hat\{W\_\{W\}\},\\hat\{W\_\{A\}\},\\hat\{W\_\{b\}\},\\hat\{W\_\{Q\}\},\\hat\{W\_\{K\}\},\\hat\{W\_\{V\}\},\\hat\{E\}\_\{E\},\\hat\{E\}\_\{U\}\\\},\\hat\{k\}\_\{m\}\\in\\\{0,\\dots,m\-1\\\}\\big\\\}\.
Finally, we quantize𝒫⁡\(𝕏n\)\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)by taking the set of quantized probability measures on𝕏n\\mathbb\{X\}\_\{n\}as

𝒫l\(𝕏n\):=\{\(pα\)α=1\|𝕏n\|:pα∈1l−1\{0,…,l−1\},∑α=1\|𝕏n\|pα=1\}\.\\mathcal\{P\}\_\{l\}\(\\mathbb\{X\}\_\{n\}\):=\\\{\(p\_\{\\alpha\}\)\_\{\\alpha=1\}^\{\|\\mathbb\{X\}\_\{n\}\|\}:p\_\{\\alpha\}\\in\\frac\{1\}\{l\-1\}\\\{0,\\dots,l\-1\\\},\\sum\_\{\\alpha=1\}^\{\|\\mathbb\{X\}\_\{n\}\|\}p\_\{\\alpha\}=1\\\}\.
We define the quantizersQn:𝕏→𝕏nQ\_\{n\}:\\mathbb\{X\}\\to\\mathbb\{X\}\_\{n\}mappingx∈𝕏x\\in\\mathbb\{X\}to the closest point in𝕏n\\mathbb\{X\}\_\{n\}, andRl:𝒫⁡\(𝕏n\)→𝒫l​\(𝕏n\)R\_\{l\}:\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\\to\\mathcal\{P\}\_\{l\}\(\\mathbb\{X\}\_\{n\}\)mappingμ^∈𝒫⁡\(𝕏n\)\\hat\{\\mu\}\\in\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)to the closest element in𝒫l​\(𝕏n\)\\mathcal\{P\}\_\{l\}\(\\mathbb\{X\}\_\{n\}\)using[Reznik \(2011\)](https://arxiv.org/html/2610.41038#bib.bib41)’s algorithm\. Quantizing the spaces in this fashion ensures a deterministic grid that is straightforward for implementation while guaranteeing the sameO⁡\(1/n\)O\(1/n\)asymptotic error rate\.

𝒫⁡\(𝕏\)\{\\lx@inpgf@ignorespaces\\mathcal\{P\}\(\\mathbb\{X\}\)\}𝒫⁡\(𝕏n\)\{\\lx@inpgf@ignorespaces\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\}𝒫\(l\)​\(𝕏n\)\{\\lx@inpgf@ignorespaces\\mathcal\{P\}^\{\(l\)\}\(\\mathbb\{X\}\_\{n\}\)\}S\{\\lx@inpgf@ignorespaces S\}𝕏\{\\lx@inpgf@ignorespaces\\mathbb\{X\}\}𝕏n\{\\lx@inpgf@ignorespaces\\mathbb\{X\}\_\{n\}\}pushforward via​Qn\\scriptstyle\{\\lx@inpgf@ignorespaces\\text\{pushforward via \}Q\_\{n\}\}Rl\\scriptstyle\{\\lx@inpgf@ignorespaces R\_\{l\}\}embed with​EE\\scriptstyle\{\\lx@inpgf@ignorespaces\\text\{embed with \}E\_\{E\}\}lift to a probability measureQn\\scriptstyle\{\\lx@inpgf@ignorespaces Q\_\{n\}\}lift to a probability measureFigure 4:Two\-stage State Quantization
### 3\.5Dynamic Programming Based Training

To guarantee that the quantized Transformer is trained optimally, we solve the finite state and action spaceMDP\(l,n,m\)\\text\{MDP\}^\{\(l,n,m\)\}exactly using backward dynamic programming rather than using gradient\-based or other training schemes that lack such a guarantee\. We defineΦ\(n\):𝒫⁡\(𝕏n\)×𝕌→𝒫⁡\(𝕏n\)\\Phi^\{\(n\)\}:\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\\times\\mathbb\{U\}\\to\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)as the pushforward on the quantized embedding space:Φ\(n\)​\(μ^,U\):=μ^∘Tμ^,U−1\\Phi^\{\(n\)\}\(\\hat\{\\mu\},U\):=\\hat\{\\mu\}\\circ T\_\{\\hat\{\\mu\},U\}^\{\-1\}, whereTTis the same as in the definition forΦ\\Phi,μ^∈𝒫⁡\(𝕏n\)\\hat\{\\mu\}\\in\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)andU∈𝕌U\\in\\mathbb\{U\}\. The quantized deterministic ensemble pushforward𝚽\(l,n,m\)\\bm\{\\Phi\}^\{\(l,n,m\)\}maps𝒫​\(𝕏n\)K×𝕌m→𝒫​\(𝕏n\)K\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}\\times\\mathbb\{U\}\_\{m\}\\to\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}by quantizing and applying the single\-measure pushforwardΦ\(n\):𝒫⁡\(𝕏n\)×𝕌→𝒫⁡\(𝕏n\)\\Phi^\{\(n\)\}:\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\\times\\mathbb\{U\}\\to\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)point\-wise and quantizing withRlR\_\{l\}, that is,𝚽\(l,n,m\)\(𝝁^,U\):=\(Rl\(Φ\(n\)\(μ^1,U\),…,Rl\(Φ\(n\)\(μ^K,U\)\\bm\{\\Phi\}^\{\(l,n,m\)\}\(\\hat\{\\bm\{\\mu\}\},U\):=\(R\_\{l\}\(\\Phi^\{\(n\)\}\(\\hat\{\\mu\}^\{1\},U\),\\dots,R\_\{l\}\(\\Phi^\{\(n\)\}\(\\hat\{\\mu\}^\{K\},U\)\. It induces the transition kernelη\(l,n,m\):𝒫⁡\(𝕏n\)×𝕌m→𝒫⁡\(𝒫⁡\(𝕏n\)\)\\eta^\{\(l,n,m\)\}:\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\\times\\mathbb\{U\}\_\{m\}\\to\\mathcal\{P\}\(\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)\)\. This characterizesMDP\(l,n,m\)\\text\{MDP\}^\{\(l,n,m\)\}with a state space𝒫​\(𝕏n\)K\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}, action space𝕌m\\mathbb\{U\}\_\{m\}, transition kernelη\(l,n,m\)\\eta^\{\(l,n,m\)\}and the discrete cross\-entropy lossVC​EV\_\{CE\}from[15](https://arxiv.org/html/2610.41038#S3.E15)\.

Note that this structure leverages all our predictive information, but does not necessarily translate into model performance in the following case: taking a smallllmay artificially reduce the loss without improving performance due to coarseness\.

For a given data setDD, we pose the dynamic programming equations forMDP\(l,n,m\)\\text\{MDP\}^\{\(l,n,m\)\}as follows

ℭl,n,m,T​\(𝝁^\):=1K​∑k=1KCE^​\(μ^Tk,N,y^k\)ℭl,n,m,t​\(𝝁^\):=min⁡∑𝝁^t\+1∈P\(l\)​\(𝕏n\)KU∈𝕌m⁡η\(l,n,m\)​\(𝝁^t\+1∣𝝁^t,U\)​ℭl,n,m,t\+1​\(𝝁^t\+1\)=minU∈𝕌m⁡ℭl,n,m,t\+1​\(𝚽\(l,n,m\)​\(𝝁^,U\)\)\\begin\{split\}\\mathfrak\{C\}\_\{l,n,m,T\}\(\\hat\{\\bm\{\\mu\}\}\)&:=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\hat\{\\mathrm\{CE\}\}\(\\hat\{\\mu\}^\{k,N\}\_\{T\},\\hat\{y\}^\{k\}\)\\\\ \\mathfrak\{C\}\_\{l,n,m,t\}\(\\hat\{\\bm\{\\mu\}\}\)&:=\\min\_\{U\\in\\mathbb\{U\}\_\{m\}\}\\sum\_\{\\hat\{\\bm\{\\mu\}\}\_\{t\+1\}\\in P^\{\(l\)\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}\}\\eta^\{\(l,n,m\)\}\(\\hat\{\\bm\{\\mu\}\}\_\{t\+1\}\\mid\\hat\{\\bm\{\\mu\}\}\_\{t\},U\)\\,\\mathfrak\{C\}\_\{l,n,m,t\+1\}\(\\hat\{\\bm\{\\mu\}\}\_\{t\+1\}\)\\\\ &\\;=\\min\_\{U\\in\\mathbb\{U\}\_\{m\}\}\\mathfrak\{C\}\_\{l,n,m,t\+1\}\(\\bm\{\\Phi\}^\{\(l,n,m\)\}\(\\hat\{\\bm\{\\mu\}\},U\)\)\\end\{split\}\(16\)
for all𝝁^=\(μ^1,…​μ^K\)∈𝒫\(l\)​\(𝕏n\)K\\hat\{\\bm\{\\mu\}\}=\(\\hat\{\\mu\}^\{1\},\\dots\\hat\{\\mu\}^\{K\}\)\\in\\mathcal\{P\}^\{\(l\)\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}andt∈𝒯t\\in\\mathcal\{T\}\. The last equality is due to the deterministic nature of𝚽\(l,n,m\)\\bm\{\\Phi\}^\{\(l,n,m\)\}\. Following\([Akman et al\., 2026b](https://arxiv.org/html/2610.41038#bib.bib3), Theorem 17\), it follows that, under the compactness of𝕏\\mathbb\{X\}and𝕌\\mathbb\{U\}, due to the weak continuity property of the constructed MDP, the quantized training procedure with the dynamic programming equations \([16](https://arxiv.org/html/2610.41038#S3.E16)\) is asymptotically globally optimal for the original problem as quantization gets finer, which then results in a near\-optimal open\-loop controls under fine enough quantization\.

## 4Construction of an Approximating Transformer

With the machinery of Section[2](https://arxiv.org/html/2610.41038#S2), we relax the next token prediction problem for observations generated by latent ergodic stochastic processes into a filtering problem with finite\-window and quantized data\. The question then becomes:*Does there exist a Transformer that can realize such filter?*The answer is affirmative and mostly expected since such predictive distribution is a lookup table\. In this part, we give a step\-by\-step recipe for how to construct such Transformer\.

Our construction in this section follows a strategy similar to the one used in\([Yun et al\., 2020a](https://arxiv.org/html/2610.41038#bib.bib51)\)to show that fixed\-width Transformers with positional encodings are universal approximators of continuous sequence\-to\-sequence functions on compact domains, see also\([Yun et al\., 2020b](https://arxiv.org/html/2610.41038#bib.bib52);[Geshkovski et al\., 2024](https://arxiv.org/html/2610.41038#bib.bib18);[Jiang and Li, 2024](https://arxiv.org/html/2610.41038#bib.bib23);[Kajitsuka and Sato, 2024](https://arxiv.org/html/2610.41038#bib.bib24);[Wang et al\., 2024](https://arxiv.org/html/2610.41038#bib.bib49);[Furuya et al\., 2025](https://arxiv.org/html/2610.41038#bib.bib16)\)\. Their argument quantizes the input, computes a*contextual mapping*with attention layers \(which is a distinct identifier for each quantized input\), and maps identifiers to outputs with feed\-forward layers\. Our inputs are already quantized, the gatherer layers of §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)compute and collect the contextual mapping at the readout token, and the ReLU network of §[4\.1\.3](https://arxiv.org/html/2610.41038#S4.SS1.SSS3)performs the value mapping by interpolation\.

Figure 5:Description of the Transformer built in this section\.### 4\.1Transformer Layers and Token Embeddings

#### 4\.1\.1Token Embeddings

Throughout this section, fix the window lengthMMand the quantization levelss\. We fix an enumerationι\\iotaof the1/s1/s\-netΛs\\Lambda\_\{s\}, meaning thatι:Λs→\{1,…,R⁡\(s\)\}\\iota:\\Lambda\_\{s\}\\to\\\{1,\\ldots,R\(s\)\\\}is a bijective map\. Recall thatR⁡\(s\)R\(s\)is the cardinality ofΛs\\Lambda\_\{s\}\. This makes each windoww=\(w1,…,wM\)w=\(w\_\{1\},\\ldots,w\_\{M\}\)an element ofΛsM\\Lambda\_\{s\}^\{M\}, which hasR​\(s\)MR\(s\)^\{M\}elements\. We set the token embedding dimension tod=2​M\+R⁡\(s\)\+1d=2M\+R\(s\)\+1with the following decomposition in mind

d=M⏟positional\+1⏟token​value\+M⏟gather\+R⁡\(s\)⏟logits\.d=\\underbrace\{M\}\_\{\\rm positional\}\\quad\+\\quad\\underbrace\{1\}\_\{\\rm token\\,value\}\\quad\+\\quad\\underbrace\{M\}\_\{\\rm gather\}\\quad\+\\quad\\underbrace\{R\(s\)\}\_\{\\rm logits\}\.\(17\)According to decomposition \([17](https://arxiv.org/html/2610.41038#S4.E17)\), we split each token embedding into four parts:x=\(xpos,xtok,xgat,xlog\)∈ℝM×ℝ1×ℝM×ℝR⁡\(s\)=ℝdx=\(x^\{\\rm pos\},x^\{\\rm tok\},x^\{\\rm gat\},x^\{\\rm log\}\)\\in\\mathbb\{R\}^\{M\}\\times\\mathbb\{R\}^\{1\}\\times\\mathbb\{R\}^\{M\}\\times\\mathbb\{R\}^\{R\(s\)\}=\\mathbb\{R\}^\{d\}\. LetΠ∙\\Pi\_\{\\bullet\}be the coordinate projections fromℝd\\mathbb\{R\}^\{d\}to∙\\bulletspace where∙∈\{pos,tok,gat,log\}\\bullet\\in\\\{\{\\rm pos\},\{\\rm tok\},\{\\rm gat\},\{\\rm log\}\\\}\. Lastly,\{ei\}i=1M\\\{e\_\{i\}\\\}\_\{i=1\}^\{M\}form the standard basis forℝM\\mathbb\{R\}^\{M\}, which we use for positional encodings and*gathering tokens*\.

For anyv∈Λsv\\in\\Lambda\_\{s\}at positioni∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}, we define

EE​\(v,pi\):=\(ei,ι⁡\(v\),0M,0R\)∈ℝdE\_\{E\}\(v,p\_\{i\}\):=\(e\_\{i\},\\iota\(v\),0\_\{M\},0\_\{R\}\)\\in\\mathbb\{R\}^\{d\}\(18\)where0M0\_\{M\}and0R0\_\{R\}are the zero vectors of dimensionsMMandR⁡\(s\)R\(s\), respectively\. The mapEEE\_\{E\}gives rise to an unembedding function satisfying the requirements of Assumption[15](https://arxiv.org/html/2610.41038#Thmtheorem15)sinceΠpos\\Pi\_\{\\rm pos\}recovers the positional encodings\. Using the embedding operator, for any windoww∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, we definex0i,\(w\)=EE​\(wi,pi\)x\_\{0\}^\{i,\(w\)\}=E\_\{E\}\(w\_\{i\},p\_\{i\}\)for alli∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}andμ0\(w\):=1M​∑i=1Mδx0i,\(w\)\\mu\_\{0\}^\{\(w\)\}:=\\frac\{1\}\{M\}\\sum\_\{i=1\}^\{M\}\\delta\_\{x\_\{0\}^\{i,\(w\)\}\}\.

#### 4\.1\.2Gatherer Layers

Let us fix a bijectionρ:\{0,…,M−1\}→\{1,…,M\}\\rho:\\\{0,\\ldots,M\-1\\\}\\to\\\{1,\\ldots,M\\\}sending each layer to a position\. We need it to copy the token at positionρ⁡\(t\)\\rho\(t\)into the gather coordinate of the readout token at each layert∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}\. We form the firstMMlayers as*gatherer layers*, which are responsible for collecting all the information in the window to the readout position at theM−1thM\-1^\{\\rm th\}layer\. As Transformers have feed\-forward parts at each layer, we need to give role to them:

###### Lemma 16\.

Letσ\\sigmabeReLUand assign

WA:=\[Id−Id\]Wb:=02​dWW:=\[Id−Id\]\.W\_\{A\}:=\\begin\{bmatrix\}I\_\{d\}\\\\ \-I\_\{d\}\\end\{bmatrix\}\\qquad W\_\{b\}:=0\_\{2d\}\\qquad W\_\{W\}:=\\begin\{bmatrix\}I\_\{d\}&\-I\_\{d\}\\end\{bmatrix\}\.Then,WW​σ​\(WA​x\+Wb\)=xW\_\{W\}\\sigma\(W\_\{A\}x\+W\_\{b\}\)=xfor allx∈ℝdx\\in\\mathbb\{R\}^\{d\}\.

We apply\(WW,WA,Wb\)\(W\_\{W\},W\_\{A\},W\_\{b\}\)in Lemma[16](https://arxiv.org/html/2610.41038#Thmtheorem16)for each gatherer layer, that is

\(WW,t,WA,t,Wb,t\)=\(WW,WA,Wb\)\(W\_\{W,t\},W\_\{A,t\},W\_\{b,t\}\)=\(W\_\{W\},W\_\{A\},W\_\{b\}\)for eacht∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}, where the right hand side as in Lemma[16](https://arxiv.org/html/2610.41038#Thmtheorem16)\. Consequently, layer\-wise updates take the form

xt\+1i=xti\+∑j=1Meβ⁡⟨WQ,t​xti,WK,t​xtj⟩∑j′=1Meβ⁡⟨WQ,t​xti,WK,t​xtj′⟩​WV,t​xtjx\_\{t\+1\}^\{i\}=x\_\{t\}^\{i\}\+\\sum\_\{j=1\}^\{M\}\\frac\{\\mathrm\{e\}^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j\}\\rangle\}\}\{\\sum\_\{j^\{\\prime\}=1\}^\{M\}\\mathrm\{e\}^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j^\{\\prime\}\}\\rangle\}\}W\_\{V,t\}\\,x\_\{t\}^\{j\}\(19\)for eachi∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}andt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}\.

We now specify the roles of the query, key and value matrices\. For keys:WK,t=ΠposW\_\{K,t\}=\\Pi\_\{\\rm pos\}so thatWK,t​xti=\(xti\)pos=eiW\_\{K,t\}x\_\{t\}^\{i\}=\(x\_\{t\}^\{i\}\)^\{\\rm pos\}=e\_\{i\}for alli∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}andt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}\. The task we assign to queries is a bit more specialized: For eacht∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}, letAt∈ℝM×MA\_\{t\}\\in\\mathbb\{R\}^\{M\\times M\}be a matrix defined on the standard basis\{ei\}i=1M\\\{e\_\{i\}\\\}\_\{i=1\}^\{M\}ofℝM\\mathbb\{R\}^\{M\}, as described in §[4\.1\.1](https://arxiv.org/html/2610.41038#S4.SS1.SSS1), whose action is defined by

At​eM=eρ⁡\(t\)At​ei:=ei​\(i≠M\)\.A\_\{t\}e\_\{M\}=e\_\{\\rho\(t\)\}\\qquad A\_\{t\}e\_\{i\}:=e\_\{i\}\\;\(i\\neq M\)\.Then we setWQ,t:=c​At​ΠposW\_\{Q,t\}:=cA\_\{t\}\\Pi\_\{\\rm pos\}for some constantc\>0c\>0satisfying

c=maxc′​‖At‖≤M𝕌0≤t≤M−1⁡c′c=\\max\_\{\\begin\{subarray\}\{c\}c^\{\\prime\}\\\|A\_\{t\}\\\|\\leq M\_\{\\mathbb\{U\}\}\\\\ 0\\leq t\\leq M\-1\\end\{subarray\}\}c^\{\\prime\}\(20\)so that query matrix is admissible in𝕌\\mathbb\{U\}\. For the value matrices: Let

Gj:=\[0M0ej0R⁡\(s\)\]T​ΠtokG\_\{j\}:=\\begin\{bmatrix\}0\_\{M\}&0&e\_\{j\}&0\_\{R\(s\)\}\\end\{bmatrix\}^\{T\}\\Pi\_\{\\rm tok\}\(21\)for eachj∈\{1,…,M\}j\\in\\\{1,\\ldots,M\\\}, which is a rank one matrix \(sinceΠtok\\Pi\_\{\\rm tok\}projects onto a11\-dimensional space\) reading the token\-value coordinate and then writing it into the gather coordinatejj\. Then we set the value matricesWV,t:=Gρ⁡\(t\)W\_\{V,t\}:=G\_\{\\rho\(t\)\}for eachj∈\{1,…,M\}j\\in\\\{1,\\ldots,M\\\}\.

To sum up, we separate the dynamics into two parts: the feed\-forward layers are responsible for carrying the token value and the positional information across the layers\. Since each token except the one at the readout position \(i=Mi=M\) has null gatherers and logits, the feed\-forward flow does not interfere with gatherer and logit positions\. On the other hand, by construction of the self\-attention layer, it is only concerned with writing the correct information to the gatherer positions of the readout token at each layer by a prescribed manner\. Note thatWV,tW\_\{V,t\}reads the token information and writes it to theρ​\(t\)th\\rho\(t\)^\{\\rm th\}position of theMthM^\{\\rm th\}token\. As a result, the self\-attention layers do not change the positional and token values\. Therefore, both layer types act as counterparts of each other\. In short, the feed\-forward layers carry the relevant information through the Transformer dynamics, while the self\-attention layers use that information in a sequential manner\. In the following, we make this high\-level discussion rigorous\.

###### Lemma 17\.

For allw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\},i∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}andt∈\{0,…,M\}t\\in\\\{0,\\ldots,M\\\}, we have

\(xti\)pos=eiand\(xti\)tok=ι⁡\(wi\)\.\(x\_\{t\}^\{i\}\)^\{\\rm pos\}=e\_\{i\}\\qquad\\text\{and\}\\qquad\(x\_\{t\}^\{i\}\)^\{\\rm tok\}=\\iota\(w\_\{i\}\)\.

Because of Lemma[17](https://arxiv.org/html/2610.41038#Thmtheorem17), the decomposition in \([17](https://arxiv.org/html/2610.41038#S4.E17)\) and choices of network weights are almost clear\. To support, this design choice also makes the error propagation analysis simpler: The attention scores depend only onxposx^\{\\rm pos\}while the values are determined byxtokx^\{\\rm tok\}in an exact way at each layer\. Before proving the complete characterization of the self\-attention block under our construction, we define a helper function

selt​\(i\):=\{ρ⁡\(t\),i=M,i,i≠M\.\\mathrm\{sel\}\_\{t\}\(i\):=\\begin\{cases\}\\rho\(t\),&i=M,\\\\ i,&i\\neq M\.\\end\{cases\}which is defined for allt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}andi∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}\.

###### Lemma 18\.

For everyt≤M−1t\\leq M\-1and alli,j∈\{1,…,M\}i,j\\in\\\{1,\\ldots,M\\\}, we have

⟨WQ,t​xti,WK,t​xtj⟩=c​χj​\(selt​\(i\)\)\.\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j\}\\rangle=c\\,\\chi\_\{j\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\.Hence,

ati​selt​\(i\)=eβ​ceβ​c\+M−1andati​j=1eβ​c\+M−1​\(j≠selt​\(i\)\),a\_\{t\}^\{i\\mathrm\{sel\}\_\{t\}\(i\)\}=\\frac\{e^\{\\beta c\}\}\{e^\{\\beta c\}\+M\-1\}\\qquad\\text\{and\}\\qquad a\_\{t\}^\{ij\}=\\frac\{1\}\{e^\{\\beta c\}\+M\-1\}\\;\\;\(j\\neq\\mathrm\{sel\}\_\{t\}\(i\)\),whereati​ja\_\{t\}^\{ij\}’s are exactly as in \([30](https://arxiv.org/html/2610.41038#A3.E30)\) andccis as in \([20](https://arxiv.org/html/2610.41038#S4.E20)\)\.

From Lemma[18](https://arxiv.org/html/2610.41038#Thmtheorem18), it is clear that we cannot recover the token information without any errors\. Ideally, we want to read the token value fromρ​\(t\)th\\rho\(t\)^\{\\rm th\}position and write it to the corresponding gatherer coordinate\. However, the self\-attention yields a weighted average of all tokens\. The next lemma quantifies this error while providing computed attention scores across all particles\.

###### Lemma 20\.

Letx¯tM\\bar\{x\}\_\{t\}^\{M\}be the readout token under \(hard\) selection, that isx¯t\+1M=x¯tM\+Gρ⁡\(t\)​x¯tρ⁡\(t\)\\bar\{x\}\_\{t\+1\}^\{M\}=\\bar\{x\}\_\{t\}^\{M\}\+G\_\{\\rho\(t\)\}\\bar\{x\}\_\{t\}^\{\\rho\(t\)\}for allt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}withx¯0M=x0M\\bar\{x\}\_\{0\}^\{M\}=x\_\{0\}^\{M\}\. Then, the final error between the hard choice and the soft choice111See Remark[22](https://arxiv.org/html/2610.41038#Thmtheorem22)for the distinction and what this buys for us in the implementation phase\.is quantified as follows:

‖xMM−x¯MM‖2≤2​M​\(M−1\)​R​\(s\)​e−β​c\.\\\|x\_\{M\}^\{M\}\-\\bar\{x\}\_\{M\}^\{M\}\\\|\_\{2\}\\leq 2M\(M\-1\)R\(s\)e^\{\-\\beta c\}\.Moreover,x¯MM:=\(eM,ι⁡\(wM\),\(ι⁡\(w1\),…,ι⁡\(wM\)\),0R\)\\bar\{x\}\_\{M\}^\{M\}:=\(e\_\{M\},\\iota\(w\_\{M\}\),\(\\iota\(w\_\{1\}\),\\ldots,\\iota\(w\_\{M\}\)\),0\_\{R\}\)\.

Remaining part of our construction is based on*distinguishability*of different tokens parametrized by different initial windows, that is, ifw≠w′∈ΛsMw\\neq w^\{\\prime\}\\in\\Lambda\_\{s\}^\{M\}, then the resultingx¯MM,\(w\)\\bar\{x\}\_\{M\}^\{M,\(w\)\}andx¯MM,\(w′\)\\bar\{x\}\_\{M\}^\{M,\(w^\{\\prime\}\)\}should be identifiable to be distinct in the metric\. This is crucial because, at the end, we want to map the readout tokens to probability distributions \(thereby we get conditional probability distribution\) without any loss of information\. The next corollary is a simple result of Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20)’s second claim\.

###### Corollary 21\.

The mapΛsM∋w↦x¯MM,\(w\)\\Lambda\_\{s\}^\{M\}\\ni w\\mapsto\\bar\{x\}\_\{M\}^\{M,\(w\)\}is injective, and‖x¯MM,\(w\)−x¯MM,\(w′\)‖2≥1\\\|\\bar\{x\}\_\{M\}^\{M,\(w\)\}\-\\bar\{x\}\_\{M\}^\{M,\(w^\{\\prime\}\)\}\\\|\_\{2\}\\geq 1wheneverw,w′∈ΛsMw,w^\{\\prime\}\\in\\Lambda\_\{s\}^\{M\}withw≠w′w\\neq w^\{\\prime\}\.

###### Proof\.

Ifw≠w′w\\neq w^\{\\prime\}, then they differ in at least one coordinate\. Sinceι\\iotais an integer\-valued enumeration ofΛsM\\Lambda\_\{s\}^\{M\}, then\|ι⁡\(w\)−ι⁡\(w′\)\|≥1\|\\iota\(w\)\-\\iota\(w^\{\\prime\}\)\|\\geq 1, which transfers to‖x¯MM,\(w\)−x¯MM,\(w′\)‖2\\\|\\bar\{x\}\_\{M\}^\{M,\(w\)\}\-\\bar\{x\}\_\{M\}^\{M,\(w^\{\\prime\}\)\}\\\|\_\{2\}directly, also establishing injectivity\. ∎

#### 4\.1\.3The Interpolator Layer

The machinery of §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)collects all data in the window into the gather block in the decomposition \([17](https://arxiv.org/html/2610.41038#S4.E17)\)\. In this part, we show that it is possible to map the information in logits part to suitable conditional distributions\. We achieve this task doing someReLUcalculus to obtain a single layerReLUnetwork applied to the readout tokensxMMx\_\{M\}^\{M\}and interpolate them into vectors inℝR⁡\(s\)\\mathbb\{R\}^\{R\(s\)\}\.

###### Lemma 23\.

Let\{ζk\}k=1n\\\{\\zeta\_\{k\}\\\}\_\{k=1\}^\{n\}be distinct set of features inℝd\\mathbb\{R\}^\{d\}and\{vk\}k=1n\\\{v\_\{k\}\\\}\_\{k=1\}^\{n\}be arbitrary targets inℝR⁡\(s\)\\mathbb\{R\}^\{R\(s\)\}\. There exists a one\-layer𝚁𝚎𝙻𝚄\{\\tt ReLU\}networkφ:ℝd→ℝR⁡\(s\)\\varphi:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{R\(s\)\}of widthn−1n\-1withφ⁡\(ζk\)=vk\\varphi\(\\zeta\_\{k\}\)=v\_\{k\}for allk∈\{1,…,n\}k\\in\\\{1,\\ldots,n\\\}with Lipschitz constantLφ∈ℝ\+L\_\{\\varphi\}\\in\\mathbb\{R\}\_\{\+\}\.

The proof of Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)chooses a genericaa, on which the whole analysis is built\. In the next result, we give an explicit choice foraaand see its effect on the Lipschitz constantLφL\_\{\\varphi\}of theReLUnetwork\.

###### Lemma 24\.

Takea=\[0M​0​⟨R⁡\(s\)⟩M​0R\]a=\[0\_\{M\}\\;0\\;\\langle R\(s\)\\rangle\_\{M\}\\;0\_\{R\}\], where⟨R⁡\(s\)⟩M=\(1,R⁡\(s\),R​\(s\)2,…,R​\(s\)M−1\)\\langle R\(s\)\\rangle\_\{M\}=\(1,R\(s\),R\(s\)^\{2\},\\ldots,R\(s\)^\{M\-1\}\), in the proof of Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)\. Then the following assertions are true:

- \(i\)⟨a,x¯MM,\(w\)⟩=∑j=1Mι⁡\(wj\)​R​\(s\)j−1\\langle a,\\bar\{x\}\_\{M\}^\{M,\(w\)\}\\rangle=\\sum\_\{j=1\}^\{M\}\\iota\(w\_\{j\}\)R\(s\)^\{j\-1\},‖a‖2≤R​\(s\)M−12\\\|a\\\|\_\{2\}\\leq R\(s\)^\{M\-\\frac\{1\}\{2\}\}andtk\+1−tk≥1t\_\{k\+1\}\-t\_\{k\}\\geq 1for allk=1,…,R⁡\(s\)−1k=1,\\ldots,R\(s\)\-1\.
- \(ii\)Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2,4\), withn=R​\(s\)Mn=R\(s\)^\{M\}and targetsvw=P^rM,s\(⋅∣w\)v\_\{w\}=\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)in Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)’s statement, we have Lφ≤4​log⁡1qmin​μmins​R​\(s\)M​\(R​\(s\)M−1\)\.L\_\{\\varphi\}\\leq 4\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\}R\(s\)^\{M\}\(R\(s\)^\{M\}\-1\)\.

### 4\.2The Approximation Theorem

Now that we have established how we construct a candidate Transformer, all is left to complete the description with an error analysis, which constitutes the purpose of this subsection\.

###### Theorem 25\.

For anyε∈\(0,1/2\]\\varepsilon\\in\(0,1/2\],r≥Mr\\geq Mand all

β≥1c​\(log⁡\(4​M​\(M−1\)​R​\(s\)\)\+log⁡Lφ\+log⁡1ε\),\\beta\\geq\\frac\{1\}\{c\}\\bigg\(\\log\{\(4M\(M\-1\)R\(s\)\)\}\+\\log\{L\_\{\\varphi\}\}\+\\log\{\\frac\{1\}\{\\varepsilon\}\}\\bigg\),the Transformer described through §[4\.1\.1](https://arxiv.org/html/2610.41038#S4.SS1.SSS1)\-[4\.1\.3](https://arxiv.org/html/2610.41038#S4.SS1.SSS3)satisfies:

- \(i\)maxw∈ΛsM∥P^rM,s\(⋅∣w\)−𝖳\(⋅∣w\)∥TV≤ε\\max\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\varepsilon\.
- \(ii\)minv∈Λs​w∈ΛsM⁡𝖳⁡\(v∣w\)≥qmin​μmins​e−2​ε\\min\_\{v\\in\\Lambda\_\{s\}\\,w\\in\\Lambda\_\{s\}^\{M\}\}\\mathsf\{T\}\(v\\mid w\)\\geq q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}e^\{\-2\\varepsilon\}\.

In this theorem,𝖳\(⋅∣w\)\\mathsf\{T\}\(\\,\\cdot\\mid w\)denotes the readout distribution of the Transformer given input sequenceww\.

## 5Main Results: Performance of Trained Transformers

In §[2](https://arxiv.org/html/2610.41038#S2), we have studied the effect of memory in next\-token prediction for observations of latent stationary ergodic andNthN^\{\\rm th\}order Markovian sources, with quantization aspects for implementable learning algorithms\. Then, §[4](https://arxiv.org/html/2610.41038#S4)is devoted to construct a Transformer that can realize finite\-memory quantized\-data predictive distribution in Definition[2](https://arxiv.org/html/2610.41038#Thmtheorem2)\.

In this section, we complete our theoretical excursion by deriving a complete error bound for Transformers trained to perform next\-token prediction tasks by the dynamic programming based training scheme presented in §[3\.5](https://arxiv.org/html/2610.41038#S3.SS5)building on[Akman et al\. \(2026b\)](https://arxiv.org/html/2610.41038#bib.bib3)\.

The final error bound consists of \(i\) optimization error due to training, \(ii\) filter stability bounds due to finite\-memory, \(iii\) sample complexity bounds due to training, and \(iv\) insufficiency of architecture\.

We need to prove some auxiliary results that lead us to the ultimate theorem\. We commence with establishing some notation\. Throughout this section, the Transformer has the architecture of §[4](https://arxiv.org/html/2610.41038#S4): it reads prompts of lengthL=ML=M, and it hasT=M\+1T=M\+1layers, namely theMMgatherer layers of §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)followed by the interpolator layer of §[4\.1\.3](https://arxiv.org/html/2610.41038#S4.SS1.SSS3)\. Hence, an open\-loop policy is a sequenceγ=\(U0,…,UM\)∈𝕌mM\+1\\gamma=\(U\_\{0\},\\ldots,U\_\{M\}\)\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}\. For suchγ\\gammaandw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, we write𝖳γ\(⋅∣w\)\\mathsf\{T\}^\{\\gamma\}\(\\,\\cdot\\mid w\)for the readout distribution of the Transformer with weights given byγ\\gammaand the input sequenceww, run asMDP\(l,n,m\)\\mathrm\{MDP\}^\{\(l,n,m\)\}of §[3\.5](https://arxiv.org/html/2610.41038#S3.SS5)\. That is, the state quantizerQnQ\_\{n\}and the measure quantizerRlR\_\{l\}act after every layer, and𝖳γ\(⋅∣w\)=softmax\(EUxM\+1M\)\\mathsf\{T\}^\{\\gamma\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(E\_\{U\}x\_\{M\+1\}^\{M\}\), wherexM\+1Mx\_\{M\+1\}^\{M\}is the token at positionMMafter the last layer\. Letγ∗\\gamma^\{\*\}be the optimal open\-loop policy that is obtained via the optimal closed\-loop policy given by the dynamic programming recursions \([16](https://arxiv.org/html/2610.41038#S3.E16)\)\. Since𝚽\(l,n,m\)\\bm\{\\Phi\}^\{\(l,n,m\)\}is deterministic,γ∗\\gamma^\{\*\}minimizesVCE​\(D,⋅\)V\_\{\\rm CE\}\(D,\\cdot\)over all open\-loop policies in𝕌mM\+1\\mathbb\{U\}\_\{m\}^\{M\+1\}\.

We also need to specify how the data set is sampled\. Fixr≥Mr\\geq M\. We assume that theK~\\tilde\{K\}sequences of the data set construction in Appendix[3](https://arxiv.org/html/2610.41038#S3)are independent copies of\(qs​\(Yr−M\+1\),…,qs​\(Yr\+1\)\)\(q^\{s\}\(Y\_\{r\-M\+1\}\),\\ldots,q^\{s\}\(Y\_\{r\+1\}\)\)\. Equivalently, the training pairs\(𝒙l,y~l\)\(\\bm\{x\}^\{l\},\\tilde\{y\}^\{l\}\),l=1,…,K~l=1,\\ldots,\\tilde\{K\}, are independent copies of\(qM,s\(Yr−M\+1:r\),qs\(Yr\+1\)\)\(q^\{M,s\}\(Y\_\{r\-M\+1:r\}\),q^\{s\}\(Y\_\{r\+1\}\)\)\. If\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is stationary, the law of this pair does not depend onrr\. For each windoww∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, we write

pr\(w\):=Pr\(qM,s\(Yr−M\+1:r\)=w\),nw:=∑l=1K~𝟏\{𝒙l=w\},p\_\{r\}\(w\):=\\mathrm\{Pr\}\\big\(q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=w\\big\),\\qquad n\_\{w\}:=\\sum\_\{l=1\}^\{\\tilde\{K\}\}\\mathbf\{1\}\_\{\\\{\\bm\{x\}^\{l\}=w\\\}\},\(22\)and, whenevernw≥1n\_\{w\}\\geq 1,yw:=1nw∑l=1K~δy~l𝟏\{𝒙l=w\}∈𝒫\(Λs\)y^\{w\}:=\\frac\{1\}\{n\_\{w\}\}\\sum\_\{l=1\}^\{\\tilde\{K\}\}\\delta\_\{\\tilde\{y\}^\{l\}\}\\mathbf\{1\}\_\{\\\{\\bm\{x\}^\{l\}=w\\\}\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)\. In words,prp\_\{r\}is the law of a test window,nwn\_\{w\}is the number of occurrences ofwwin the data set andywy^\{w\}is the empirical distribution of its labels\. Ifw=𝒙kw=\\bm\{x\}^\{k\}, thennw=nkn\_\{w\}=n\_\{k\}andyw=yky^\{w\}=y^\{k\}in the notation of Appendix[3](https://arxiv.org/html/2610.41038#S3)\.

Finally, the Transformer of §[4](https://arxiv.org/html/2610.41038#S4)has to be representable on the grids of §[3\.4](https://arxiv.org/html/2610.41038#S3.SS4)\. To this end, we take𝕏=\[−M𝕏,M𝕏\]d\\mathbb\{X\}=\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]^\{d\}and let𝕌\\mathbb\{U\}be the set of weights whose entries lie in\[−M𝕌,M𝕌\]\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\], whereM𝕏,M𝕌∈ℕM\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{U\}\}\\in\\mathbb\{N\}satisfy

M𝕏≥max⁡\{R⁡\(s\),1\+log⁡1qmin​μmins\}andM𝕌≥max⁡\{2​R​\(s\)M,2​log​1qmin​μmins\}\.M\_\{\\mathbb\{X\}\}\\geq\\max\\Big\\\{R\(s\),\\;1\+\\log\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\\Big\\\}\\qquad\\text\{and\}\\qquad M\_\{\\mathbb\{U\}\}\\geq\\max\\Big\\\{2R\(s\)^\{M\},\\;2\\log\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\\Big\\\}\.\(23\)The first condition keeps the positional encodings, the token values, the gather coordinates and the logits of the readout token inside\[−M𝕏,M𝕏\]\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]\. The second one makes the weights of the interpolator layer admissible; see the proof of Lemma[35](https://arxiv.org/html/2610.41038#Thmtheorem35)\.

The cross\-entropy loss can be decomposed into two parts:

CE\(P1,P2\)=H\(P1\)\+DKL\(P1∥P2\)\\mathrm\{CE\}\(P\_\{1\},P\_\{2\}\)=H\(P\_\{1\}\)\+D\_\{\\rm KL\}\(P\_\{1\}\\parallel P\_\{2\}\)for two probability measuresP1,P2∈𝒫⁡\(Λs\)P\_\{1\},P\_\{2\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\), whereH\(P1\):=−∑v∈ΛsP1\(v\)logP1\(v\)H\(P\_\{1\}\):=\-\\sum\_\{v\\in\\Lambda\_\{s\}\}P\_\{1\}\(v\)\\log P\_\{1\}\(v\)is the Shannon entropy\. Therefore, unless the true distribution, in this caseP1P\_\{1\}, is deterministic \(a Dirac mass\),CE⁡\(P1,P2\)\>0\\mathrm\{CE\}\(P\_\{1\},P\_\{2\}\)\>0\. HoweverP1P\_\{1\}is often modeled as a non\-deterministic distribution\. For this reason, we want to get rid of the entropy term, and define the*excess loss*

ℰ\(γ\):=VCE\(D,γ\)−∑k=1KnkK~H\(yk\)=∑k=1KnkK~DKL\(yk∥𝖳γ\(⋅∣𝒙k\)\),\\mathcal\{E\}\(\\gamma\):=V\_\{\\rm CE\}\(D,\\gamma\)\-\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}H\(y^\{k\}\)=\\sum\_\{k=1\}^\{K\}\\frac\{n\_\{k\}\}\{\\tilde\{K\}\}D\_\{\\rm KL\}\\big\(y^\{k\}\\parallel\\mathsf\{T\}^\{\\gamma\}\(\\,\\cdot\\mid\\bm\{x\}^\{k\}\)\\big\),whereγ∈𝕌mM\+1\\gamma\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}is an open\-loop policy\.ℰ\\mathcal\{E\}is always a nonnegative quantity due to the aforementioned decomposition of the cross\-entropy loss\. Since we do not consider any other data set, we also do not explicitly write the data set dependency for a simpler notation\. Note that the optimization task is not changed since the data set is not changed: For fixed data setDD,ℰ\\mathcal\{E\}is an affine transformation ofVCE​\(D,⋅\)V\_\{\\rm CE\}\(D,\\cdot\)\. Therefore, any minimizer ofℰ\\mathcal\{E\}is a minimizer ofVCE​\(D,⋅\)V\_\{\\rm CE\}\(D,\\cdot\)and vice versa\. In particular,γ∗\\gamma^\{\*\}minimizesℰ\\mathcal\{E\}over𝕌mM\+1\\mathbb\{U\}\_\{m\}^\{M\+1\}\.

### 5\.1The Unified Bound for Stationary Ergodic Sources

###### Theorem 26\.

Suppose Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)and[6](https://arxiv.org/html/2610.41038#Thmtheorem6)hold\. LetY1,Y2,…Y\_\{1\},Y\_\{2\},\\dotsbe a single trajectory of the source, and letU¯K~∗\\underline\{U\}^\{\*\}\_\{\\tilde\{K\}\}be an open\-loop policy returned by \([16](https://arxiv.org/html/2610.41038#S3.E16)\) for the training pairs\(qM,s\(Yj:j\+M−1\),qs\(Yj\+M\)\)\\big\(q^\{M,s\}\(Y\_\{j:j\+M\-1\}\),\\,q^\{s\}\(Y\_\{j\+M\}\)\\big\),j=1,…,K~j=1,\\dots,\\tilde\{K\}\. Let the test sequenceY0:rY\_\{0:r\}be independent of this trajectory\. Then, almost surely,

lim supM→∞lim sups→∞lim sup\(n,m,β\)→∞lim supK~→∞supr≥M𝔼Y0:r∥Pr−Ξs\(𝖳U¯K~∗\(⋅∣qM,s\(Yr−M\+1:r\)\)\)∥TV=0,\\limsup\_\{M\\to\\infty\}\\;\\limsup\_\{s\\to\\infty\}\\;\\limsup\_\{\(n,m,\\beta\)\\to\\infty\}\\;\\limsup\_\{\\tilde\{K\}\\to\\infty\}\\;\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\Big\\\|P\_\{r\}\-\\Xi\_\{s\}\\big\(\\mathsf\{T\}^\{\\underline\{U\}^\{\*\}\_\{\\tilde\{K\}\}\}\(\\,\\cdot\\mid q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)\)\\big\)\\Big\\\|\_\{\\rm\\scriptscriptstyle TV\}=0,

### 5\.2The Unified Bound for Markovian Sources

###### Theorem 27\.

Suppose that Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)holds\. Letr≥M≥Nr\\geq M\\geq N, lets∈ℕs\\in\\mathbb\{N\}withR⁡\(s\)≥2R\(s\)\\geq 2, letδ∈\(0,1\)\\delta\\in\(0,1\), and let the data set be sampled as above with

K~≥8​\(qmin​μmins\)−M​\(M​log⁡R⁡\(s\)\+log⁡3δ\)\.\\tilde\{K\}\\geq 8\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{\-M\}\\Big\(M\\log\{R\(s\)\}\+\\log\{\\frac\{3\}\{\\delta\}\}\\Big\)\.Let\(n,m,l\)∈ℕ3\(n,m,l\)\\in\\mathbb\{N\}^\{3\}be aligning quantization levels \(Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\), letM𝕏M\_\{\\mathbb\{X\}\}andM𝕌M\_\{\\mathbb\{U\}\}satisfy \([23](https://arxiv.org/html/2610.41038#S5.E23)\), and letβ\>0\\beta\>0be such thatη⁡\(n,m,β\)≤1\\eta\(n,m,\\beta\)\\leq 1, where

η⁡\(n,m,β\):=2​Lφ​M​\(M−1\)​R​\(s\)​e−β​c\+\(1\+Lφ​M​d\)​M𝕏n−1\+M𝕏​d​R​\(s\)3​M−12​M𝕌​dsm−1\.\\eta\(n,m,\\beta\):=2L\_\{\\varphi\}M\(M\-1\)R\(s\)e^\{\-\\beta c\}\+\\big\(1\+L\_\{\\varphi\}M\\sqrt\{d\}\\big\)\\frac\{M\_\{\\mathbb\{X\}\}\}\{n\-1\}\+M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}\.\(24\)Here,LφL\_\{\\varphi\}is the Lipschitz constant of the interpolator layer \(Lemmata[23](https://arxiv.org/html/2610.41038#Thmtheorem23)and[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\),c=M𝕌c=M\_\{\\mathbb\{U\}\}as in Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\-\(ii\),d=2​M\+R⁡\(s\)\+1d=2M\+R\(s\)\+1is the embedding dimension in \([17](https://arxiv.org/html/2610.41038#S4.E17)\) andds:=R​\(s\)M\+12d\_\{s\}:=R\(s\)^\{\\frac\{M\+1\}\{2\}\}\. Then, with probability at least1−δ1\-\\deltaover the data set,

𝔼Y0:r∥Pr−Ξs\(𝖳γ∗\(⋅∣qM,s\(Yr−M\+1:r\)\)\)∥TV≤𝜶⌊M/N⌋⏟memory\+\(1\+2​Mqmin\)​Lψs⏟quantization\+\(1\+2\)​R​\(s\)M​Cδqmin​μmins​K~⏟sample complexity\+η⁡\(n,m,β\)2⏟architecture and training,\\begin\{split\}&\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\Big\\\|P\_\{r\}\-\\Xi\_\{s\}\\big\(\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)\)\\big\)\\Big\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\qquad\\leq\\underbrace\{\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}\}\_\{\\text\{memory\}\}\+\\underbrace\{\\Big\(1\+\\frac\{2M\}\{q\_\{\\rm min\}\}\\Big\)\\frac\{L\_\{\\psi\}\}\{s\}\}\_\{\\text\{quantization\}\}\+\\underbrace\{\(1\+\\sqrt\{2\}\)\\sqrt\{\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\\,\\tilde\{K\}\}\}\}\_\{\\text\{sample complexity\}\}\+\\underbrace\{\\frac\{\\eta\(n,m,\\beta\)\}\{\\sqrt\{2\}\}\}\_\{\\text\{architecture and training\}\},\\end\{split\}\(25\)whereY0:rY\_\{0:r\}is independent of the data set,Cδ:=R⁡\(s\)​log⁡2\+M​log⁡R⁡\(s\)\+log⁡\(3/δ\)C\_\{\\delta\}:=R\(s\)\\log 2\+M\\log R\(s\)\+\\log\(3/\\delta\), and𝛂=\(1−δ⁡\(𝐓\)\)​\(2−δ⁡\(𝐐\)\)\\bm\{\\alpha\}=\(1\-\\delta\(\\mathbf\{T\}\)\)\(2\-\\delta\(\\mathbf\{Q\}\)\)is as in Proposition[10](https://arxiv.org/html/2610.41038#Thmtheorem10)\.

### 5\.3Asymptotic Performance under Filter Stability and Weak Feller Continuity

Theorem[27](https://arxiv.org/html/2610.41038#Thmtheorem27)is quantitative in its nature\. If one only asks that the error vanish asMM,ss,\(n,m,β\)\(n,m,\\beta\)andK~\\tilde\{K\}grow, then neither geometric filter stability \(α<1\\alpha<1\) nor the Lipschitz condition \(Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\)\)is needed, and two changes in the proof suffice\.

*On memory\.*Letν^\\hat\{\\nu\}be the uniform distribution onSS, letπnλ\\pi^\{\\lambda\}\_\{n\}denote the filter started fromZ0∼λZ\_\{0\}\\sim\\lambda, and setεnfs:=supλ∈𝒫⁡\(S\)𝔼λ​‖πnλ−πnν^‖TV\\varepsilon^\{\\mathrm\{fs\}\}\_\{n\}:=\\sup\_\{\\lambda\\in\\mathcal\{P\}\(S\)\}\\mathbb\{E\}^\{\\lambda\}\\\|\\pi^\{\\lambda\}\_\{n\}\-\\pi^\{\\hat\{\\nu\}\}\_\{n\}\\\|\_\{\\mathrm\{TV\}\}\. Conditioning onY0:r−MY\_\{0:r\-M\}as in the proof of Proposition 2\.9 givessupr≥M𝔼​‖Pr−PrM‖TV≤2​εM−1fs\\sup\_\{r\\geq M\}\\mathbb\{E\}\\\|P\_\{r\}\-P^\{M\}\_\{r\}\\\|\_\{\\mathrm\{TV\}\}\\leq 2\\varepsilon^\{\\mathrm\{fs\}\}\_\{M\-1\}, without assuming stationarity\.

Under \(A1\),εnfs→0\\varepsilon^\{\\mathrm\{fs\}\}\_\{n\}\\to 0whenever the filter is stable in total variation in expectation for every prior\([McDonald and Yüksel, 2024](https://arxiv.org/html/2610.41038#bib.bib34), Definition I\.5\(iii\)\)\. Indeed, sinceD\(λ∥ν^\)≤log\|S\|D\(\\lambda\\\|\\hat\{\\nu\}\)\\leq\\log\|S\|, Theorem II\.20 of[McDonald and Yüksel \(2024\)](https://arxiv.org/html/2610.41038#bib.bib34)upgrades this to stability in relative entropy\. The expected relative entropy is non\-increasing innnand continuous inλ\\lambda, so Dini’s theorem on the compact simplex𝒫⁡\(S\)\\mathcal\{P\}\(S\)makes the convergence uniform, and Pinsker’s inequality returns to total variation\.

By Theorems II\.7 and II\.19 of[McDonald and Yüksel \(2024\)](https://arxiv.org/html/2610.41038#bib.bib34), such stability holds whenever the model is observable in their sense \(which in turn builds on[Handel \(2008\)](https://arxiv.org/html/2610.41038#bib.bib22)\)\. For finiteSS, this means that the lawsPr\(Y1:N∈⋅∣Z1=z\)\\mathrm\{Pr\}\(Y\_\{1:N\}\\in\\cdot\\mid Z\_\{1\}=z\),z∈Sz\\in S, are linearly independent for someNN; linearly independent observation densities suffice\. As an additional remark,NNth\-order sources the sliding lift can be used here, since no contraction coefficient is involved\.

*On quantization\.*If Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\) is replaced by continuity ofψ\(⋅∣z\)\\psi\(\\cdot\\mid z\)with modulusωψ\\omega\_\{\\psi\}, the arguments of Proposition[5](https://arxiv.org/html/2610.41038#Thmtheorem5)and Lemma[28](https://arxiv.org/html/2610.41038#Thmtheorem28)hold with2​Lψ/s2L\_\{\\psi\}/sreplaced byωψ​\(2/s\)\\omega\_\{\\psi\}\(2/s\)\.

Consequently, under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1,A3\), the bound of Theorem[27](https://arxiv.org/html/2610.41038#Thmtheorem27)holds withα⌊M/N⌋\\alpha^\{\\lfloor M/N\\rfloor\}and\(1\+2​M/qmin\)​Lψ/s\(1\+2M/q\_\{\\min\}\)L\_\{\\psi\}/sreplaced by2​εM−1fs2\\varepsilon^\{\\mathrm\{fs\}\}\_\{M\-1\}and\(12\+M/qmin\)​ωψ​\(2/s\)\(\\tfrac\{1\}\{2\}\+M/q\_\{\\min\}\)\\,\\omega\_\{\\psi\}\(2/s\), respectively\. This bound is uniform inrrand vanishes in the iterated limit\.

This covers cases outside the scope of Theorem[27](https://arxiv.org/html/2610.41038#Thmtheorem27)\. A deterministic cyclic transition kernel hasδ⁡\(κ\)=0\\delta\(\\kappa\)=0, henceα≥1\\alpha\\geq 1\. Yet the observation matricesθk​J\+\(1−θ\)​A\\frac\{\\theta\}\{k\}J\+\(1\-\\theta\)A\-see §[6\.2\.1](https://arxiv.org/html/2610.41038#S6.SS2.SSS1)are invertible for everyθ<1\\theta<1, so the model is observable for all0<θ<10<\\theta<1\. Finally, sinceSSis finite, the weak Feller\-type conditions in[McDonald and Yüksel \(2024\)](https://arxiv.org/html/2610.41038#bib.bib34)hold automatically, so continuity is needed only in the observation variable\.

## 6Numerical Experiments

For the following trials, unless stated otherwise, we useT=2T=2layers,l=9l=9probability quantization levels,n=5n=5state space quantization levels ranging from−2\.5\-2\.5to2\.52\.5, attention temperatureβ=0\.3\\beta=0\.3,N=1N=1,L=2L=2withK~=400\\tilde\{K\}=400training samples\. We usem=5m=5action quantization levels, ranging entries inW,A,b,Q,VW,A,b,Q,Vfrom−2\.5\-2\.5to2\.52\.5, keepingK=IK=Ias done by[Geshkovski et al\. \(2025\)](https://arxiv.org/html/2610.41038#bib.bib19), since we are using scalars inℝ\\mathbb\{R\}whereQQabsorbsKKby commutativity\. We takeEE=IE\_\{E\}=IandEU=eU⋅IE\_\{U\}=e\_\{U\}\\cdot Ifor a trainable parametereUe\_\{U\}\. This ensures that Assumption[15](https://arxiv.org/html/2610.41038#Thmtheorem15)holds and allows us to vary the divergence of the predictive distribution viaeUe\_\{U\}to approximate the empirical distribution\. We therefore havem6m^\{6\}trainable parameters per layer\. We use cross\-entropy with a floor ofε=10−3\\varepsilon=10^\{\-3\}to cap the maximum prompt\-wise loss at−log⁡\(ε\)≈6\.91\-\\log\(\\varepsilon\)\\approx 6\.91nats, to avoid takinglog⁡\(0\)\\log\(0\)\. First, we test the effect of memory on prediction by varying the observation channel noise, which affects the decay rate ofΔM\\Delta\_\{M\}\. For the stability tests, we run three trials: varying the sources mixing rateα=\(1−δ⁡\(T\)\)​\(2−δ⁡\(Q\)\)\\alpha=\(1\-\\delta\(T\)\)\(2\-\\delta\(Q\)\)by sweepingT,QT,Qvalues, varying the granularity of the action space viann, and varying the granularity of the state space viamm\.

### 6\.1Memory of data in Transformer training

We conduct three trials with varying Markov characteristics of the input\-generating processes by setting a garbling rateξ=0,0\.5,1\\xi=0,0\.5,1\.

For each trial, we test the model on a distinct data set from the training data set, with the same number of data points and quantization levels\. The testing data generation uses the same transition kernel as the training data, which the model is expected to learn\. We plot the last index input \(before\), predicted \(after\) and label states on theyy\-axis for each input state on the x\-axis, which can be seen in the top row of Figure[6](https://arxiv.org/html/2610.41038#S6.F6)\. Note that thexx\-axis has fewer thanK~=98\\tilde\{K\}=98ticks in all cases when we combine identical input labels by converting state labels into distribution labels\. We also plot the difference between the label state and the last index input state \(before\), and the label state and the predicted state \(after\), which can be seen in the bottom row of Figure[6](https://arxiv.org/html/2610.41038#S6.F6)\.

![[Uncaptioned image]](https://arxiv.org/html/2610.41038v1/Figures/xi0-1.png)

![[Uncaptioned image]](https://arxiv.org/html/2610.41038v1/Figures/xi05-1.png)

![[Uncaptioned image]](https://arxiv.org/html/2610.41038v1/Figures/xi1-1.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/xi0-2.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/xi05-2.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/xi1-2.png)

Figure 6:The first group shows the value of the input, prediction, and the label\. The second group illustrates the losses calculated from discrepancy between the predicted states and the labels\.Final loss decreases as the source becomes more Markovian\. However, the variation between the initial data set losses show that the difference between the original loss and the final loss is highest \(even increases after training\) in the completely non\-Markovian case, but remains relatively unchanging between the partially non\-Markovian and completely Markovian case\.

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/example_display.png)Figure 7:The radius of the label circle corresponds to occurrences in the distribution
### 6\.2Filtering via Transformers Trained with Optimal Control

#### 6\.2\.1Effect of mixing rate

Our goal is to control the Markovianness of our generated input sequence\{Yt\}t=0K~\\\{Y\_\{t\}\\\}\_\{t=0\}^\{\\tilde\{K\}\}viaα\\alphadescribed by[McDonald and Yüksel \(2020\)](https://arxiv.org/html/2610.41038#bib.bib33)\.α=\(1−δ⁡\(T\)\)​\(2−δ⁡\(Q\)\)\\alpha=\(1\-\\delta\(T\)\)\(2\-\\delta\(Q\)\)signifies the rate of agreement between two posterior estimates beginning at different starting state distributions given observationsY\[0,t\]Y\_\{\[0,t\]\}, whereα≈1\\alpha\\approx 1has slow or no convergence andα<<1\\alpha<<1has fast convergence to a single estimate\.

We hand\-pick matrices for transition and measure kernelsTTandQQon finite spaces to obtain the desired Dobrushin coefficientsδ⁡\(T\)\\delta\(T\)andδ⁡\(Q\)\\delta\(Q\), determiningα\\alpha\. We tookT,Q∈ℝ3×3T,Q\\in\\mathbb\{R\}^\{3\\times 3\}, mapping betweenn=3n=3quantized states\. Several examples forTTare shown below:

δ⁡\(\[010001100\]\)=0,δ⁡\(\[1/62/31/61/61/62/32/31/61/6\]\)=12,δ⁡\(\[1/31/31/31/31/31/31/31/31/3\]\)=1\\delta\\left\(\\begin\{bmatrix\}0&1&0\\\\ 0&0&1\\\\ 1&0&0\\\\ \\end\{bmatrix\}\\right\)=0,\\qquad\\delta\\left\(\\begin\{bmatrix\}1/6&2/3&1/6\\\\ 1/6&1/6&2/3\\\\ 2/3&1/6&1/6\\\\ \\end\{bmatrix\}\\right\)=\\frac\{1\}\{2\},\\qquad\\delta\\left\(\\begin\{bmatrix\}1/3&1/3&1/3\\\\ 1/3&1/3&1/3\\\\ 1/3&1/3&1/3\\\\ \\end\{bmatrix\}\\right\)=1with the matrices used for different Dobrushin coefficientsδ⁡\(T\)\\delta\(T\)following a similar pattern of uniformly distributing mass outside the single\-step transition matrix, which is shown above on the left\. We construct theQQmatrix entry values to obtain the desired Dobrushin coefficientsδ⁡\(Q\)\\delta\(Q\)with a similar matrix pattern but different row ordering\.

This setup allows us to varyδ⁡\(T\)\\delta\(T\)andδ⁡\(Q\)\\delta\(Q\)independently by selecting the corresponding matrix to observe the independent effect of the transition and observation kernel on the Transformer’s performance\. We test six different values for eachδ⁡\(T\),δ⁡\(Q\)\\delta\(T\),\\delta\(Q\), for 36 total trials\.

We generate an input sequence by taking observationYt∼Q​δXtY\_\{t\}\\sim Q\\delta\_\{X\_\{t\}\}of the underlying Markov process\(Xt\)t=0K~\(X\_\{t\}\)\_\{t=0\}^\{\\tilde\{K\}\}, beginning atX0X\_\{0\}and evolving asXt\+1∼T​δXtX\_\{t\+1\}\\sim T\\delta\_\{X\_\{t\}\}\. The numerical experiment ran for 90 hours\. Each training set was comprised ofK~=98\\tilde\{K\}=98\(prompt, label\) pairs\. The sweep varied the transition and observation kernel independently over the irregularly spaced Dobrushin coefficient set\{0,16,13,12,23,1\}\\\{0,\\frac\{1\}\{6\},\\frac\{1\}\{3\},\\frac\{1\}\{2\},\\frac\{2\}\{3\},1\\\}selected to show interesting behaviour at small Dobrushin coefficients\. The matrices ranged from fully deterministic cyclic to fully uniform/mixing\. We conduct 3 trials and average the loss for each value ofδ⁡\(T\)\\delta\(T\)andδ⁡\(Q\)\\delta\(Q\)\. In doing so, we obtain the losses for Dobrushin coefficients ranging from 0 to 1, as shown in Figure[8](https://arxiv.org/html/2610.41038#S6.F8)\.

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/long-trial.png)Figure 8:Test prediction loss for varying values ofδ⁡\(Q\)\\delta\(Q\)andδ⁡\(T\)\\delta\(T\)Note that the right\-most column corresponds toα=0\\alpha=0, the bottom\-left cell toα=2\\alpha=2, and other values can be found via the equation forα\\alpha\. The loss is the lowest at purely non\-mixing, deterministic kernels at\(δ⁡\(T\),δ⁡\(Q\)\)=\(0,0\)\(\\delta\(T\),\\delta\(Q\)\)=\(0,0\)\. Atδ⁡\(T\)≥0\.67\\delta\(T\)\\geq 0\.67the loss is consistently low regardless ofQQ\.

The near\-deterministic region is prone to occasional confident incorrect predictions, which is strongly punished by cross\-entropy loss\. This is seen at\(0\.17,0\.00\)\(0\.17,0\.00\)and\(0\.50,0\.33\)\(0\.50,0\.33\)with loss 7\.35 and 4\.99, respectively\. These occasional errors may be due to the relatively low sample size of 3; thus, regional trends may be more indicative of Transformer performance\.

Overall, the bottom\-left quadrant \(highα\\alpha\) attains the highest average loss \(2\.84\), while the top\-right quadrant \(lowα\\alpha\) attains the lowest \(1\.26\), suggesting a positive relationship between stability and Transformer performance\. While this trend is partially driven by the outlier at\(δ⁡\(T\),δ⁡\(Q\)\)=\(0\.17,0\)\(\\delta\(T\),\\delta\(Q\)\)=\(0\.17,0\), if we exclude it, the bottom left quadrant average is still elevated \(2\.42\), so the overall direction still holds\.

Thus, asδ⁡\(T\),δ⁡\(Q\)\\delta\(T\),\\delta\(Q\)increase andα\\alphadecreases, loss tends to decrease while stability increases\. This experiment suggests that stability is positively related to Transformer performance\.

#### 6\.2\.2Effect of action space quantization

For this trial, we keepδ⁡\(Q\)=δ⁡\(T\)=13\\delta\(Q\)=\\delta\(T\)=\\frac\{1\}\{3\}to attainα=109\\alpha=\\frac\{10\}\{9\}, and sweepmm, the quantization level of the action space parameters, through\{1,2,3,4,5\}\\\{1,2,3,4,5\\\}\. We keep the action space constant withn=3n=3for all trials\. The resulting loss is displayed in Figure[9](https://arxiv.org/html/2610.41038#S6.F9)\.

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/action_space_loss.png)Figure 9:Test prediction loss for varying values of mThere is a nonlinear dependence on the granularity of the action space for smaller values ofmm\. Loss is higher for very coarse action spaces, and actually increases as we refine the action space, only to decrease for the highest tested value ofm=5m=5\. Note that due to computational challenges of using dynamic programming, we only tested relatively coarse action spaces\.

A possible explanation for the non\-monotonic pattern is the following: for coarse action spaces \(m∈\{1,…,4\}m\\in\\\{1,\\dots,4\\\}\), the model struggles to cleanly fit to the training data set, and distributes the output for uniformly to hedge against badly punished incorrect guesses\. All coarse\-spaced losses cluster around1\.12±0\.021\.12\\pm 0\.02, so the difference is small, and any increase withmmmay reflect overfitting with insufficient generalization due to the smallmm\. That is, when more fine\-grained models make more confident, non\-distributional guesses, they end up even more incorrect because the parameters overfit to loose data trends\. Form=5m=5, the model begins to confidently and correctly model more localized data set tendencies, not solely distributional\.

#### 6\.2\.3Effect of state space quantization

We now sweep varying values ofnn, the quantization level of the state space, keeping the transition kernel Dobrushin coefficientsδ⁡\(T\)\\delta\(T\)andδ⁡\(Q\)\\delta\(Q\), and the action space constant\. We takem=5m=5,K~=400\\tilde\{K\}=400, andδ⁡\(Q\)=δ⁡\(T\)=13\\delta\(Q\)=\\delta\(T\)=\\frac\{1\}\{3\}for all trials\. Note that althoughδ⁡\(Q\)\\delta\(Q\)andδ⁡\(T\)\\delta\(T\)are constant, the transition kernels have different coefficients due to the difference in state space shape\. We create a general kernel matrixΩ\\Omegausing the equation shown below\. Given an action space withnnlevels and a desired Dobrushin coefficientδ⁡\(Ω\)\\delta\(\\Omega\), we takeΩ=δ⁡\(Ω\)n​J\+\(1−δ⁡\(Ω\)\)​A\\Omega=\\frac\{\\delta\(\\Omega\)\}\{n\}J\+\(1\-\\delta\(\\Omega\)\)AwhereJJis the all\-ones matrix andAAis any permutation matrix\. Note that constructingΩ\\Omegathis way ensures that the desiredδ⁡\(Ω\)\\delta\(\\Omega\)is always obtained\. Note also that this corresponds to theT,QT,Qused in[6\.2\.1](https://arxiv.org/html/2610.41038#S6.SS2.SSS1)withn=3n=3, taking

A=\[010001100\]\.A=\\begin\{bmatrix\}0&1&0\\\\ 0&0&1\\\\ 1&0&0\\end\{bmatrix\}\.Figure[10](https://arxiv.org/html/2610.41038#S6.F10)shows the trained model’s predicted distributions for varying state space quantizations, where the size of the squares corresponds to the mass in the predicted measure\.

![[Uncaptioned image]](https://arxiv.org/html/2610.41038v1/Figures/n=2.png)

![[Uncaptioned image]](https://arxiv.org/html/2610.41038v1/Figures/n=3.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n=4.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n=5.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n=6.png)

Figure 10:Shape of the predicted distributions with respect to varying state quantization levelnn\.These predictions resulted in actual test losses and per\-state losses plotted with linear interpolation in Figure[11](https://arxiv.org/html/2610.41038#S6.F11)for varying state quantization levels:

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n-loss.png)

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n-loss-per-state.png)

Figure 11:Test prediction loss \(left\) and per\-state loss \(right\) for varying values of nActual test loss tends to increase withnnin Figure[11](https://arxiv.org/html/2610.41038#S6.F11)\(left\), except atn=6n=6, where the trend breaks and test loss decreases\. However, since cross\-entropy has no intrinsic normalization for the size of the state space, this loss increase reflects the increased difficulty in the task of classification, which is expected since there are more possible states to classify\. Once we normalize by the number of possible states \(nNn^\{N\}\) in Figure[11](https://arxiv.org/html/2610.41038#S6.F11)\(right\), per\-state loss monotonically decreases near\-linearly\. Thus, model performance outpaces the growth rate of the state space itself\.

The cumulative training runs took 75 hours\. The runtime breakdown for each trial is shown in Figure[12](https://arxiv.org/html/2610.41038#S6.F12)\.

![Refer to caption](https://arxiv.org/html/2610.41038v1/Figures/n-runtime.png)Figure 12:Numerical experiment runtime for varying values of nA least\-squares quadratic fit of the runtime in seconds as a function ofnnyieldstruntime​\(n\)=8542\.9​n2−31702\.9​n\+27200\.0t\_\{\\text\{runtime\}\}\(n\)=8542\.9n^\{2\}\-31702\.9n\+27200\.0withR2≈0\.994R^\{2\}\\approx 0\.994\.

## 7Conclusion

We studied next\-token prediction by optimally trained Transformers for observations generated by a latent process\. We truncated the context to a lengthMM, and quantified the resulting error with entropy gaps for stationary ergodic process; with filter stability results forNNth order Markov sources with the explicit rate𝜶⌊M/N⌋\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}, where𝜶\\bm\{\\alpha\}is a mixing rate determined by the Dobrushin coefficients of the transition kernel and the observation kernel\. We then constructed a Transformer that realizes the quantized finite\-window predictor, trained it with dynamic programming over a quantized set of weights\. We finally showed that the resulting learning error can be decomposed into a sum of four factors: loss due to finite memory, quantization error, sample complexity error, and error due to insufficiency of architecture\. Finally, through extensive experiments, we measured how important the memory of data is in Transformer training, and the effect of the mixing rate in filtering via Transformers\.

## Funding

The first author was supported by the Scientific and Technological Research Council of Türkiye \(TÜBİTAK\) through the 2211\-E program\. The third author was supported by Turkish Academy of Sciences through the 2025 GEBIP Award\. The last author was supported in part by the Natural Sciences and Engineering Research Council \(NSERC\) of Canada\.

## Appendix AThe Training Algorithm

Algorithm 1Next Token Prediction with Dynamic Programming1:Input:data set

D~=\{\(𝒙k,y~k\)\}k=1K~\\tilde\{D\}=\\\{\(\\bm\{x\}^\{k\},\\tilde\{y\}^\{k\}\)\\\}\_\{k=1\}^\{\\tilde\{K\}\}, horizon

TT, quantization levels

\(l,n,m\)\(l,n,m\)
2:Construct embedding space grid

𝕏n\\mathbb\{X\}\_\{n\}from

𝕏\\mathbb\{X\}and for

x∈𝕏x\\in\\mathbb\{X\}define

Qn\(⋅\)=argminx′∈𝕏n\|\|⋅−x′\|\|Q\_\{n\}\(\\cdot\)=\\arg\\min\_\{x^\{\\prime\}\\in\\mathbb\{X\}\_\{n\}\}\|\|\\cdot\-x^\{\\prime\}\|\|
3:Construct finite action set

𝕌m=\{U1,…​Um\}⊂𝕌\\mathbb\{U\}\_\{m\}=\\\{U^\{1\},\\dots U^\{m\}\\\}\\subset\\mathbb\{U\}
4:Construct measure grid

𝒫\(l\)​\(𝕏n\)\\mathcal\{P\}^\{\(l\)\}\(\\mathbb\{X\}\_\{n\}\)and for

ν∈𝒫⁡\(𝕏n\)\\nu\\in\\mathcal\{P\}\(\\mathbb\{X\}\_\{n\}\)define

Rl​\(ν\)=arg⁡minν^∈𝒫\(l\)​\(𝕏n\)​W2​\(ν,ν^\)R\_\{l\}\(\\nu\)=\\arg\\min\_\{\\hat\{\\nu\}\\in\\mathcal\{P\}^\{\(l\)\}\(\\mathbb\{X\}\_\{n\}\)\}W\_\{2\}\(\\nu,\\hat\{\\nu\}\)
5:for

k=1,…,K~k=1,\\dots,\\tilde\{K\}do

6:

μ^0k←Rl​\(1N​∑i=1NδQn​\(pi,xi,k\)\)\\hat\{\\mu\}\_\{0\}^\{k\}\\leftarrow R\_\{l\}\(\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}\\delta\_\{Q\_\{n\}\(p\_\{i\},x^\{i,k\}\)\}\)
7:

y^k←Rl​\(∑j=1K~δy~j𝟙\{𝒙j=𝒙k\}∑j=1K~𝟙\{𝒙j=𝒙k\}\)\\hat\{y\}^\{k\}\\leftarrow R\_\{l\}\(\{\\frac\{\\sum\_\{j=1\}^\{\\tilde\{K\}\}\\delta\_\{\\tilde\{y\}^\{j\}\}\\mathbbm\{1\}\_\{\\\{\\bm\{x\}^\{j\}=\\bm\{x\}^\{k\}\\\}\}\}\{\\sum\_\{j=1\}^\{\\tilde\{K\}\}\\mathbbm\{1\}\_\{\\\{\\bm\{x\}^\{j\}=\\bm\{x\}^\{k\}\\\}\}\}\}\)
8:end for

9:Construct data set

D=\{\(𝒙k,y^k\)\}k=1KD=\\\{\(\\bm\{x\}^\{k\},\\hat\{y\}^\{k\}\)\\\}\_\{k=1\}^\{K\}
10:Define dynamics for

U∈𝕌mU\\in\\mathbb\{U\}\_\{m\}Φ\(l,n,m\)​\(μ^,U\)=Rl​\(Φ\(n\)​\(μ^,U\)\)\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\},U\)=R\_\{l\}\(\\Phi^\{\(n\)\}\(\\hat\{\\mu\},U\)\)
11:Define terminal cost

ℭl,n,m,T​\(𝝁^\)=1K​∑k=1KCE⁡\(yk,softmax​\(EU​xT,γk,L\)\)\\mathfrak\{C\}\_\{l,n,m,T\}\(\\hat\{\\bm\{\\mu\}\}\)=\\frac\{1\}\{K\}\\sum\_\{k=1\}^\{K\}\\mathrm\{CE\}\(y^\{k\},\\text\{softmax\}\(E\_\{U\}\{x\_\{T,\\gamma\}^\{k,L\}\}\)\)
12:for

t=T−1t=T\-1down to

00do

13:for all

𝝁^=\(μ^1,…,μ^K\)∈𝒫l​\(𝕏n\)K\\hat\{\\bm\{\\mu\}\}=\(\\hat\{\\mu\}^\{1\},\\dots,\\hat\{\\mu\}^\{K\}\)\\in\\mathcal\{P\}^\{l\}\(\\mathbb\{X\}\_\{n\}\)^\{K\}do

14:

ℭl,n,m,t​\(𝝁^\)=minU∈𝕌m⁡ℭl,n,m,t\+1​\(Φ\(l,n,m\)​\(μ^1,U\),…,Φ\(l,n,m\)​\(μ^K,U\)\)\\mathfrak\{C\}\_\{l,n,m,t\}\(\\hat\{\\bm\{\\mu\}\}\)=\\min\_\{U\\in\\mathbb\{U\}\_\{m\}\}\\mathfrak\{C\}\_\{l,n,m,t\+1\}\(\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\}^\{1\},U\),\\dots,\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\}^\{K\},U\)\)
15:

γt∗​\(𝝁^\)=arg⁡minU∈𝕌m​ℭl,n,m,t\+1​\(Φ\(l,n,m\)​\(μ^1,U\),…,Φ\(l,n,m\)​\(μ^K,U\)\)\\gamma^\{\*\}\_\{t\}\(\\hat\{\\bm\{\\mu\}\}\)=\\arg\\min\_\{U\\in\\mathbb\{U\}\_\{m\}\}\\mathfrak\{C\}\_\{l,n,m,t\+1\}\(\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\}^\{1\},U\),\\dots,\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\}^\{K\},U\)\)
16:end for

17:end for

18:Forward Pass:

19:for

t=0,…,T−1t=0,\\dots,T\-1do

20:

Ut∗←γt∗​\(μ^t1,…,μ^tK\)U^\{\*\}\_\{t\}\\leftarrow\\gamma^\{\*\}\_\{t\}\(\\hat\{\\mu\}\_\{t\}^\{1\},\\dots,\\hat\{\\mu\}\_\{t\}^\{K\}\)
21:for

k=1,…​Kk=1,\\dots Kdo

22:

μ^t\+1k←Φ\(l,n,m\)​\(μ^K,tW,Ut∗\)\\hat\{\\mu\}^\{k\}\_\{t\+1\}\\leftarrow\\Phi^\{\(l,n,m\)\}\(\\hat\{\\mu\}^\{W\}\_\{K,t\},U^\{\*\}\_\{t\}\)
23:end for

24:end for

25:Output:

\(Ut∗\)t=0T−1\(U^\{\*\}\_\{t\}\)^\{T\-1\}\_\{t=0\}

## Appendix BProofs of the Results in Section[2](https://arxiv.org/html/2610.41038#S2)

### B\.1Auxiliary Results

###### Lemma 28\.

Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2, A3\), for allr∈ℤ\+r\\in\\mathbb\{Z\}\_\{\+\}ands∈ℕs\\in\\mathbb\{N\},maxY0:r∈𝕐r∥Pr\(⋅∣Y0:r\)−Ξs\(q\#sPr\)\(⋅∣Y0:r\)∥TV≤Lψ/s\\max\_\{Y\_\{0:r\}\\in\\mathbb\{Y\}^\{r\}\}\\\|P\_\{r\}\(\\,\\cdot\\mid Y\_\{0:r\}\)\-\\Xi\_\{s\}\(q^\{s\}\_\{\\\#\}P\_\{r\}\)\(\\,\\cdot\\mid Y\_\{0:r\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq L\_\{\\psi\}/s\.

###### Proof\.

Taking the Radon\-Nikodym derivative ofPrP\_\{r\}with respect toμ\\mu, we get aμ\\mu\-integrable functionpr\(y\):=∑z∈Sψ\(y∣z\)Pr\(Zr\+1=z∣Y0:r\)p\_\{r\}\(y\):=\\sum\_\{z\\in S\}\\psi\(y\\mid z\)\\,\\mathrm\{Pr\}\(Z\_\{r\+1\}=z\\mid Y\_\{0:r\}\)\. Then, for ally,y′∈𝕐y,y^\{\\prime\}\\in\\mathbb\{Y\},

\|pr​\(y\)−pr​\(y\)\|\\displaystyle\|p\_\{r\}\(y\)\-p\_\{r\}\(y\)\|=\|∑z∈S\(ψ\(y∣z\)−ψ\(y′∣z\)\)Pr\(Zr\+1=z∣Y0:r\)\|\\displaystyle=\\big\|\\sum\_\{z\\in S\}\(\\psi\(y\\mid z\)\-\\psi\(y^\{\\prime\}\\mid z\)\)\\,\\mathrm\{Pr\}\(Z\_\{r\+1\}=z\\mid Y\_\{0:r\}\)\\big\|≤∑z∈S\|ψ\(y∣z\)−ψ\(y′∣z\)\|Pr\(Zr\+1=z∣Y0:r\\displaystyle\\leq\\sum\_\{z\\in S\}\|\\psi\(y\\mid z\)\-\\psi\(y^\{\\prime\}\\mid z\)\|\\mathrm\{Pr\}\(Z\_\{r\+1\}=z\\mid Y\_\{0:r\}≤Lψρ\(y,y′\)∑z∈SPr\(Zr\+1=z∣Y0:r\)⏟1\\displaystyle\\leq L\_\{\\psi\}\\rho\(y,y^\{\\prime\}\)\\underbrace\{\\sum\_\{z\\in S\}\\mathrm\{Pr\}\(Z\_\{r\+1\}=z\\mid Y\_\{0:r\}\)\}\_\{1\}\(by Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\)\)=Lψ​ρ​\(y,y′\)\.\\displaystyle=L\_\{\\psi\}\\rho\(y,y^\{\\prime\}\)\.So,prp\_\{r\}is alsoLψL\_\{\\psi\}\-Lipschitz\. For eachi=1,…,R⁡\(s\)i=1,\\ldots,R\(s\), on the cellC⁡\(s,i\)C\(s,i\), theμ\\mu\-density ofΞs\(q\#sPr\)\(⋅∣Y0:r\)\\Xi\_\{s\}\(q^\{s\}\_\{\\\#\}P\_\{r\}\)\(\\,\\cdot\\mid Y\_\{0:r\}\), denotedp^r\\hat\{p\}\_\{r\}, is constant and equal to theμ\\mu\-average ofprp\_\{r\}\. Hence,

supy∈C⁡\(s,i\)\|pr​\(y\)−p^r​\(y\)\|≤Lψ​diam⁡\(C⁡\(s,i\)\)≤2​Lψs\.\\sup\_\{y\\in C\(s,i\)\}\|p\_\{r\}\(y\)\-\\hat\{p\}\_\{r\}\(y\)\|\\leq L\_\{\\psi\}\\operatorname\{diam\}\(C\(s,i\)\)\\leq\\frac\{2L\_\{\\psi\}\}\{s\}\.We use the density representation of the total variation distance:

∥Pr\(⋅∣Y0:r\)−Ξs\(q\#sPr\)\(⋅∣Y0:r\)∥TV=12∫𝕐μ\(dy\)\|pr\(y\)−p^r\(y\)\|≤12⋅2​Lψs=Lψs\.\\\|P\_\{r\}\(\\,\\cdot\\mid Y\_\{0:r\}\)\-\\Xi\_\{s\}\(q^\{s\}\_\{\\\#\}P\_\{r\}\)\(\\,\\cdot\\mid Y\_\{0:r\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}=\\frac\{1\}\{2\}\\int\_\{\\mathbb\{Y\}\}\\mu\(\\mathrm\{d\}y\)\\,\|p\_\{r\}\(y\)\-\\hat\{p\}\_\{r\}\(y\)\|\\leq\\frac\{1\}\{2\}\\cdot\\frac\{2L\_\{\\psi\}\}\{s\}=\\frac\{L\_\{\\psi\}\}\{s\}\.The above bound does not depend on the choice of trajectoryY0:rY\_\{0:r\}at any step of this proof, thereby, the claim holds and the proof is now concluded\. ∎

###### Lemma 29\.

Under Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\) and \(A2\), for all𝐲=\(y1,…,yM\),𝐲′=\(y1′,…,yM′\)∈𝕐M\\mathbf\{y\}=\(y\_\{1\},\\ldots,y\_\{M\}\),\\mathbf\{y\}^\{\\prime\}=\(y\_\{1\}^\{\\prime\},\\ldots,y\_\{M\}^\{\\prime\}\)\\in\\mathbb\{Y\}^\{M\}, for allr≥Mr\\geq M, we have

∥Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲\)−Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲′\)∥TV≤Lψqmin​∑i=1Mρ⁡\(yi,yi′\)\.\\begin\{split\}\\\|\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{y\}\)\-\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid&Y\_\{r\-M\+1:r\}=\\mathbf\{y\}^\{\\prime\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\leq\\frac\{L\_\{\\psi\}\}\{q\_\{\\rm min\}\}\\sum\_\{i=1\}^\{M\}\\rho\(y\_\{i\},y\_\{i\}^\{\\prime\}\)\.\\end\{split\}

###### Proof\.

LetP~\(𝐳\):=Pr\{Zr−M\+1:r=𝐳\}\\tilde\{P\}\(\\mathbf\{z\}\):=\\mathrm\{Pr\}\\\{Z\_\{r\-M\+1:r\}=\\mathbf\{z\}\\\}for all𝐳∈SM\\mathbf\{z\}\\in S^\{M\}\. Conditional independence ofYr−M\+1:rY\_\{r\-M\+1:r\}gives the likelihood

𝝍𝐲~​\(𝐳\):=∏i=1Mψ⁡\(y~i∣zi\)\\bm\{\\psi\}\_\{\\tilde\{\\mathbf\{y\}\}\}\(\\mathbf\{z\}\):=\\prod\_\{i=1\}^\{M\}\\psi\(\\tilde\{y\}\_\{i\}\\mid z\_\{i\}\)for all𝐲~:=\(y~1,…,y~M\)∈𝕐M\\tilde\{\\mathbf\{y\}\}:=\(\\tilde\{y\}\_\{1\},\\ldots,\\tilde\{y\}\_\{M\}\)\\in\\mathbb\{Y\}^\{M\}\. The posterior is then

μ𝐲~​\(𝐳\):=P~​\(𝐳\)​𝝍𝐲~​\(𝐳\)∑𝐳′∈SMP~​\(𝐳′\)​𝝍𝐲~​\(𝐳′\)\.\\mu\_\{\\tilde\{\\mathbf\{y\}\}\}\(\\mathbf\{z\}\):=\\frac\{\\tilde\{P\}\(\\mathbf\{z\}\)\\,\\bm\{\\psi\}\_\{\\tilde\{\\mathbf\{y\}\}\}\(\\mathbf\{z\}\)\}\{\\sum\_\{\\mathbf\{z\}^\{\\prime\}\\in S^\{M\}\}\\tilde\{P\}\(\\mathbf\{z\}^\{\\prime\}\)\\,\\bm\{\\psi\}\_\{\\tilde\{\\mathbf\{y\}\}\}\(\\mathbf\{z\}^\{\\prime\}\)\}\.We continue after the following*intermezzo*: For any two vectorsu,v∈ℝ\+d∖\{0\}u,v\\in\\mathbb\{R\}\_\{\+\}^\{d\}\\setminus\\\{0\\\}for somed∈ℕd\\in\\mathbb\{N\}, we denote their normalizations byu^=u/‖u‖1\\hat\{u\}=u/\\\|u\\\|\_\{1\}andv^=v/‖v‖1\\hat\{v\}=v/\\\|v\\\|\_\{1\}\. Then, we can write

u^−v^=u−v‖u‖1\+v⁡\(1‖u‖1−1‖v‖1\)\.\\hat\{u\}\-\\hat\{v\}=\\frac\{u\-v\}\{\\\|u\\\|\_\{1\}\}\+v\\left\(\\frac\{1\}\{\\\|u\\\|\_\{1\}\}\-\\frac\{1\}\{\\\|v\\\|\_\{1\}\}\\right\)\.\(26\)The term on the right hand side has the norm

‖v⁡\(1‖u‖1−1‖v‖1\)‖1≤\|∥v∥1−∥u∥1‖u‖1≤‖v−u‖1‖u‖1\.\\left\\\|v\\left\(\\frac\{1\}\{\\\|u\\\|\_\{1\}\}\-\\frac\{1\}\{\\\|v\\\|\_\{1\}\}\\right\)\\right\\\|\_\{1\}\\leq\\frac\{\|\\\|v\\\|\_\{1\}\-\\\|u\\\|\_\{1\}\}\{\\\|u\\\|\_\{1\}\}\\leq\\frac\{\\\|v\-u\\\|\_\{1\}\}\{\\\|u\\\|\_\{1\}\}\.\(27\)Therefore, by \([26](https://arxiv.org/html/2610.41038#A2.E26)\) and \([27](https://arxiv.org/html/2610.41038#A2.E27)\), we have

‖u^−v^‖TV=12​‖u^−v^‖1≤‖v−u‖1‖u‖1\.\\\|\\hat\{u\}\-\\hat\{v\}\\\|\_\{\\rm\\scriptscriptstyle TV\}=\\frac\{1\}\{2\}\\\|\\hat\{u\}\-\\hat\{v\}\\\|\_\{1\}\\leq\\frac\{\\\|v\-u\\\|\_\{1\}\}\{\\\|u\\\|\_\{1\}\}\.\(28\)We resume from where we left: Let𝐚,𝐛∈𝕐M\\mathbf\{a\},\\mathbf\{b\}\\in\\mathbb\{Y\}^\{M\}differ at the coordinatekkwithρ⁡\(ak,bk\)=δ\\rho\(a\_\{k\},b\_\{k\}\)=\\delta\. We definec⁡\(𝐳\):=P~​\(𝐳\)​∏i≠kψ⁡\(ai∣zi\)≥0c\(\\mathbf\{z\}\):=\\tilde\{P\}\(\\mathbf\{z\}\)\\prod\_\{i\\neq k\}\\psi\(a\_\{i\}\\mid z\_\{i\}\)\\geq 0\(as a matter of fact, Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\) gives that‖c‖1≥qminM−1\>0\\\|c\\\|\_\{1\}\\geq q\_\{\\rm min\}^\{M\-1\}\>0\)\. Writingu⁡\(𝐳\)=c⁡\(𝐳\)​ψ​\(ak∣zk\)u\(\\mathbf\{z\}\)=c\(\\mathbf\{z\}\)\\psi\(a\_\{k\}\\mid z\_\{k\}\)andv⁡\(𝐳\)=c⁡\(𝐳\)​ψ​\(bk∣zk\)v\(\\mathbf\{z\}\)=c\(\\mathbf\{z\}\)\\psi\(b\_\{k\}\\mid z\_\{k\}\)\. We have

- •from Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\) that‖u−v‖1=∑𝐳∈SMc⁡\(𝐳\)\|ψ⁡\(ak∣zk\)−ψ⁡\(bk∣zk\)\|≤Lψ​δ​‖c‖1\\\|u\-v\\\|\_\{1\}=\\sum\_\{\\mathbf\{z\}\\in S^\{M\}\}c\(\\mathbf\{z\}\)\|\\psi\(a\_\{k\}\\mid z\_\{k\}\)\-\\psi\(b\_\{k\}\\mid z\_\{k\}\)\|\\leq L\_\{\\psi\}\\delta\\\|c\\\|\_\{1\}\.
- •from Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\) that‖u‖1=∑𝐳∈SMc⁡\(𝐳\)​ψ​\(ak∣zk\)≥qmin​‖c‖1\\\|u\\\|\_\{1\}=\\sum\_\{\\mathbf\{z\}\\in S^\{M\}\}c\(\\mathbf\{z\}\)\\psi\(a\_\{k\}\\mid z\_\{k\}\)\\geq q\_\{\\rm min\}\\\|c\\\|\_\{1\}\.

Sinceccis*non\-degenerate*\(‖c‖1\>0\\\|c\\\|\_\{1\}\>0\), from the observation above and from \([28](https://arxiv.org/html/2610.41038#A2.E28)\), we have

‖μ𝐚−μ𝐛‖TV≤Lψ​δqmin\.\\\|\\mu\_\{\\mathbf\{a\}\}\-\\mu\_\{\\mathbf\{b\}\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{L\_\{\\psi\}\\delta\}\{q\_\{\\rm min\}\}\.\(29\)
Now define𝐲\(0\)=𝐲\\mathbf\{y\}^\{\(0\)\}=\\mathbf\{y\},𝐲\(k\)=\(y1′,…,yk′,yk\+1,…,yM\)\\mathbf\{y\}^\{\(k\)\}=\(y\_\{1\}^\{\\prime\},\\ldots,y\_\{k\}^\{\\prime\},y\_\{k\+1\},\\ldots,y\_\{M\}\)for allk=1,…,Mk=1,\\ldots,Mand𝐲\(M\)=𝐲′\\mathbf\{y\}^\{\(M\)\}=\\mathbf\{y\}^\{\\prime\}\. Then, from \([29](https://arxiv.org/html/2610.41038#A2.E29)\), we have

‖μ𝐲−μ𝐲′‖TV≤∑k=1M‖μ𝐲\(k−1\)−μ𝐲\(k\)‖TV≤Lψqmin​∑k=1Mρ⁡\(yk,yk′\)\.\\\|\\mu\_\{\\mathbf\{y\}\}\-\\mu\_\{\\mathbf\{y\}^\{\\prime\}\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sum\_\{k=1\}^\{M\}\\\|\\mu\_\{\\mathbf\{y\}^\{\(k\-1\)\}\}\-\\mu\_\{\\mathbf\{y\}^\{\(k\)\}\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{L\_\{\\psi\}\}\{q\_\{\\rm min\}\}\\sum\_\{k=1\}^\{M\}\\rho\(y\_\{k\},y\_\{k\}^\{\\prime\}\)\.Seeingμ𝐚\\mu\_\{\\mathbf\{a\}\}is equivalent withPr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐚\)\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{a\}\)concludes this proof\. ∎

###### Lemma 30\.

Under Assumption[6](https://arxiv.org/html/2610.41038#Thmtheorem6),\{Yr\}r∈ℤ\+\\\{Y\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}is stationary and ergodic\.

###### Proof\.

By the randomization lemma\([Kallenberg, 2021](https://arxiv.org/html/2610.41038#bib.bib25), Lemma 2\.22\), there existsH:S×\[0,1\]→𝕐H:S\\times\[0,1\]\\to\\mathbb\{Y\}and an i\.i\.d\.\[0,1\]\[0,1\]\-valued noise process\{Vr\}r∈ℤ\+\\\{V\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}independent of\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}such thatYr=H⁡\(Zr,Vr\)Y\_\{r\}=H\(Z\_\{r\},V\_\{r\}\)for allr∈ℤ\+r\\in\\mathbb\{Z\}\_\{\+\}\. Since mixing of an ergodic process with a noise is still ergodic\([Walters, 2000](https://arxiv.org/html/2610.41038#bib.bib48), Theorem 1\.23\), the claim follows\. ∎

###### Lemma 31\.

The block process\{\(𝐗i,𝐘i\)\}i≥0\\\{\(\\mathbf\{X\}\_\{i\},\\mathbf\{Y\}\_\{i\}\)\\\}\_\{i\\geq 0\}is a partially observed Markov process with the state transition kernel𝐓\\mathbf\{T\}and the observation channel𝐐\\mathbf\{Q\}\.

###### Proof\.

Fixi≥0i\\geq 0\. From \([10](https://arxiv.org/html/2610.41038#S2.E10)\) and the definition of𝐓\\mathbf\{T\},𝐗i\+1\\mathbf\{X\}\_\{i\+1\}is generated from𝐗i\\mathbf\{X\}\_\{i\}via the kernel𝐓\\mathbf\{T\}, hence it is Markov\. By the stochastic realization results, eachYiY\_\{i\}is a function ofZiZ\_\{i\}and an independent noiseViV\_\{i\}\. As we block,𝐘i\\mathbf\{Y\}\_\{i\}is then generated only by𝐗i\\mathbf\{X\}\_\{i\}via a noise block𝐕i\\mathbf\{V\}\_\{i\}for allii\. Hence,\{\(𝐗i,𝐘i\)\}i≥0\\\{\(\\mathbf\{X\}\_\{i\},\\mathbf\{Y\}\_\{i\}\)\\\}\_\{i\\geq 0\}is indeed a POMP\. ∎

### B\.2Proof of Proposition[5](https://arxiv.org/html/2610.41038#Thmtheorem5)

Since the prediction kernel is non\-expansive in total variation distance \(as all stochastic kernels are\), writing𝐲′:=qM,s​\(𝐲\)\\mathbf\{y\}^\{\\prime\}:=q^\{M,s\}\(\\mathbf\{y\}\)we get

∥PrM\(⋅∣𝐲\)−PrM,s\(⋅∣𝐪M,s\(𝐲\)\)∥TV=∥Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲\)Pred−Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲′\)Pred∥TV≤∥Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲\)−Pr\(Zr−M\+1:r∈⋅∣Yr−M\+1:r=𝐲′\)∥TV\.\\begin\{split\}\\\|P\_\{r\}^\{M\}\(\\,\\cdot\\mid\\mathbf\{y\}\)\-P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid\\mathbf\{q\}^\{M,s\}\(\\mathbf\{y\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&=\\\|\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{y\}\)\\mathrm\{Pred\}\\\\ &\\qquad\-\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{y\}^\{\\prime\}\)\\mathrm\{Pred\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\leq\\\|\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{y\}\)\\\\ &\\qquad\-\\mathrm\{Pr\}\(Z\_\{r\-M\+1:r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}=\\mathbf\{y\}^\{\\prime\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\.\\end\{split\}Then, we apply Lemma[29](https://arxiv.org/html/2610.41038#Thmtheorem29)with the fact thatρ⁡\(yi,qs​\(yi\)\)≤1/s\\rho\(y\_\{i\},q^\{s\}\(y\_\{i\}\)\)\\leq 1/sfor alli=1,…,Mi=1,\\ldots,M\. The factor22is due to each cell having diameter at most2/s2/sbecause of quantization\.■\\blacksquare

### B\.3Proof of Proposition[7](https://arxiv.org/html/2610.41038#Thmtheorem7)

By Lemma[30](https://arxiv.org/html/2610.41038#Thmtheorem30), for allr≥Mr\\geq M,H\(Yr\+1∣Yr−M\+1:r\)=H\(YM∣Y0:M−1\)=HMH\(Y\_\{r\+1\}\\mid Y\_\{r\-M\+1:r\}\)=H\(Y\_\{M\}\\mid Y\_\{0:M\-1\}\)=H\_\{M\}andHr\+1=H\(Yr\+1∣Y0:r\)H\_\{r\+1\}=H\(Y\_\{r\+1\}\\mid Y\_\{0:r\}\)by definition\. ThenΔM,r=HM−Hr\+1\\Delta\_\{M,r\}=H\_\{M\}\-H\_\{r\+1\}for allr≥Mr\\geq M\. So,supr≥MΔM,r=HM−H∞\\sup\_\{r\\geq M\}\\Delta\_\{M,r\}=H\_\{M\}\-H\_\{\\infty\}\. Note thatHM≥H∞H\_\{M\}\\geq H\_\{\\infty\}for allM∈ℕM\\in\\mathbb\{N\}andHM→H∞H\_\{M\}\\to H\_\{\\infty\}asM→∞M\\to\\infty\. Then, by monotone convergence theorem for a real\-valued non\-increasing sequence,ΔM↓0\\Delta\_\{M\}\\downarrow 0\.

For the second claim, for anyr≥Mr\\geq M, we have from Pinsker’s inequality\([Polyanskiy and Wu, 2025](https://arxiv.org/html/2610.41038#bib.bib38), Theorem 7\.10\)

𝔼Y0:r∥Pr−PrM∥TV\\displaystyle\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}≤𝔼Y0:r12DKL\(Pr∥PrM\)\\displaystyle\\leq\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\sqrt\{\\frac\{1\}\{2\}D\_\{\\rm KL\}\(P\_\{r\}\\parallel P\_\{r\}^\{M\}\)\}≤12𝔼Y0:rDKL\(Pr∥PrM\)=12​ΔM,r,\\displaystyle\\leq\\sqrt\{\\frac\{1\}\{2\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,D\_\{\\rm KL\}\(P\_\{r\}\\parallel P\_\{r\}^\{M\}\)\}=\\sqrt\{\\frac\{1\}\{2\}\\Delta\_\{M,r\}\},where the last inequality follows from Jensen’s inequality for concave functions applied to⋅\\sqrt\{\\cdot\}and the last equality holds by \([6](https://arxiv.org/html/2610.41038#S2.E6)\) and \([7](https://arxiv.org/html/2610.41038#S2.E7)\) applied toA=Yr\+1A=Y\_\{r\+1\},B=Y0:r−MB=Y\_\{0:r\-M\}andC=Yr−M\+1:rC=Y\_\{r\-M\+1:r\}and by the definition ofΔM,r\\Delta\_\{M,r\}in \([3](https://arxiv.org/html/2610.41038#S2.E3)\)\. Taking the suprema overr≥Mr\\geq Mof both sides concludes this proof\.■\\blacksquare

### B\.4Proof of Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9)

Define the one\-step Bayesian update map

F⁡\(λ,y\)​\(z\):=ψ​\(y∣z\)​\(λ​κ​\(z\)\)∑z′∈Sψ⁡\(y∣z′\)​\(λ​κ​\(z′\)\)\.F\(\\lambda,y\)\(z\):=\\frac\{\\psi\(y\\mid z\)\(\\lambda\\kappa\(z\)\)\}\{\\sum\_\{z^\{\\prime\}\\in S\}\\psi\(y\\mid z^\{\\prime\}\)\(\\lambda\\kappa\(z^\{\\prime\}\)\)\}\.We define the initializations of filters:

π0\(⋅\):=Pr\(Zr−M∈⋅∣Y0:r−M\)\\displaystyle\\pi\_\{0\}\(\\cdot\):=\\mathrm\{Pr\}\(Z\_\{r\-M\}\\in\\cdot\\mid Y\_\{0:r\-M\}\)π0′:=πZ\\displaystyle\\pi\_\{0\}^\{\\prime\}:=\\pi\_\{Z\}using which we write the filter update equations

πi:=F⁡\(πi−1,Yr−M\+i\)πi′:=F⁡\(πi−1′,Yr−M\+i\)for all​i∈\{1,…,M\}\\begin\{aligned\} &\\pi\_\{i\}:=F\(\\pi\_\{i\-1\},Y\_\{r\-M\+i\}\)\\\\ &\\pi\_\{i\}^\{\\prime\}:=F\(\\pi\_\{i\-1\}^\{\\prime\},Y\_\{r\-M\+i\}\)\\end\{aligned\}\\quad\\text\{ for all \}i\\in\\\{1,\\ldots,M\\\}so that we have

πM\(⋅\):=Pr\(Zr∈⋅∣Y0:r\)\\displaystyle\\pi\_\{M\}\(\\cdot\):=\\mathrm\{Pr\}\(Z\_\{r\}\\in\\cdot\\mid Y\_\{0:r\}\)πM′\(⋅\):=Pr\(Zr∈⋅∣Yr−M\+1:r\)\\displaystyle\\pi\_\{M\}^\{\\prime\}\(\\cdot\):=\\mathrm\{Pr\}\(Z\_\{r\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}\)If we definePred⁡\(y∣z\):=∑z′∈SQ⁡\(y∣z′\)​κ​\(z′∣z\)\\mathrm\{Pred\}\(y\\mid z\):=\\sum\_\{z^\{\\prime\}\\in S\}Q\(y\\mid z^\{\\prime\}\)\\,\\kappa\(z^\{\\prime\}\\mid z\)for allz∈Sz\\in Sandy∈𝕐y\\in\\mathbb\{Y\}, then we havePr=πM​PredP\_\{r\}=\\pi\_\{M\}\\mathrm\{Pred\}andPrM=πM′​PredP\_\{r\}^\{M\}=\\pi\_\{M\}^\{\\prime\}\\mathrm\{Pred\}\.

Using non\-expansiveness of stochastic kernel in total variation distance, we get

‖Pr−PrM‖TV≤‖πM​Pred−πM′​Pred‖TV≤‖πM−πM′‖TV\.\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\\|\\pi\_\{M\}\\mathrm\{Pred\}\-\\pi\_\{M\}^\{\\prime\}\\mathrm\{Pred\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\\|\\pi\_\{M\}\-\\pi\_\{M\}^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\.
Sinceπ0′=𝔼​π0\\pi\_\{0\}^\{\\prime\}=\\mathbb\{E\}\\pi\_\{0\}, thenπ0′≪π0\\pi\_\{0\}^\{\\prime\}\\ll\\pi\_\{0\}almost surely, and henceπ0′​κ≪π0​κ\\pi\_\{0\}^\{\\prime\}\\kappa\\ll\\pi\_\{0\}\\kappaa\.s\. Now, conditioning onσ\(Y0:r−M\)\\sigma\(Y\_\{0:r\-M\}\), the process\{\(Zr−M\+1\+i,Yr−M\+1\+i\)\}i≥0\\\{\(Z\_\{r\-M\+1\+i\},Y\_\{r\-M\+1\+i\}\)\\\}\_\{i\\geq 0\}is then a partially observed Markov process whose initial stateZr−M\+1Z\_\{r\-M\+1\}has the lawπ0​κ\\pi\_\{0\}\\kappa\. Then the sequences\{πi\}i≥1\\\{\\pi\_\{i\}\\\}\_\{i\\geq 1\}and\{πi′\}i≥1\\\{\\pi\_\{i\}^\{\\prime\}\\\}\_\{i\\geq 1\}are its filters started from the true priorπ0​κ\\pi\_\{0\}\\kappaand the false priorπ0′​κ\\pi\_\{0\}^\{\\prime\}\\kappa\. Then,\([McDonald and Yüksel, 2020](https://arxiv.org/html/2610.41038#bib.bib33), Corollary 3\.7\)applies:

𝔼\[∥πM−πM′∥TV∣Y0:r−M\]≤αM∥π0−π0′∥TV≤αM\.\\mathbb\{E\}\[\\\|\\pi\_\{M\}\-\\pi\_\{M\}^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\mid Y\_\{0:r\-M\}\]\\leq\\alpha^\{M\}\\\|\\pi\_\{0\}\-\\pi\_\{0\}^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\alpha^\{M\}\.Taking the expectations of both sides concludes this proof\.■\\hfill\\blacksquare

### B\.5Proof of Proposition[10](https://arxiv.org/html/2610.41038#Thmtheorem10)

Define the one\-step Bayesian update map

F⁡\(λ,𝐲\)​\(𝐳\):=𝝍​\(𝐲∣𝐳\)​\(λ​𝐓​\(𝐳\)\)∑𝐳′∈SN𝝍⁡\(𝐲∣𝐳′\)​\(λ​𝐓​\(𝐳′\)\)F\(\\lambda,\\mathbf\{y\}\)\(\\mathbf\{z\}\):=\\frac\{\\bm\{\\psi\}\(\\mathbf\{y\}\\mid\\mathbf\{z\}\)\(\\lambda\\mathbf\{T\}\(\\mathbf\{z\}\)\)\}\{\\sum\_\{\\mathbf\{z\}^\{\\prime\}\\in S^\{N\}\}\\bm\{\\psi\}\(\\mathbf\{y\}\\mid\\mathbf\{z\}^\{\\prime\}\)\(\\lambda\\mathbf\{T\}\(\\mathbf\{z\}^\{\\prime\}\)\)\}for allλ∈𝒫⁡\(SN\)\\lambda\\in\\mathcal\{P\}\(S^\{N\}\)and𝐲∈𝕐N\\mathbf\{y\}\\in\\mathbb\{Y\}^\{N\}where𝝍⁡\(𝐲∣𝐳\):=∏i=1Nψ⁡\(yi∣zi\)\\bm\{\\psi\}\(\\mathbf\{y\}\\mid\\mathbf\{z\}\):=\\prod\_\{i=1\}^\{N\}\\psi\(y\_\{i\}\\mid z\_\{i\}\)\. We also defineλ0′:=Pr\(𝐗0∈⋅∣𝐘r−M\+1:res\)\\lambda\_\{0\}^\{\\prime\}:=\\mathrm\{Pr\}\(\\mathbf\{X\}\_\{0\}\\in\\cdot\\mid\\mathbf\{Y\}\_\{r\-M\+1:\\mathrm\{res\}\}\)andλ0:=Pr\(𝐗0∈⋅∣𝐘0:res\)\\lambda\_\{0\}:=\\mathrm\{Pr\}\(\\mathbf\{X\}\_\{0\}\\in\\cdot\\mid\\mathbf\{Y\}\_\{0:\\mathrm\{res\}\}\)\. Using these, we write the filter update equations:

πi:=F⁡\(πi−1,𝐲i\)​with​π0=λ0\\pi\_\{i\}:=F\(\\pi\_\{i\-1\},\\mathbf\{y\}\_\{i\}\)\\text\{ with \}\\pi\_\{0\}=\\lambda\_\{0\}and

πi′:=F⁡\(πi−1′,𝐲i\)​with​π0′=λ0′\.\\pi\_\{i\}^\{\\prime\}:=F\(\\pi\_\{i\-1\}^\{\\prime\},\\mathbf\{y\}\_\{i\}\)\\text\{ with \}\\pi\_\{0\}^\{\\prime\}=\\lambda\_\{0\}^\{\\prime\}\.Conditionally on𝐗0\\mathbf\{X\}\_\{0\}, the pastY0:resY\_\{0:\\mathrm\{res\}\}is independent of\(𝐗1:m,𝐘1:m\)\(\\mathbf\{X\}\_\{1:m\},\\mathbf\{Y\}\_\{1:m\}\)\. Hence,

πm=Pr\(𝐗m∈⋅∣Y0:r\)andπm′=Pr\(𝐗m∈⋅∣Yr−M\+1:r\)\.\\pi\_\{m\}=\\mathrm\{Pr\}\(\\mathbf\{X\}\_\{m\}\\in\\cdot\\mid Y\_\{0:r\}\)\\text\{ and \}\\pi\_\{m\}^\{\\prime\}=\\mathrm\{Pr\}\(\\mathbf\{X\}\_\{m\}\\in\\cdot\\mid Y\_\{r\-M\+1:r\}\)\.AsYr\+1Y\_\{r\+1\}depends on the past only through𝐗m\\mathbf\{X\}\_\{m\}, the predictive distributions take the shapePr=πm​𝐏𝐫𝐞𝐝P\_\{r\}=\\pi\_\{m\}\\mathbf\{Pred\}andPrM=πm′​𝐏𝐫𝐞𝐝P\_\{r\}^\{M\}=\\pi\_\{m\}^\{\\prime\}\\mathbf\{Pred\}\. As stochastic kernels are non\-expansive in total variation, we have

‖Pr−PrM‖TV=‖πm​𝐏𝐫𝐞𝐝−πm′​𝐏𝐫𝐞𝐝‖TV≤‖πm−πm′‖TV\.\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}=\\\|\\pi\_\{m\}\\mathbf\{Pred\}\-\\pi\_\{m\}^\{\\prime\}\\mathbf\{Pred\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\\|\\pi\_\{m\}\-\\pi\_\{m\}^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\.
Condition onY0:resY\_\{0:\\mathrm\{res\}\}and apply the same steps in the proof of Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9)\(see §[B\.5](https://arxiv.org/html/2610.41038#A2.SS5)\): applying\([McDonald and Yüksel, 2020](https://arxiv.org/html/2610.41038#bib.bib33), Corollary 3\.7\), we get:

𝔼λ0′\[∥πm−πm′∥TV∣Y0:res\]≤𝜶⌊M/N⌋\\mathbb\{E\}\_\{\\lambda\_\{0\}^\{\\prime\}\}\\,\\left\[\\\|\\pi\_\{m\}\-\\pi\_\{m\}^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\mid Y\_\{0:\\mathrm\{res\}\}\\right\]\\leq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}Taking the expectation and writingm=⌊M/N⌋m=\\lfloor M/N\\rfloorconclude this proof\.■\\blacksquare

### B\.6Proof of Theorem[12](https://arxiv.org/html/2610.41038#Thmtheorem12)

For eachr≥Mr\\geq M, we decompose𝔼​‖Pr−PrM,s‖TV\\mathbb\{E\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}into two parts:

𝔼Y0:r∥Pr−PrM,s∥TV≤𝔼Y0:r∥Pr−PrM∥TV\+𝔼Y0:r∥PrM−PrM,s∥TV\.\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}\-P\_\{r\}^\{M\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\+\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\,\\\|P\_\{r\}^\{M\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\.On the right hand side, the first summand is bounded by Proposition[9](https://arxiv.org/html/2610.41038#Thmtheorem9)for \(i\) and by Proposition[10](https://arxiv.org/html/2610.41038#Thmtheorem10)for \(ii\)\. The second summand is handled by Proposition[5](https://arxiv.org/html/2610.41038#Thmtheorem5)\. As a note, expectation in the second term is not needed since the objects vary only because of the quantized data\. Since the bound holds independent ofr≥Mr\\geq M, the statement follows\.■\\blacksquare

### B\.7Proof of Corollary[13](https://arxiv.org/html/2610.41038#Thmtheorem13)

We allocateε\\varepsilonequally to the two summand in the bound of Theorem[12](https://arxiv.org/html/2610.41038#Thmtheorem12)\. First, we solve the inequality

ε2≥𝜶⌊M/N⌋\\frac\{\\varepsilon\}\{2\}\\geq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}forMMto get:

M≥N​log⁡\(2/ε\)log⁡\(1/𝜶\)\.M\\geq N\\frac\{\\log\{\(2/\\varepsilon\)\}\}\{\\log\(1/\\bm\{\\alpha\}\)\}\.Then we solve

ε2≥2​M​Lψqmin​s\\frac\{\\varepsilon\}\{2\}\\geq\\frac\{2ML\_\{\\psi\}\}\{q\_\{\\rm min\}s\}forss\. This concludes the proof for \(ii\)\. The argument for \(i\) is same with the exception thatNNdoes not appear since the latent source is Markovian, i\.e\.N=1N=1\.■\\blacksquare

## Appendix CProofs of the Results in Section[4](https://arxiv.org/html/2610.41038#S4)

### C\.1Proof of Lemma[16](https://arxiv.org/html/2610.41038#Thmtheorem16)

Writex\+:=σ⁡\(x\)x\_\{\+\}:=\\sigma\(x\)andx−:=σ⁡\(−x\)x\_\{\-\}:=\\sigma\(\-x\)for the coordinate\-wise positive and negative parts ofxx, respectively\. A sequence of simple calculations shows that

WW​σ​\(WA​x\+Wb\)\\displaystyle W\_\{W\}\\sigma\(W\_\{A\}x\+W\_\{b\}\)=WW​σ​\(\[x−x\]\)\\displaystyle=W\_\{W\}\\,\\sigma\\bigg\(\\begin\{bmatrix\}x\\\\ \-x\\end\{bmatrix\}\\bigg\)=\[Id−Id\]​\[x\+x−\]=x\+−x−=x,\\displaystyle=\\begin\{bmatrix\}I\_\{d\}&\-I\_\{d\}\\end\{bmatrix\}\\begin\{bmatrix\}x\_\{\+\}\\\\ x\_\{\-\}\\end\{bmatrix\}=x\_\{\+\}\-x\_\{\-\}=x,which concludes this proof\.■\\blacksquare

### C\.2Proof of Lemma[17](https://arxiv.org/html/2610.41038#Thmtheorem17)

We prove by induction\. Att=0t=0, this is true by construction in \([18](https://arxiv.org/html/2610.41038#S4.E18)\)\. Suppose that the claim is true fort∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}\. Then we get

xt\+1i=xti\+∑j=1Mati​j​Gρ⁡\(t\)​xtjx\_\{t\+1\}^\{i\}=x\_\{t\}^\{i\}\+\\sum\_\{j=1\}^\{M\}a\_\{t\}^\{ij\}G\_\{\\rho\(t\)\}\\,x\_\{t\}^\{j\}as in \([19](https://arxiv.org/html/2610.41038#S4.E19)\) for alli∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}, where

ati​j:=eβ⁡⟨WQ,t​xti,WK,t​xtj⟩∑j′=1Meβ⁡⟨WQ,t​xti,WK,t​xtj′⟩,i,j∈\{1,…,M\}\.a\_\{t\}^\{ij\}:=\\frac\{\\mathrm\{e\}^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j\}\\rangle\}\}\{\\sum\_\{j^\{\\prime\}=1\}^\{M\}\\mathrm\{e\}^\{\\beta\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j^\{\\prime\}\}\\rangle\}\},\\quad i,j\\in\\\{1,\\ldots,M\\\}\.\(30\)By the construction ofGρ⁡\(t\)G\_\{\\rho\(t\)\}in \([21](https://arxiv.org/html/2610.41038#S4.E21)\),∑j=1Mati​j​Gρ⁡\(t\)​xtj\\sum\_\{j=1\}^\{M\}a\_\{t\}^\{ij\}G\_\{\\rho\(t\)\}\\,x\_\{t\}^\{j\}has no non\-zero component in positional encodings and token values, regardless of the attention scores\. Therefore, the positional encodings and token values ofxt\+1ix\_\{t\+1\}^\{i\}are equal to those ofxtix\_\{t\}^\{i\}\.■\\blacksquare

### C\.3Proof of Lemma[18](https://arxiv.org/html/2610.41038#Thmtheorem18)

Using the definitions ofWQ,tW\_\{Q,t\},WK,tW\_\{K,t\},AtA\_\{t\}andselt\\mathrm\{sel\}\_\{t\}, we have

⟨WQ,t​xti,WK,t​xtj⟩=⟨c​At​Πpos​xti,Πpos​xtj⟩=c⁡⟨At​ei,ej⟩=\{c⁡⟨eρ⁡\(t\),ej⟩,i=M,c⁡⟨ei,ej⟩,i≠M\.=c​χj​\(selt​\(i\)\)\.\\begin\{split\}\\langle W\_\{Q,t\}x\_\{t\}^\{i\},W\_\{K,t\}x\_\{t\}^\{j\}\\rangle&=\\langle cA\_\{t\}\\Pi\_\{\\rm pos\}x\_\{t\}^\{i\},\\Pi\_\{\\rm pos\}x\_\{t\}^\{j\}\\rangle\\\\ &=c\\langle A\_\{t\}e\_\{i\},e\_\{j\}\\rangle=\\begin\{cases\}c\\langle e\_\{\\rho\(t\)\},e\_\{j\}\\rangle,&i=M,\\\\ c\\langle e\_\{i\},e\_\{j\}\\rangle,&i\\neq M\.\\end\{cases\}=c\\,\\chi\_\{j\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\.\\end\{split\}Consequently, the attention scores are in the following form: Ifj=selt​\(i\)j=\\mathrm\{sel\}\_\{t\}\(i\), then

ati​selt​\(i\)=exp⁡\(β​c​χj​\(selt​\(i\)\)\)∑j′=1Mexp⁡\(β​c​χj′​\(selt​\(i\)\)\)=exp⁡\(β​c​χj​\(selt​\(i\)\)\)exp⁡\(β​c​χj​\(selt​\(i\)\)\)\+∑j′=1,j′≠jMexp⁡\(β​c​χj′​\(selt​\(i\)\)\)=exp⁡\(β​c\)exp⁡\(β​c\)\+M−1\.\\begin\{split\}a\_\{t\}^\{i\\mathrm\{sel\}\_\{t\}\(i\)\}&=\\frac\{\\exp\(\\beta c\\,\\chi\_\{j\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\)\}\{\\sum\_\{j^\{\\prime\}=1\}^\{M\}\\exp\(\\beta c\\,\\chi\_\{j^\{\\prime\}\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\)\}\\\\ &=\\frac\{\\exp\(\\beta c\\,\\chi\_\{j\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\)\}\{\\exp\(\\beta c\\,\\chi\_\{j\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\)\+\\sum\_\{j^\{\\prime\}=1,j^\{\\prime\}\\neq j\}^\{M\}\\exp\(\\beta c\\,\\chi\_\{j^\{\\prime\}\}\(\\mathrm\{sel\}\_\{t\}\(i\)\)\)\}=\\frac\{\\exp\(\\beta c\)\}\{\\exp\(\\beta c\)\+M\-1\}\.\\end\{split\}Ifj≠selt​\(i\)j\\neq\\mathrm\{sel\}\_\{t\}\(i\), then there existsj′∈\{1,…,M\}j^\{\\prime\}\\in\\\{1,\\ldots,M\\\}withj′≠jj^\{\\prime\}\\neq jsuch thatj′=selt​\(i\)j^\{\\prime\}=\\mathrm\{sel\}\_\{t\}\(i\)sinceρ\\rhois a bijection\. Hence, we have

ati​j=exp⁡\(β​c\)exp⁡\(β​c\)\+M−1\.a\_\{t\}^\{ij\}=\\frac\{\\exp\(\\beta c\)\}\{\\exp\(\\beta c\)\+M\-1\}\.This conclude the proof\.■\\blacksquare

### C\.4Proof of Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20)

Fixt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}\. By Lemma[17](https://arxiv.org/html/2610.41038#Thmtheorem17), the valuesGρ⁡\(t\)​xtj=ι⁡\(wj\)​\(0M,0,ej,0R\)G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}=\\iota\(w\_\{j\}\)\(0\_\{M\},0,e\_\{j\},0\_\{R\}\)are deterministically obtained for allj∈\{1,…,M\}j\\in\\\{1,\\ldots,M\\\}, and they are bounded byR⁡\(s\)R\(s\)in22\-norm\. Then the single layer can be calculated by

‖∑j=1MatM​j​Gρ⁡\(t\)​xtj−Gρ⁡\(t\)​xtρ⁡\(t\)‖2\\displaystyle\\bigg\\\|\\sum\_\{j=1\}^\{M\}a\_\{t\}^\{Mj\}G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}\-G\_\{\\rho\(t\)\}x\_\{t\}^\{\\rho\(t\)\}\\bigg\\\|\_\{2\}=‖\(atM​ρ​\(t\)−1\)​Gρ⁡\(t\)​xtρ⁡\(t\)\+∑j=1j≠ρ⁡\(t\)MatM​j​Gρ⁡\(t\)​xtj‖2\\displaystyle=\\bigg\\\|\(a\_\{t\}^\{M\\rho\(t\)\}\-1\)G\_\{\\rho\(t\)\}x\_\{t\}^\{\\rho\(t\)\}\+\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\neq\\rho\(t\)\\end\{subarray\}\}^\{M\}a\_\{t\}^\{Mj\}G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}\\bigg\\\|\_\{2\}≤\(1−atM​ρ​\(t\)\)​‖Gρ⁡\(t\)​xtρ⁡\(t\)‖2\+∑j=1,j≠ρ⁡\(t\)MatM​j​‖Gρ⁡\(t\)​xtj‖2\\displaystyle\\leq\(1\-a\_\{t\}^\{M\\rho\(t\)\}\)\\\|G\_\{\\rho\(t\)\}x\_\{t\}^\{\\rho\(t\)\}\\\|\_\{2\}\+\\sum\_\{j=1,\\,j\\neq\\rho\(t\)\}^\{M\}a\_\{t\}^\{Mj\}\\\|G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}\\\|\_\{2\}By Lemma[17](https://arxiv.org/html/2610.41038#Thmtheorem17),‖Gρ⁡\(t\)​xtj‖2=ι⁡\(wj\)≤R⁡\(s\)\\\|G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}\\\|\_\{2\}=\\iota\(w\_\{j\}\)\\leq R\(s\)for allj∈\{1,…,M\}j\\in\\\{1,\\ldots,M\\\},∑j=1j≠ρ⁡\(t\)MatM​j=1−atM​ρ​\(t\)\\sum\_\{\\begin\{subarray\}\{c\}j=1\\\\ j\\neq\\rho\(t\)\\end\{subarray\}\}^\{M\}a\_\{t\}^\{Mj\}=1\-a\_\{t\}^\{M\\rho\(t\)\}, and1−atM⁡\(t\)≤\(M−1\)​e−β​c1\-a\_\{t\}^\{M\(t\)\}\\leq\(M\-1\)e^\{\-\\beta c\}by Lemma[18](https://arxiv.org/html/2610.41038#Thmtheorem18), then≤R⁡\(s\)​\(M−1\)​e−β​c\+R⁡\(s\)​\(M−1\)​e−β​c\\displaystyle\\leq R\(s\)\(M\-1\)e^\{\-\\beta c\}\+R\(s\)\(M\-1\)e^\{\-\\beta c\}=2​\(M−1\)​R​\(s\)​e−β​c\.\\displaystyle=2\(M\-1\)R\(s\)e^\{\-\\beta c\}\.Since both recursions of the formzt\+1=zt\+⋆z\_\{t\+1\}=z\_\{t\}\+\\star, the errors are added:

xMM−x¯MM=∑t=0M−1∑j=1MatM​j​Gρ⁡\(t\)​xtj−Gρ⁡\(t\)​xtρ⁡\(t\)\.x\_\{M\}^\{M\}\-\\bar\{x\}\_\{M\}^\{M\}=\\sum\_\{t=0\}^\{M\-1\}\\sum\_\{j=1\}^\{M\}a\_\{t\}^\{Mj\}G\_\{\\rho\(t\)\}x\_\{t\}^\{j\}\-G\_\{\\rho\(t\)\}x\_\{t\}^\{\\rho\(t\)\}\.Passing to the norm and applying the triangle inequality, we obtain

‖xMM−x¯MM‖2≤2​M​\(M−1\)​R​\(s\)​e−β​c,\\\|x\_\{M\}^\{M\}\-\\bar\{x\}\_\{M\}^\{M\}\\\|\_\{2\}\\leq 2M\(M\-1\)R\(s\)e^\{\-\\beta c\},which verifies the statement’s claim\.■\\blacksquare

### C\.5Proof of Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)

We writeσ\\sigmafor𝚁𝚎𝙻𝚄\{\\tt ReLU\}\. Sinceζk\\zeta\_\{k\}’s are distinct, the setHk,j:=\{a∈ℝd:⟨a,ζk−ζj⟩=0\}H\_\{k,j\}:=\\\{a\\in\\mathbb\{R\}^\{d\}:\\langle a,\\zeta\_\{k\}\-\\zeta\_\{j\}\\rangle=0\\\}is a hyperplane inℝd\\mathbb\{R\}^\{d\}, therefore, is ofpp\-dimensional Lebesgue measure00\. So, the finite unionH=⋃k≠jHk,jH=\\bigcup\_\{k\\neq j\}H\_\{k,j\}is also ofdd\-dimensional Lebesgue measure00\. Let us picka∈ℝd∖Ha\\in\\mathbb\{R\}^\{d\}\\setminus Hso thattkt\_\{k\}’s defined bytk:=⟨a,ζk⟩t\_\{k\}:=\\langle a,\\zeta\_\{k\}\\rangleare pairwise distinct\. Relabel them so thatt1<⋯<tnt\_\{1\}<\\cdots<t\_\{n\}\.

We define the slopes

sk=vk\+1−vktk\+1−tk∈ℝR⁡\(s\),t∈ℝ\+s\_\{k\}=\\frac\{v\_\{k\+1\}\-v\_\{k\}\}\{t\_\{k\+1\}\-t\_\{k\}\}\\in\\mathbb\{R\}^\{R\(s\)\},\\quad t\\in\\mathbb\{R\}\_\{\+\}for allk=1,…,nk=1,\\ldots,nwiths0=0s\_\{0\}=0\. Set

g⁡\(t\):=v1\+∑k=1n−1\(sk−sk−1\)​σ​\(t−tk\)\.g\(t\):=v\_\{1\}\+\\sum\_\{k=1\}^\{n\-1\}\(s\_\{k\}\-s\_\{k\-1\}\)\\,\\sigma\(t\-t\_\{k\}\)\.Then, with an induction argument onkk, one can easily show thatg⁡\(tk\)=vkg\(t\_\{k\}\)=v\_\{k\}for everyk=1,…,n−1k=1,\\ldots,n\-1\.

Finally, we composeφ⁡\(ζ\):=g⁡\(⟨a,ζ⟩\)\\varphi\(\\zeta\):=g\(\\langle a,\\zeta\\rangle\), which gives the desired interpolation propertyφ⁡\(ζk\)=g⁡\(tk\)=vk\\varphi\(\\zeta\_\{k\}\)=g\(t\_\{k\}\)=v\_\{k\}\. For the Lipschitz constant, takeζ,ζ′∈ℝd\\zeta,\\zeta^\{\\prime\}\\in\\mathbb\{R\}^\{d\}

‖φ⁡\(ζ\)−φ⁡\(ζ′\)‖2≤∑k=0n−1‖sk−sk−1‖2​\|σ⁡\(⟨a,ζ⟩\)−σ⁡\(⟨a,ζ′⟩\)\|≤∑k=0n−1∥sk−sk−1∥2\|⟨a,ζ⟩−⟨a,ζ′⟩\|\(⋆\)≤∑k=0n−1‖sk−sk−1‖2​\|⟨a,ζ−ζ′⟩\|≤∑k=0n−1‖sk−sk−1‖2​‖a‖2​‖ζ−ζ′‖2\.\\begin\{split\}\\\|\\varphi\(\\zeta\)\-\\varphi\(\\zeta^\{\\prime\}\)\\\|\_\{2\}&\\leq\\sum\_\{k=0\}^\{n\-1\}\\\|s\_\{k\}\-s\_\{k\-1\}\\\|\_\{2\}\\,\|\\sigma\(\\langle a,\\zeta\\rangle\)\-\\sigma\(\\langle a,\\zeta^\{\\prime\}\\rangle\)\|\\\\ &\\leq\\sum\_\{k=0\}^\{n\-1\}\\\|s\_\{k\}\-s\_\{k\-1\}\\\|\_\{2\}\\,\|\\langle a,\\zeta\\rangle\-\\langle a,\\zeta^\{\\prime\}\\rangle\|\\quad\(\\star\)\\\\ &\\leq\\sum\_\{k=0\}^\{n\-1\}\\\|s\_\{k\}\-s\_\{k\-1\}\\\|\_\{2\}\\,\|\\langle a,\\zeta\-\\zeta^\{\\prime\}\\rangle\|\\leq\\sum\_\{k=0\}^\{n\-1\}\\\|s\_\{k\}\-s\_\{k\-1\}\\\|\_\{2\}\\,\\\|a\\\|\_\{2\}\\,\\\|\\zeta\-\\zeta^\{\\prime\}\\\|\_\{2\}\.\\end\{split\}ThereforeLφ:=‖a‖2​∑k=0n−1‖sk−sk−1‖2<\+∞L\_\{\\varphi\}:=\\\|a\\\|\_\{2\}\\sum\_\{k=0\}^\{n\-1\}\\\|s\_\{k\}\-s\_\{k\-1\}\\\|\_\{2\}<\+\\inftysince it is a finite sum\. In above,\(⋆\)\(\\star\)is due to11\-Lipschitzness of𝚁𝚎𝙻𝚄\{\\tt ReLU\}and the last majorization is due to the Cauchy\-Bunyakovsky\-Schwarz inequality\.■\\blacksquare

### C\.6Proof of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)

We prove step\-by\-step:

Proof of \(i\)\.By Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20), we havex¯MM:=\[eM​ι​\(wM\)​\(ι⁡\(w1\),…,ι⁡\(wM\)\)​0R\]\\bar\{x\}\_\{M\}^\{M\}:=\[e\_\{M\}\\;\\iota\(w\_\{M\}\)\\;\(\\iota\(w\_\{1\}\),\\ldots,\\iota\(w\_\{M\}\)\)\\;0\_\{R\}\]\. With the choice ofaa, the result is evident\. For the norm:

‖a‖22=∑j=1M\(R​\(s\)2\)j−1=R​\(s\)2​M−1R​\(s\)2−1≤R​\(s\)2​M−1R⁡\(s\)≤R​\(s\)2​M−1\.\\\|a\\\|\_\{2\}^\{2\}=\\sum\_\{j=1\}^\{M\}\(R\(s\)^\{2\}\)^\{j\-1\}=\\frac\{R\(s\)^\{2M\}\-1\}\{R\(s\)^\{2\}\-1\}\\leq\\frac\{R\(s\)^\{2M\}\-1\}\{R\(s\)\}\\leq R\(s\)^\{2M\-1\}\.This gives that‖a‖2≤R​\(s\)M−1/2≤R​\(s\)M\\\|a\\\|\_\{2\}\\leq R\(s\)^\{M\-1/2\}\\leq R\(s\)^\{M\}\. Finally, considering thatι⁡\(w\)↦ι⁡\(w\)−1\\iota\(w\)\\mapsto\\iota\(w\)\-1is an injective map from\{1,…,R​\(s\)M\}\\\{1,\\ldots,R\(s\)^\{M\}\\\}to\{0,…,R​\(s\)M−1\}\\\{0,\\ldots,R\(s\)^\{M\}\-1\\\}for anyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\},tk\+1−tk≥1t\_\{k\+1\}\-t\_\{k\}\\geq 1can be obtained via the same argument in Corollary[21](https://arxiv.org/html/2610.41038#Thmtheorem21)\.

Proof of \(ii\)\.First we note thatP^rM,s\\hat\{P\}\_\{r\}^\{M,s\}satisfies:miny^∈Λs,w^∈ΛsM⁡P^rM,s​\(y^∣w^\)≥qmin​μmins\\min\_\{\\hat\{y\}\\in\\Lambda\_\{s\},\\,\\hat\{w\}\\in\\Lambda\_\{s\}^\{M\}\}\\hat\{P\}\_\{r\}^\{M,s\}\(\\hat\{y\}\\mid\\hat\{w\}\)\\geq q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}since for anyy^∈Λs\\hat\{y\}\\in\\Lambda\_\{s\}andw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}we have

P^rM,s​\(y^∣w^\)\\displaystyle\\hat\{P\}\_\{r\}^\{M,s\}\(\\hat\{y\}\\mid\\hat\{w\}\)=𝔼\[Pr\(Yr\+1∈C\(s,i\)∣Zr\+1\)∣qM,s\(Yr−M\+1:r\)=w^\]\\displaystyle=\\mathbb\{E\}\[\\mathrm\{Pr\}\(Y\_\{r\+1\}\\in C\(s,i\)\\mid Z\_\{r\+1\}\)\\mid q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=\\hat\{w\}\]=∫C⁡\(s,i\)μ⁡\(𝑑y\)​ψ​\(y∣Zr\+1\)\\displaystyle=\\int\_\{C\(s,i\)\}\\mu\(\\mathrm\{d\}y\)\\,\\psi\(y\\mid Z\_\{r\+1\}\)≥qmin​∫C⁡\(s,i\)μ⁡\(𝑑y\)\\displaystyle\\geq q\_\{\\rm min\}\\int\_\{C\(s,i\)\}\\mu\(\\mathrm\{d\}y\)\(by Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\)\)≥qmin​μmins\.\\displaystyle\\geq q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\.\(by Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A4\)\)Since the targets are of the formvw=logP^rM,s\(⋅∣w\)v\_\{w\}=\\log\{\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\}, we have\{\(vw\)y\}y∈Λs∈\[−log⁡1qmin​μmins,0\]\\\{\(v\_\{w\}\)\_\{y\}\\\}\_\{y\\in\\Lambda\_\{s\}\}\\in\[\-\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\},0\]\. Fixk=1,…,R​\(s\)M−1k=1,\\ldots,R\(s\)^\{M\}\-1, then

‖vw‖2=∑y∈Λs\(vw\)y2≤log⁡1qmin​μmins​R⁡\(s\)\.\\\|v\_\{w\}\\\|\_\{2\}=\\sqrt\{\\sum\_\{y\\in\\Lambda\_\{s\}\}\(v\_\{w\}\)\_\{y\}^\{2\}\}\\leq\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\}\\sqrt\{R\(s\)\}\.From here, by an application of the triangle inequality, we obtain

‖vk\+1−vk‖2≤2​log⁡1qmin​μmins​R⁡\(s\)\.\\\|v\_\{k\+1\}\-v\_\{k\}\\\|\_\{2\}\\leq 2\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\}\\sqrt\{R\(s\)\}\.Withtk\+1−tk≥1t\_\{k\+1\}\-t\_\{k\}\\geq 1from \(i\), it follows that‖sk‖2≤2​log⁡1qmin​μmins​R⁡\(s\)\.\\\|s\_\{k\}\\\|\_\{2\}\\leq 2\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\}\\sqrt\{R\(s\)\}\.Likewise, this leads us to‖sk\+1−sk‖2≤4​log⁡1qmin​μmins​R⁡\(s\)\\\|s\_\{k\+1\}\-s\_\{k\}\\\|\_\{2\}\\leq 4\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\}\\sqrt\{R\(s\)\}\. Hence,

Lφ≤‖a‖2​∑k=1R​\(s\)M−1‖sk\+1−sk‖2≤R​\(s\)M−1/2​\(R​\(s\)M−1\)​4​R⁡\(s\)​log⁡1qmin​μmins=4​R​\(s\)M​\(R​\(s\)M−1\)​log⁡1qmin​μmins,\\begin\{split\}L\_\{\\varphi\}&\\leq\\\|a\\\|\_\{2\}\\sum\_\{k=1\}^\{R\(s\)^\{M\}\-1\}\\\|s\_\{k\+1\}\-s\_\{k\}\\\|\_\{2\}\\leq R\(s\)^\{M\-1/2\}\(R\(s\)^\{M\}\-1\)4\\sqrt\{R\(s\)\}\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\}\\\\ &=4R\(s\)^\{M\}\(R\(s\)^\{M\}\-1\)\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\}\},\\end\{split\}which gives the bound in the statement\.■\\blacksquare

### C\.7Proof of Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)

Fixw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\. Letz⁡\(w\):=Πlog​xM\+1Mz\(w\):=\\Pi\_\{\\rm log\}x\_\{M\+1\}^\{M\}for the logits produced from the interpolator layer andz¯\(w\):=logP^rM,s\(⋅∣w\)\\bar\{z\}\(w\):=\\log\{\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\}for the log\-targets\. Let\{xtM,\(w\)\}t=0M\+1\\\{x\_\{t\}^\{M,\(w\)\}\\\}\_\{t=0\}^\{M\+1\}be the hard\-choice trajectory as in Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20)\. By Corollary[21](https://arxiv.org/html/2610.41038#Thmtheorem21), the readouts are separated by at least11in norm, which enables us to invoke Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)to construct a one layerReLUnetworkφ:ℝd→ℝR⁡\(s\)\\varphi:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{R\(s\)\}withφ⁡\(x¯MM,\(w\)\)=z¯​\(w\)\\varphi\(\\bar\{x\}\_\{M\}^\{M,\(w\)\}\)=\\bar\{z\}\(w\)\. SinceP^rM,s\(⋅∣w\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)is a probability vector, we get

softmax​\(z¯​\(w\)\)v=exp⁡\(log⁡P^rM,s​\(v∣w\)\)∑v′∈Λsexp⁡\(log⁡P^rM,s​\(v′∣w\)\)=P^rM,s​\(v∣w\)∑v′∈ΛsP^rM,s​\(v′∣w\)=P^rM,s​\(v∣w\)\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}=\\frac\{\\exp\(\\log\{\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)\}\)\}\{\\sum\_\{v^\{\\prime\}\\in\\Lambda\_\{s\}\}\\exp\(\\log\{\\hat\{P\}\_\{r\}^\{M,s\}\(v^\{\\prime\}\\mid w\)\}\)\}=\\frac\{\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)\}\{\\sum\_\{v^\{\\prime\}\\in\\Lambda\_\{s\}\}\\hat\{P\}\_\{r\}^\{M,s\}\(v^\{\\prime\}\\mid w\)\}=\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)exactly\. Let us compute the error now: By Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20), we already know

‖xMM,\(w\)−x¯MM,\(w\)‖2≤2​M​\(M−1\)​R​\(s\)​e−β​c\.\\\|x\_\{M\}^\{M,\(w\)\}\-\\bar\{x\}\_\{M\}^\{M,\(w\)\}\\\|\_\{2\}\\leq 2M\(M\-1\)R\(s\)e^\{\-\\beta c\}\.Sincez⁡\(w\)=φ⁡\(xM\(w\)\)z\(w\)=\\varphi\(x\_\{M\}^\{\(w\)\}\)andx¯​\(w\)=φ⁡\(x¯MM,\(w\)\)\\bar\{x\}\(w\)=\\varphi\(\\bar\{x\}\_\{M\}^\{M,\(w\)\}\),

‖z⁡\(w\)−z¯​\(w\)‖∞≤‖z⁡\(w\)−z¯​\(w\)‖2≤Lφ​‖xMM,\(w\)−x¯MM,\(w\)‖2≤8​log⁡1qmin​μmins​R​\(s\)M\+1​\(R​\(s\)M−1\)​M​\(M−1\)​e−β​c\\begin\{split\}\\\|z\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}&\\leq\\\|z\(w\)\-\\bar\{z\}\(w\)\\\|\_\{2\}\\\\ &\\leq L\_\{\\varphi\}\\\|x\_\{M\}^\{M,\(w\)\}\-\\bar\{x\}\_\{M\}^\{M,\(w\)\}\\\|\_\{2\}\\\\ &\\leq 8\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\}R\(s\)^\{M\+1\}\(R\(s\)^\{M\}\-1\)M\(M\-1\)e^\{\-\\beta c\}\\end\{split\}\(31\)by Lemmata[24](https://arxiv.org/html/2610.41038#Thmtheorem24)and[23](https://arxiv.org/html/2610.41038#Thmtheorem23), which are invoked to obtain the fact that theReLUnetwork is Lipschitz and the explicit Lipschitz constantLφL\_\{\\varphi\}\. We chooseβ\\betalarge enough to ensure that

8​log⁡1qmin​μmins​R​\(s\)M\+1​\(R​\(s\)M−1\)​M​\(M−1\)​e−β​c≤ε2\.8\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\}R\(s\)^\{M\+1\}\(R\(s\)^\{M\}\-1\)M\(M\-1\)e^\{\-\\beta c\}\\leq\\frac\{\\varepsilon\}\{2\}\.\(32\)Solving forβ\\betagives exactly the same bound in the hypothesis\.

Recall that𝖳\(⋅∣w\)=softmax\(z\(w\)\)\\mathsf\{T\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(z\(w\)\)andP^rM,s\(⋅∣w\):=softmax\(z¯\(w\)\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\):=\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\. Then

∥P^rM,s\(⋅∣w\)−𝖳\(⋅∣w\)∥TV\\displaystyle\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}=‖softmax⁡\(z¯​\(w\)\)−softmax⁡\(z⁡\(w\)\)‖TV\\displaystyle=\\\|\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\-\\mathrm\{softmax\}\(z\(w\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}=12​‖softmax⁡\(z¯​\(w\)\)−softmax⁡\(z⁡\(w\)\)‖1\\displaystyle=\\frac\{1\}\{2\}\\\|\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\-\\mathrm\{softmax\}\(z\(w\)\)\\\|\_\{1\}=12​∑v∈Λs\|softmax​\(z¯​\(w\)\)v−softmax​\(z⁡\(w\)\)v\|\\displaystyle=\\frac\{1\}\{2\}\\sum\_\{v\\in\\Lambda\_\{s\}\}\|\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}\-\\mathrm\{softmax\}\(z\(w\)\)\_\{v\}\|=12​∑v∈Λssoftmax​\(z¯​\(w\)\)v​\|softmax​\(z⁡\(w\)\)vsoftmax​\(z¯​\(w\)\)v−1\|\\displaystyle=\\frac\{1\}\{2\}\\sum\_\{v\\in\\Lambda\_\{s\}\}\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}\\bigg\|\\frac\{\\mathrm\{softmax\}\(z\(w\)\)\_\{v\}\}\{\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}\}\-1\\bigg\|=12​∑v∈Λssoftmax​\(z¯​\(w\)\)v​\|ez​\(w\)v∑u∈Λsez​\(w\)u⋅∑u∈Λsez¯​\(w\)uez¯​\(w\)v−1\|\\displaystyle=\\frac\{1\}\{2\}\\sum\_\{v\\in\\Lambda\_\{s\}\}\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}\\bigg\|\\frac\{e^\{z\(w\)\_\{v\}\}\}\{\\sum\_\{u\\in\\Lambda\_\{s\}\}e^\{z\(w\)\_\{u\}\}\}\\cdot\\frac\{\\sum\_\{u\\in\\Lambda\_\{s\}\}e^\{\\bar\{z\}\(w\)\_\{u\}\}\}\{e^\{\\bar\{z\}\(w\)\_\{v\}\}\}\-1\\bigg\|=12​∑v∈Λssoftmax​\(z¯​\(w\)\)v​\|ez​\(w\)v−z¯​\(w\)v⏟\(a\)​∑u∈Λsez¯​\(w\)u∑u∈Λsez​\(w\)u⏟\(b\)−1\|\\displaystyle=\\frac\{1\}\{2\}\\sum\_\{v\\in\\Lambda\_\{s\}\}\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}\\bigg\|\\underbrace\{e^\{z\(w\)\_\{v\}\-\\bar\{z\}\(w\)\_\{v\}\}\}\_\{\(a\)\}\\underbrace\{\\frac\{\\sum\_\{u\\in\\Lambda\_\{s\}\}e^\{\\bar\{z\}\(w\)\_\{u\}\}\}\{\\sum\_\{u\\in\\Lambda\_\{s\}\}e^\{z\(w\)\_\{u\}\}\}\}\_\{\(b\)\}\-1\\bigg\|\(⋆\\star\): Writingε′:=8​log⁡1qmin​μmins​R​\(s\)M\+1​\(R​\(s\)M−1\)​M​\(M−1\)​e−β​c\\varepsilon^\{\\prime\}:=8\\log\{\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\}R\(s\)^\{M\+1\}\(R\(s\)^\{M\}\-1\)M\(M\-1\)e^\{\-\\beta c\}from \([31](https://arxiv.org/html/2610.41038#A3.E31)\), it follows that\(a\)∈\[e−ε′,eε′\]\(a\)\\in\[e^\{\-\\varepsilon^\{\\prime\}\},e^\{\\varepsilon^\{\\prime\}\}\], which also givese−ε′​ez¯​\(w\)u≤ez​\(w\)u≤eε′​ez¯​\(w\)ue^\{\-\\varepsilon^\{\\prime\}\}e^\{\\bar\{z\}\(w\)\_\{u\}\}\\leq e^\{z\(w\)\_\{u\}\}\\leq e^\{\\varepsilon^\{\\prime\}\}e^\{\\bar\{z\}\(w\)\_\{u\}\}\. Summing overuuand taking the reciprocals yield the fact hat\(b\)∈\[e−ε′,eε′\]\(b\)\\in\[e^\{\-\\varepsilon^\{\\prime\}\},e^\{\\varepsilon^\{\\prime\}\}\]as well\. Thus,≤max⁡\{e2​ε′−1,1−e2​ε′\}=e2​ε′−1\.\\displaystyle\\leq\\max\\\{e^\{2\\varepsilon^\{\\prime\}\}\-1,1\-e^\{2\\varepsilon^\{\\prime\}\}\\\}=e^\{2\\varepsilon^\{\\prime\}\}\-1\.By the convexity of the exponential functione2​ε′−1≤2​ε′e^\{2\\varepsilon^\{\\prime\}\}\-1\\leq 2\\varepsilon^\{\\prime\}\. Hence,

∥P^rM,s\(⋅∣w\)−𝖳\(⋅∣w\)∥TV≤2ε′\.\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq 2\\varepsilon^\{\\prime\}\.From \([32](https://arxiv.org/html/2610.41038#A3.E32)\) and the definition ofε′\\varepsilon^\{\\prime\}, we have

∥P^rM,s\(⋅∣w\)−𝖳\(⋅∣w\)∥TV≤ε,\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\varepsilon,which concludes the part \(i\) since the bounds are independent of the choice ofw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\.

Proof of \(ii\)\.By \(⋆\\star\),𝖳⁡\(v∣w\)=softmax​\(z⁡\(w\)\)v≥softmax​\(z¯​\(w\)\)v​e−2​ε′\\mathsf\{T\}\(v\\mid w\)=\\mathrm\{softmax\}\(z\(w\)\)\_\{v\}\\geq\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}e^\{\-2\\varepsilon^\{\\prime\}\}for allv∈Λsv\\in\\Lambda\_\{s\}\. Sincesoftmax\(z¯\(w\)\)=P^rM,s\(⋅∣w\)\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)=\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)and from the proof of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24),

𝖳⁡\(v∣w\)≥softmax​\(z¯​\(w\)\)v​e−2​ε′=P^rM,s​\(v∣w\)​e−2​ε′≥qmin​μmins​e−2​ε′≥qmin​μmins​e−2​ε\.\\mathsf\{T\}\(v\\mid w\)\\geq\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\_\{v\}e^\{\-2\\varepsilon^\{\\prime\}\}=\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)e^\{\-2\\varepsilon^\{\\prime\}\}\\geq q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}e^\{\-2\\varepsilon^\{\\prime\}\}\\geq q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}e^\{\-2\\varepsilon\}\.The bound is uniform for allv∈Λsv\\in\\Lambda\_\{s\}, thereby, the claim follows\.■\\blacksquare

## Appendix DProofs of the Results in Section[5](https://arxiv.org/html/2610.41038#S5)

### D\.1Auxiliary Results

In this part, we give all the necessary tools for proving Theorem[27](https://arxiv.org/html/2610.41038#Thmtheorem27)\. Throughout, we fixMMandssin accordance with Corollary[13](https://arxiv.org/html/2610.41038#Thmtheorem13)and the conditions therein, that are; under Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)and𝜶=\(1−δ⁡\(𝐓\)\)​\(2−δ⁡\(𝐐\)\)<1\\bm\{\\alpha\}=\(1\-\\delta\(\\mathbf\{T\}\)\)\(2\-\\delta\(\\mathbf\{Q\}\)\)<1,MMandsshave to satisfy:

- •M≥N⁡⌈log⁡\(2/ε\)/log⁡\(1/𝜶\)⌉M\\geq N\\lceil\\log\(2/\\varepsilon\)/\\log\(1/\\bm\{\\alpha\}\)\\rceil,
- •s≥4​M​Lψ/\(qmin​ε\)s\\geq 4ML\_\{\\psi\}/\(q\_\{\\rm min\}\\varepsilon\)\.

###### Definition 32\.

LetM𝕏,M𝕌∈ℕM\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{U\}\}\\in\\mathbb\{N\}\. We say that the triple\(n,m,l\)∈ℕ3\(n,m,l\)\\in\\mathbb\{N\}^\{3\}are*aligning quantization levels*if the following hold:

- \(i\)2​M𝕏n−1=1K𝕏\\frac\{2M\_\{\\mathbb\{X\}\}\}\{n\-1\}=\\frac\{1\}\{K\_\{\\mathbb\{X\}\}\}for someK𝕏∈ℕK\_\{\\mathbb\{X\}\}\\in\\mathbb\{N\};
- \(ii\)2​M𝕌m−1=1K𝕌\\frac\{2M\_\{\\mathbb\{U\}\}\}\{m\-1\}=\\frac\{1\}\{K\_\{\\mathbb\{U\}\}\}for someK𝕌∈ℕK\_\{\\mathbb\{U\}\}\\in\\mathbb\{N\}andc=M𝕌c=M\_\{\\mathbb\{U\}\}, whereccis \([20](https://arxiv.org/html/2610.41038#S4.E20)\);
- \(iii\)MMdividesl−1l\-1\.

Due to Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32), we have that𝕏n⊃\(\[−M𝕏,M𝕏\]∩ℤ\)d\\mathbb\{X\}\_\{n\}\\supset\(\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]\\cap\\mathbb\{Z\}\)^\{d\}and the grid of𝕌m\\mathbb\{U\}\_\{m\}contains\[−M𝕌,M𝕌\]∩ℤ\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\]\\cap\\mathbb\{Z\}in each entry\. Indeed, sinceM𝕏∈ℕM\_\{\\mathbb\{X\}\}\\in\\mathbb\{N\}, every integerp∈\[−M𝕏,M𝕏\]p\\in\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]is the grid point−M𝕏\+2​M𝕏​k^n−1\-M\_\{\\mathbb\{X\}\}\+\\frac\{2M\_\{\\mathbb\{X\}\}\\hat\{k\}\}\{n\-1\}withk^=\(p\+M𝕏\)​K𝕏∈\{0,…,n−1\}\\hat\{k\}=\(p\+M\_\{\\mathbb\{X\}\}\)K\_\{\\mathbb\{X\}\}\\in\\\{0,\\ldots,n\-1\\\}, and the same argument applies to𝕌m\\mathbb\{U\}\_\{m\}\. In other words, the state and action quantization preserve integer values within the diameter of the ambient spaces\. This is an important feature since the weights in the gatherer layers in §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)are\{−1,0,c,1\}\\\{\-1,0,c,1\\\}\-valued\. If\(n,m,l\)\(n,m,l\)are aligning quantization levels, then the weights of §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)live in𝕌m\\mathbb\{U\}\_\{m\}\. On the other hand, the condition \(iii\) results in no error due to quantization of measure\-states sinceM\|l−1M\\mid l\-1ensures that any mass, which is an integer multiple of1M\\frac\{1\}\{M\}, is a grid value of𝒫l​\(𝕏n\)\\mathcal\{P\}\_\{l\}\(\\mathbb\{X\}\_\{n\}\), and hence remains invariant under the measure quantization\.

###### Lemma 33\.

Assume that\(n,m,l\)∈ℕ3\(n,m,l\)\\in\\mathbb\{N\}^\{3\}are chosen so that they are aligning quantization levels and thatM𝕏≥R⁡\(s\)M\_\{\\mathbb\{X\}\}\\geq R\(s\)\. Then, along the trajectory of the constructed Transformer, for anyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, the following assertions are true:

- \(a\)The weights defined in §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2)live in𝕌m\\mathbb\{U\}\_\{m\};
- \(b\)Πpos​Qn​\(x\)=Πpos​x\\Pi\_\{\\rm pos\}Q\_\{n\}\(x\)=\\Pi\_\{\\rm pos\}xandΠtok​Qn​\(x\)=Πtok​x\\Pi\_\{\\rm tok\}Q\_\{n\}\(x\)=\\Pi\_\{\\rm tok\}xfor every tokenxxalong the trajectory;
- \(c\)the attention scores are exactlyc​χselt​\(i\)​\(j\)c\\chi\_\{\\mathrm\{sel\}\_\{t\}\(i\)\}\(j\), and the measure states passing through the Transformer layers have atoms carrying mass exactly1/M1/M;
- \(d\)the only quantization error is due toQnQ\_\{n\}being applied on the gather coordinates, which is of size at mostM𝕏​d/\(n−1\)M\_\{\\mathbb\{X\}\}\\sqrt\{d\}/\(n\-1\)per layer\.

###### Proof\.

We prove step\-by\-step:

Proof of \(a\)\.Recall thatWW=\[Id−Id\]W\_\{W\}=\[I\_\{d\}\\;\-I\_\{d\}\]andWA=WWTW\_\{A\}=W\_\{W\}^\{T\}, therefore, all components ofWWW\_\{W\}andWAW\_\{A\}are in\{−1,0,1\}\\\{\-1,0,1\\\}\. In addition,WbW\_\{b\}is the zero vector\.WK,t=ΠposW\_\{K,t\}=\\Pi\_\{\\rm pos\}is a coordinate projection, hence, all components are0,10,1as well\. The value matrixWV,t=Gρ⁡\(t\)=\[0M​0​eρ⁡\(t\)​0R⁡\(s\)\]T​ΠtokW\_\{V,t\}=G\_\{\\rho\(t\)\}=\[0\_\{M\}\\;0\\;e\_\{\\rho\(t\)\}\\;0\_\{R\(s\)\}\]^\{T\}\\Pi\_\{\\rm tok\}has components with values in\{0,1\}\\\{0,1\\\}too\. Finally,WQ,tW\_\{Q,t\}isc​At​ΠposcA\_\{t\}\\Pi\_\{\\rm pos\}whereAtA\_\{t\}is a0,10,1matrix\. Because of the factorcc,WQ,tW\_\{Q,t\}is component\-wise\{0,c\}\\\{0,c\\\}\-valued\. By Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\-\(ii\),c=M𝕌c=M\_\{\\mathbb\{U\}\}, so−1,0,1,c\-1,0,1,care integers in\[−M𝕌,M𝕌\]\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\], hence grid points of𝕌m\\mathbb\{U\}\_\{m\}\. Therefore,\(WW,t,WA,t,Wb,t,WQ,t,WK,t,WV,t\)∈𝕌m\(W\_\{W,t\},W\_\{A,t\},W\_\{b,t\},W\_\{Q,t\},W\_\{K,t\},W\_\{V,t\}\)\\in\\mathbb\{U\}\_\{m\}\.

Proof of \(b\)\.From Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32),𝕏n\\mathbb\{X\}\_\{n\}contains all integers in\[−M𝕏,M𝕏\]\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]in each coordinate\. SinceM𝕏≥R⁡\(s\)M\_\{\\mathbb\{X\}\}\\geq R\(s\)andι⁡\(wi\)∈\{1,…,R⁡\(s\)\}\\iota\(w\_\{i\}\)\\in\\\{1,\\ldots,R\(s\)\\\}for allwi∈Λsw\_\{i\}\\in\\Lambda\_\{s\}, the token values are such integers, and so are the positional coordinates, which lie in\{0,1\}\\\{0,1\\\}\. These coordinates are not changed by the attention blocks \(Lemma[17](https://arxiv.org/html/2610.41038#Thmtheorem17), whose argument applies verbatim to the quantized trajectory\)\. Since𝕏n\\mathbb\{X\}\_\{n\}is a product grid,QnQ\_\{n\}acts coordinate\-wise\. Therefore the state quantizerQnQ\_\{n\}must fix positional and token value components of each token\.

Proof of \(c\)\.By \(b\), the attention scores are computed from the exact positional blocks with exactWQ,t,WK,tW\_\{Q,t\},W\_\{K,t\}by \(a\)\. Therefore, Lemma[18](https://arxiv.org/html/2610.41038#Thmtheorem18)applies directly\. For the weights, each measure is supported onMMatoms of mass1/M1/M, since the positional encodings of the tokens are distinct, and1/M=\(l−1\)/Ml−11/M=\\frac\{\(l\-1\)/M\}\{l\-1\}is a grid value of𝒫l​\(𝕏n\)\\mathcal\{P\}\_\{l\}\(\\mathbb\{X\}\_\{n\}\)by Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\-\(iii\)\. ThereforeRlR\_\{l\}fixes the measure states\.

Proof of \(d\)\.Each gather coordinate is of the form∑j=1Ma^ti​j​ι​\(wj\)\\sum\_\{j=1\}^\{M\}\\hat\{a\}\_\{t\}^\{ij\}\\iota\(w\_\{j\}\), which is not an integer in general\. Therefore,QnQ\_\{n\}does not fix it, but moves it by at most the grid spacings, which isM𝕏​d/\(n−1\)M\_\{\\mathbb\{X\}\}\\sqrt\{d\}/\(n\-1\)\. ∎

###### Lemma 34\.

Assume that\(n,m,l\)∈ℕ3\(n,m,l\)\\in\\mathbb\{N\}^\{3\}are chosen so that they are aligning quantization levels and thatM𝕏,M𝕌M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{U\}\}satisfy \([23](https://arxiv.org/html/2610.41038#S5.E23)\)\. Then for everyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\},

‖z^​\(w\)−z¯​\(w\)‖∞≤M𝕏​d​R​\(s\)3​M−12​M𝕌​dsm−1\+Lφ​\(M​M𝕏​dn−1\+2​M​\(M−1\)​R​\(s\)​e−β​c\),\\\|\\hat\{z\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}\+L\_\{\\varphi\}\\bigg\(M\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}\+2M\(M\-1\)R\(s\)e^\{\-\\beta c\}\\bigg\),wherez¯\\bar\{z\}is as in Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)andz^\\hat\{z\}is the logits produced by the quantized weights before the last application ofQnQ\_\{n\}\.

###### Proof\.

Letx¯ti\\bar\{x\}\_\{t\}^\{i\}be the unquantized trajectory given by Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20),x^ti\\hat\{x\}\_\{t\}^\{i\}is given by the quantized weights, and letxtix\_\{t\}^\{i\}be the usual trajectory \(neither a hard choice as in Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20)nor under quantization likex^ti\\hat\{x\}\_\{t\}^\{i\}\)\.

There are two immediate observations: \(i\) The attention scores are computed from the positional encodings\. From Lemma[33](https://arxiv.org/html/2610.41038#Thmtheorem33)\-\(b,c\) and Lemma[18](https://arxiv.org/html/2610.41038#Thmtheorem18), we know that both the quantized and the unquantized trajectories are computed under the same attention scores\. We writeati​ja\_\{t\}^\{ij\}for the common attention scores\. \(ii\) From Lemma[33](https://arxiv.org/html/2610.41038#Thmtheorem33)\-\(b\), the token coordinates are invariant under the state quantizerQnQ\_\{n\}\. ThereforeWV,t​x¯ti=WV,t​x^tiW\_\{V,t\}\\bar\{x\}\_\{t\}^\{i\}=W\_\{V,t\}\\hat\{x\}\_\{t\}^\{i\}for allt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}andi∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}\.

For convenience, we writeAi=∑j=1Mati​j​WV,t​x¯tjA\_\{i\}=\\sum\_\{j=1\}^\{M\}a\_\{t\}^\{ij\}W\_\{V,t\}\\bar\{x\}\_\{t\}^\{j\}for theiith token’s attention part, which is same for bothx¯ti\\bar\{x\}\_\{t\}^\{i\}andx^ti\\hat\{x\}\_\{t\}^\{i\}by \(i\) and \(ii\)\. Then

xt\+1i−x^t\+1i=\(xti\+Ai\)−Qn​\(x^ti\+Ai\)=xti\+Ai−Qn​\(x^ti\+Ai\)\+x^ti−x^ti=\(xti−x^ti\)\+\(\(x^ti\+Ai\)−Qn​\(x^ti\+Ai\)\)\.\\begin\{split\}x\_\{t\+1\}^\{i\}\-\\hat\{x\}\_\{t\+1\}^\{i\}&=\(x\_\{t\}^\{i\}\+A\_\{i\}\)\-Q\_\{n\}\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\\\\ &=x\_\{t\}^\{i\}\+A\_\{i\}\-Q\_\{n\}\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\+\\hat\{x\}\_\{t\}^\{i\}\-\\hat\{x\}\_\{t\}^\{i\}\\\\ &=\(x\_\{t\}^\{i\}\-\\hat\{x\}\_\{t\}^\{i\}\)\+\\big\(\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\-Q\_\{n\}\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\\big\)\.\\end\{split\}Passing to the norm, we get

‖xt\+1i−x^t\+1i‖≤‖xti−x^ti‖\+‖\(x^ti\+Ai\)−Qn​\(x^ti\+Ai\)‖⏟≤M𝕏​dn−1,\\begin\{split\}\\\|x\_\{t\+1\}^\{i\}\-\\hat\{x\}\_\{t\+1\}^\{i\}\\\|&\\leq\\\|x\_\{t\}^\{i\}\-\\hat\{x\}\_\{t\}^\{i\}\\\|\+\\underbrace\{\\\|\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\-Q\_\{n\}\(\\hat\{x\}\_\{t\}^\{i\}\+A\_\{i\}\)\\\|\}\_\{\\leq\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}\},\\end\{split\}\(33\)for allt∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\}and alli∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}\. Initial tokens given by\[ei​ι​\(wi\)​0M​0R⁡\(s\)\]\[e\_\{i\}\\;\\iota\(w\_\{i\}\)\\;0\_\{M\}\\;0\_\{R\(s\)\}\]have integer coordinates, hence, invariant underQnQ\_\{n\}\. So‖x0i−x^0i‖=0\\\|x\_\{0\}^\{i\}\-\\hat\{x\}\_\{0\}^\{i\}\\\|=0for alli∈\{1,…,M\}i\\in\\\{1,\\ldots,M\\\}\. Consequently, the error in the last gatherer layer’s readout position is upper bounded as

‖x^MM−x¯MM‖≤‖x^MM−xMM‖\+‖xMM−x¯MM‖≤M​M𝕏​dn−1\+2​M​\(M−1\)​R​\(s\)​e−β​c,\\\|\\hat\{x\}\_\{M\}^\{M\}\-\\bar\{x\}\_\{M\}^\{M\}\\\|\\leq\\\|\\hat\{x\}\_\{M\}^\{M\}\-x\_\{M\}^\{M\}\\\|\+\\\|x\_\{M\}^\{M\}\-\\bar\{x\}\_\{M\}^\{M\}\\\|\\leq M\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}\+2M\(M\-1\)R\(s\)e^\{\-\\beta c\},\(34\)where the first summand in the last bound is due to \([33](https://arxiv.org/html/2610.41038#A4.E33)\) accumulated overMMlayers and the the second summand is due to Lemma[20](https://arxiv.org/html/2610.41038#Thmtheorem20)\.

Now we arrive at the interpolator layer, in which we haveWV,MW\_\{V,M\}is the zero matrix, hence the attention part does not contribute the final token\. The logit part of the final layer is then given in terms of a one\-layer feed\-forward network of the formφ⁡\(ζ\)=WW,M​σ​\(WA,M​ζ\+Wb,M\)\\varphi\(\\zeta\)=W\_\{W,M\}\\sigma\(W\_\{A,M\}\\zeta\+W\_\{b,M\}\)\. Letφ^\\hat\{\\varphi\}be the corresponding feed\-forward with quantized weights, i\.e\.φ^​\(ζ\)=W^W,M​σ​\(WA,M​ζ\+Wb,M\)\\hat\{\\varphi\}\(\\zeta\)=\\hat\{W\}\_\{W,M\}\\sigma\(W\_\{A,M\}\\zeta\+W\_\{b,M\}\)\. We deliberately writeWA,MW\_\{A,M\}andWb,MW\_\{b,M\}without hats because of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24), which implies that their entries are integers in\[−M𝕌,M𝕌\]\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\]by \([23](https://arxiv.org/html/2610.41038#S5.E23)\); hence, they are not affected by the quantization due to Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\-\(ii\) \(see the proof of Lemma[35](https://arxiv.org/html/2610.41038#Thmtheorem35)\)\. So, for anyζ∈𝕏\\zeta\\in\\mathbb\{X\}

φ^​\(ζ\)−φ⁡\(ζ\)=\(WW,M−W^W,M\)​σ​\(WA,M​ζ\+Wb,M\)\.\\hat\{\\varphi\}\(\\zeta\)\-\\varphi\(\\zeta\)=\(W\_\{W,M\}\-\\hat\{W\}\_\{W,M\}\)\\sigma\(W\_\{A,M\}\\zeta\+W\_\{b,M\}\)\.In norm, we have

‖φ^​\(ζ\)−φ⁡\(ζ\)‖≤‖WW,M−W^W,M‖F​\(∑k=1R​\(s\)Mσ​\(⟨a,ζ⟩−tk\)2\)12\.\\\|\\hat\{\\varphi\}\(\\zeta\)\-\\varphi\(\\zeta\)\\\|\\leq\\\|W\_\{W,M\}\-\\hat\{W\}\_\{W,M\}\\\|\_\{F\}\\bigg\(\\sum\_\{k=1\}^\{R\(s\)^\{M\}\}\\sigma\(\\langle a,\\zeta\\rangle\-t\_\{k\}\)^\{2\}\\bigg\)^\{\\frac\{1\}\{2\}\}\.‖WW,M−W^W,M‖F≤M𝕌​dsm−1\\\|W\_\{W,M\}\-\\hat\{W\}\_\{W,M\}\\\|\_\{F\}\\leq\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}due to action quantization\. Indeed, the only entries ofWW,MW\_\{W,M\}that are not integers are those of its logit block, which hasR⁡\(s\)R\(s\)rows andR​\(s\)MR\(s\)^\{M\}columns, so that there are at mostds2=R​\(s\)M\+1d\_\{s\}^\{2\}=R\(s\)^\{M\+1\}of them\. By the proof of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\-\(ii\), they lie in\[−2​log⁡1qmin​μmins,2​log⁡1qmin​μmins\]⊂\[−M𝕌,M𝕌\]\[\-2\\log\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\},2\\log\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\]\\subset\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\], and each of them is moved by at most half of the grid spacing2​M𝕌m−1\\frac\{2M\_\{\\mathbb\{U\}\}\}\{m\-1\}\. On the other hand,

\(∑k=1R​\(s\)Mσ​\(⟨a,ζ⟩−tk\)2\)12≤R​\(s\)M​‖a‖2​M𝕏​d\.\\bigg\(\\sum\_\{k=1\}^\{R\(s\)^\{M\}\}\\sigma\(\\langle a,\\zeta\\rangle\-t\_\{k\}\)^\{2\}\\bigg\)^\{\\frac\{1\}\{2\}\}\\leq\\sqrt\{R\(s\)^\{M\}\}\\\|a\\\|\_\{2\}M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\.By Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24),a=\(1,R⁡\(s\),…,R​\(s\)M−1\)a=\(1,R\(s\),\\ldots,R\(s\)^\{M\-1\}\), hence‖a‖2≤R​\(s\)M−1/2\\\|a\\\|\_\{2\}\\leq R\(s\)^\{M\-1/2\}, which implies

‖φ−φ^‖∞≤M𝕏​d​R​\(s\)3​M−12​M𝕌​dsm−1\.\\\|\\varphi\-\\hat\{\\varphi\}\\\|\_\{\\infty\}\\leq M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}\.\(35\)By definition,z^​\(w\)=φ^​\(x^MM\)\\hat\{z\}\(w\)=\\hat\{\\varphi\}\(\\hat\{x\}\_\{M\}^\{M\}\)andz¯​\(w\)=φ⁡\(x¯MM\)\\bar\{z\}\(w\)=\\varphi\(\\bar\{x\}\_\{M\}^\{M\}\)\. With an “add and subtract” trick:

‖z^​\(w\)−z¯​\(w\)‖∞≤‖z^​\(w\)−φ⁡\(x^MM\)‖∞\+‖φ⁡\(x^MM\)−z¯​\(w\)‖∞\.\\\|\\hat\{z\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\\|\\hat\{z\}\(w\)\-\\varphi\(\\hat\{x\}\_\{M\}^\{M\}\)\\\|\_\{\\infty\}\+\\\|\\varphi\(\\hat\{x\}\_\{M\}^\{M\}\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\.For the first term, we have

‖z^​\(w\)−φ⁡\(x^MM\)‖∞≤M𝕏​d​R​\(s\)3​M−12​M𝕌​dsm−1\\\|\\hat\{z\}\(w\)\-\\varphi\(\\hat\{x\}\_\{M\}^\{M\}\)\\\|\_\{\\infty\}\\leq M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}from \([35](https://arxiv.org/html/2610.41038#A4.E35)\), and the second term is bounded as

‖φ⁡\(x^MM\)−z¯​\(w\)‖∞≤Lφ​\(M​M𝕏​dn−1\+2​M​\(M−1\)​R​\(s\)​e−β​c\)\\\|\\varphi\(\\hat\{x\}\_\{M\}^\{M\}\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq L\_\{\\varphi\}\\bigg\(M\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}\+2M\(M\-1\)R\(s\)e^\{\-\\beta c\}\\bigg\)by Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)and \([34](https://arxiv.org/html/2610.41038#A4.E34)\)\. Therefore, we get

‖z^​\(w\)−z¯​\(w\)‖∞≤M𝕏​d​R​\(s\)3​M−12​M𝕌​dsm−1\+Lφ​\(M​M𝕏​dn−1\+2​M​\(M−1\)​R​\(s\)​e−β​c\)\.∎\\\|\\hat\{z\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{m\-1\}\+L\_\{\\varphi\}\\bigg\(M\\frac\{M\_\{\\mathbb\{X\}\}\\sqrt\{d\}\}\{n\-1\}\+2M\(M\-1\)R\(s\)e^\{\-\\beta c\}\\bigg\)\.\\qed

###### Lemma 35\.

Assume that\(n,m,l\)∈ℕ3\(n,m,l\)\\in\\mathbb\{N\}^\{3\}are chosen so that they are aligning quantization levels, thatM𝕏,M𝕌M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{U\}\}satisfy \([23](https://arxiv.org/html/2610.41038#S5.E23)\), thatR⁡\(s\)≥2R\(s\)\\geq 2, and let Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1, A3\) hold\. Letβ\>0\\beta\>0be such thatη⁡\(n,m,β\)≤1\\eta\(n,m,\\beta\)\\leq 1, whereη⁡\(n,m,β\)\\eta\(n,m,\\beta\)is as in \([24](https://arxiv.org/html/2610.41038#S5.E24)\)\. Then there exists an open\-loop policyU¯^∈𝕌mM\+1\\underline\{\\hat\{U\}\}\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}such that, for everyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\},𝖳U¯^\(⋅∣w\)=softmax\(z^out\(w\)\)\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(\\hat\{z\}\_\{\\rm out\}\(w\)\)for a logit vectorz^out​\(w\)∈ℝR⁡\(s\)\\hat\{z\}\_\{\\rm out\}\(w\)\\in\\mathbb\{R\}^\{R\(s\)\}with

‖z^out​\(w\)−z¯​\(w\)‖∞≤η⁡\(n,m,β\),\\\|\\hat\{z\}\_\{\\rm out\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\eta\(n,m,\\beta\),\(36\)wherez¯\(w\)=logP^rM,s\(⋅∣w\)\\bar\{z\}\(w\)=\\log\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)is as in Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)\. Consequently,

maxw∈ΛsM∥P^rM,s\(⋅∣w\)−𝖳U¯^\(⋅∣w\)∥TV≤η⁡\(n,m,β\)2\.\\max\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{\\eta\(n,m,\\beta\)\}\{2\}\.In particular, ifβ\\betais as in Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)for someε0∈\(0,1/2\]\\varepsilon\_\{0\}\\in\(0,1/2\], then

maxw∈ΛsM∥P^rM,s\(⋅∣w\)−𝖳U¯^\(⋅∣w\)∥TV≤ε04\+\(1\+Lφ​M​d\)​M𝕏2​\(n−1\)\+M𝕏​d​R​\(s\)3​M−12​M𝕌​ds2​\(m−1\)\.\\begin\{split\}\\max\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&\\leq\\frac\{\\varepsilon\_\{0\}\}\{4\}\+\\big\(1\+L\_\{\\varphi\}M\\sqrt\{d\}\\big\)\\frac\{M\_\{\\mathbb\{X\}\}\}\{2\(n\-1\)\}\\\\ &\\qquad\\qquad\+M\_\{\\mathbb\{X\}\}\\sqrt\{d\}R\(s\)^\{\\frac\{3M\-1\}\{2\}\}\\frac\{M\_\{\\mathbb\{U\}\}d\_\{s\}\}\{2\(m\-1\)\}\.\\end\{split\}

###### Proof\.

We prove step\-by\-step:

Step 1\.LetU¯^=\(U^0,…,U^M\)\\underline\{\\hat\{U\}\}=\(\\hat\{U\}\_\{0\},\\ldots,\\hat\{U\}\_\{M\}\)be the weights of §[4](https://arxiv.org/html/2610.41038#S4)in which only the output matrix of the interpolator layer is moved onto the grid\. That is, fort∈\{0,…,M−1\}t\\in\\\{0,\\ldots,M\-1\\\},U^t\\hat\{U\}\_\{t\}consists of the gatherer weights of §[4\.1\.2](https://arxiv.org/html/2610.41038#S4.SS1.SSS2), andU^0\\hat\{U\}\_\{0\}also holds the embeddingEEE\_\{E\}in \([18](https://arxiv.org/html/2610.41038#S4.E18)\)\. The last layerU^M\\hat\{U\}\_\{M\}consists ofWV,M=0W\_\{V,M\}=0, so that its attention block is switched off, of the matricesWA,MW\_\{A,M\}andWb,MW\_\{b,M\}of theReLUnetworkφ\\varphiof Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)with the choice ofaain Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24), ofW^W,M\\hat\{W\}\_\{W,M\}, the entry\-wise closest grid point toWW,MW\_\{W,M\}, and of the unembeddingEU=ΠlogE\_\{U\}=\\Pi\_\{\\rm log\}\. Since the feed\-forward block has no output bias, the constant termv1v\_\{1\}of the interpolantggin the proof of Lemma[23](https://arxiv.org/html/2610.41038#Thmtheorem23)is produced by one additional hidden unit with zero input weights and bias11, whose output column isv1v\_\{1\}\. In addition, the interpolator layer carries the positional block through:WA,MW\_\{A,M\}has the additional rows\(eiT,0,0MT,0RT\)\(e\_\{i\}^\{T\},0,0\_\{M\}^\{T\},0\_\{R\}^\{T\}\)andWW,MW\_\{W,M\}the corresponding columns\(ei,0,0M,0R\)\(e\_\{i\},0,0\_\{M\},0\_\{R\}\),i=1,…,Mi=1,\\ldots,M\. Since the positional coordinates are nonnegative,σ\\sigmaacts on them as the identity\. These additional weights do not interact with the logit block, and they keep Assumption[15](https://arxiv.org/html/2610.41038#Thmtheorem15)in force after the last layer, so that the readout token is identified by its position\.

Step 2\.We show thatU¯^∈𝕌mM\+1\\underline\{\\hat\{U\}\}\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}\. The gatherer weights live in𝕌m\\mathbb\{U\}\_\{m\}by Lemma[33](https://arxiv.org/html/2610.41038#Thmtheorem33)\-\(a\), and the entries ofEEE\_\{E\}, ofEUE\_\{U\}and of the positional entries are in\{0,1,…,R⁡\(s\)\}\\\{0,1,\\ldots,R\(s\)\\\}\. The entries ofWA,MW\_\{A,M\}are00,11orR​\(s\)j−1≤R​\(s\)M−1R\(s\)^\{j\-1\}\\leq R\(s\)^\{M\-1\},j=1,…,Mj=1,\\ldots,M, the latter coming fromaain Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\. The entries ofWb,MW\_\{b,M\}are11\(for the constant unit\) and−tk\-t\_\{k\}, where, by Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\-\(i\) andR⁡\(s\)≥2R\(s\)\\geq 2,

0≤tk≤∑j=1MR⁡\(s\)​R​\(s\)j−1=R⁡\(s\)​R​\(s\)M−1R⁡\(s\)−1≤2​R​\(s\)M\.0\\leq t\_\{k\}\\leq\\sum\_\{j=1\}^\{M\}R\(s\)\\,R\(s\)^\{j\-1\}=R\(s\)\\frac\{R\(s\)^\{M\}\-1\}\{R\(s\)\-1\}\\leq 2R\(s\)^\{M\}\.Therefore, all these entries are integers in\[−M𝕌,M𝕌\]\[\-M\_\{\\mathbb\{U\}\},M\_\{\\mathbb\{U\}\}\]by \([23](https://arxiv.org/html/2610.41038#S5.E23)\), hence grid points of𝕌m\\mathbb\{U\}\_\{m\}by Definition[32](https://arxiv.org/html/2610.41038#Thmtheorem32)\-\(ii\)\. Finally,W^W,M\\hat\{W\}\_\{W,M\}is a grid point by construction\. So,U¯^∈𝕌mM\+1\\underline\{\\hat\{U\}\}\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}\.

Step 3\.Fixw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\. The positional carry uses exact integer weights, so Lemma[34](https://arxiv.org/html/2610.41038#Thmtheorem34)applies without change to the logit vectorz^​\(w\)=φ^​\(x^MM\)\\hat\{z\}\(w\)=\\hat\{\\varphi\}\(\\hat\{x\}\_\{M\}^\{M\}\)produced by the interpolator layer before the last application ofQnQ\_\{n\}\. By the definition \([24](https://arxiv.org/html/2610.41038#S5.E24)\), we have

‖z^​\(w\)−z¯​\(w\)‖∞≤η⁡\(n,m,β\)−M𝕏n−1\.\\\|\\hat\{z\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\eta\(n,m,\\beta\)\-\\frac\{M\_\{\\mathbb\{X\}\}\}\{n\-1\}\.Sincez¯​\(w\)∈\[−log⁡1qmin​μmins,0\]R⁡\(s\)\\bar\{z\}\(w\)\\in\[\-\\log\\frac\{1\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\},0\]^\{R\(s\)\}by the proof of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\-\(ii\) andη⁡\(n,m,β\)≤1\\eta\(n,m,\\beta\)\\leq 1, the logit coordinates of the readout token lie in\[−M𝕏,M𝕏\]\[\-M\_\{\\mathbb\{X\}\},M\_\{\\mathbb\{X\}\}\]by \([23](https://arxiv.org/html/2610.41038#S5.E23)\)\. Hence, the last application ofQnQ\_\{n\}moves each of them by at most half of the grid spacing, that is, by at mostM𝕏/\(n−1\)M\_\{\\mathbb\{X\}\}/\(n\-1\), and it fixes the positional block by Lemma[33](https://arxiv.org/html/2610.41038#Thmtheorem33)\-\(b\)\. The atoms of the last measure state still carry mass exactly1/M1/Msince their positional encodings are distinct, soRlR\_\{l\}does not move them either by Lemma[33](https://arxiv.org/html/2610.41038#Thmtheorem33)\-\(c\)\. Consequently,𝖳U¯^\(⋅∣w\)=softmax\(z^out\(w\)\)\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(\\hat\{z\}\_\{\\rm out\}\(w\)\)withz^out​\(w\):=Πlog​Qn​\(x^M\+1M\)\\hat\{z\}\_\{\\rm out\}\(w\):=\\Pi\_\{\\rm log\}Q\_\{n\}\(\\hat\{x\}\_\{M\+1\}^\{M\}\), and

‖z^out​\(w\)−z¯​\(w\)‖∞≤‖z^out​\(w\)−z^​\(w\)‖∞\+‖z^​\(w\)−z¯​\(w\)‖∞≤M𝕏n−1\+η⁡\(n,m,β\)−M𝕏n−1=η⁡\(n,m,β\),\\\|\\hat\{z\}\_\{\\rm out\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\\|\\hat\{z\}\_\{\\rm out\}\(w\)\-\\hat\{z\}\(w\)\\\|\_\{\\infty\}\+\\\|\\hat\{z\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\frac\{M\_\{\\mathbb\{X\}\}\}\{n\-1\}\+\\eta\(n,m,\\beta\)\-\\frac\{M\_\{\\mathbb\{X\}\}\}\{n\-1\}=\\eta\(n,m,\\beta\),which is \([36](https://arxiv.org/html/2610.41038#A4.E36)\)\.

Step 4\.SinceP^rM,s\(⋅∣w\)=softmax\(z¯\(w\)\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\(see the proof of Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)\), Lemma[37](https://arxiv.org/html/2610.41038#Thmtheorem37)\-\(i\) below and \([36](https://arxiv.org/html/2610.41038#A4.E36)\) give

∥P^rM,s\(⋅∣w\)−𝖳U¯^\(⋅∣w\)∥TV≤12∥z¯\(w\)−z^out\(w\)∥∞≤η⁡\(n,m,β\)2\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{1\}\{2\}\\\|\\bar\{z\}\(w\)\-\\hat\{z\}\_\{\\rm out\}\(w\)\\\|\_\{\\infty\}\\leq\\frac\{\\eta\(n,m,\\beta\)\}\{2\}for everyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\. Finally, ifβ\\betais as in Theorem[25](https://arxiv.org/html/2610.41038#Thmtheorem25)withε=ε0\\varepsilon=\\varepsilon\_\{0\}, thenβ​c≥log⁡\(4​M​\(M−1\)​R​\(s\)​Lφ/ε0\)\\beta c\\geq\\log\(4M\(M\-1\)R\(s\)L\_\{\\varphi\}/\\varepsilon\_\{0\}\), that is,2​Lφ​M​\(M−1\)​R​\(s\)​e−β​c≤ε0/22L\_\{\\varphi\}M\(M\-1\)R\(s\)e^\{\-\\beta c\}\\leq\\varepsilon\_\{0\}/2\. Plugging this into \([24](https://arxiv.org/html/2610.41038#S5.E24)\) gives the last claim\. ∎

###### Lemma 36\.

Suppose Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1, A3\) hold and that the data set is sampled as in the beginning of this section\. Then the following assertions are true:

- \(i\)pr​\(w\)≥\(qmin​μmins\)Mp\_\{r\}\(w\)\\geq\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{M\}andminv∈Λs⁡P^rM,s​\(v∣w\)≥qmin​μmins\\min\_\{v\\in\\Lambda\_\{s\}\}\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)\\geq q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}for allw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\.
- \(ii\)Letδ∈\(0,1\)\\delta\\in\(0,1\)andK~≥8​\(qmin​μmins\)−M​\(M​log⁡R⁡\(s\)\+log⁡3δ\)\\tilde\{K\}\\geq 8\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{\-M\}\(M\\log\{R\(s\)\}\+\\log\{\\frac\{3\}\{\\delta\}\}\)\. Then, with probability at least1−δ1\-\\delta, the following two events occur simultaneously: 𝒢1:=\{nw≥12K~pr\(w\)for allw∈ΛsM\},\\mathcal\{G\}\_\{1\}:=\\Big\\\{n\_\{w\}\\geq\\frac\{1\}\{2\}\\tilde\{K\}p\_\{r\}\(w\)\\ \\text\{ for all \}w\\in\\Lambda\_\{s\}^\{M\}\\Big\\\},𝒢2:=\{∥yw−P^rM,s\(⋅∣w\)∥TV≤Cδ2​nwfor allw∈ΛsMwithnw≥1\},\\mathcal\{G\}\_\{2\}:=\\bigg\\\{\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sqrt\{\\frac\{C\_\{\\delta\}\}\{2n\_\{w\}\}\}\\ \\text\{ for all \}w\\in\\Lambda\_\{s\}^\{M\}\\text\{ with \}n\_\{w\}\\geq 1\\bigg\\\},whereCδ:=R⁡\(s\)​log⁡2\+M​log⁡R⁡\(s\)\+log⁡\(3/δ\)C\_\{\\delta\}:=R\(s\)\\log 2\+M\\log R\(s\)\+\\log\(3/\\delta\)\. On𝒢1\\mathcal\{G\}\_\{1\}, every windoww∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}appears in the data set; hence,K=R​\(s\)MK=R\(s\)^\{M\}and\{𝒙k:k∈𝒦\}=ΛsM\\\{\\bm\{x\}^\{k\}:k\\in\\mathcal\{K\}\\\}=\\Lambda\_\{s\}^\{M\}\.
- \(iii\)On𝒢1∩𝒢2\\mathcal\{G\}\_\{1\}\\cap\\mathcal\{G\}\_\{2\}, we have ∑w∈ΛsMpr\(w\)∥yw−P^rM,s\(⋅∣w\)∥TV≤R​\(s\)M​CδK~\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sqrt\{\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{\\tilde\{K\}\}\}and ∑w∈ΛsMnwK~∥yw−P^rM,s\(⋅∣w\)∥TV2≤R​\(s\)M​Cδ2​K~\.\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}^\{2\}\\leq\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{2\\tilde\{K\}\}\.

###### Proof\.

We prove step\-by\-step:

Proof of \(i\)\.Conditionally onZ0:r\+1Z\_\{0:r\+1\}, the observationsY0,…,Yr\+1Y\_\{0\},\\ldots,Y\_\{r\+1\}are independent withYt∼Q\(⋅∣Zt\)Y\_\{t\}\\sim Q\(\\,\\cdot\\mid Z\_\{t\}\); see the representationYt=H⁡\(Zt,Vt\)Y\_\{t\}=H\(Z\_\{t\},V\_\{t\}\)with i\.i\.d\. noise independent of\{Zr\}r∈ℤ\+\\\{Z\_\{r\}\\\}\_\{r\\in\\mathbb\{Z\}\_\{\+\}\}in the proof of Lemma[30](https://arxiv.org/html/2610.41038#Thmtheorem30)\. For everyv∈Λsv\\in\\Lambda\_\{s\}, the preimage\(qs\)−1​\(v\)\(q^\{s\}\)^\{\-1\}\(v\)is one of the cellsC⁡\(s,i\)C\(s,i\), so that, for allz∈Sz\\in S,

Q⁡\(\(qs\)−1​\(v\)∣z\)=∫\(qs\)−1​\(v\)μ⁡\(𝑑y\)​ψ​\(y∣z\)≥qmin​μ​\(\(qs\)−1​\(v\)\)≥qmin​μminsQ\\big\(\(q^\{s\}\)^\{\-1\}\(v\)\\mid z\\big\)=\\int\_\{\(q^\{s\}\)^\{\-1\}\(v\)\}\\mu\(\\mathrm\{d\}y\)\\,\\psi\(y\\mid z\)\\geq q\_\{\\rm min\}\\,\\mu\\big\(\(q^\{s\}\)^\{\-1\}\(v\)\\big\)\\geq q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}by Assumptions[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A1\) and \(A3\)\. Therefore, forw=\(w1,…,wM\)∈ΛsMw=\(w\_\{1\},\\ldots,w\_\{M\}\)\\in\\Lambda\_\{s\}^\{M\},

pr​\(w\)=𝔼⁡\[∏j=1MQ⁡\(\(qs\)−1​\(wj\)∣Zr−M\+j\)\]≥\(qmin​μmins\)M\.p\_\{r\}\(w\)=\\mathbb\{E\}\\bigg\[\\prod\_\{j=1\}^\{M\}Q\\big\(\(q^\{s\}\)^\{\-1\}\(w\_\{j\}\)\\mid Z\_\{r\-M\+j\}\\big\)\\bigg\]\\geq\(q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}\)^\{M\}\.In particular,\{qM,s\(Yr−M\+1:r\)=w\}\\\{q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=w\\\}has positive probability, and, by the same conditional independence,

P^rM,s\(v∣w\)=𝔼\[Q\(\(qs\)−1\(v\)∣Zr\+1\)\|qM,s\(Yr−M\+1:r\)=w\]≥qminμmins\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)=\\mathbb\{E\}\\big\[Q\\big\(\(q^\{s\}\)^\{\-1\}\(v\)\\mid Z\_\{r\+1\}\\big\)\\,\\big\|\\,q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=w\\big\]\\geq q\_\{\\rm min\}\\,\\mu\_\{\\rm min\}^\{s\}for allv∈Λsv\\in\\Lambda\_\{s\}, as in the proof of Lemma[24](https://arxiv.org/html/2610.41038#Thmtheorem24)\-\(ii\)\.

Proof of \(ii\)\.Since the prompts are independent copies ofqM,s\(Yr−M\+1:r\)q^\{M,s\}\(Y\_\{r\-M\+1:r\}\),nwn\_\{w\}is binomially distributed withK~\\tilde\{K\}trials and success probabilitypr​\(w\)p\_\{r\}\(w\)\. The multiplicative Chernoff bound\([Mulzer, 2018](https://arxiv.org/html/2610.41038#bib.bib37), Corollary 4\.2\)and \(i\) give

Pr⁡\(nw<12​K~​pr​\(w\)\)≤exp⁡\(−K~​pr​\(w\)8\)≤exp⁡\(−K~​\(qmin​μmins\)M8\)\.\\mathrm\{Pr\}\\Big\(n\_\{w\}<\\frac\{1\}\{2\}\\tilde\{K\}p\_\{r\}\(w\)\\Big\)\\leq\\exp\\Big\(\-\\frac\{\\tilde\{K\}p\_\{r\}\(w\)\}\{8\}\\Big\)\\leq\\exp\\Big\(\-\\frac\{\\tilde\{K\}\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{M\}\}\{8\}\\Big\)\.A union bound over theR​\(s\)MR\(s\)^\{M\}windows and the lower bound onK~\\tilde\{K\}yield

Pr⁡\(𝒢1c\)≤R​\(s\)M​exp⁡\(−K~​\(qmin​μmins\)M8\)≤R​\(s\)M​exp⁡\(−M​log⁡R⁡\(s\)−log⁡3δ\)=δ3\.\\mathrm\{Pr\}\(\\mathcal\{G\}\_\{1\}^\{c\}\)\\leq R\(s\)^\{M\}\\exp\\Big\(\-\\frac\{\\tilde\{K\}\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{M\}\}\{8\}\\Big\)\\leq R\(s\)^\{M\}\\exp\\Big\(\-M\\log R\(s\)\-\\log\\frac\{3\}\{\\delta\}\\Big\)=\\frac\{\\delta\}\{3\}\.For𝒢2\\mathcal\{G\}\_\{2\}, since the training pairs are independent and identically distributed, conditionally on the prompts𝒙1,…,𝒙K~\\bm\{x\}^\{1\},\\ldots,\\bm\{x\}^\{\\tilde\{K\}\}, the labels are independent andy~l∼P^rM,s\(⋅∣𝒙l\)\\tilde\{y\}^\{l\}\\sim\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid\\bm\{x\}^\{l\}\)\. Hence, conditionally on the prompts, for everywwwithnw≥1n\_\{w\}\\geq 1, the label counts\(nw​yw​\(v\)\)v∈Λs\(n\_\{w\}y^\{w\}\(v\)\)\_\{v\\in\\Lambda\_\{s\}\}form a multinomial vector withnwn\_\{w\}trials and cell probabilitiesP^rM,s\(⋅∣w\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\. By the Bretagnolle\-Huber\-Carol inequality\([van der Vaart and Wellner, 1996](https://arxiv.org/html/2610.41038#bib.bib47), Proposition A\.6\.6\), such a vector satisfies

Pr⁡\(∑v∈Λs\|nw​yw​\(v\)−nw​P^rM,s​\(v∣w\)\|≥2​nw​λ\)≤2R⁡\(s\)​e−2​λ2,λ\>0\.\\mathrm\{Pr\}\\bigg\(\\sum\_\{v\\in\\Lambda\_\{s\}\}\\big\|n\_\{w\}y^\{w\}\(v\)\-n\_\{w\}\\hat\{P\}\_\{r\}^\{M,s\}\(v\\mid w\)\\big\|\\geq 2\\sqrt\{n\_\{w\}\}\\lambda\\bigg\)\\leq 2^\{R\(s\)\}e^\{\-2\\lambda^\{2\}\},\\quad\\lambda\>0\.Since∥⋅∥TV=12∥⋅∥1\\\|\\cdot\\\|\_\{\\rm\\scriptscriptstyle TV\}=\\frac\{1\}\{2\}\\\|\\cdot\\\|\_\{1\}on𝒫⁡\(Λs\)\\mathcal\{P\}\(\\Lambda\_\{s\}\), the event on the left hand side is\{∥yw−P^rM,s\(⋅∣w\)∥TV≥λ/nw\}\\\{\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\geq\\lambda/\\sqrt\{n\_\{w\}\}\\\}\. Choosingλ=Cδ/2\\lambda=\\sqrt\{C\_\{\\delta\}/2\}, we get

Pr\(∥yw−P^rM,s\(⋅∣w\)∥TV\>Cδ2​nw\|𝒙1,…,𝒙K~\)≤2R⁡\(s\)e−Cδ=δ3​R​\(s\)M\.\\mathrm\{Pr\}\\bigg\(\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\>\\sqrt\{\\frac\{C\_\{\\delta\}\}\{2n\_\{w\}\}\}\\;\\bigg\|\\;\\bm\{x\}^\{1\},\\ldots,\\bm\{x\}^\{\\tilde\{K\}\}\\bigg\)\\leq 2^\{R\(s\)\}e^\{\-C\_\{\\delta\}\}=\\frac\{\\delta\}\{3R\(s\)^\{M\}\}\.A union bound over at mostR​\(s\)MR\(s\)^\{M\}windows and taking the expectation givePr⁡\(𝒢2c\)≤δ/3\\mathrm\{Pr\}\(\\mathcal\{G\}\_\{2\}^\{c\}\)\\leq\\delta/3\. Therefore,Pr⁡\(𝒢1∩𝒢2\)≥1−2​δ/3≥1−δ\\mathrm\{Pr\}\(\\mathcal\{G\}\_\{1\}\\cap\\mathcal\{G\}\_\{2\}\)\\geq 1\-2\\delta/3\\geq 1\-\\delta\. Finally, on𝒢1\\mathcal\{G\}\_\{1\}, we havenw≥K~​pr​\(w\)/2\>0n\_\{w\}\\geq\\tilde\{K\}p\_\{r\}\(w\)/2\>0for everyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}by \(i\), that is, every window appears in the data set\.

Proof of \(iii\)\.On𝒢1∩𝒢2\\mathcal\{G\}\_\{1\}\\cap\\mathcal\{G\}\_\{2\}, we havenw≥K~​pr​\(w\)/2n\_\{w\}\\geq\\tilde\{K\}p\_\{r\}\(w\)/2, and therefore

∥yw−P^rM,s\(⋅∣w\)∥TV≤Cδ2​nw≤CδK~​pr​\(w\)\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sqrt\{\\frac\{C\_\{\\delta\}\}\{2n\_\{w\}\}\}\\leq\\sqrt\{\\frac\{C\_\{\\delta\}\}\{\\tilde\{K\}p\_\{r\}\(w\)\}\}for everyw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\. By the Cauchy\-Bunyakovsky\-Schwarz inequality and∑wpr​\(w\)=1\\sum\_\{w\}p\_\{r\}\(w\)=1,

∑w∈ΛsMpr\(w\)∥yw−P^rM,s\(⋅∣w\)∥TV≤CδK~∑w∈ΛsMpr​\(w\)≤CδK~R​\(s\)M\.\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\sqrt\{\\frac\{C\_\{\\delta\}\}\{\\tilde\{K\}\}\}\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\sqrt\{p\_\{r\}\(w\)\}\\leq\\sqrt\{\\frac\{C\_\{\\delta\}\}\{\\tilde\{K\}\}\}\\sqrt\{R\(s\)^\{M\}\}\.For the second claim,

∑w∈ΛsMnwK~∥yw−P^rM,s\(⋅∣w\)∥TV2≤∑w∈ΛsMnwK~Cδ2​nw=R​\(s\)M​Cδ2​K~\.∎\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}^\{2\}\\leq\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}\\frac\{C\_\{\\delta\}\}\{2n\_\{w\}\}=\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{2\\tilde\{K\}\}\.\\qed

The next lemma collects three elementary facts on the softmax map and the Kullback\-Leibler divergence, which convert errors in the logits into errors in total variation and in the excess loss\.

###### Lemma 37\.

Letz,z′∈ℝR⁡\(s\)z,z^\{\\prime\}\\in\\mathbb\{R\}^\{R\(s\)\}, indexed byΛs\\Lambda\_\{s\}, and letν,P,P′∈𝒫⁡\(Λs\)\\nu,P,P^\{\\prime\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)\. Then the following assertions are true:

- \(i\)‖softmax⁡\(z\)−softmax⁡\(z′\)‖TV≤12​‖z−z′‖∞\\\|\\mathrm\{softmax\}\(z\)\-\\mathrm\{softmax\}\(z^\{\\prime\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{1\}\{2\}\\\|z\-z^\{\\prime\}\\\|\_\{\\infty\};
- \(ii\)DKL\(ν∥softmax\(z′\)\)≤DKL\(ν∥softmax\(z\)\)\+2∥ν−softmax\(z\)∥TV∥z−z′∥∞\+12∥z−z′∥∞2D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z^\{\\prime\}\)\)\\leq D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z\)\)\+2\\\|\\nu\-\\mathrm\{softmax\}\(z\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\|z\-z^\{\\prime\}\\\|\_\{\\infty\}\+\\frac\{1\}\{2\}\\\|z\-z^\{\\prime\}\\\|\_\{\\infty\}^\{2\};
- \(iii\)ifminv∈Λs⁡P′​\(v\)≥p\>0\\min\_\{v\\in\\Lambda\_\{s\}\}P^\{\\prime\}\(v\)\\geq p\>0, thenDKL\(P∥P′\)≤4p∥P−P′∥TV2D\_\{\\rm KL\}\(P\\parallel P^\{\\prime\}\)\\leq\\frac\{4\}\{p\}\\\|P\-P^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}^\{2\}\.

###### Proof\.

Throughout, writeh:=z′−zh:=z^\{\\prime\}\-zandπτ:=softmax⁡\(z\+τ​h\)\\pi\_\{\\tau\}:=\\mathrm\{softmax\}\(z\+\\tau h\)forτ∈\[0,1\]\\tau\\in\[0,1\]\. Forπ∈𝒫⁡\(Λs\)\\pi\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\), write⟨π,h⟩:=∑vπ⁡\(v\)​hv\\langle\\pi,h\\rangle:=\\sum\_\{v\}\\pi\(v\)h\_\{v\}andVarπ​\(h\):=∑vπ⁡\(v\)​\(hv−⟨π,h⟩\)2\\mathrm\{Var\}\_\{\\pi\}\(h\):=\\sum\_\{v\}\\pi\(v\)\(h\_\{v\}\-\\langle\\pi,h\\rangle\)^\{2\}for the mean and the variance ofhhunderπ\\pi\. We have that

Varπ​\(h\)≤⟨π,h2⟩≤‖h‖∞2\\mathrm\{Var\}\_\{\\pi\}\(h\)\\leq\\langle\\pi,h^\{2\}\\rangle\\leq\\\|h\\\|\_\{\\infty\}^\{2\}\(37\)sinceπ\\piis a probability vector andhhis bounded\.

Moreover, a direct computation givesdd​τ​πτ​\(v\)=πτ​\(v\)​\(hv−⟨πτ,h⟩\)\\frac\{\\mathrm\{d\}\}\{\\mathrm\{d\}\\tau\}\\pi\_\{\\tau\}\(v\)=\\pi\_\{\\tau\}\(v\)\(h\_\{v\}\-\\langle\\pi\_\{\\tau\},h\\rangle\)for allv∈Λsv\\in\\Lambda\_\{s\}\.

Proof of \(i\)\.By the fundamental theorem of calculus, the Cauchy\-Bunyakovsky\-Schwarz inequality and \([37](https://arxiv.org/html/2610.41038#A4.E37)\),

‖softmax⁡\(z′\)−softmax⁡\(z\)‖1≤∫01∑v∈Λsπτ​\(v\)​\|hv−⟨πτ,h⟩\|​𝑑τ≤∫01Varπτ​\(h\)12​dτ≤‖h‖∞\.\\begin\{split\}\\\|\\mathrm\{softmax\}\(z^\{\\prime\}\)\-\\mathrm\{softmax\}\(z\)\\\|\_\{1\}&\\leq\\int\_\{0\}^\{1\}\\sum\_\{v\\in\\Lambda\_\{s\}\}\\pi\_\{\\tau\}\(v\)\\,\|h\_\{v\}\-\\langle\\pi\_\{\\tau\},h\\rangle\|\\,\\mathrm\{d\}\\tau\\\\ &\\leq\\int\_\{0\}^\{1\}\\mathrm\{Var\}\_\{\\pi\_\{\\tau\}\}\(h\)^\{\\frac\{1\}\{2\}\}\\,\\mathrm\{d\}\\tau\\leq\\\|h\\\|\_\{\\infty\}\.\\end\{split\}Since∥⋅∥TV=12∥⋅∥1\\\|\\cdot\\\|\_\{\\rm\\scriptscriptstyle TV\}=\\frac\{1\}\{2\}\\\|\\cdot\\\|\_\{1\}on𝒫⁡\(Λs\)\\mathcal\{P\}\(\\Lambda\_\{s\}\), \(i\) follows\.

Proof of \(ii\)\.WriteLSE\(u\):=log∑veuv\\mathrm\{LSE\}\(u\):=\\log\\sum\_\{v\}e^\{u\_\{v\}\}, so thatlog⁡softmax​\(u\)v=uv−LSE⁡\(u\)\\log\\mathrm\{softmax\}\(u\)\_\{v\}=u\_\{v\}\-\\mathrm\{LSE\}\(u\)\. Since the softmax map has full support, both divergences in \(ii\) are finite, and

DKL\(ν∥softmax\(z′\)\)−DKL\(ν∥softmax\(z\)\)=∑v∈Λsν⁡\(v\)​log⁡softmax​\(z\)vsoftmax​\(z′\)v=−⟨ν,h⟩\+LSE⁡\(z\+h\)−LSE⁡\(z\)\.\\begin\{split\}D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z^\{\\prime\}\)\)\-D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z\)\)&=\\sum\_\{v\\in\\Lambda\_\{s\}\}\\nu\(v\)\\log\\frac\{\\mathrm\{softmax\}\(z\)\_\{v\}\}\{\\mathrm\{softmax\}\(z^\{\\prime\}\)\_\{v\}\}\\\\ &=\-\\langle\\nu,h\\rangle\+\\mathrm\{LSE\}\(z\+h\)\-\\mathrm\{LSE\}\(z\)\.\\end\{split\}The mapτ↦LSE⁡\(z\+τ​h\)\\tau\\mapsto\\mathrm\{LSE\}\(z\+\\tau h\)has the first derivative⟨πτ,h⟩\\langle\\pi\_\{\\tau\},h\\rangleand, by the computation above, the second derivativeVarπτ​\(h\)≤‖h‖∞2\\mathrm\{Var\}\_\{\\pi\_\{\\tau\}\}\(h\)\\leq\\\|h\\\|\_\{\\infty\}^\{2\}\. By Taylor’s theorem, we have the first order approximation:LSE⁡\(z\+h\)−LSE⁡\(z\)≤⟨π0,h⟩\+12​‖h‖∞2\\mathrm\{LSE\}\(z\+h\)\-\\mathrm\{LSE\}\(z\)\\leq\\langle\\pi\_\{0\},h\\rangle\+\\frac\{1\}\{2\}\\\|h\\\|\_\{\\infty\}^\{2\}withπ0=softmax⁡\(z\)\\pi\_\{0\}=\\mathrm\{softmax\}\(z\)\. Therefore,

DKL\(ν∥softmax\(z′\)\)−DKL\(ν∥softmax\(z\)\)≤⟨softmax⁡\(z\)−ν,h⟩\+12​‖h‖∞2≤‖softmax⁡\(z\)−ν‖1​‖h‖∞\+12​‖h‖∞2,\\begin\{split\}D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z^\{\\prime\}\)\)\-D\_\{\\rm KL\}\(\\nu\\parallel\\mathrm\{softmax\}\(z\)\)&\\leq\\langle\\mathrm\{softmax\}\(z\)\-\\nu,h\\rangle\+\\frac\{1\}\{2\}\\\|h\\\|\_\{\\infty\}^\{2\}\\\\ &\\leq\\\|\\mathrm\{softmax\}\(z\)\-\\nu\\\|\_\{1\}\\\|h\\\|\_\{\\infty\}\+\\frac\{1\}\{2\}\\\|h\\\|\_\{\\infty\}^\{2\},\\end\{split\}and∥⋅∥1=2∥⋅∥TV\\\|\\cdot\\\|\_\{1\}=2\\\|\\cdot\\\|\_\{\\rm\\scriptscriptstyle TV\}concludes \(ii\)\.

Proof of \(iii\)\.By Jensen’s inequality for the concave logarithm,

DKL\(P∥P′\)=∑v:P⁡\(v\)\>0P\(v\)logP⁡\(v\)P′​\(v\)≤log∑v∈ΛsP​\(v\)2P′​\(v\)=log⁡\(1\+∑v∈Λs\(P⁡\(v\)−P′​\(v\)\)2P′​\(v\)\),\\begin\{split\}D\_\{\\rm KL\}\(P\\parallel P^\{\\prime\}\)&=\\sum\_\{v:\\,P\(v\)\>0\}P\(v\)\\log\\frac\{P\(v\)\}\{P^\{\\prime\}\(v\)\}\\\\ &\\leq\\log\\sum\_\{v\\in\\Lambda\_\{s\}\}\\frac\{P\(v\)^\{2\}\}\{P^\{\\prime\}\(v\)\}\\\\ &=\\log\\bigg\(1\+\\sum\_\{v\\in\\Lambda\_\{s\}\}\\frac\{\(P\(v\)\-P^\{\\prime\}\(v\)\)^\{2\}\}\{P^\{\\prime\}\(v\)\}\\bigg\),\\end\{split\}where the last equality follows from expandingP​\(v\)2=\(P⁡\(v\)−P′​\(v\)\)2\+2​P​\(v\)​P′​\(v\)−P′​\(v\)2P\(v\)^\{2\}=\(P\(v\)\-P^\{\\prime\}\(v\)\)^\{2\}\+2P\(v\)P^\{\\prime\}\(v\)\-P^\{\\prime\}\(v\)^\{2\}and summing overvv\. Usinglog⁡\(1\+x\)≤x\\log\(1\+x\)\\leq x,P′​\(v\)≥pP^\{\\prime\}\(v\)\\geq pand∑v\(P⁡\(v\)−P′​\(v\)\)2≤\(∑v\|P⁡\(v\)−P′​\(v\)\|\)2\\sum\_\{v\}\(P\(v\)\-P^\{\\prime\}\(v\)\)^\{2\}\\leq\(\\sum\_\{v\}\|P\(v\)\-P^\{\\prime\}\(v\)\|\)^\{2\}, we obtain

DKL\(P∥P′\)≤1p\(∑v∈Λs\|P\(v\)−P′\(v\)\|\)2=4p∥P−P′∥TV2\.∎D\_\{\\rm KL\}\(P\\parallel P^\{\\prime\}\)\\leq\\frac\{1\}\{p\}\\bigg\(\\sum\_\{v\\in\\Lambda\_\{s\}\}\|P\(v\)\-P^\{\\prime\}\(v\)\|\\bigg\)^\{2\}=\\frac\{4\}\{p\}\\\|P\-P^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}^\{2\}\.\\qed

### D\.2Proof of Theorem[27](https://arxiv.org/html/2610.41038#Thmtheorem27)

Fixr≥Mr\\geq Mand writewr:=qM,s\(Yr−M\+1:r\)w\_\{r\}:=q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)for the quantized test window\. ThenPrM,s=PrM,s\(⋅∣wr\)P\_\{r\}^\{M,s\}=P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)andP^rM,s=P^rM,s\(⋅∣wr\)\\hat\{P\}\_\{r\}^\{M,s\}=\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)in Definition[2](https://arxiv.org/html/2610.41038#Thmtheorem2), and the law ofwrw\_\{r\}isprp\_\{r\}in \([22](https://arxiv.org/html/2610.41038#S5.E22)\)\. The data set enters only throughγ∗\\gamma^\{\*\}and it is independent ofY0:rY\_\{0:r\}; so, in Steps 1–3, we fix the data set and take the expectation overY0:rY\_\{0:r\}only\. We prove step\-by\-step:

Step 1\.For every realization ofY0:rY\_\{0:r\}, all measures below are probability measures on𝕐\\mathbb\{Y\}, and the triangle inequality gives

∥Pr−Ξs\(𝖳γ∗\(⋅∣wr\)\)∥TV≤∥Pr−PrM,s∥TV\+∥PrM,s\(⋅∣wr\)−Ξs\(P^rM,s\(⋅∣wr\)\)∥TV\+∥Ξs\(P^rM,s\(⋅∣wr\)\)−Ξs\(𝖳γ∗\(⋅∣wr\)\)∥TV\.\\begin\{split\}\\\|P\_\{r\}\-\\Xi\_\{s\}\(\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\_\{r\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&\\leq\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\+\\\|P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)\-\\Xi\_\{s\}\(\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\qquad\+\\\|\\Xi\_\{s\}\(\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)\)\-\\Xi\_\{s\}\(\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\_\{r\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\.\\end\{split\}\(38\)By Theorem[12](https://arxiv.org/html/2610.41038#Thmtheorem12)\-\(ii\), the first term satisfies

𝔼Y0:r∥Pr−PrM,s∥TV≤𝜶⌊M/N⌋\+2​M​Lψqmin​s\.\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-P\_\{r\}^\{M,s\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}\+\\frac\{2ML\_\{\\psi\}\}\{q\_\{\\rm min\}s\}\.
Step 2\.Fixw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}\. Conditionally onZr\+1Z\_\{r\+1\}, the next observationYr\+1Y\_\{r\+1\}is independent ofYr−M\+1:rY\_\{r\-M\+1:r\}and has the lawQ\(⋅∣Zr\+1\)Q\(\\,\\cdot\\mid Z\_\{r\+1\}\)\. Hence, theμ\\mu\-density ofPrM,s\(⋅∣w\)P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)is

prM,s\(y∣w\):=∑z∈Sψ\(y∣z\)Pr\(Zr\+1=z∣qM,s\(Yr−M\+1:r\)=w\),y∈𝕐,p\_\{r\}^\{M,s\}\(y\\mid w\):=\\sum\_\{z\\in S\}\\psi\(y\\mid z\)\\,\\mathrm\{Pr\}\\big\(Z\_\{r\+1\}=z\\mid q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=w\\big\),\\qquad y\\in\\mathbb\{Y\},which is a convex combination of\{ψ\(⋅∣z\)\}z∈S\\\{\\psi\(\\,\\cdot\\mid z\)\\\}\_\{z\\in S\}\. Exactly as forprp\_\{r\}in the proof of Lemma[28](https://arxiv.org/html/2610.41038#Thmtheorem28), Assumption[3](https://arxiv.org/html/2610.41038#Thmtheorem3)\-\(A2\) shows thatprM,s\(⋅∣w\)p\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)isLψL\_\{\\psi\}\-Lipschitz\. SinceP^rM,s\(⋅∣w\)=q\#sPrM,s\(⋅∣w\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)=q^\{s\}\_\{\\\#\}P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)by Definition[2](https://arxiv.org/html/2610.41038#Thmtheorem2), theμ\\mu\-density ofΞs\(P^rM,s\(⋅∣w\)\)\\Xi\_\{s\}\(\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\)is constant on each cellC⁡\(s,i\)C\(s,i\)and equal to theμ\\mu\-average ofprM,s\(⋅∣w\)p\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)overC⁡\(s,i\)C\(s,i\)\. The remaining lines of the proof of Lemma[28](https://arxiv.org/html/2610.41038#Thmtheorem28)then apply verbatim and give

∥PrM,s\(⋅∣w\)−Ξs\(P^rM,s\(⋅∣w\)\)∥TV≤Lψs,w∈ΛsM\.\\\|P\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\Xi\_\{s\}\(\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\frac\{L\_\{\\psi\}\}\{s\},\\qquad w\\in\\Lambda\_\{s\}^\{M\}\.So, the second term in \([38](https://arxiv.org/html/2610.41038#A4.E38)\) is at mostLψ/sL\_\{\\psi\}/sfor every realization ofY0:rY\_\{0:r\}\.

Step 3\.Forν,ν′∈𝒫⁡\(Λs\)\\nu,\\nu^\{\\prime\}\\in\\mathcal\{P\}\(\\Lambda\_\{s\}\)and a Borel setA⊂𝕐A\\subset\\mathbb\{Y\}, Definition[4](https://arxiv.org/html/2610.41038#Thmtheorem4)gives

Ξs​\(ν\)​\(A\)−Ξs​\(ν′\)​\(A\)=∑i=1R⁡\(s\)\(ν⁡\(ys,i\)−ν′​\(ys,i\)\)​μ⁡\(A∩C⁡\(s,i\)\)μ⁡\(C⁡\(s,i\)\)\.\\Xi\_\{s\}\(\\nu\)\(A\)\-\\Xi\_\{s\}\(\\nu^\{\\prime\}\)\(A\)=\\sum\_\{i=1\}^\{R\(s\)\}\\big\(\\nu\(y\_\{s,i\}\)\-\\nu^\{\\prime\}\(y\_\{s,i\}\)\\big\)\\frac\{\\mu\(A\\cap C\(s,i\)\)\}\{\\mu\(C\(s,i\)\)\}\.Each ratio on the right hand side is in the interval\[0,1\]\[0,1\], and∑i\(ν⁡\(ys,i\)−ν′​\(ys,i\)\)=0\\sum\_\{i\}\(\\nu\(y\_\{s,i\}\)\-\\nu^\{\\prime\}\(y\_\{s,i\}\)\)=0\. Therefore, the absolute value of the right hand side is at most∑i\(ν⁡\(ys,i\)−ν′​\(ys,i\)\)\+=‖ν−ν′‖TV\\sum\_\{i\}\(\\nu\(y\_\{s,i\}\)\-\\nu^\{\\prime\}\(y\_\{s,i\}\)\)^\{\+\}=\\\|\\nu\-\\nu^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}, wherex\+:=max⁡\{x,0\}x^\{\+\}:=\\max\\\{x,0\\\}\. Hence,‖Ξs​\(ν\)−Ξs​\(ν′\)‖TV≤‖ν−ν′‖TV\\\|\\Xi\_\{s\}\(\\nu\)\-\\Xi\_\{s\}\(\\nu^\{\\prime\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\leq\\\|\\nu\-\\nu^\{\\prime\}\\\|\_\{\\rm\\scriptscriptstyle TV\}, and the third term in \([38](https://arxiv.org/html/2610.41038#A4.E38)\) is at most∥P^rM,s\(⋅∣wr\)−𝖳γ∗\(⋅∣wr\)∥TV\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\_\{r\}\)\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\_\{r\}\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\. Taking the expectation and collecting Steps 1–3, we have shown

𝔼Y0:r∥Pr−Ξs\(𝖳γ∗\(⋅∣wr\)\)∥TV≤𝜶⌊M/N⌋\+\(1\+2​Mqmin\)Lψs\+∑w∈ΛsMpr\(w\)∥P^rM,s\(⋅∣w\)−𝖳γ∗\(⋅∣w\)∥TV\.\\begin\{split\}&\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\\|P\_\{r\}\-\\Xi\_\{s\}\(\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\_\{r\}\)\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\qquad\\leq\\bm\{\\alpha\}^\{\\lfloor M/N\\rfloor\}\+\\Big\(1\+\\frac\{2M\}\{q\_\{\\rm min\}\}\\Big\)\\frac\{L\_\{\\psi\}\}\{s\}\+\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\.\\end\{split\}\(39\)
Step 4\.Let𝒢1\\mathcal\{G\}\_\{1\}and𝒢2\\mathcal\{G\}\_\{2\}be the events of Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(ii\)\. Their intersection has probability at least1−δ1\-\\delta, and we fix a realization of the data set in𝒢1∩𝒢2\\mathcal\{G\}\_\{1\}\\cap\\mathcal\{G\}\_\{2\}for the rest of the proof\. By Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(ii\), every window appears in the data set, so the sums overk∈𝒦k\\in\\mathcal\{K\}in \([15](https://arxiv.org/html/2610.41038#S3.E15)\) and in the definition ofℰ\\mathcal\{E\}are sums overw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, and

ℰ\(γ\)=∑w∈ΛsMnwK~DKL\(yw∥𝖳γ\(⋅∣w\)\),γ∈𝕌mM\+1\.\\mathcal\{E\}\(\\gamma\)=\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\mathsf\{T\}^\{\\gamma\}\(\\,\\cdot\\mid w\)\\big\),\\qquad\\gamma\\in\\mathbb\{U\}\_\{m\}^\{M\+1\}\.By the triangle inequality,

∑w∈ΛsMpr\(w\)∥P^rM,s\(⋅∣w\)−𝖳γ∗\(⋅∣w\)∥TV≤∑w∈ΛsMpr\(w\)∥P^rM,s\(⋅∣w\)−yw∥TV\+∑w∈ΛsMpr\(w\)∥yw−𝖳γ∗\(⋅∣w\)∥TV\.\\begin\{split\}\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&\\leq\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-y^\{w\}\\\|\_\{\\rm\\scriptscriptstyle TV\}\\\\ &\\qquad\+\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|y^\{w\}\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\.\\end\{split\}\(40\)The first sum is at mostR​\(s\)M​Cδ/K~\\sqrt\{R\(s\)^\{M\}C\_\{\\delta\}/\\tilde\{K\}\}by Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(iii\)\. For the second sum, we use Pinsker’s inequality\([Polyanskiy and Wu, 2025](https://arxiv.org/html/2610.41038#bib.bib38), Theorem 7\.10\), Jensen’s inequality for the concave function⋅\\sqrt\{\\cdot\}with the weightspr​\(w\)p\_\{r\}\(w\), which sum to11, and the inequalitypr​\(w\)≤2​nw/K~p\_\{r\}\(w\)\\leq 2n\_\{w\}/\\tilde\{K\}, which holds on𝒢1\\mathcal\{G\}\_\{1\}:

∑w∈ΛsMpr\(w\)∥yw−𝖳γ∗\(⋅∣w\)∥TV≤∑w∈ΛsMpr​\(w\)​12DKL\(yw∥𝖳γ∗\(⋅∣w\)\)≤12∑w∈ΛsMpr\(w\)DKL\(yw∥𝖳γ∗\(⋅∣w\)\)≤∑w∈ΛsMnwK~DKL\(yw∥𝖳γ∗\(⋅∣w\)\)=ℰ⁡\(γ∗\)\.\\begin\{split\}\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|y^\{w\}\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&\\leq\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\sqrt\{\\frac\{1\}\{2\}D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\big\)\}\\\\ &\\leq\\sqrt\{\\frac\{1\}\{2\}\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\big\)\}\\\\ &\\leq\\sqrt\{\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\big\)\}=\\sqrt\{\\mathcal\{E\}\(\\gamma^\{\*\}\)\}\.\\end\{split\}\(41\)Finally,γ∗\\gamma^\{\*\}minimizesℰ\\mathcal\{E\}over𝕌mM\+1\\mathbb\{U\}\_\{m\}^\{M\+1\}, and the policyU¯^\\underline\{\\hat\{U\}\}of Lemma[35](https://arxiv.org/html/2610.41038#Thmtheorem35)belongs to𝕌mM\+1\\mathbb\{U\}\_\{m\}^\{M\+1\}\. Therefore,

ℰ⁡\(γ∗\)≤ℰ⁡\(U¯^\)\.\\mathcal\{E\}\(\\gamma^\{\*\}\)\\leq\\mathcal\{E\}\(\\underline\{\\hat\{U\}\}\)\.\(42\)
Step 5\.Fixw∈ΛsMw\\in\\Lambda\_\{s\}^\{M\}, writeη:=η⁡\(n,m,β\)\\eta:=\\eta\(n,m,\\beta\)andaw:=∥yw−P^rM,s\(⋅∣w\)∥TVa\_\{w\}:=\\\|y^\{w\}\-\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}\. By Lemma[35](https://arxiv.org/html/2610.41038#Thmtheorem35),𝖳U¯^\(⋅∣w\)=softmax\(z^out\(w\)\)\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(\\hat\{z\}\_\{\\rm out\}\(w\)\)with‖z^out​\(w\)−z¯​\(w\)‖∞≤η\\\|\\hat\{z\}\_\{\\rm out\}\(w\)\-\\bar\{z\}\(w\)\\\|\_\{\\infty\}\\leq\\eta, andP^rM,s\(⋅∣w\)=softmax\(z¯\(w\)\)\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)=\\mathrm\{softmax\}\(\\bar\{z\}\(w\)\)\. We apply Lemma[37](https://arxiv.org/html/2610.41038#Thmtheorem37)\-\(ii\) withν=yw\\nu=y^\{w\},z=z¯​\(w\)z=\\bar\{z\}\(w\)andz′=z^out​\(w\)z^\{\\prime\}=\\hat\{z\}\_\{\\rm out\}\(w\), and then Lemma[37](https://arxiv.org/html/2610.41038#Thmtheorem37)\-\(iii\) withP=ywP=y^\{w\}andP′=P^rM,s\(⋅∣w\)P^\{\\prime\}=\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\), whose entries are at leastqmin​μminsq\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}by Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(i\):

DKL\(yw∥𝖳U¯^\(⋅∣w\)\)≤DKL\(yw∥P^rM,s\(⋅∣w\)\)\+2awη\+η22≤4​aw2qmin​μmins\+2awη\+η22\.D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\mathsf\{T\}^\{\\underline\{\\hat\{U\}\}\}\(\\,\\cdot\\mid w\)\\big\)\\leq D\_\{\\rm KL\}\\big\(y^\{w\}\\parallel\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\\big\)\+2a\_\{w\}\\eta\+\\frac\{\\eta^\{2\}\}\{2\}\\leq\\frac\{4a\_\{w\}^\{2\}\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\+2a\_\{w\}\\eta\+\\frac\{\\eta^\{2\}\}\{2\}\.We multiply bynw/K~n\_\{w\}/\\tilde\{K\}and sum overww\. WriteA:=\(∑wnwK~​aw2\)1/2A:=\(\\sum\_\{w\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}a\_\{w\}^\{2\}\)^\{1/2\}\. Since∑wnw/K~=1\\sum\_\{w\}n\_\{w\}/\\tilde\{K\}=1, the Cauchy\-Bunyakovsky\-Schwarz inequality gives∑wnwK~​aw≤A\\sum\_\{w\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}a\_\{w\}\\leq A, and therefore

ℰ⁡\(U¯^\)≤4​A2qmin​μmins\+2​A​η\+η22≤\(2​Aqmin​μmins\+η2\)2,\\mathcal\{E\}\(\\underline\{\\hat\{U\}\}\)\\leq\\frac\{4A^\{2\}\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\+2A\\eta\+\\frac\{\\eta^\{2\}\}\{2\}\\leq\\bigg\(\\frac\{2A\}\{\\sqrt\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\}\+\\frac\{\\eta\}\{\\sqrt\{2\}\}\\bigg\)^\{2\},where, for the last inequality, we expanded the square and used2​A​η≤2​2​A​η/qmin​μmins2A\\eta\\leq 2\\sqrt\{2\}A\\eta/\\sqrt\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\}\. The latter holds sinceqmin​μmins≤Q⁡\(\(qs\)−1​\(v\)∣z\)≤1q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\\leq Q\(\(q^\{s\}\)^\{\-1\}\(v\)\\mid z\)\\leq 1for allv∈Λsv\\in\\Lambda\_\{s\}andz∈Sz\\in Sby the proof of Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(i\)\. By Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(iii\),A2≤R​\(s\)M​Cδ/\(2​K~\)A^\{2\}\\leq R\(s\)^\{M\}C\_\{\\delta\}/\(2\\tilde\{K\}\), hence

ℰ⁡\(U¯^\)≤2​R​\(s\)M​Cδqmin​μmins​K~\+η2\.\\sqrt\{\\mathcal\{E\}\(\\underline\{\\hat\{U\}\}\)\}\\leq\\sqrt\{\\frac\{2R\(s\)^\{M\}C\_\{\\delta\}\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\\tilde\{K\}\}\}\+\\frac\{\\eta\}\{\\sqrt\{2\}\}\.\(43\)
Step 6\.Combining \([40](https://arxiv.org/html/2610.41038#A4.E40)\)–\([43](https://arxiv.org/html/2610.41038#A4.E43)\) with the bound on the first sum in \([40](https://arxiv.org/html/2610.41038#A4.E40)\), and using1≤\(qminμmins\)−1/21\\leq\(q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\)^\{\-1/2\}, we get

∑w∈ΛsMpr\(w\)∥P^rM,s\(⋅∣w\)−𝖳γ∗\(⋅∣w\)∥TV≤R​\(s\)M​CδK~\+2​R​\(s\)M​Cδqmin​μmins​K~\+η2≤\(1\+2\)​R​\(s\)M​Cδqmin​μmins​K~\+η2\.\\begin\{split\}\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\_\{r\}\(w\)\\\|\\hat\{P\}\_\{r\}^\{M,s\}\(\\,\\cdot\\mid w\)\-\\mathsf\{T\}^\{\\gamma^\{\*\}\}\(\\,\\cdot\\mid w\)\\\|\_\{\\rm\\scriptscriptstyle TV\}&\\leq\\sqrt\{\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{\\tilde\{K\}\}\}\+\\sqrt\{\\frac\{2R\(s\)^\{M\}C\_\{\\delta\}\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\\tilde\{K\}\}\}\+\\frac\{\\eta\}\{\\sqrt\{2\}\}\\\\ &\\leq\(1\+\\sqrt\{2\}\)\\sqrt\{\\frac\{R\(s\)^\{M\}C\_\{\\delta\}\}\{q\_\{\\rm min\}\\mu\_\{\\rm min\}^\{s\}\\tilde\{K\}\}\}\+\\frac\{\\eta\}\{\\sqrt\{2\}\}\.\\end\{split\}Plugging this into \([39](https://arxiv.org/html/2610.41038#A4.E39)\) gives \([25](https://arxiv.org/html/2610.41038#S5.E25)\)\. Since the realization of the data set in𝒢1∩𝒢2\\mathcal\{G\}\_\{1\}\\cap\\mathcal\{G\}\_\{2\}is arbitrary, the bound holds with probability at least1−δ1\-\\delta, and the proof is now concluded\.■\\blacksquare

### D\.3Proof of Theorem[26](https://arxiv.org/html/2610.41038#Thmtheorem26)

By Lemma[30](https://arxiv.org/html/2610.41038#Thmtheorem30),\{\(Zr,Yr\)\}\\\{\(Z\_\{r\},Y\_\{r\}\)\\\}is stationary\. Hence the window lawp\(w\):=Pr\(qM,s\(Yr−M\+1:r\)=w\)p\(w\):=\\mathrm\{Pr\}\\big\(q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)=w\\big\)and the objectsPrMP^\{M\}\_\{r\},PrM,sP^\{M,s\}\_\{r\}andP^rM,s=:P^\(⋅∣w\)\\hat\{P\}^\{M,s\}\_\{r\}=:\\hat\{P\}\(\\cdot\\mid w\)do not depend onr≥Mr\\geq M\. FixMMandsswithR⁡\(s\)≥2R\(s\)\\geq 2, and letU¯\\underline\{U\}be any open\-loop policy\. We proceed in four steps\.

*Step 1\.*Replacing the memory bound of Theorem[12](https://arxiv.org/html/2610.41038#Thmtheorem12)with that of Proposition[7](https://arxiv.org/html/2610.41038#Thmtheorem7), we obtain

supr≥M𝔼Y0:r∥Pr−Ξs\(TU¯\(⋅∣wr\)\)∥TV≤ΔM/2\+\(1\+2​Mqmin\)​Lψs\+∑w∈ΛsMp\(w\)∥P^\(⋅∣w\)−TU¯\(⋅∣w\)∥TV,\\begin\{split\}\\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\Big\\\|P\_\{r\}\-\\Xi\_\{s\}\\big\(T^\{\\underline\{U\}\}\(\\cdot\\mid w\_\{r\}\)\\big\)\\Big\\\|\_\{\{\\rm\\scriptscriptstyle TV\}\}\\leq\\sqrt\{\\Delta\_\{M\}/2\}&\+\\Big\(1\+\\frac\{2M\}\{q\_\{\\min\}\}\\Big\)\\frac\{L\_\{\\psi\}\}\{s\}\\\\ &\+\\sum\_\{w\\in\\Lambda\_\{s\}^\{M\}\}p\(w\)\\,\\big\\\|\\hat\{P\}\(\\cdot\\mid w\)\-T^\{\\underline\{U\}\}\(\\cdot\\mid w\)\\big\\\|\_\{\{\\rm\\scriptscriptstyle TV\}\},\\end\{split\}\(⋆\\star\)wherewr:=qM,s\(Yr−M\+1:r\)w\_\{r\}:=q^\{M,s\}\(Y\_\{r\-M\+1:r\}\)\.

*Step 2\.*Letξj:=\(qM,s\(Yj:j\+M−1\),qs\(Yj\+M\)\)\\xi\_\{j\}:=\\big\(q^\{M,s\}\(Y\_\{j:j\+M\-1\}\),q^\{s\}\(Y\_\{j\+M\}\)\\big\)\. This is a fixed measurable function of finitely many coordinates of the shifted sequence\(Yj\+i\)i≧0\(Y\_\{j\+i\}\)\_\{i\\geqq 0\}\. Since\{Yr\}\\\{Y\_\{r\}\\\}is stationary and ergodic \(Lemma[30](https://arxiv.org/html/2610.41038#Thmtheorem30)\), so is\{ξj\}j\\\{\\xi\_\{j\}\\\}\_\{j\}\. By Birkhoff’s ergodic theorem, applied to the finitely many indicators𝟏\{ξj=\(w,v\)\}\\mathbf\{1\}\\\{\\xi\_\{j\}=\(w,v\)\\\}, there is an eventΩM,s\\Omega\_\{M,s\}of probability one on which

nwK~→p\(w\)and1K~∑j=1K~𝟏\{ξj=\(w,v\)\}→p\(w\)P^\(v∣w\)\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}\\to p\(w\)\\qquad\\text\{and\}\\qquad\\frac\{1\}\{\\tilde\{K\}\}\\sum\_\{j=1\}^\{\\tilde\{K\}\}\\mathbf\{1\}\\\{\\xi\_\{j\}=\(w,v\)\\\}\\to p\(w\)\\,\\hat\{P\}\(v\\mid w\)for all\(w,v\)∈ΛsM×Λs\(w,v\)\\in\\Lambda\_\{s\}^\{M\}\\times\\Lambda\_\{s\}\. The proof of Lemma[36](https://arxiv.org/html/2610.41038#Thmtheorem36)\-\(i\) uses only the conditional independence of the observations, sop⁡\(w\)≥\(qmin​μmins\)M\>0p\(w\)\\geq\(q\_\{\\min\}\\mu^\{s\}\_\{\\min\}\)^\{M\}\>0\. Hence, onΩM,s\\Omega\_\{M,s\}:

- •there existsK~0​\(ω\)\\tilde\{K\}\_\{0\}\(\\omega\)such that, for allK~≥K~0​\(ω\)\\tilde\{K\}\\geq\\tilde\{K\}\_\{0\}\(\\omega\), every window appears in the data set andp⁡\(w\)≤2​nw/K~p\(w\)\\leq 2n\_\{w\}/\\tilde\{K\}for everyww;
- •aw:=∥yw−P^\(⋅∣w\)∥TV→0a\_\{w\}:=\\\|y^\{w\}\-\\hat\{P\}\(\\cdot\\mid w\)\\\|\_\{\{\\rm\\scriptscriptstyle TV\}\}\\to 0for everyww, asK~→∞\\tilde\{K\}\\to\\infty\.

*Step 3\.*Fix aligning levels\(n,m,l\)\(n,m,l\)andβ\\betawithη:=η⁡\(n,m,β\)≤1\\eta:=\\eta\(n,m,\\beta\)\\leq 1, and fixω∈ΩM,s\\omega\\in\\Omega\_\{M,s\}andK~≥K~0​\(ω\)\\tilde\{K\}\\geq\\tilde\{K\}\_\{0\}\(\\omega\)\. Steps 4 and 5 of §[D\.2](https://arxiv.org/html/2610.41038#A4.SS2)use only the properties listed at the end of Step 2, the optimality ofU¯K~∗\\underline\{U\}^\{\*\}\_\{\\tilde\{K\}\}, and Lemmata[34](https://arxiv.org/html/2610.41038#Thmtheorem34)and[37](https://arxiv.org/html/2610.41038#Thmtheorem37)\. These steps therefore give

∑wp\(w\)∥P^\(⋅∣w\)−TU¯K~∗\(⋅∣w\)∥TV≤∑wp\(w\)aw\+2​AK~qmin​μmins\+η2,\\sum\_\{w\}p\(w\)\\big\\\|\\hat\{P\}\(\\cdot\\mid w\)\-T^\{\\underline\{U\}^\{\*\}\_\{\\tilde\{K\}\}\}\(\\cdot\\mid w\)\\big\\\|\_\{\{\\rm\\scriptscriptstyle TV\}\}\\leq\\sum\_\{w\}p\(w\)\\,a\_\{w\}\+\\frac\{2A\_\{\\tilde\{K\}\}\}\{\\sqrt\{q\_\{\\min\}\\mu^\{s\}\_\{\\min\}\}\}\+\\frac\{\\eta\}\{\\sqrt\{2\}\},withAK~2:=∑wnwK~​aw2A\_\{\\tilde\{K\}\}^\{2\}:=\\sum\_\{w\}\\frac\{n\_\{w\}\}\{\\tilde\{K\}\}\\,a\_\{w\}^\{2\}\.

By Step 2,aw→0a\_\{w\}\\to 0for each of the finitely manyww, andnw/K~≤1n\_\{w\}/\\tilde\{K\}\\leq 1\. Hence the first two terms vanish asK~→∞\\tilde\{K\}\\to\\infty\. Combining with \([⋆‣D\.3](https://arxiv.org/html/2610.41038#A4.Ex173)\), onΩM,s\\Omega\_\{M,s\}we obtain

lim supK~→∞supr≥M𝔼Y0:r∥Pr−Ξs\(TU¯K~∗\(⋅∣wr\)\)∥TV≤ΔM/2\+\(1\+2​Mqmin\)Lψs\+η⁡\(n,m,β\)2\.\\limsup\_\{\\tilde\{K\}\\to\\infty\}\\ \\sup\_\{r\\geq M\}\\mathbb\{E\}\_\{Y\_\{0:r\}\}\\Big\\\|P\_\{r\}\-\\Xi\_\{s\}\\big\(T^\{\\underline\{U\}^\{\*\}\_\{\\tilde\{K\}\}\}\(\\cdot\\mid w\_\{r\}\)\\big\)\\Big\\\|\_\{\{\\rm\\scriptscriptstyle TV\}\}\\leq\\sqrt\{\\Delta\_\{M\}/2\}\+\\Big\(1\+\\frac\{2M\}\{q\_\{\\min\}\}\\Big\)\\frac\{L\_\{\\psi\}\}\{s\}\+\\frac\{\\eta\(n,m,\\beta\)\}\{\\sqrt\{2\}\}\.\(⋆′\\star^\{\\prime\}\)
*Step 4\.*The eventΩ∗:=⋂M,s∈ℕΩM,s\\Omega^\{\*\}:=\\bigcap\_\{M,s\\in\\mathbb\{N\}\}\\Omega\_\{M,s\}is a countable intersection, so it has probability one\. It does not depend on\(n,m,l,β\)\(n,m,l,\\beta\), and on it \([⋆ ′‣D\.3](https://arxiv.org/html/2610.41038#A4.Ex176)\) holds for allMM,ss, aligning\(n,m,l\)\(n,m,l\)andβ\\betawithη≤1\\eta\\leq 1\. For fixedMMandss, we havec=M𝕌c=M\_\{\\mathbb\{U\}\}, so each term of \([24](https://arxiv.org/html/2610.41038#S5.E24)\) tends to00asβ,n,m→∞\\beta,n,m\\to\\inftyalong aligning levels; in particularη≤1\\eta\\leq 1eventually\. Letting\(n,m,β\)→∞\(n,m,\\beta\)\\to\\infty, thens→∞s\\to\\infty, thenM→∞M\\to\\infty, and usingΔM↓0\\Delta\_\{M\}\\downarrow 0\(Proposition[7](https://arxiv.org/html/2610.41038#Thmtheorem7)\), proves the claim onΩ∗\\Omega^\{\*\}\.■\\blacksquare

## References

- Agrachev and Letrouit \[2025\]A\. Agrachev and C\. Letrouit\.Generic controllability of equivariant systems and applications to particle systems and neural networks\.*Annales de l’Institut Henri Poincaré C*, 43\(3\):639–668, 2025\.
- Akman et al\. \[2026a\]K\. Akman, N\. Saldi, and S\. Yüksel\.Generalization bounds on optimal control for Transformer training and Wasserstein distributional robustness\.*arXiv preprint arXiv:2607\.27975*, 2026a\.
- Akman et al\. \[2026b\]K\. Akman, N\. Saldi, and S\. Yüksel\.An optimal control approach to Transformer training\.*arXiv preprint arXiv:2603\.09571*, 2026b\.
- Anderson and Moore \[2007\]B\. D\. O\. Anderson and J\. B\. Moore\.*Optimal control: Linear quadratic methods*\.Courier Corporation, 2007\.
- Atar and Zeitouni \[1997\]R\. Atar and O\. Zeitouni\.Exponential stability for nonlinear filtering\.In*Annales de l’Institut Henri Poincare \(B\) Probability and Statistics*, volume 33, pages 697–725\. Elsevier, 1997\.
- Barbero et al\. \[2024\]F\. Barbero, A\. Vitvitskyi, C\. Perivolaropoulos, R\. Pascanu, and P\. Veličković\.Round and round we go\! what makes rotary positional encodings useful?*arXiv preprint arXiv:2410\.06205*, 2024\.
- Butcher \[2016\]J\. C\. Butcher\.*Numerical methods for ordinary differential equations*\.John Wiley & Sons, 2016\.
- Chang and Mehta \[2025\]H\.\-S\. Chang and P\. G\. Mehta\.Dual filter: A mathematical framework for inference using transformer\-like architectures\.*arXiv preprint arXiv:2505\.00818*, 2025\.
- Chen et al\. \[2018\]R\. T\. Chen, Y\. Rubanova, J\. Bettencourt, and D\. K\. Duvenaud\.Neural ordinary differential equations\.*Advances in neural information processing systems*, 31, 2018\.
- Chigansky and Liptser \[2004\]P\. Chigansky and R\. Liptser\.Stability of nonlinear filters in non\-mixing case\.*Annals App\. Prob*, 14:2038–2056, 2004\.
- Chigansky et al\. \[2009\]P\. Chigansky, R\. Liptser, and R\. van Handel\.Intrinsic methods in filter stability\.*Handbook of Nonlinear Filtering*, 2009\.
- Ciampiconi et al\. \[2026\]L\. Ciampiconi, A\. Elwood, M\. Leonardi, A\. Mohamed, and A\. Rozza\.A survey and taxonomy of loss functions in machine learning\.*AI*, 7\(4\), 2026\.ISSN 2673\-2688\.doi:10\.3390/ai7040128\.URL[https://www\.mdpi\.com/2673\-2688/7/4/128](https://www.mdpi.com/2673-2688/7/4/128)\.
- Devlin et al\. \[2019\]J\. Devlin, M\.\-W\. Chang, K\. Lee, and K\. Toutanova\.Bert: Pre\-training of deep bidirectional transformers for language understanding, 2019\.URL[https://arxiv\.org/abs/1810\.04805](https://arxiv.org/abs/1810.04805)\.
- Dobrushin \[1956\]R\. L\. Dobrushin\.Central limit theorem for nonstationary markov chains\. i\.*Theory of Probability & Its Applications*, 1\(1\):65–80, 1956\.
- E \[2017\]W\. E\.A proposal on machine learning via dynamical systems\.*Communications in Mathematics and Statistics*, 5\(1\):1–11, 2017\.
- Furuya et al\. \[2025\]T\. Furuya, M\. V\. de Hoop, and G\. Peyré\.Transformers are universal in\-context learners\.In*International Conference on Learning Representations*, volume 2025, pages 80820–80845, 2025\.
- Geshkovski et al\. \[2023\]B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. Rigollet\.The emergence of clusters in self\-attention dynamics\.*Advances in Neural Information Processing Systems*, 36:57026–57037, 2023\.
- Geshkovski et al\. \[2024\]B\. Geshkovski, P\. Rigollet, and D\. Ruiz\-Balet\.Measure\-to\-measure interpolation using transformers\.*arXiv preprint arXiv:2411\.04551*, 2024\.
- Geshkovski et al\. \[2025\]B\. Geshkovski, C\. Letrouit, Y\. Polyanskiy, and P\. Rigollet\.A mathematical perspective on transformers\.*Bulletin of the American Mathematical Society*, 62\(3\):427–479, 2025\.
- Haber and Ruthotto \[2018\]E\. Haber and L\. Ruthotto\.Stable architectures for deep neural networks\.*Inverse problems*, 34\(1\):014004, 2018\.
- Hahn \[2020\]M\. Hahn\.Theoretical limitations of self\-attention in neural sequence models\.*Transactions of the Association for Computational Linguistics*, 8:156–171, 2020\.
- Handel \[2008\]R\. V\. Handel\.Discrete time nonlinear filters with informative observations are stable\.*Electron\. Commun\. Probab*, 13:562–575, 2008\.
- Jiang and Li \[2024\]H\. Jiang and Q\. Li\.Approximation rate of the transformer architecture for sequence modeling\.*Advances in Neural Information Processing Systems*, 37:68926–68955, 2024\.
- Kajitsuka and Sato \[2024\]T\. Kajitsuka and I\. Sato\.Are transformers with one layer self\-attention using low\-rank weight matrices universal approximators?In*International Conference on Learning Representations*, volume 2024, pages 35177–35205, 2024\.
- Kallenberg \[2021\]O\. Kallenberg\.*Foundations of modern probability*\.Springer, 2021\.
- Kara and Yuksel \[2022\]A\. Kara and S\. Yuksel\.Near optimality of finite memory feedback policies in partially observed Markov decision processes\.*Journal of Machine Learning Research*, 23\(11\):1–46, 2022\.
- Kara and Yüksel \[2024\]A\. D\. Kara and S\. Yüksel\.Partially observed optimal stochastic control: Regularity, optimality, approximations, and learning\.In*2024 IEEE 63rd Conference on Decision and Control \(CDC\)*, pages 6709–6721\. IEEE, 2024\.
- Kara et al\. \[2025\]A\. D\. Kara, E\. Bayraktar, and S\. Yüksel\.Near optimal approximations and finite memory policies for POMDPs with continuous spaces\.*Journal of Systems Science and Complexity*, 38\(1\):238–270, 2025\.
- Kudre et al\. \[2026\]A\. Kudre, H\.\-S\. Chang, and P\. G\. Mehta\.Transformer\-like inference from optimal control\.*arXiv preprint arXiv:2605\.15608*, 2026\.
- Le Gland and Oudjane \[2004\]F\. Le Gland and N\. Oudjane\.Stability and uniform approximation of nonlinear filters using the hilbert metric and application to particle filters\.*The Annals of Applied Probability*, 14\(1\):144–187, 2004\.
- Li and Khaneja \[2009\]J\.\-S\. Li and N\. Khaneja\.Ensemble control of bloch equations\.*IEEE Transactions on Automatic Control*, 54\(3\):528–536, 2009\.
- Lu et al\. \[2018\]Y\. Lu, A\. Zhong, Q\. Li, and B\. Dong\.Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations\.In*International conference on machine learning*, pages 3276–3285\. PMLR, 2018\.
- McDonald and Yüksel \[2020\]C\. McDonald and S\. Yüksel\.Exponential filter stability via Dobrushin’s coefficient\.*Electronic Communations in Probability*, 2020\.
- McDonald and Yüksel \[2024\]C\. McDonald and S\. Yüksel\.Stochastic observability and filter stability under several criteria\.*IEEE Transactions on Automatic Control*, 69\(5\):2931–2946, 2024\.
- McKean Jr \[1966\]H\. P\. McKean Jr\.A class of Markov processes associated with nonlinear parabolic equations\.*Proceedings of the National Academy of Sciences*, 56\(6\):1907–1911, 1966\.
- Merrill and Sabharwal \[2023\]W\. Merrill and A\. Sabharwal\.The parallelism tradeoff: Limitations of log\-precision transformers\.*Transactions of the Association for Computational Linguistics*, 11:531–545, 2023\.
- Mulzer \[2018\]W\. Mulzer\.Five proofs of chernoff’s bound with applications\.*arXiv preprint arXiv:1801\.03365*, 2018\.
- Polyanskiy and Wu \[2025\]Y\. Polyanskiy and Y\. Wu\.*Information theory: From coding to learning*\.Cambridge university press, 2025\.
- Radford et al\. \[2018\]A\. Radford, K\. Narasimhan, T\. Salimans, I\. Sutskever, et al\.Improving language understanding by generative pre\-training\.2018\.
- Rajaraman et al\. \[2024\]N\. Rajaraman, M\. Bondaschi, K\. Ramchandran, M\. Gastpar, and A\. V\. Makkuva\.Transformers on markov data: Constant depth suffices\.*Advances in Neural Information Processing Systems*, 37:137521–137556, 2024\.
- Reznik \[2011\]Y\. A\. Reznik\.An algorithm for quantization of discrete probability distributions\.In*2011 Data Compression Conference*, pages 333–342\. IEEE, 2011\.
- Ruiz\-Balet and Zuazua \[2023\]D\. Ruiz\-Balet and E\. Zuazua\.Neural ODE control for classification, approximation, and transport\.*SIAM Review*, 65\(3\):735–773, 2023\.
- Ruiz\-Balet and Zuazua \[2024\]D\. Ruiz\-Balet and E\. Zuazua\.Control of neural transport for normalising flows\.*Journal de Mathématiques Pures et Appliquées*, 181:58–90, 2024\.
- Ruthotto and Haber \[2020\]L\. Ruthotto and E\. Haber\.Deep neural networks motivated by partial differential equations\.*Journal of Mathematical Imaging and Vision*, 62\(3\):352–364, 2020\.
- Saldi et al\. \[2018\]N\. Saldi, T\. Linder, and S\. Yüksel\.*Finite approximations in discrete\-time stochastic control*\.Springer, 2018\.
- Strobl et al\. \[2024\]L\. Strobl, W\. Merrill, G\. Weiss, D\. Chiang, and D\. Angluin\.What formal languages can transformers express? a survey\.*Transactions of the Association for Computational Linguistics*, 12:543–561, 2024\.
- van der Vaart and Wellner \[1996\]A\. W\. van der Vaart and J\. A\. Wellner\.*Weak Convergence and Empirical Processes: With Applications to Statistics*\.Springer Series in Statistics\. Springer, New York, 1996\.ISBN 978\-0\-387\-94640\-5\.doi:10\.1007/978\-1\-4757\-2545\-2\.
- Walters \[2000\]P\. Walters\.*An introduction to ergodic theory*, volume 79\.Springer Science & Business Media, 2000\.
- Wang et al\. \[2024\]M\. Wang et al\.Understanding the expressive power and mechanisms of transformer for sequence modeling\.*Advances in Neural Information Processing Systems*, 37:25781–25856, 2024\.
- Yüksel \[2025\]S\. Yüksel\.Another look at partially observed optimal stochastic control: Existence, ergodicity, and approximations without belief\-reduction\.*Applied Mathematics & Optimization*, 91\(1\):16, 2025\.
- Yun et al\. \[2020a\]C\. Yun, S\. Bhojanapalli, A\. S\. Rawat, S\. Reddi, and S\. Kumar\.Are transformers universal approximators of sequence\-to\-sequence functions?In*International Conference on Learning Representations*, 2020a\.URL[https://openreview\.net/forum?id=ByxRM0Ntvr](https://openreview.net/forum?id=ByxRM0Ntvr)\.
- Yun et al\. \[2020b\]C\. Yun, Y\.\-W\. Chang, S\. Bhojanapalli, A\. S\. Rawat, S\. Reddi, and S\. Kumar\.O \(n\) connections are expressive enough: Universal approximability of sparse transformers\.*Advances in Neural Information Processing Systems*, 33:13783–13794, 2020b\.
- Zhao et al\. \[2024\]Y\. Zhao, Y\. Qu, K\. Staniszewski, S\. Tworkowski, W\. Liu, P\. Miłoś, Y\. Wu, and P\. Minervini\.Analysing the impact of sequence composition on language model pre\-training\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 7897–7912, 2024\.