Transformers as Bayesian In-Context Experimenters: Smoothness-Adaptive Efficient ATE Estimation

# 1 Introduction
Source: [https://arxiv.org/html/2606.31184](https://arxiv.org/html/2606.31184)
\\RRHSecondLine\\LRHSecondLine

\\RUNTITLE

Transformers as Bayesian In\-Context Experimenters\\TITLETransformers as Bayesian In\-Context Experimenters: Smoothness\-Adaptive Efficient ATE Estimation

\\RUNAUTHOR

Li and Simchi\-Levi\\ARTICLEAUTHORS\\AUTHORJiachun Li\\AFFLaboratory for Information and Decision Systems, MIT,[jiach334@mit\.edu](https://arxiv.org/html/2606.31184v1/[email protected])\\AUTHORDavid Simchi\-Levi\\AFFLaboratory for Information and Decision Systems, MIT,[dslevi@mit\.edu](https://arxiv.org/html/2606.31184v1/[email protected])

###### Abstract

Adaptive experiments for average treatment effects \(ATE\) require randomized allocations balancing valid inference with statistical efficiency\. The oracle design is a covariate\-dependent Neyman rule governed by unknown arm\-conditional outcome variances\. We investigate whether this sequential variance\-estimation and allocation process can be amortized via in\-context learning\. We introduce Bayesian in\-context experimenters: transformer policies trained to imitate a Bayesian posterior Neyman teacher\. The teacher updates nonparametric beliefs over potential outcomes using experimental history to assign posterior Neyman treatment probabilities\. This design converges to the oracle rule, supporting efficient ATE inference\. Transformers constructively implement this mapping through attention\-based sufficient statistics and projected gradient descent, imitating Bayesian updating for Gaussian\-series priors\. To address unknown outcome smoothness, we combine smoothness\-indexed experimenters using a mixture\-of\-experts transformer\. The gate acts as a hierarchical posterior over smoothness classes, concentrating on near\-oracle experts\. By bounding the complexity of the transformer class, we prove this amortized policy can be learned via empirical risk minimization using supervised pretraining\. Experiments confirm accurate teacher imitation, adaptive allocation, and improved ATE precision over baselines\.

Adaptive experimentation is widely used in clinical trials and online platforms to improve the precision of average treatment effect \(ATE\) estimation\(Dai et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib8); Hu and Rosenberger,[2006](https://arxiv.org/html/2606.31184#bib.bib12)\)\. Bayesian experimental design is particularly natural in these settings, as it incorporates prior domain knowledge and historical data into sequential decision\-making\(Chaloner and Verdinelli,[1995](https://arxiv.org/html/2606.31184#bib.bib6); Berry et al\.,[2010](https://arxiv.org/html/2606.31184#bib.bib4)\)\. For ATE inference, the optimal rule is an oracle covariate\-dependent Neyman allocation determined by unknown arm\-conditional residual variances\(Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17); Zhao,[2023](https://arxiv.org/html/2606.31184#bib.bib30); Armstrong,[2022](https://arxiv.org/html/2606.31184#bib.bib1); Kato et al\.,[2020](https://arxiv.org/html/2606.31184#bib.bib15)\)\. A properly specified Bayesian experimenter can recover this rule by updating nonparametric beliefs over potential outcomes: posterior contraction of outcome moments drives the sequential assignment probabilities toward the oracle Neyman design\(Dai et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib8); Cook et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib7); Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17)\)\.

Implementing this Bayesian update\-and\-design process requires both accurate estimation of arm\-conditional outcome moments and efficient translation of posterior uncertainty into treatment\-assignment probabilities\. In high\-dimensional covariate spaces, the main bottleneck is specifying the smoothness of the potential\-outcome models\(van der Vaart and van Zanten,[2009](https://arxiv.org/html/2606.31184#bib.bib27); van Waaij and van Zanten,[2017](https://arxiv.org/html/2606.31184#bib.bib28)\)\. Overly smooth specifications induce bias and misdirect allocation, while overly rough specifications converge slowly\. Adapting to unknown smoothness is therefore essential for posterior accuracy and experimental efficiency\(Szabó et al\.,[2015](https://arxiv.org/html/2606.31184#bib.bib21)\)\.

At the deployment level, Bayesian adaptive design calls for an automated system that maps incoming experimental histories directly into statistically principled assignment probabilities, without per\-experiment engineering\(Tian et al\.,[2021](https://arxiv.org/html/2606.31184#bib.bib22)\)\. Recent work shows that pretrained transformers can act as in\-context Bayesian learners, inferring latent structure from histories without parameter updates\(Bai et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib2); Lin et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib18); Lee et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib16)\)\. We extend this idea to causal experimental design, proposing transformers not as treatment\-effect estimators but as amortized Bayesian experimenters\. By balancing bias and variance, the required nonparametric Bayesian update can be truncated through dynamic effective\-dimension masking, reducing an intractable inference problem to finite\-rank masked ridge regression\. This update is constructively realized by attention\-based sufficient statistics and projected gradient\-descent blocks\. To handle model selection, we use a mixture\-of\-experts \(MoE\) architecture whose gate approximates a hierarchical Bayesian posterior over smoothness classes and concentrates on the appropriate complexity scale as data accumulate\. Together, these mechanisms allow the transformer to learn the history\-to\-propensity map and adapt to the unknown complexity of the potential outcomes\.

Although our construction shows that transformers can represent the Bayesian experimenter, practical models are learned through supervised pretraining rather than manual engineering\. We therefore analyze supervised empirical risk minimization \(ERM\) over finite pretraining trajectories\(Huang et al\.,[2025](https://arxiv.org/html/2606.31184#bib.bib14); Hu et al\.,[2025](https://arxiv.org/html/2606.31184#bib.bib13); Lin et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib18)\)\. Our theory gives an algorithm\-imitation guarantee: the trained transformer mimics the teacher’s history\-to\-propensity behavior up to approximation and estimation errors\. Importantly, the learned model need not recover the exact posterior state or explicitly execute Neyman allocation; pretraining allows it to learn the design map directly from experimental histories\.

Our contributions are as follows:

Bayesian design and smoothness adaptation\.We define and analyze a Bayesian posterior Neyman teacher for efficient ATE\-oriented adaptive experimentation\. The teacher maintains nonparametric Bayesian beliefs over arm\-conditional means and second moments, translating posterior residual\-variance estimates into known assignment probabilities for valid inference\. We further introduce a hierarchical Bayesian extension over smoothness classes and prove that the resulting design adapts to unknown outcome complexity, achieving near\-oracle contraction rates and optimal efficiency\.

Transformer expressivity and generalization\.We show that the Bayesian update\-and\-design map can be represented and learned by transformer architectures\. Constructively, attention\-based sufficient statistics and projected gradient\-descent blocks realize the finite\-rank masked ridge regression update, while an MoE gate implements smoothness adaptation through a hierarchical posterior\. We then establish an ERM generalization theory showing that supervised pretraining over finite trajectories enables the learned transformer to imitate the teacher’s history\-to\-propensity mapping\.

Empirical validation\.On synthetic adaptive trajectories, a single direct\-estimation transformer without an explicit MoE head achieves smoothness\-adaptive minimax rates across seven unseen smoothness levels\. The trained design transformer reproduces Bayesian\-Neyman allocation behavior, including step\-level fluctuation patterns matching the Bayesian teacher during online deployment, without explicitly learning outcome moments\. End\-to\-end, it reduces AIPW ATE estimation MSE relative to uniform randomization and approaches oracle Neyman allocation\.

## 2Formulation and Bayesian Statistical Guarantees

### 2\.1Sequential experiment formulation

We consider a sequential two\-arm experiment with covariates\. At each roundt=1,…,nt=1,\\ldots,n, a unit arrives with covariatesXt∼PXX\_\{t\}\\sim P\_\{X\}on𝒳⊂\[0,1\]d\\mathcal\{X\}\\subset\\mathbb\{\[\}0,1\]^\{d\}with bounded, continuous density\. Conditional on the historyℋt−1=\{\(Xs,Ws,Ys\):s<t\}\\mathcal\{H\}\_\{t\-1\}=\\\{\(X\_\{s\},W\_\{s\},Y\_\{s\}\):s<t\\\}and onXtX\_\{t\}, the experimenter chooses a propensityet​\(Xt\)∈\[η,1−η\],η∈\(0,1/2\),e\_\{t\}\(X\_\{t\}\)\\in\[\\eta,1\-\\eta\],\\,\\eta\\in\(0,1/2\),and assigns

Wt∣Xt,ℋt−1∼Bernoulli​\(et​\(Xt\)\)\.W\_\{t\}\\mid X\_\{t\},\\mathcal\{H\}\_\{t\-1\}\\sim\\mathrm\{Bernoulli\}\(e\_\{t\}\(X\_\{t\}\)\)\.The observed outcome isYt=Yt\(Wt\),Y\_\{t\}=Y\_\{t\}^\{\(W\_\{t\}\)\},whereYt\(0\),Yt\(1\)Y\_\{t\}^\{\(0\)\},Y\_\{t\}^\{\(1\)\}are the potential outcomes\. For each armw∈\{0,1\}w\\in\\\{0,1\\\}, define

μw⋆​\(x\)=𝔼​\[Y\(w\)∣X=x\],qw⋆​\(x\)=𝔼​\[\(Y\(w\)\)2∣X=x\],\\mu\_\{w\}^\{\\star\}\(x\)=\\mathbb\{E\}\[Y^\{\(w\)\}\\mid X=x\],\\qquad q\_\{w\}^\{\\star\}\(x\)=\\mathbb\{E\}\[\(Y^\{\(w\)\}\)^\{2\}\\mid X=x\],and variancevw⋆​\(x\)=qw⋆​\(x\)−\{μw⋆​\(x\)\}2\.v\_\{w\}^\{\\star\}\(x\)=q\_\{w\}^\{\\star\}\(x\)\-\\\{\\mu\_\{w\}^\{\\star\}\(x\)\\\}^\{2\}\.The estimand is the marginal average treatment effect

τ⋆=𝔼X∼PX​\[μ1⋆​\(X\)−μ0⋆​\(X\)\]\.\\tau^\{\\star\}=\\mathbb\{E\}\_\{X\\sim P\_\{X\}\}\[\\mu\_\{1\}^\{\\star\}\(X\)\-\\mu\_\{0\}^\{\\star\}\(X\)\]\.
###### Assumption 1\(Basic regularity\)

We assume that the conditional variances satisfy0<cV≤vw⋆​\(x\)≤CV<∞,w∈\{0,1\}\.0<c\_\{V\}\\leq v\_\{w\}^\{\\star\}\(x\)\\leq C\_\{V\}<\\infty,\\,w\\in\\\{0,1\\\}\.Also assume the fourth conditional moment𝔼​\[\(Y\(w\)\)4∣X=x\]≤Cq\\mathbb\{E\}\[\(Y^\{\(w\)\}\)^\{4\}\\mid X=x\]\\leq C\_\{q\}is bounded\.

### 2\.2A nonparametric Bayesian moment belief

The Bayesian teacher maintains beliefs over the conditional meanμw⋆​\(x\)=𝔼​\[Y\(w\)∣X=x\]\\mu\_\{w\}^\{\\star\}\(x\)=\\mathbb\{E\}\[Y^\{\(w\)\}\\mid X=x\]and the conditional second momentqw⋆​\(x\)=𝔼​\[\(Y\(w\)\)2∣X=x\]q\_\{w\}^\{\\star\}\(x\)=\\mathbb\{E\}\[\(Y^\{\(w\)\}\)^\{2\}\\mid X=x\]\. The former is needed for ATE inference; the latter induces the residual variancevw⋆​\(x\)=qw⋆​\(x\)−\(μw⋆​\(x\)\)2v\_\{w\}^\{\\star\}\(x\)=q\_\{w\}^\{\\star\}\(x\)\-\(\\mu\_\{w\}^\{\\star\}\(x\)\)^\{2\}used in posterior Neyman allocation\. WheneverWt=wW\_\{t\}=w,

Yt=μw⋆​\(Xt\)\+ξt,μ,w,Yt2=qw⋆​\(Xt\)\+ξt,q,w,Y\_\{t\}=\\mu\_\{w\}^\{\\star\}\(X\_\{t\}\)\+\\xi\_\{t,\\mu,w\},\\qquad Y\_\{t\}^\{2\}=q\_\{w\}^\{\\star\}\(X\_\{t\}\)\+\\xi\_\{t,q,w\},with

𝔼​\[ξt,μ,w∣Xt,Wt=w,ℋt−1\]=𝔼​\[ξt,q,w∣Xt,Wt=w,ℋt−1\]=0\.\\mathbb\{E\}\[\\xi\_\{t,\\mu,w\}\\mid X\_\{t\},W\_\{t\}=w,\\mathcal\{H\}\_\{t\-1\}\]=\\mathbb\{E\}\[\\xi\_\{t,q,w\}\\mid X\_\{t\},W\_\{t\}=w,\\mathcal\{H\}\_\{t\-1\}\]=0\.Note that the noise in the first and second moment equations can generally be non\-Gaussian and heteroskedastic\.

Regular Basis and Smoothness prior\.Fix an orthonormal basis\{ψj:j≥1\}\\\{\\psi\_\{j\}:j\\geq 1\\\}ofL2​\(PX\)L\_\{2\}\(P\_\{X\}\), ordered by a frequency scaleλj≍j2/d\\lambda\_\{j\}\\asymp j^\{2/d\}, such as a Fourier or Laplacian eigenbasis\(Shen and Ghosal,[2015](https://arxiv.org/html/2606.31184#bib.bib20)\)\. Everyf∈L2​\(PX\)f\\in L\_\{2\}\(P\_\{X\}\)has an expansionf​\(x\)=∑j=1∞θj​ψj​\(x\)\.f\(x\)=\\sum\_\{j=1\}^\{\\infty\}\\theta\_\{j\}\\psi\_\{j\}\(x\)\.We say thatffhas smoothnessβ\\betaif∑j=1∞\(1\+λj\)β​θj2<∞\.\\sum\_\{j=1\}^\{\\infty\}\(1\+\\lambda\_\{j\}\)^\{\\beta\}\\theta\_\{j\}^\{2\}<\\infty\.Thus smoothness is encoded by decay of high\-frequency coefficients: largerβ\\betameans faster decay and hence smoother functions\.

###### Assumption 2\(Moment smoothness and tails\)

For eachw∈\{0,1\}w\\in\\\{0,1\\\},μw⋆\\mu\_\{w\}^\{\\star\}andqw⋆q\_\{w\}^\{\\star\}have smoothness levelsβw,μ⋆,βw,q⋆\>0\\beta\_\{w,\\mu\}^\{\\star\},\\beta\_\{w,q\}^\{\\star\}\>0in the basis above\. The noisesξt,μ,w=Yt−μw⋆​\(Xt\),ξt,q,w=Yt2−qw⋆​\(Xt\)\\xi\_\{t,\\mu,w\}=Y\_\{t\}\-\\mu\_\{w\}^\{\\star\}\(X\_\{t\}\),\\,\\xi\_\{t,q,w\}=Y\_\{t\}^\{2\}\-q\_\{w\}^\{\\star\}\(X\_\{t\}\)are conditionally sub\-exponential, uniformly overt,wt,w\.

For fixedβ\>0\\beta\>0, place independent Gaussian\-series priors onμw\\mu\_\{w\}andqwq\_\{w\}\. For example,

μw​\(x\)=∑j=1∞aw,μ,j​ψj​\(x\),aw,μ,j​∼ind​N​\(0,ρj​\(β\)\),\\mu\_\{w\}\(x\)=\\sum\_\{j=1\}^\{\\infty\}a\_\{w,\\mu,j\}\\psi\_\{j\}\(x\),\\qquad a\_\{w,\\mu,j\}\\overset\{\\mathrm\{ind\}\}\{\\sim\}N\(0,\\rho\_\{j\}\(\\beta\)\),

and analogously forqwq\_\{w\}, whereρj​\(β\)≍j−1−2​β/d\.\\rho\_\{j\}\(\\beta\)\\asymp j^\{\-1\-2\\beta/d\}\.Changingβ\\betachanges the prior variance assigned to high\-frequency coordinates\. Smallerβ\\betaallows rougher functions, while largerβ\\betashrinks high\-frequency coordinates more aggressively\.

Given arm\-wwobservations, the teacher updates its mean belief through the simplified Gaussian model

Yt∣Xt,Wt=w,μw​∼model​N​\(μw​\(Xt\),σ¯μ2\),Y\_\{t\}\\mid X\_\{t\},W\_\{t\}=w,\\mu\_\{w\}\\overset\{\\mathrm\{model\}\}\{\\sim\}N\(\\mu\_\{w\}\(X\_\{t\}\),\\bar\{\\sigma\}\_\{\\mu\}^\{2\}\),whereσ¯μ2\>0\\bar\{\\sigma\}\_\{\\mu\}^\{2\}\>0is a user\-chosen tuning constant for the update, not the true noise variance\. This gives

d​Πμ,w,t\(β\)​\(μ\)∝exp⁡\{−12​σ¯μ2​∑s≤t:Ws=w\(Ys−μ​\(Xs\)\)2\}​d​Πμ,w,0\(β\)​\(μ\)\.d\\Pi\_\{\\mu,w,t\}^\{\(\\beta\)\}\(\\mu\)\\propto\\exp\\left\\\{\-\\frac\{1\}\{2\\bar\{\\sigma\}\_\{\\mu\}^\{2\}\}\\sum\_\{s\\leq t:W\_\{s\}=w\}\\bigl\(Y\_\{s\}\-\\mu\(X\_\{s\}\)\\bigr\)^\{2\}\\right\\\}d\\Pi\_\{\\mu,w,0\}^\{\(\\beta\)\}\(\\mu\)\.

The second\-moment belief is updated analogously by replacingYsY\_\{s\},μ\\mu, andσ¯μ2\\bar\{\\sigma\}\_\{\\mu\}^\{2\}withYs2Y\_\{s\}^\{2\},qq, andσ¯q2\\bar\{\\sigma\}\_\{q\}^\{2\}\. While these Gaussian models may be misspecified, the key point is that their squared\-loss population targets are still the correct conditional momentsμw⋆\\mu\_\{w\}^\{\\star\}andqw⋆q\_\{w\}^\{\\star\}; under Assumption[2](https://arxiv.org/html/2606.31184#Thmassumption2), the resulting beliefs contract to these targets at the usual nonparametric rates\.

### 2\.3Posterior contraction

LetNt,wN\_\{t,w\}be the arm\-specific sample size up to periodtt\. The clipped design impliesNt,w≍tN\_\{t,w\}\\asymp twith high probability\. Sinceete\_\{t\}is predictable, the arm\-wwobservations form an adaptive regression sample with martingale difference noise and overlap bounded away from zero\.

###### Proposition 1\(Moment posterior contraction\)

Suppose Assumptions[1](https://arxiv.org/html/2606.31184#Thmassumption1)and[2](https://arxiv.org/html/2606.31184#Thmassumption2)hold\. Consider any predictable adaptive design withet​\(x\)∈\[η,1−η\]e\_\{t\}\(x\)\\in\[\\eta,1\-\\eta\]\. If the prior smoothness in \([2\.2](https://arxiv.org/html/2606.31184#S2.SS2)\) is matched to the corresponding true smoothness, then for sufficiently large constantMM,

Πμ,w,t\(β\)\(∥μw−μw⋆∥L2​\(PX\)\>Mϵμ,w,t\|ℋt\)→0,Πq,w,t\(β\)\(∥qw−qw⋆∥L2​\(PX\)\>Mϵq,w,t\|ℋt\)→0,\\Pi\_\{\\mu,w,t\}^\{\(\\beta\)\}\\left\(\\\|\\mu\_\{w\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\epsilon\_\{\\mu,w,t\}\\,\\middle\|\\,\\mathcal\{H\}\_\{t\}\\right\)\\to 0,\\quad\\Pi\_\{q,w,t\}^\{\(\\beta\)\}\\left\(\\\|q\_\{w\}\-q\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\epsilon\_\{q,w,t\}\\,\\middle\|\\,\\mathcal\{H\}\_\{t\}\\right\)\\to 0,where

ϵμ,w,t≍Nt,w−βw,μ⋆/\(2​βw,μ⋆\+d\),ϵq,w,t≍Nt,w−βw,q⋆/\(2​βw,q⋆\+d\)\\epsilon\_\{\\mu,w,t\}\\asymp N\_\{t,w\}^\{\-\\beta\_\{w,\\mu\}^\{\\star\}/\(2\\beta\_\{w,\\mu\}^\{\\star\}\+d\)\},\\qquad\\epsilon\_\{q,w,t\}\\asymp N\_\{t,w\}^\{\-\\beta\_\{w,q\}^\{\\star\}/\(2\\beta\_\{w,q\}^\{\\star\}\+d\)\}are the nonparametric convergence rate\. In other words, the posterior belief will concentrate on the local ball\{μω:‖μω−μω∗‖L2≲ϵμ,ω,t\}\\\{\\mu\_\{\\omega\}:\\\|\\mu\_\{\\omega\}\-\\mu^\{\*\}\_\{\\omega\}\\\|\_\{L\_\{2\}\}\\lesssim\\epsilon\_\{\\mu,\\omega,t\}\\\}\.

The proof is given in Appendix[8](https://arxiv.org/html/2606.31184#S8)\. It adapts standard Gaussian\-series posterior contraction arguments to squared\-loss Bayesian beliefs under predictable adaptive sampling\. Non\-Gaussian and heteroskedastic noise does not affect the contraction rate, provided the sub\-exponential tail condition holds\.

### 2\.4Posterior Neyman Allocation and Semiparametric Efficiency

We now turn posterior moment consistency into an adaptive design guarantee and an efficient ATE inference guarantee\. Given posterior draws\(μw,qw\)\(\\mu\_\{w\},q\_\{w\}\), define the induced residual\-variance draw by

vw​\(x\)=Π\[cV,CV\]​\{qw​\(x\)−μw​\(x\)2\},v\_\{w\}\(x\)=\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\\left\\\{q\_\{w\}\(x\)\-\\mu\_\{w\}\(x\)^\{2\}\\right\\\},whereΠ\[cV,CV\]\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}is pointwise projection onto\[cV,CV\]\[c\_\{V\},C\_\{V\}\]\. Letv¯w,t​\(x\)=𝔼Πt​\[vw​\(x\)∣ℋt\]\\bar\{v\}\_\{w,t\}\(x\)=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[v\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]denote the posterior variance belief\. The Bayesian teacher assigns the next unit according to the posterior Neyman propensity

etB​\(x\)=clip\[η,1−η\]​\{v¯1,t−1​\(x\)v¯0,t−1​\(x\)\+v¯1,t−1​\(x\)\}\.e\_\{t\}^\{\\mathrm\{B\}\}\(x\)=\\mathrm\{clip\}\_\{\[\\eta,1\-\\eta\]\}\\left\\\{\\frac\{\\sqrt\{\\bar\{v\}\_\{1,t\-1\}\(x\)\}\}\{\\sqrt\{\\bar\{v\}\_\{0,t\-1\}\(x\)\}\+\\sqrt\{\\bar\{v\}\_\{1,t\-1\}\(x\)\}\}\\right\\\}\.

The oracle counterpart ise⋆​\(x\)=v1⋆​\(x\)v0⋆​\(x\)\+v1⋆​\(x\)\.e^\{\\star\}\(x\)=\\frac\{\\sqrt\{v\_\{1\}^\{\\star\}\(x\)\}\}\{\\sqrt\{v\_\{0\}^\{\\star\}\(x\)\}\+\\sqrt\{v\_\{1\}^\{\\star\}\(x\)\}\}\.Since the variance has upper and lower bound, the oracle propensity does not need a clip\.

###### Proposition 2\(Design convergence\)

Under the conditions of Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1),

‖v¯w,t−vw⋆‖L2​\(PX\)→0,w∈\{0,1\}\.\\\|\\bar\{v\}\_\{w,t\}\-v\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\to 0,\\qquad w\\in\\\{0,1\\\}\.Consequently, the posterior propensity score converges to Neyman allocation‖etB−e⋆‖L2​\(PX\)→0\.\\\|e\_\{t\}^\{\\mathrm\{B\}\}\-e^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\to 0\.

The adaptive design produces propensitieset​\(Xt\)e\_\{t\}\(X\_\{t\}\)at every assignment time\. After the experiment, we conduct inference using the standard augmented inverse\-probability weighted \(AIPW\) estimator\. Given posterior mean estimatesμ^0,μ^1\\widehat\{\\mu\}\_\{0\},\\widehat\{\\mu\}\_\{1\}, define

τ^AIPW=1n​∑t=1n\[μ^1​\(Xt\)−μ^0​\(Xt\)\+Wt​\{Yt−μ^1​\(Xt\)\}et​\(Xt\)−\(1−Wt\)​\{Yt−μ^0​\(Xt\)\}1−et​\(Xt\)\]\.\\widehat\{\\tau\}\_\{\\rm AIPW\}=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\left\[\\widehat\{\\mu\}\_\{1\}\(X\_\{t\}\)\-\\widehat\{\\mu\}\_\{0\}\(X\_\{t\}\)\+\\frac\{W\_\{t\}\\\{Y\_\{t\}\-\\widehat\{\\mu\}\_\{1\}\(X\_\{t\}\)\\\}\}\{e\_\{t\}\(X\_\{t\}\)\}\-\\frac\{\(1\-W\_\{t\}\)\\\{Y\_\{t\}\-\\widehat\{\\mu\}\_\{0\}\(X\_\{t\}\)\\\}\}\{1\-e\_\{t\}\(X\_\{t\}\)\}\\right\]\.The use of known, clipped propensities ensures valid inverse\-probability correction, while posterior contraction of the moment beliefs makes the nuisance estimation error asymptotically negligible\. Equivalently, one may view this as an adjusted Bayesian belief overτ⋆\\tau^\{\\star\}by applying the same one\-step AIPW correction to posterior draws of\(μ0,μ1\)\(\\mu\_\{0\},\\mu\_\{1\}\)\. Thus the Bayesian posterior Neyman rule is used to collect the data efficiently; the final reported ATE can be formed by the usual frequentist one\-step estimator or by its adjusted Bayesian analogue\.

###### Theorem 3\(Efficient ATE inference under posterior Neyman allocation\)

Suppose the moment posteriors contract as in Proposition 1 and the posterior Neyman propensities converge to the oracle allocation as in Proposition 2\. Then the AIPW estimator based on the adaptive known propensities satisfies

n​\(τ^AIPW−τ⋆\)↝N​\(0,V⋆\),\\sqrt\{n\}\(\\widehat\{\\tau\}\_\{\\rm AIPW\}\-\\tau^\{\\star\}\)\\rightsquigarrow N\(0,V^\{\\star\}\),whereV⋆V^\{\\star\}is the oracle semiparametric efficiency bound attained by the Neyman allocation, which is the smallest variance that can be possibly achieved by any regular design\-estimator pair\.

The proof of the asymptotic normality and efficiency is in Appendix[10](https://arxiv.org/html/2606.31184#S10)\.

### 2\.5Smoothness adaptation by hierarchical Bayes

The contraction result above assumes that the prior smoothness is matched to the truth\. We now allow thesmoothness to be unknown\. Letβ∼Πβ,\\beta\\sim\\Pi\_\{\\beta\},whereΠβ\\Pi\_\{\\beta\}has a density bounded away from zero and infinity on a compact interval\[βmin,βmax\]⊂\(0,∞\)\.\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]\\subset\(0,\\infty\)\.Conditional onβ\\beta, the mean function prior is the Gaussian\-series prior

μw​\(x\)=∑j≥1aw,μ,j​ψj​\(x\),aw,μ,j∣β​∼ind​N​\(0,ρj​\(β\)\),ρj​\(β\)≍j−1−2​β/d\.\\mu\_\{w\}\(x\)=\\sum\_\{j\\geq 1\}a\_\{w,\\mu,j\}\\psi\_\{j\}\(x\),\\qquad a\_\{w,\\mu,j\}\\mid\\beta\\overset\{\\mathrm\{ind\}\}\{\\sim\}N\(0,\\rho\_\{j\}\(\\beta\)\),\\qquad\\rho\_\{j\}\(\\beta\)\\asymp j^\{\-1\-2\\beta/d\}\.

The hierarchical posterior forμw\\mu\_\{w\}is therefore

d​Πμ,w,t​\(μ,β\)∝exp⁡\{−12​σ¯μ2​∑s≤t:Ws=w\(Ys−μ​\(Xs\)\)2\}​d​Πμ,w,0\(β\)​\(μ\)​d​Πβ​\(β\)\.d\\Pi\_\{\\mu,w,t\}\(\\mu,\\beta\)\\propto\\exp\\bigg\\\{\-\\frac\{1\}\{2\\bar\{\\sigma\}\_\{\\mu\}^\{2\}\}\\sum\_\{s\\leq t:W\_\{s\}=w\}\\bigl\(Y\_\{s\}\-\\mu\(X\_\{s\}\)\\bigr\)^\{2\}\\bigg\\\}d\\Pi\_\{\\mu,w,0\}^\{\(\\beta\)\}\(\\mu\)\\,d\\Pi\_\{\\beta\}\(\\beta\)\.

The same hierarchy is used forqwq\_\{w\}, replacingYsY\_\{s\},μ\\mu, andσ¯μ2\\bar\{\\sigma\}\_\{\\mu\}^\{2\}byYs2Y\_\{s\}^\{2\},qq, andσ¯q2\\bar\{\\sigma\}\_\{q\}^\{2\}\.

###### Theorem 4\(Contraction rate adaptive to smoothness\)

Suppose Assumptions[1](https://arxiv.org/html/2606.31184#Thmassumption1)and[2](https://arxiv.org/html/2606.31184#Thmassumption2)hold, and supposeβw,μ⋆,βw,q⋆∈\[βmin,βmax\]\.\\beta\_\{w,\\mu\}^\{\\star\},\\ \\beta\_\{w,q\}^\{\\star\}\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]\.Then the hierarchical posterior forμw\\mu\_\{w\}contracts at the oracle smoothness rate:

Πμ,w,t\(∥μw−μw⋆∥L2​\(PX\)\>Mϵμ,w,tad\|ℋt\)→0,\\Pi\_\{\\mu,w,t\}\\left\(\\\|\\mu\_\{w\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\epsilon\_\{\\mu,w,t\}^\{\\mathrm\{ad\}\}\\,\\middle\|\\,\\mathcal\{H\}\_\{t\}\\right\)\\to 0,where

ϵμ,w,tad≍Nt,w−βw,μ⋆/\(2​βw,μ⋆\+d\)\\epsilon\_\{\\mu,w,t\}^\{\\mathrm\{ad\}\}\\asymp N\_\{t,w\}^\{\-\\beta\_\{w,\\mu\}^\{\\star\}/\(2\\beta\_\{w,\\mu\}^\{\\star\}\+d\)\}up to logarithmic factors\. The same statement holds forqwq\_\{w\}\. Equivalently, the posterior overβ\\betaconcentrates on smoothness values whose induced contraction rate is within logarithmic factors of the oracle rate\.

This adaptation is useful because smoothness determines the bias–variance trade\-off in the moment beliefs: rough priors are robust but slow, while overly smooth priors can be biased\. Hierarchical Bayes lets the design automatically borrow information at the right scale, yielding near\-oracle rates without knowing the smoothness in advance\. Combining Theorem[4](https://arxiv.org/html/2606.31184#Thmtheorem4)with Proposition[2](https://arxiv.org/html/2606.31184#Thmtheorem2)gives smoothness\-adaptive design consistency:‖etHB−e⋆‖L2​\(PX\)→0,\\\|e\_\{t\}^\{\\mathrm\{HB\}\}\-e^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\to 0,whereetHBe\_\{t\}^\{\\mathrm\{HB\}\}is the posterior Neyman propensity formed under the hierarchical smoothness posterior\.

## 3Transformer Expressivity and Generalization

Section 2 defined a Bayesian teacher that maps experimental histories to posterior moment beliefs and Neyman propensities\. We now connect this teacher to transformer computation\. First, we constructively show that a transformer can explicitly represent this experimenter: self\-attention compiles sufficient statistics, while tokenwise FFNs execute masked ridge updates and Neyman allocation\. Second, embedding this realization into a standard trainable transformer class ensures supervised pretraining can learn the adaptive\-experimentation map via empirical risk minimization \(ERM\)\. Crucially, the trained model only needs to imitate the teacher’s input\-output behavior, without recovering its internal posterior state\.

### 3\.1Expressive power: estimation and adaptive design

Transformer construction overview\.Figure 1 illustrates this correspondence\. After pointwise input encoding, attention blocks aggregate historical tokens to compute empirical response statistics and covariance\-vector products\. Tokenwise FFNs then execute coordinate masking, prior shrinkage, and projected\-gradient updates for the finite\-rank Bayesian ridge problem\. ChainingO​\(log⁡n\)O\(\\log n\)blocks yields a posterior\-mean approximation for a generic targetm⋆​\(x\)=𝔼​\[Z∣X=x\]m^\{\\star\}\(x\)=\\mathbb\{E\}\[Z\\mid X=x\]\. We reuse this exact module to estimate arm\-conditional moments \(settingZ=YZ=Yforμw\\mu\_\{w\}andZ=Y2Z=Y^\{2\}forqwq\_\{w\}\)\. Finally, a lightweight head maps the estimates\(μ^0,μ^1,q^0,q^1\)\(\\widehat\{\\mu\}\_\{0\},\\widehat\{\\mu\}\_\{1\},\\widehat\{q\}\_\{0\},\\widehat\{q\}\_\{1\}\)to a clipped Neyman propensity\. A detailed description of the transformer architecture is in Appendix[12](https://arxiv.org/html/2606.31184#S12)\.

![Refer to caption](https://arxiv.org/html/2606.31184v1/transformer_illustration.png)Figure 1:An illustration of the transformer and its correspondence with the Bayesian experimenterCommon basis and finite truncation\.While the Bayesian teacher is infinite\-dimensional, for a finite horizonnnit can be truncated to a maximum dimensionJmax,nJ\_\{\\max,n\}\. This maximum dimension corresponds to the roughest smoothness levelβmin\\beta\_\{\\min\}in the expert library:

Jmax,n≍nd/\(2​βmin\+d\)\.J\_\{\\max,n\}\\asymp n^\{d/\(2\\beta\_\{\\min\}\+d\)\}\.\(1\)Coordinates beyond this threshold contribute only higher\-order truncation bias relative to the nonparametric rate, eliminating the need for infinitely long features\. All smoothness experts share this exact same feature vectorϕn​\(x\)\\phi\_\{n\}\(x\); different smoothness levels alter only the prior variances \(regularization\), not the input representation\.

Dynamic effective\-dimension mask\.For a given timett, armww, and expertkk, the architecture activates a dynamic number of coordinates:

Jt,w,k=max⁡\{J≤Jmax,n:Nt,w≳J​log⁡J,ρJ,k−1≲Nt,w\}\.J\_\{t,w,k\}=\\max\\\{J\\leq J\_\{\\max,n\}:N\_\{t,w\}\\gtrsim J\\log J,\\rho\_\{J,k\}^\{\-1\}\\lesssim N\_\{t,w\}\\\}\.\(2\)This deterministic mask naturally implements a sample\-size\-dependent bias\-variance tradeoff and reflects the Bayesian effective dimension: a smalltt, a smoother expert, or a poorly sampled arm will activate fewer coordinates\. The first condition guarantees that the Gram matrix is well conditioned, and the second condition guarantees that the information learned from data is not dominated by prior belief\. Under these two conditions, the finite dimension update is a well\-conditioned ridge regression\.

Masked ridge update\.The finite\-rank Bayesian posterior mean update can be written as a masked ridge objective:

a^t,w,h,k=arg⁡mina=Pt,w,k​a⁡\{12​Nt,w​∑s≤t:Ws=w\(Zs,h−ϕn​\(Xs\)⊤​a\)2\+σ¯h22​Nt,w​a⊤​Λk−1​a\}\.\\widehat\{a\}\_\{t,w,h,k\}=\\arg\\min\_\{a=P\_\{t,w,k\}a\}\\left\\\{\\frac\{1\}\{2N\_\{t,w\}\}\\sum\_\{s\\leq t:W\_\{s\}=w\}\(Z\_\{s,h\}\-\\phi\_\{n\}\(X\_\{s\}\)^\{\\top\}a\)^\{2\}\+\\frac\{\\bar\{\\sigma\}\_\{h\}^\{2\}\}\{2N\_\{t,w\}\}a^\{\\top\}\\Lambda\_\{k\}^\{\-1\}a\\right\\\}\.

\(3\)Because of the common basis, the sufficient statisticsSt,wS\_\{t,w\}and the response statisticbt,w,hb\_\{t,w,h\}are shared across all smoothness experts\. The only components specific to expertkkare the dynamic maskPt,w,kP\_\{t,w,k\}and the prior precisionΛk−1\\Lambda\_\{k\}^\{\-1\}\.

Transformer implementation\.Because the objective is quadratic, the gradient is linear in the coefficient vector\. The termSt,w​aS\_\{t,w\}acan be computed using linear attention or read from cached sufficient\-statistic memory, while the prior termΛk−1​a\\Lambda\_\{k\}^\{\-1\}ais a fixed diagonal map\. Thus, one block implements a projected gradient descent \(GD\) update:

aℓ\+1=Pt,w,k​\[aℓ−η​\(Ht,w,h,k​aℓ−ct,w,h,k\)\]\.a^\{\\ell\+1\}=P\_\{t,w,k\}\\left\[a^\{\\ell\}\-\\eta\(H\_\{t,w,h,k\}a^\{\\ell\}\-c\_\{t,w,h,k\}\)\\right\]\.\(4\)The projection is applied at every step to ensure the optimization remains strictly on the active coordinates\.

Depth and conditioning\.The active\-dimension rule guarantees that the masked Hessian is well\-conditioned: the conditionNt,w≳Jt,w,k​log⁡Jt,w,kN\_\{t,w\}\\gtrsim J\_\{t,w,k\}\\log J\_\{t,w,k\}stabilizes the empirical covariance, whileρJ,k−1≲Nt,w\\rho\_\{J,k\}^\{\-1\}\\lesssim N\_\{t,w\}prevents the prior precision from dominating the curvature\. Therefore, projected GD exhibits geometric convergence, andO​\(log⁡n\)O\(\\log n\)blocks are sufficient to make the optimization error strictly below the statistical contraction rate\.

MoE for smoothness\-adaptive estimation\.To adapt to unknown smoothness for our generic targetm⋆m^\{\\star\}, we use a mixture\-of\-experts \(MoE\) architecture\. Each expertkkcorresponds to a smoothnessβk\\beta\_\{k\}, using the sameϕn\\phi\_\{n\}but a different mask andΛk\\Lambda\_\{k\}\. The MoE gate acts as an efficient approximation to the hierarchical posterior over smoothness: each expert is weighted by an approximate marginal likelihood score that can be computed from attention\-based sufficient statistics and a simple active\-complexity penalty\. The output is the hierarchical posterior mean:m^t​\(x\)=∑k=1Knωk,t​m^k,t​\(x\)\.\\widehat\{m\}\_\{t\}\(x\)=\\sum\_\{k=1\}^\{K\_\{n\}\}\\omega\_\{k,t\}\\widehat\{m\}\_\{k,t\}\(x\)\.

With all these parts, in the following theorem we show that the transformer we construct can conduct Bayesian adaptive experiment with negligible error\.

###### Theorem 5\(Smoothness\-adaptive estimation transformer\)

Under bounded basis functions, overlap, dynamic masks, andO​\(log⁡n\)O\(\\log n\)projected\-GD depth, there exists a structured estimation transformer whose output approximates the finite\-rank smoothness\-mixture posterior mean uniformly over arms, moment targets, smoothness experts, and time points, with negligible approximation errorδn\\delta\_\{n\}\. Therefore, for each armw∈\{0,1\}w\\in\\\{0,1\\\}and moment targeth∈\{μ,q\}h\\in\\\{\\mu,q\\\},

‖m^w,h,t−mw,h⋆‖L2​\(PX\)=O~p​\(Nt,w−βw,h⋆/\(2​βw,h⋆\+d\)\)\.\\\|\\widehat\{m\}\_\{w,h,t\}\-m^\{\\star\}\_\{w,h\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=\\widetilde\{O\}\_\{p\}\\\!\\left\(N\_\{t,w\}^\{\-\\beta^\{\\star\}\_\{w,h\}/\(2\\beta^\{\\star\}\_\{w,h\}\+d\)\}\\right\)\.\(5\)Heremw,μ⋆=μw⋆m^\{\\star\}\_\{w,\\mu\}=\\mu^\{\\star\}\_\{w\},mw,q⋆=qw⋆m^\{\\star\}\_\{w,q\}=q^\{\\star\}\_\{w\},Zs,μ=YsZ\_\{s,\\mu\}=Y\_\{s\}, andZs,q=Ys2Z\_\{s,q\}=Y\_\{s\}^\{2\}\. Thus the statement applies to both arm\-conditional means and second moments\.

Neyman allocation layer\.The estimation transformer producesUt​\(x\)=\(μ^0,t,μ^1,t,q^0,t,q^1,t\)​\(x\)U\_\{t\}\(x\)=\(\\widehat\{\\mu\}\_\{0,t\},\\widehat\{\\mu\}\_\{1,t\},\\widehat\{q\}\_\{0,t\},\\widehat\{q\}\_\{1,t\}\)\(x\)\. In the structured realization, an admissible head is the fixed Neyman map: it computesvw=Π\[cV,CV\]​\(qw−μw2\)v\_\{w\}=\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\(q\_\{w\}\-\\mu\_\{w\}^\{2\}\), formse=v1/\(v0\+v1\)e=\\sqrt\{v\_\{1\}\}/\(\\sqrt\{v\_\{0\}\}\+\\sqrt\{v\_\{1\}\}\), and clips the result to\[η,1−η\]\[\\eta,1\-\\eta\]\. By standard ReLU approximation results, a feedforward ReLU network can uniformly approximate it\(Yarotsky,[2017](https://arxiv.org/html/2606.31184#bib.bib29)\)\. At deployment, treatment is sampled from the learned Bernoulli policy:Wt∼Bernoulli​\(πθ​\(1∣ℋt−1,Xt\)\)W\_\{t\}\\sim\{\\rm Bernoulli\}\(\\pi\_\{\\theta\}\(1\\mid\\mathcal\{H\}\_\{t\-1\},X\_\{t\}\)\)\.

### 3\.2Generalization ability of pretrained transformer via ERM

While the preceding construction proves that a standard transformer*can*explicitly compute Bayesian updates via sufficient\-statistic attention and projected gradient descent, we do not enforce this internal mechanism during training\. Instead, the constructed network serves strictly as an expressivity comparator—guaranteeing that the model class contains at least one parameter configuration capable of approximating the Bayesian teacher\. The trained model is ultimately only required to imitate the teacher’s input\-output behavior\.

Given a regression prefix𝒟N=\{\(Xi,Zi\)\}i=1N\\mathcal\{D\}\_\{N\}=\\\{\(X\_\{i\},Z\_\{i\}\)\\\}\_\{i=1\}^\{N\}and a queryxx, we construct history tokensTi=\(Xi,Zi,i/N,N/n,history\)T\_\{i\}=\(X\_\{i\},Z\_\{i\},i/N,N/n,\\mathrm\{history\}\)and a query tokenTQ=\(x,N/n,query\)T\_\{Q\}=\(x,N/n,\\mathrm\{query\}\)\. A pointwise encoder \(withPencP\_\{\\rm enc\}parameters\) maps these raw inputs toDnD\_\{n\}\-dimensional embeddings, which are then processed byLnL\_\{n\}shared transformer blocks\. Intuitively,*the self\-attention mechanism acts as a learned in\-context kernel smoother*, aggregating information from historical tokens that are most relevant to the query\. While our constructive realization relies on attention heads to compute empirical moments \(e\.g\.,N−1​∑iϕ​\(Xi\)​ZiN^\{\-1\}\\sum\_\{i\}\\phi\(X\_\{i\}\)Z\_\{i\}\) and FFN layers to execute coordinatewise projections and gradient steps, the trained network is never explicitly supervised to recover these intermediate mathematical objects\.

To enable smoothness adaptation, the network produces a2​Kn2K\_\{n\}\-dimensional readout\. The firstKnK\_\{n\}coordinates supply smoothness\-indexed predictionsm^θ,1​\(x\),…,m^θ,Kn​\(x\)\\widehat\{m\}\_\{\\theta,1\}\(x\),\\ldots,\\widehat\{m\}\_\{\\theta,K\_\{n\}\}\(x\), while the remainingKnK\_\{n\}coordinates provide corresponding evidence logitsℓ^θ,1,…,ℓ^θ,Kn\\widehat\{\\ell\}\_\{\\theta,1\},\\ldots,\\widehat\{\\ell\}\_\{\\theta,K\_\{n\}\}\. We combine these via a parameter\-free softmax to yield mixture weightsω^θ,k=exp⁡\(ℓ^θ,k\)/∑r=1Knexp⁡\(ℓ^θ,r\),\\widehat\{\\omega\}\_\{\\theta,k\}=\\exp\(\\widehat\{\\ell\}\_\{\\theta,k\}\)/\\sum\_\{r=1\}^\{K\_\{n\}\}\\exp\(\\widehat\{\\ell\}\_\{\\theta,r\}\),producing the final estimatorm^θ​\(x\)=∑k=1Knω^θ,k​m^θ,k​\(x\)\\widehat\{m\}\_\{\\theta\}\(x\)=\\sum\_\{k=1\}^\{K\_\{n\}\}\\widehat\{\\omega\}\_\{\\theta,k\}\\widehat\{m\}\_\{\\theta,k\}\(x\)\.

Parameter scaleWe first detail the parameter count of our explicitly constructed transformer before embedding it into the broader trainable class optimized via empirical risk minimization \(ERM\)\.

Beyond the initial pointwise token encoder, the constructive architecture comprises three distinct stages\. First, a constant number of statistic\-attention blocks aggregate history\-level summaries, such as the empirical moments and the covariance\-vector products

N−1​∑iϕ​\(Xi\)​Zi,N−1​∑iϕ​\(Xi\)​\{ϕ​\(Xi\)⊤​a\}\.N^\{\-1\}\\sum\_\{i\}\\phi\(X\_\{i\}\)Z\_\{i\},\\quad N^\{\-1\}\\sum\_\{i\}\\phi\(X\_\{i\}\)\\\{\\phi\(X\_\{i\}\)^\{\\top\}a\\\}\.

Second, a mask\-preparation block dynamically constructs the sample\-size\-dependent coordinate masks\. Third, a sequence ofLnL\_\{n\}projected\-GD blocks iteratively refines the active coefficient vector\. Within each GD block, the attention sublayer handles cross\-token aggregation, while the tokenwise FFN executes the within\-token algebraic steps: dynamic masking, diagonal prior shrinkage, linear gradient updates, and active\-coordinate projections\. Because these within\-token operations are coordinatewise or linear piecewise\-linear functions, they are realized by ReLU FFNs with a width proportional to the block dimension\. Therefore, each of theLnL\_\{n\}consecutive GD block containsO​\(Dn2\)O\(D\_\{n\}^\{2\}\)parameters\. The smoothness\-adaptive readout requires an affine matrix of size2​Kn×Dn2K\_\{n\}\\times D\_\{n\}for expert predictions and evidence logits, addingO​\(Kn​Dn\)O\(K\_\{n\}D\_\{n\}\)\. Thus, the total parameter count isPn=Penc\+O​\(Ln​Dn2\)\+O​\(Kn​Dn\)\.P\_\{n\}=P\_\{\\rm enc\}\+O\(L\_\{n\}D\_\{n\}^\{2\}\)\+O\(K\_\{n\}D\_\{n\}\)\.

Crucially, this constructed network represents just one specific configuration within the broader trainable transformer class with norm\-bounded transformers sharing the same token encoder, widthDnD\_\{n\}, depth proportional toLnL\_\{n\}, and2​Kn2K\_\{n\}\-dimensional readout\. Substituting the structural requirementsDn≍JnD\_\{n\}\\asymp J\_\{n\}, whereJn≍ndx/\(2​βmin\+dx\)J\_\{n\}\\asymp n^\{d\_\{x\}/\(2\\beta\_\{\\min\}\+d\_\{x\}\)\}, alongside the logarithmic depthsLn=O​\(log⁡n\)L\_\{n\}=O\(\\log n\)andKn=O​\(log⁡n\)K\_\{n\}=O\(\\log n\), the total parameter complexity simplifies to:Pn=Penc\+O~​\(Jn2\)\.P\_\{n\}=P\_\{\\rm enc\}\+\\widetilde\{O\}\(J\_\{n\}^\{2\}\)\.

Supervised pretraining\.During supervised pretraining, we observeNpreN\_\{\\rm pre\}independent training trajectories\{𝒟i\}i=1Npre\\\{\\mathcal\{D\}\_\{i\}\\\}\_\{i=1\}^\{N\_\{\\rm pre\}\}, each of lengthnn\. For theiith trajectory, let𝒟i,t\\mathcal\{D\}\_\{i,t\}denote its prefix of lengthtt\. Given a query covariatexx, the Bayesian teacher provides the posterior\-mean labelmi,tB​\(x\)m^\{B\}\_\{i,t\}\(x\)\. The transformer is trained to imitate this prefix\-to\-prediction map through the empirical risk

R^est​\(θ\)=1Npre​∑i=1Npre1n​∑t=1n𝔼x∼PX​\[Fθ​\(𝒟i,t,x\)−mi,tB​\(x\)\]2\.\\widehat\{R\}\_\{\\rm est\}\(\\theta\)=\\frac\{1\}\{N\_\{\\rm pre\}\}\\sum\_\{i=1\}^\{N\_\{\\rm pre\}\}\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\_\{x\\sim P\_\{X\}\}\\left\[F\_\{\\theta\}\(\\mathcal\{D\}\_\{i,t\},x\)\-m^\{B\}\_\{i,t\}\(x\)\\right\]^\{2\}\.

\(6\)In practice, the expectation overxxis approximated by sampled query covariates\. For the smoothness\-MoE readout, the network’s adaptation to smoothness can be driven either implicitly or explicitly\. We can train the model purely end\-to\-end via the final mixture outputFθF\_\{\\theta\}, allowing it to organically learn the routing strategy without internal constraints\. Conversely, we can explicitly supervise the individual expert estimates and their posterior likelihoods, forcing the network to explicitly implement the theoretical MoE mechanism\. Both approaches are valid, as explicit auxiliary losses are not strictly required for the fundamental performance ERM guarantee\.

Let𝒯n\\mathcal\{T\}\_\{n\}denote the resulting norm\-bounded transformer class\. Existing covering\-number bounds ensure its statistical complexity isComp​\(𝒯n\)=O~​\(Pn\)\\mathrm\{Comp\}\(\\mathcal\{T\}\_\{n\}\)=\\widetilde\{O\}\(P\_\{n\}\), absorbing logarithmic and depth\-dependent Lipschitz factors\(Trauger and Tewari,[2024](https://arxiv.org/html/2606.31184#bib.bib23); Bai et al\.,[2023](https://arxiv.org/html/2606.31184#bib.bib2); Bartlett et al\.,[2017](https://arxiv.org/html/2606.31184#bib.bib3)\)\. Ifθ^\\widehat\{\\theta\}minimizes the empirical pretraining risk overNpreN\_\{\\rm pre\}independent trajectories, learning theory guarantees that, with probability at least1−δ1\-\\delta:

Rest​\(θ^\)≤infθ∈𝒯nRest​\(θ\)\+C​Comp​\(𝒯n\)\+log⁡\(1/δ\)Npre\.R\_\{\\rm est\}\(\\widehat\{\\theta\}\)\\leq\\inf\_\{\\theta\\in\\mathcal\{T\}\_\{n\}\}R\_\{\\rm est\}\(\\theta\)\+C\\sqrt\{\\frac\{\\mathrm\{Comp\}\(\\mathcal\{T\}\_\{n\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.

Because our explicitly constructed Bayesian\-update transformer lives within𝒯n\\mathcal\{T\}\_\{n\}, it serves as a valid comparator\. Ultimately, supervised pretraining successfully distills the Bayesian estimation algorithm into an efficient input\-output mapping, completely bypassing the need for the network to explicitly track internal posterior states, sufficient statistics, or projected\-gradient iterates\. The guarantee for the design transformer follows similarly, with one additional FFN layer for Neyman allocation and the supervised pretraining loss function

R^pol​\(θ\)=1Npre​∑i=1Npre1n​∑t=1n\[−Wi,tB​log⁡e^θ​\(Xi,t∣ℋi,t−1\)−\(1−Wi,tB\)​log⁡\{1−e^θ​\(Xi,t∣ℋi,t−1\)\}\],\\widehat\{R\}\_\{\\rm pol\}\(\\theta\)=\\frac\{1\}\{N\_\{\\rm pre\}\}\\sum\_\{i=1\}^\{N\_\{\\rm pre\}\}\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\left\[\-W^\{B\}\_\{i,t\}\\log\\widehat\{e\}\_\{\\theta\}\(X\_\{i,t\}\\mid\\mathcal\{H\}\_\{i,t\-1\}\)\-\(1\-W^\{B\}\_\{i,t\}\)\\log\\\{1\-\\widehat\{e\}\_\{\\theta\}\(X\_\{i,t\}\\mid\\mathcal\{H\}\_\{i,t\-1\}\)\\\}\\right\],

whereWi,tB∈\{0,1\}W^\{B\}\_\{i,t\}\\in\\\{0,1\\\}is the Bayesian teacher\-sampled treatment action, ande^θ​\(Xi,t∣ℋi,t−1\)=πθ​\(1∣ℋi,t−1,Xi,t\)\\widehat\{e\}\_\{\\theta\}\(X\_\{i,t\}\\mid\\mathcal\{H\}\_\{i,t\-1\}\)=\\pi\_\{\\theta\}\(1\\mid\\mathcal\{H\}\_\{i,t\-1\},X\_\{i,t\}\)is the model’s predicted propensity\.

## 4Experiments

We empirically validate three claims: \(i\) a single transformer, trained only on Bayesian mixture posterior labels and without an explicit MoE head, attains smoothness\-adaptive rates; \(ii\) a design transformer reproduces Bayesian\-Neyman allocation behavior without linearly representing the outcome moments used by the teacher; \(iii\) when deployed online, the learned policy improves downstream ATE estimation relative to uniform randomization, including under out\-of\-distribution variance ratios\.

Setup\.We generate one\-dimensional covariatesX∼Unif​\(\[0,1\]\)X\\sim\\mathrm\{Unif\}\(\[0,1\]\)and smooth potential outcomes from a cosine basis with coefficient decayθj∝j−\(β/d\+1/2\)​ξj\\theta\_\{j\}\\propto j^\{\-\(\\beta/d\+1/2\)\}\\xi\_\{j\},ξj∼N​\(0,1\)\\xi\_\{j\}\\sim N\(0,1\), whereβ∈\{0\.5,0\.75,1\.0,1\.5,2\.0,2\.5,3\.0\}\\beta\\in\\\{0\.5,0\.75,1\.0,1\.5,2\.0,2\.5,3\.0\\\}is sampled per task\. For design experiments, the arm\-conditional variancesσw2​\(⋅\)\\sigma\_\{w\}^\{2\}\(\\cdot\)are independent smooth functions with random arm asymmetry and clipping\. We train two direct transformers: an estimation transformer labeled by the Bayesian mixture posterior mean, and a design transformer labeled by the Bayesian\-Neyman propensity\. Both useD=512D=512,L=8L=8, andH=8H=8heads, with prefixes up tonest=256n\_\{\\rm est\}=256andndes=512n\_\{\\rm des\}=512\. Both transformers are optimized with AdamW \(weight decay10−410^\{\-4\}, gradient clip1\.01\.0, 500\-step warmup then cosine decay\) on a single A100 GPU; Estimation uses lr3×10−43\{\\times\}10^\{\-4\}, batch 256, BF16 autocast for 5,000 steps, while design uses lr3×10−53\{\\times\}10^\{\-5\}, batch 64, FP32 for 7,000 steps\.

Smoothness adaptation without an explicit MoE\.Figure[3](https://arxiv.org/html/2606.31184#S4.F3)shows that the estimation transformer achieves the correct smoothness\-adaptive scaling on held\-out trajectories\. Across the seven smoothness levels, the empirical log–log slopes\{−0\.45,−0\.55,−0\.59,−0\.77,−0\.87,−0\.93,−0\.95\}\\\{\-0\.45,\-0\.55,\-0\.59,\-0\.77,\-0\.87,\-0\.93,\-0\.95\\\}closely match the Bayesian\-minimax rates \. This includes interpolation smoothness levels unseen during training\. Thus, smoothness selection emerges from the Bayesian\-mixture training signal itself: the model is never givenβ\\betaand has no explicit smoothness mixture head\.

Neyman behavior without explicit moment representation\.We next test whether the design transformer has learned variance\-sensitive Neyman allocation rather than a shortcut rule\. In a counterfactual intervention, we inflate residuals in one arm while holdingXX,WW, and empirical arm means fixed\. Increasing arm\-11variance raises the propensity in95\.6%95\.6\\%of samples, while increasing arm\-0variance lowers it in99\.8%99\.8\\%of samples\. This confirms that the learned policy respondsdirectionally to arm\-specific residual variance\.

Layerwise probes reveal a sharper insight\. The propensityeebecomes linearly decodable in late layers, withR2R^\{2\}up to0\.780\.78, but the conditional meansμ0,μ1\\mu\_\{0\},\\mu\_\{1\}and the second momentsq0,q1q\_\{0\},q\_\{1\}are only not correlated, thus not explicitly learned in latent states\. Hence the transformer reproduces Neyman\-style allocation without explicitly representing the teacher’s moment\-estimation pipeline\. It learns the history\-to\-propensity map directly\.

Online deployment and ATE precision\.We deploy the trained design transformer forn=256n=256steps onK=50K=50fresh environments and compare it with the Bayesian teacher on identical covariate streams\. Figure[3](https://arxiv.org/html/2606.31184#S4.F3)plots the error to the oracle Neyman propensity\. The transformer and teacher follow nearly identical convergence paths through early and intermediate time steps\. The transformer is slightly better at very small samples because its meta\-learned prior is informative, while the teacher initially defaults toward0\.50\.5\. After roughlyt=100t=100, the teacher continues improving, whereas the transformer plateaus around an imitation gap of0\.040\.04\. Notably, the two policies display matching step\-level fluctuations, suggesting that the transformer has learnedthe teacher’s conditional response pattern, not merely its average behavior\.

![Refer to caption](https://arxiv.org/html/2606.31184v1/rate_true_direct_estimation.png)Figure 2:Smoothness\-adaptive estimation rates\.
![Refer to caption](https://arxiv.org/html/2606.31184v1/convergence.png)Figure 3:Online convergence to oracle Neyman allocation\.

Finally, we evaluate downstream ATE estimation on a outcome distribution family with variance ratios up to approximately100×100\\times, outside the training distribution\. Table[1](https://arxiv.org/html/2606.31184#S4.T1)reports MSE overK=100K=100adaptive experiments\. The design transformer improves over uniform randomization by1\.101\.10–1\.26×1\.26\\times, showing that the learned allocation policy remains value\-relevant out of distribution\. The performance is close to the oracle Neyman allocation, which assigns the optimal propensity score\.

Table 1:ATE estimation MSE on a held\-out environment family with variance ratios up to∼100×\{\\sim\}100\\times\.
## 5Conclusion, discussion, and broader impact

We study transformers as amortized Bayesian in\-context experimenters for efficient ATE estimation\. The proposed policy imitates a posterior Neyman teacher, adapts to unknown outcome smoothness through a smoothness\-mixture mechanism, and preserves valid inference through known clipped propensities\. Our results suggest that pretrained sequence models can learn statistically principled experimental designs directly from histories, reducing the need to solve a full Bayesian design problem separately for each deployment\. The current study is limited to two\-arm experiments and synthetic environments; extending the approach to richer action spaces, delayed outcomes, real experimental logs, and misspecified pretraining distributions remains important\. The broader social benefit is improved data efficiency in domains such as digital health, education, and online services\.

\{APPENDICES\}

## 6Notation, regularity preliminaries, and probabilistic toolkit

This appendix collects the conventions, the predictable filtration, and the concentration inequalities used in every later appendix\. Every later appendix quotes a numbered lemma stated here so that the proofs are self\-contained\.

### 6\.1Notation

For a random variableUUwith values in a Euclidean space,‖U‖2\\\|U\\\|\_\{2\}denotes its Euclidean norm and‖U‖op\\\|U\\\|\_\{\\rm op\}denotes the operator norm ifUUis matrix\-valued\. The Orliczψ1\\psi\_\{1\}\-norm is

‖U‖ψ1=inf\{λ\>0:𝔼​exp⁡\(\|U\|/λ\)≤2\},\\\|U\\\|\_\{\\psi\_\{1\}\}=\\inf\\\{\\lambda\>0:\\mathbb\{E\}\\exp\(\|U\|/\\lambda\)\\leq 2\\\},and the conditional Orlicz norm given aσ\\sigma\-algebra𝒢\\mathcal\{G\}is defined analogously by replacing𝔼\\mathbb\{E\}with𝔼\[⋅∣𝒢\]\\mathbb\{E\}\[\\cdot\\mid\\mathcal\{G\}\]and the infimum by an essential infimum\. We writean≲bna\_\{n\}\\lesssim b\_\{n\}if there exists a constantC\>0C\>0, depending only on the structural constants\(d,cX,CX,η,βmin,βmax,cV,CV,Cq,τ\)\(d,c\_\{X\},C\_\{X\},\\eta,\\beta\_\{\\min\},\\beta\_\{\\max\},c\_\{V\},C\_\{V\},C\_\{q\},\\tau\), such thatan≤C​bna\_\{n\}\\leq Cb\_\{n\}for allnnlarge enough;an≍bna\_\{n\}\\asymp b\_\{n\}if bothan≲bna\_\{n\}\\lesssim b\_\{n\}andbn≲anb\_\{n\}\\lesssim a\_\{n\}\. The notationO~\\widetilde\{O\}hides at most polylogarithmic factors innn\. For a functiong:𝒳→ℝg:\\mathcal\{X\}\\to\\mathbb\{R\},‖g‖L2​\(PX\)2=∫g2​𝑑PX\\\|g\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}=\\int g^\{2\}\\,dP\_\{X\}and‖g‖N2=N−1​∑i=1Ng​\(Xi\)2\\\|g\\\|\_\{N\}^\{2\}=N^\{\-1\}\\sum\_\{i=1\}^\{N\}g\(X\_\{i\}\)^\{2\}\.

### 6\.2Predictable filtration

We work on a single probability space supporting the entire experimental trajectory and the Bayesian teacher\. Define the filtration

ℱ0=σ​\(∅\),ℱt=σ​\(\{Xs,Ws,Ys\}s≤t−1\),ℱtX=σ​\(ℱt∪\{Xt\}\),ℱtW=σ​\(ℱtX∪\{Wt\}\)\.\\mathcal\{F\}\_\{0\}=\\sigma\(\\emptyset\),\\quad\\mathcal\{F\}\_\{t\}=\\sigma\\bigl\(\\\{X\_\{s\},W\_\{s\},Y\_\{s\}\\\}\_\{s\\leq t\-1\}\\bigr\),\\quad\\mathcal\{F\}\_\{t\}^\{X\}=\\sigma\\bigl\(\\mathcal\{F\}\_\{t\}\\cup\\\{X\_\{t\}\\\}\\bigr\),\\quad\\mathcal\{F\}\_\{t\}^\{W\}=\\sigma\\bigl\(\\mathcal\{F\}\_\{t\}^\{X\}\\cup\\\{W\_\{t\}\\\}\\bigr\)\.\(7\)Equivalently,ℱt=σ​\(ℋt−1\)\\mathcal\{F\}\_\{t\}=\\sigma\(\\mathcal\{H\}\_\{t\-1\}\)\. The propensityet​\(⋅\)e\_\{t\}\(\\cdot\)isℱt\\mathcal\{F\}\_\{t\}\-measurable \(it is a function of the past\),XtX\_\{t\}isℱtX\\mathcal\{F\}\_\{t\}^\{X\}\-measurable,WtW\_\{t\}isℱtW\\mathcal\{F\}\_\{t\}^\{W\}\-measurable, andYtY\_\{t\}isℱt\+1\\mathcal\{F\}\_\{t\+1\}\-measurable\. By construction𝔼​\[Wt∣ℱtX\]=et​\(Xt\),\\mathbb\{E\}\[W\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]=e\_\{t\}\(X\_\{t\}\),and𝔼​\[Yt∣ℱtW\]=μWt⋆​\(Xt\)\.\\mathbb\{E\}\[Y\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{W\}\]=\\mu\_\{W\_\{t\}\}^\{\\star\}\(X\_\{t\}\)\.This is the predictable filtration on which all martingale arguments below are based\.

### 6\.3Sub\-exponential calculus

A random variableUUis\(σ,K\)\(\\sigma,K\)\-sub\-exponential conditional on𝒢\\mathcal\{G\}if

𝔼​\[exp⁡\{λ​\(U−𝔼​\[U∣𝒢\]\)\}\|𝒢\]≤exp⁡\(σ2​λ2/2\)∀\|λ\|≤1/K\.\\mathbb\{E\}\\bigl\[\\exp\\\{\\lambda\(U\-\\mathbb\{E\}\[U\\mid\\mathcal\{G\}\]\)\\\}\\,\\big\|\\,\\mathcal\{G\}\\bigr\]\\leq\\exp\\bigl\(\\sigma^\{2\}\\lambda^\{2\}/2\\bigr\)\\qquad\\forall\\,\|\\lambda\|\\leq 1/K\.Under Assumption[2](https://arxiv.org/html/2606.31184#Thmassumption2), bothξt,μ,w=Yt−μw⋆​\(Xt\)\\xi\_\{t,\\mu,w\}=Y\_\{t\}\-\\mu\_\{w\}^\{\\star\}\(X\_\{t\}\)andξt,q,w=Yt2−qw⋆​\(Xt\)\\xi\_\{t,q,w\}=Y\_\{t\}^\{2\}\-q\_\{w\}^\{\\star\}\(X\_\{t\}\)are conditionally sub\-exponential given\(ℱtX,Wt=w\)\(\\mathcal\{F\}\_\{t\}^\{X\},W\_\{t\}=w\), with parameters\(σξ,Kξ\)\(\\sigma\_\{\\xi\},K\_\{\\xi\}\)uniform int,wt,w\. Throughoutτ:=max⁡\(σξ,Kξ\)\\tau:=\\max\(\\sigma\_\{\\xi\},K\_\{\\xi\}\)denotes the corresponding constant\.

###### Lemma 6\(Bernstein for sub\-exponential MDS\)

Let\(Di,𝒢i\)i≥1\(D\_\{i\},\\mathcal\{G\}\_\{i\}\)\_\{i\\geq 1\}be a martingale\-difference sequence:𝔼​\[Di∣𝒢i−1\]=0\\mathbb\{E\}\[D\_\{i\}\\mid\\mathcal\{G\}\_\{i\-1\}\]=0and‖Di‖ψ1∣𝒢i−1≤K\\\|D\_\{i\}\\\|\_\{\\psi\_\{1\}\\mid\\mathcal\{G\}\_\{i\-1\}\}\\leq Ka\.s\. LetVt=∑i=1t𝔼​\[Di2∣𝒢i−1\]V\_\{t\}=\\sum\_\{i=1\}^\{t\}\\mathbb\{E\}\[D\_\{i\}^\{2\}\\mid\\mathcal\{G\}\_\{i\-1\}\]be the predictable quadratic variation\. Then for everyu\>0u\>0,

Pr⁡\(\|∑i=1tDi\|\>u,Vt≤V\)≤2​exp⁡\(−c​min⁡\{u2V,uK\}\)\\Pr\\\!\\left\(\\Bigl\|\\sum\_\{i=1\}^\{t\}D\_\{i\}\\Bigr\|\>u,\\,V\_\{t\}\\leq V\\right\)\\leq 2\\exp\\\!\\left\(\-c\\min\\\!\\left\\\{\\frac\{u^\{2\}\}\{V\},\\,\\frac\{u\}\{K\}\\right\\\}\\right\)for an absolute constantc\>0c\>0\.

Sketch\.Standard Freedman\-type Bernstein bound forψ1\\psi\_\{1\}MDS; see, e\.g\.,\(Geer,[2000](https://arxiv.org/html/2606.31184#bib.bib9)\)Lemma 8\.2 or\(Boucheron et al\.,[2003](https://arxiv.org/html/2606.31184#bib.bib5)\)Theorem 6\.1 adapted to the predictable quadratic variation via a stopping\-time argument\.□\\square

###### Lemma 7\(Hoeffding/Azuma for bounded MDS\)

If\(Di,𝒢i\)\(D\_\{i\},\\mathcal\{G\}\_\{i\}\)is an MDS with\|Di\|≤K\|D\_\{i\}\|\\leq Ka\.s\., thenPr⁡\(\|∑i=1tDi\|\>u\)≤2​exp⁡\(−u2/\(2​t​K2\)\)\\Pr\(\|\\sum\_\{i=1\}^\{t\}D\_\{i\}\|\>u\)\\leq 2\\exp\(\-u^\{2\}/\(2tK^\{2\}\)\)\.

### 6\.4Matrix Freedman

###### Lemma 8\(Matrix Freedman,Tropp \([2011](https://arxiv.org/html/2606.31184#bib.bib24)\)\)

Let\(Δi,𝒢i\)i=1t\(\\Delta\_\{i\},\\mathcal\{G\}\_\{i\}\)\_\{i=1\}^\{t\}be a sequence of self\-adjointJ×JJ\\times Jmatrices with𝔼​\[Δi∣𝒢i−1\]=0\\mathbb\{E\}\[\\Delta\_\{i\}\\mid\\mathcal\{G\}\_\{i\-1\}\]=0and‖Δi‖op≤R\\\|\\Delta\_\{i\}\\\|\_\{\\rm op\}\\leq Ra\.s\. SetWt=∑i=1t𝔼​\[Δi2∣𝒢i−1\]W\_\{t\}=\\sum\_\{i=1\}^\{t\}\\mathbb\{E\}\[\\Delta\_\{i\}^\{2\}\\mid\\mathcal\{G\}\_\{i\-1\}\]andσ2=‖Wt‖op\\sigma^\{2\}=\\\|W\_\{t\}\\\|\_\{\\rm op\}\. Then for everyu\>0u\>0,

Pr⁡\(‖∑i=1tΔi‖op≥u,‖Wt‖op≤σ2\)≤2​J​exp⁡\(−u2/2σ2\+R​u/3\)\.\\Pr\\\!\\left\(\\Bigl\\\|\\sum\_\{i=1\}^\{t\}\\Delta\_\{i\}\\Bigr\\\|\_\{\\rm op\}\\geq u,\\,\\\|W\_\{t\}\\\|\_\{\\rm op\}\\leq\\sigma^\{2\}\\right\)\\leq 2J\\exp\\\!\\left\(\-\\frac\{u^\{2\}/2\}\{\\sigma^\{2\}\+Ru/3\}\\right\)\.

### 6\.5Le Cam–Birgé tests

###### Lemma 9\(Adaptive Le Cam–Birgé test\)

Letℱ\\mathcal\{F\}be a class of measurable functionsf:𝒳→ℝf:\\mathcal\{X\}\\to\\mathbb\{R\}uniformly bounded byBB\. Supposeξi\\xi\_\{i\}is conditionally\(σξ,Kξ\)\(\\sigma\_\{\\xi\},K\_\{\\xi\}\)\-sub\-exponential givenℱi−1X\\mathcal\{F\}\_\{i\-1\}^\{X\}, with𝔼​\[ξi∣ℱi−1X\]=0\\mathbb\{E\}\[\\xi\_\{i\}\\mid\\mathcal\{F\}\_\{i\-1\}^\{X\}\]=0andZi=m⋆​\(Xi\)\+ξiZ\_\{i\}=m^\{\\star\}\(X\_\{i\}\)\+\\xi\_\{i\}\. For everyρ\>0\\rho\>0and anyf∈ℱf\\in\\mathcal\{F\}with‖f−m⋆‖L2​\(PX\)\>ρ\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>\\rho, there exists a testTf∈\{0,1\}T\_\{f\}\\in\\\{0,1\\\}measurable with respect to\(Xi,Zi\)i≤N\(X\_\{i\},Z\_\{i\}\)\_\{i\\leq N\}satisfying

ℙm⋆​\(Tf=1\)≤2​exp⁡\(−c​N​ρ2\),ℙf​\(Tf=0\)≤2​exp⁡\(−c​N​ρ2\),\\mathbb\{P\}\_\{m^\{\\star\}\}\(T\_\{f\}=1\)\\leq 2\\exp\(\-cN\\rho^\{2\}\),\\quad\\mathbb\{P\}\_\{f\}\(T\_\{f\}=0\)\\leq 2\\exp\(\-cN\\rho^\{2\}\),providedρ2≤σξ2\\rho^\{2\}\\leq\\sigma\_\{\\xi\}^\{2\}andN​ρ2≥Kξ2N\\rho^\{2\}\\geq K\_\{\\xi\}^\{2\}\. The constantccdepends only on\(B,σξ,Kξ\)\(B,\\sigma\_\{\\xi\},K\_\{\\xi\}\)\.

The proof is the squared\-residual likelihood ratioTf=𝟏​\{N−1​∑\(Zi−f\)2−\(Zi−m⋆\)2≤−12​ρ2\}T\_\{f\}=\\mathbf\{1\}\\\{N^\{\-1\}\\sum\(Z\_\{i\}\-f\)^\{2\}\-\(Z\_\{i\}\-m^\{\\star\}\)^\{2\}\\leq\-\\frac\{1\}\{2\}\\rho^\{2\}\\\}, combined with Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)on∑ξi​\(f​\(Xi\)−m⋆​\(Xi\)\)\\sum\\xi\_\{i\}\\bigl\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\\bigr\), the empirical\-norm equivalence from Lemma[10](https://arxiv.org/html/2606.31184#Thmtheorem10)below, and the squared\-loss bias identity\. The detailed argument is reproduced inside Appendix[8](https://arxiv.org/html/2606.31184#S8)\.

### 6\.6Empirical norm equivalence on a finite\-dimensional class

###### Lemma 10\(Empirical\-vs\-population norm equivalence\)

Let𝒱⊂L2​\(PX\)\\mathcal\{V\}\\subset L\_\{2\}\(P\_\{X\}\)be aJJ\-dimensional linear space of bounded functions admitting an orthonormal basisϕJ\\phi\_\{J\}such thatsupx‖ϕJ​\(x\)‖22≤κ​J\\sup\_\{x\}\\\|\\phi\_\{J\}\(x\)\\\|\_\{2\}^\{2\}\\leq\\kappa J\. Suppose the design\(Xi,Wi\)\(X\_\{i\},W\_\{i\}\)satisfies the predictable conditional Riesz bound

η​ΓJ⪯𝔼​\[Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤∣ℱi−1\]⪯\(1−η\)​ΓJ,\\eta\\,\\Gamma\_\{J\}\\preceq\\mathbb\{E\}\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]\\preceq\(1\-\\eta\)\\,\\Gamma\_\{J\},withcX​I⪯ΓJ⪯CX​Ic\_\{X\}I\\preceq\\Gamma\_\{J\}\\preceq C\_\{X\}I\. Then there exist constantsC,c\>0C,c\>0depending only on\(cX,CX,η,κ\)\(c\_\{X\},C\_\{X\},\\eta,\\kappa\)such that, on an event of probability at least1−2​J​exp⁡\(−c​Nt,w/J\)1\-2J\\exp\(\-cN\_\{t,w\}/J\),

12​‖g‖L2​\(PX\)2≤1Nt,w​∑i≤t:Wi=wg​\(Xi\)2≤32​‖g‖L2​\(PX\)2∀g∈𝒱\.\\tfrac\{1\}\{2\}\\\|g\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq\\tfrac\{1\}\{N\_\{t,w\}\}\\sum\_\{i\\leq t:W\_\{i\}=w\}g\(X\_\{i\}\)^\{2\}\\leq\\tfrac\{3\}\{2\}\\\|g\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\qquad\\forall g\\in\\mathcal\{V\}\.

The proof is Appendix[7](https://arxiv.org/html/2606.31184#S7)\(Lemma[14](https://arxiv.org/html/2606.31184#Thmtheorem14)\)\.

### 6\.7Triangular\-array martingale CLT

###### Lemma 11\(Hall–Heyde Theorem 3\.2\(Hall and Heyde,[2014](https://arxiv.org/html/2606.31184#bib.bib11)\)\)

Let\(ξn,t,𝒢n,t\)1≤t≤n\(\\xi\_\{n,t\},\\mathcal\{G\}\_\{n,t\}\)\_\{1\\leq t\\leq n\}be a triangular array of square\-integrable martingale differences\. Suppose:

1. 1\.*Conditional variance:*there is a positive random variableVVwith∑t=1n𝔼​\[ξn,t2∣𝒢n,t−1\]→𝑃V\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\[\\xi\_\{n,t\}^\{2\}\\mid\\mathcal\{G\}\_\{n,t\-1\}\]\\xrightarrow\{P\}V\.
2. 2\.*Conditional Lindeberg:*for everyε\>0\\varepsilon\>0,∑t=1n𝔼​\[ξn,t2​𝟏​\{\|ξn,t\|\>ε\}∣𝒢n,t−1\]→𝑃0\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\[\\xi\_\{n,t\}^\{2\}\\mathbf\{1\}\\\{\|\\xi\_\{n,t\}\|\>\\varepsilon\\\}\\mid\\mathcal\{G\}\_\{n,t\-1\}\]\\xrightarrow\{P\}0\.

Then∑t=1nξn,t↝Z\\sum\_\{t=1\}^\{n\}\\xi\_\{n,t\}\\rightsquigarrow Zstably, whereZ∣V∼N​\(0,V\)Z\\mid V\\sim N\(0,V\)\. In particular ifVVis constant, then∑t=1nξn,t↝N​\(0,V\)\\sum\_\{t=1\}^\{n\}\\xi\_\{n,t\}\\rightsquigarrow N\(0,V\)\.

### 6\.8Standing assumptions and quantities

We collect the constants used downstream:

cX,CX\\displaystyle c\_\{X\},C\_\{X\}∈\(0,∞\)\\displaystyle\\in\(0,\\infty\)density bounds on​PX\\displaystyle\\text\{density bounds on \}P\_\{X\}cV,CV\\displaystyle c\_\{V\},C\_\{V\}∈\(0,∞\)\\displaystyle\\in\(0,\\infty\)variance bounds \(Assumption[1](https://arxiv.org/html/2606.31184#Thmassumption1)\)η\\displaystyle\\eta∈\(0,1/2\)\\displaystyle\\in\(0,1/2\)clip thresholdβmin,βmax\\displaystyle\\beta\_\{\\min\},\\beta\_\{\\max\}smoothness rangeτ\\displaystyle\\tausub\-exponential constant for​ξt,μ,w,ξt,q,w\\displaystyle\\text\{sub\-exponential constant for \}\\xi\_\{t,\\mu,w\},\\xi\_\{t,q,w\}κ\\displaystyle\\kappabasis envelope:​supx‖ϕJ​\(x\)‖22≤κ​J\\displaystyle\\text\{basis envelope: \}\\sup\_\{x\}\\\|\\phi\_\{J\}\(x\)\\\|\_\{2\}^\{2\}\\leq\\kappa JR\\displaystyle RSobolev radius:​m⋆∈ℋβ⋆​\(R\)\.\\displaystyle\\text\{Sobolev radius: \}m^\{\\star\}\\in\\mathcal\{H\}^\{\\beta^\{\\star\}\}\(R\)\.For the Fourier/Laplacian basis on\[0,1\]d\[0,1\]^\{d\},κ\\kappais an absolute constant\. Together with bounded density these constants yield the geometric facts of Appendix[7](https://arxiv.org/html/2606.31184#S7)\.

## 7Basis geometry and adaptive Gram concentration

##### Roadmap\.

The Bayesian update \([2\.2](https://arxiv.org/html/2606.31184#S2.Ex8)\) is, on the active subspace of sizeJJ, a finite\-dimensional ridge regression with design matrixΦ=\[ϕJ​\(Xi\)\]i\\Phi=\[\\phi\_\{J\}\(X\_\{i\}\)\]\_\{i\}and GramΓ^J=N−1​Φ⊤​Φ\\widehat\{\\Gamma\}\_\{J\}=N^\{\-1\}\\Phi^\{\\top\}\\Phi\. Every later contraction or rate argument needs this Gram to be uniformly well\-conditioned — bounded eigenvalues both above and below — so that inversion is stable and empirical norms are equivalent toL2​\(PX\)L\_\{2\}\(P\_\{X\}\)norms on the active class\. This appendix proves uniform well\-conditionedness in three escalating settings:

1. 1\.*Population*\(Lemma[12](https://arxiv.org/html/2606.31184#Thmtheorem12)\): bounded density on\[0,1\]d\[0,1\]^\{d\}pulls back the Lebesgue orthogonality of the cosine basis to a Riesz bound onΓJ\\Gamma\_\{J\}\.
2. 2\.*IID empirical*\(Lemma[13](https://arxiv.org/html/2606.31184#Thmtheorem13)\): matrix Bernstein on iid rank\-one matrices, valid as soon asN≳J​log⁡JN\\gtrsim J\\log J\.
3. 3\.*Adaptive martingale*\(Lemma[14](https://arxiv.org/html/2606.31184#Thmtheorem14)\): for arm\-specific covariates collected under the predictable Bayesian propensity, the Gram increments form a matrix MDS; matrix Freedman delivers the same conclusion underNt,w≳J​log⁡JN\_\{t,w\}\\gtrsim J\\log J\.

Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)unions these across all relevant\(J,t,w\)\(J,t,w\)to produce the single high\-probability event𝒢n\\mathcal\{G\}\_\{n\}on which every subsequent appendix argument operates\.

##### Why the thresholdN≳J​log⁡JN\\gtrsim J\\log J\.

Each rank\-one summandϕJ​\(Xi\)​ϕJ​\(Xi\)⊤\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}has operator normO​\(J\)O\(J\)but population mean operator normO​\(1\)O\(1\); the variance proxy is therefore orderN​JNJrather thanNN\. To get a≤12\\leq\\tfrac\{1\}\{2\}operator\-norm relative error one needs the standard deviationN​J/N\\sqrt\{NJ\}/Nto dominate, i\.e\.N≳JN\\gtrsim J, and an extralog⁡J\\log Jfrom union\-bounding acrossJJmatrix dimensions \(matrix Bernstein\)\. ThisJ​log⁡JJ\\log Jthreshold is exactly the dynamic\-mask condition used by the transformer in Appendix[12](https://arxiv.org/html/2606.31184#S12): it is the minimum sample size at which finite\-rank ridge updates are stable\.

### 7\.1Reference basis on\[0,1\]d\[0,1\]^\{d\}

Order the tensor\-product cosine basis on\[0,1\]d\[0,1\]^\{d\}by increasing Laplacian eigenvalue and call it\{ψj:j≥1\}\\\{\\psi\_\{j\}:j\\geq 1\\\}\. By Weyl’s law,λj≍j2/d\.\\lambda\_\{j\}\\asymp j^\{2/d\}\.This basis is orthonormal under Lebesgue measure on\[0,1\]d\[0,1\]^\{d\}:∫ψj​ψℓ=𝟏​\{j=ℓ\}\\int\\psi\_\{j\}\\psi\_\{\\ell\}=\\mathbf\{1\}\\\{j=\\ell\\\}\. It is uniformly bounded coordinatewise —supx\|ψj​\(x\)\|≤2d/2\\sup\_\{x\}\|\\psi\_\{j\}\(x\)\|\\leq 2^\{d/2\}— so writingϕJ​\(x\)=\(ψ1​\(x\),…,ψJ​\(x\)\)⊤\\phi\_\{J\}\(x\)=\(\\psi\_\{1\}\(x\),\\ldots,\\psi\_\{J\}\(x\)\)^\{\\top\},

supx∈\[0,1\]d∥ϕJ\(x\)∥22≤2dJ=:κJ\.\\sup\_\{x\\in\[0,1\]^\{d\}\}\\\|\\phi\_\{J\}\(x\)\\\|\_\{2\}^\{2\}\\leq 2^\{d\}J=:\\kappa J\.\(8\)The basis\-Sobolev ball matches the spectral Sobolev ball:ℋβ​\(R\)=\{f=∑jθj​ψj:∑jj2​β/d​θj2≤R2\}\.\\mathcal\{H\}^\{\\beta\}\(R\)=\\\{f=\\sum\_\{j\}\\theta\_\{j\}\\psi\_\{j\}:\\sum\_\{j\}j^\{2\\beta/d\}\\theta\_\{j\}^\{2\}\\leq R^\{2\}\\\}\.

### 7\.2Population Gram

###### Lemma 12\(Population Gram\)

LetPXP\_\{X\}have densitypXp\_\{X\}with0<cX≤pX≤CX<∞0<c\_\{X\}\\leq p\_\{X\}\\leq C\_\{X\}<\\inftyon\[0,1\]d\[0,1\]^\{d\}\. Then the population Gram matrixΓJ=𝔼PX​\[ϕJ​\(X\)​ϕJ​\(X\)⊤\]\\Gamma\_\{J\}=\\mathbb\{E\}\_\{P\_\{X\}\}\[\\phi\_\{J\}\(X\)\\phi\_\{J\}\(X\)^\{\\top\}\]satisfiescX​IJ⪯ΓJ⪯CX​IJ\.c\_\{X\}I\_\{J\}\\preceq\\Gamma\_\{J\}\\preceq C\_\{X\}I\_\{J\}\.

Proof\.For anya∈ℝJa\\in\\mathbb\{R\}^\{J\},ga​\(x\):=∑j≤Jaj​ψj​\(x\)g\_\{a\}\(x\):=\\sum\_\{j\\leq J\}a\_\{j\}\\psi\_\{j\}\(x\)has LebesgueL2L\_\{2\}norm‖a‖2\\\|a\\\|\_\{2\}, since the cosine basis is Lebesgue\-orthonormal\. Thus

a⊤​ΓJ​a=∫\[0,1\]dga​\(x\)2​pX​\(x\)​𝑑x∈\[cX​‖a‖22,CX​‖a‖22\]\.a^\{\\top\}\\Gamma\_\{J\}a=\\int\_\{\[0,1\]^\{d\}\}g\_\{a\}\(x\)^\{2\}p\_\{X\}\(x\)\\,dx\\in\\bigl\[c\_\{X\}\\\|a\\\|\_\{2\}^\{2\},\\,C\_\{X\}\\\|a\\\|\_\{2\}^\{2\}\\bigr\]\.□\\square

### 7\.3Empirical Gram, iid case

###### Lemma 13\(iid empirical Gram concentration\)

LetX1,…,XN∼iidPXX\_\{1\},\\ldots,X\_\{N\}\\stackrel\{\{\\scriptstyle\\rm iid\}\}\{\{\\sim\}\}P\_\{X\}andΓ^J=N−1​∑i=1NϕJ​\(Xi\)​ϕJ​\(Xi\)⊤\\widehat\{\\Gamma\}\_\{J\}=N^\{\-1\}\\sum\_\{i=1\}^\{N\}\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\. There exist constantsC,c\>0C,c\>0depending only on\(cX,CX,κ\)\(c\_\{X\},C\_\{X\},\\kappa\)such that, ifN≥C​J​log⁡\(J/δ\)N\\geq CJ\\log\(J/\\delta\), then with probability at least1−δ1\-\\delta,12​ΓJ⪯Γ^J⪯32​ΓJ\.\\tfrac\{1\}\{2\}\\Gamma\_\{J\}\\preceq\\widehat\{\\Gamma\}\_\{J\}\\preceq\\tfrac\{3\}\{2\}\\Gamma\_\{J\}\.

Proof\.SetYi=ΓJ−1/2​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤​ΓJ−1/2−IJY\_\{i\}=\\Gamma\_\{J\}^\{\-1/2\}\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\Gamma\_\{J\}^\{\-1/2\}\-I\_\{J\}\. Then𝔼​Yi=0\\mathbb\{E\}Y\_\{i\}=0and, using \([8](https://arxiv.org/html/2606.31184#S7.E8)\) and Lemma[12](https://arxiv.org/html/2606.31184#Thmtheorem12),

∥Yi∥op≤∥ΓJ−1/2ϕJ\(Xi\)∥22\+1≤κ​JcX\+1≤2​κ​JcX=:R\.\\\|Y\_\{i\}\\\|\_\{\\rm op\}\\leq\\\|\\Gamma\_\{J\}^\{\-1/2\}\\phi\_\{J\}\(X\_\{i\}\)\\\|\_\{2\}^\{2\}\+1\\leq\\frac\{\\kappa J\}\{c\_\{X\}\}\+1\\leq\\frac\{2\\kappa J\}\{c\_\{X\}\}=:R\.The matrix variance is bounded by‖𝔼​\[Yi2\]‖op≤𝔼​‖Yi‖op2≤R2\.\\\|\\mathbb\{E\}\[Y\_\{i\}^\{2\}\]\\\|\_\{\\rm op\}\\leq\\mathbb\{E\}\\\|Y\_\{i\}\\\|\_\{\\rm op\}^\{2\}\\leq R^\{2\}\.Apply the matrix Bernstein inequality\(Tropp,[2015](https://arxiv.org/html/2606.31184#bib.bib25)\)to the iid sum∑iYi\\sum\_\{i\}Y\_\{i\}: withσ2=N​R2\\sigma^\{2\}=NR^\{2\}andu=N/2u=N/2,

Pr⁡\(‖1N​∑iYi‖op\>12\)≤2​J​exp⁡\(−N/8R2/\(cX2\)\+R/3\)≤2​J​exp⁡\(−c​N/J\)\.\\Pr\\\!\\left\(\\Bigl\\\|\\tfrac\{1\}\{N\}\\sum\_\{i\}Y\_\{i\}\\Bigr\\\|\_\{\\rm op\}\>\\tfrac\{1\}\{2\}\\right\)\\leq 2J\\exp\\\!\\left\(\-\\frac\{N/8\}\{R^\{2\}/\(c\_\{X\}^\{2\}\)\+R/3\}\\right\)\\leq 2J\\exp\(\-cN/J\)\.Equivalently12​IJ⪯ΓJ−1/2​Γ^J​ΓJ−1/2⪯32​IJ\\tfrac\{1\}\{2\}I\_\{J\}\\preceq\\Gamma\_\{J\}^\{\-1/2\}\\widehat\{\\Gamma\}\_\{J\}\\Gamma\_\{J\}^\{\-1/2\}\\preceq\\tfrac\{3\}\{2\}I\_\{J\}\. Translating back gives the stated bound\. The thresholdN≥C​J​log⁡\(J/δ\)N\\geq CJ\\log\(J/\\delta\)makes the right\-hand side≤δ\\leq\\delta\.□\\square

### 7\.4Adaptive martingale Gram

###### Lemma 14\(Adaptive Gram concentration\)

For armw∈\{0,1\}w\\in\\\{0,1\\\}and the predictable filtration \([7](https://arxiv.org/html/2606.31184#S6.E7)\), letIi​\(w\)=𝟏​\{Wi=w\}I\_\{i\}\(w\)=\\mathbf\{1\}\\\{W\_\{i\}=w\\\},eiB=eiB​\(Xi\)e\_\{i\}^\{B\}=e\_\{i\}^\{B\}\(X\_\{i\}\)\(treatingw=0w=0via1−eiB1\-e\_\{i\}^\{B\}\)\. SetNt,w=∑i≤tIi​\(w\)N\_\{t,w\}=\\sum\_\{i\\leq t\}I\_\{i\}\(w\)and

Γ^t,w,J=1Nt,w​∑i≤t:Wi=wϕJ​\(Xi\)​ϕJ​\(Xi\)⊤\.\\widehat\{\\Gamma\}\_\{t,w,J\}=\\frac\{1\}\{N\_\{t,w\}\}\\sum\_\{i\\leq t:W\_\{i\}=w\}\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\.Suppose Assumption[1](https://arxiv.org/html/2606.31184#Thmassumption1)andeiB∈\[η,1−η\]e\_\{i\}^\{B\}\\in\[\\eta,1\-\\eta\]a\.s\. There exist constantsC1,C2,c\>0C\_\{1\},C\_\{2\},c\>0depending only on\(cX,CX,η,κ\)\(c\_\{X\},C\_\{X\},\\eta,\\kappa\)such that for everyJ≤Jmax,nJ\\leq J\_\{\\max,n\},δ\>0\\delta\>0, and everyttwithη​t≥C1​J​log⁡\(J/δ\)\\eta t\\geq C\_\{1\}J\\log\(J/\\delta\), on an event of probability at least1−2​δ1\-2\\delta,

η2​cX​IJ⪯Γ^t,w,J⪯2​\(1−η\)​CX​IJ,Nt,w≥η​t/2\.\\tfrac\{\\eta\}\{2\}\\,c\_\{X\}I\_\{J\}\\preceq\\widehat\{\\Gamma\}\_\{t,w,J\}\\preceq 2\(1\-\\eta\)\\,C\_\{X\}I\_\{J\},\\qquad N\_\{t,w\}\\geq\\eta t/2\.\(9\)In particular, on the same event,12​ΓJ⪯Γ^t,w,J⪯32​ΓJ\.\\tfrac\{1\}\{2\}\\Gamma\_\{J\}\\preceq\\widehat\{\\Gamma\}\_\{t,w,J\}\\preceq\\tfrac\{3\}\{2\}\\Gamma\_\{J\}\.

Proof\.*Step 1 \(sample\-size lower bound\)\.*DefineΔicnt=Ii​\(w\)−eiB​\(Xi\)​𝟏​\{w=1\}−\(1−eiB​\(Xi\)\)​𝟏​\{w=0\}\\Delta\_\{i\}^\{\\rm cnt\}=I\_\{i\}\(w\)\-e\_\{i\}^\{B\}\(X\_\{i\}\)\\mathbf\{1\}\\\{w=1\\\}\-\(1\-e\_\{i\}^\{B\}\(X\_\{i\}\)\)\\mathbf\{1\}\\\{w=0\\\}\. Then𝔼​\[Δicnt∣ℱiX\]=0\\mathbb\{E\}\[\\Delta\_\{i\}^\{\\rm cnt\}\\mid\\mathcal\{F\}\_\{i\}^\{X\}\]=0and\|Δicnt\|≤1\|\\Delta\_\{i\}^\{\\rm cnt\}\|\\leq 1\. By the bounded\-difference Azuma inequality \(Lemma[7](https://arxiv.org/html/2606.31184#Thmtheorem7)\),

Pr⁡\(\|∑i≤tΔicnt\|\>u\)≤2​exp⁡\(−u2/\(2​t\)\)\.\\Pr\\\!\\left\(\\Bigl\|\\sum\_\{i\\leq t\}\\Delta\_\{i\}^\{\\rm cnt\}\\Bigr\|\>u\\right\)\\leq 2\\exp\(\-u^\{2\}/\(2t\)\)\.Since the predictable conditional mean satisfies𝔼​\[Ii​\(w\)∣ℱiX\]≥η\\mathbb\{E\}\[I\_\{i\}\(w\)\\mid\\mathcal\{F\}\_\{i\}^\{X\}\]\\geq\\eta,𝔼​\[Nt,w∣ℱt\]≥η​t\.\\mathbb\{E\}\[N\_\{t,w\}\\mid\\mathcal\{F\}\_\{t\}\]\\geq\\eta t\.Withu=η​t/2u=\\eta t/2, the Azuma bound givesPr⁡\(Nt,w<η​t/2\)≤2​exp⁡\(−c​η2​t\)≤δ\\Pr\(N\_\{t,w\}<\\eta t/2\)\\leq 2\\exp\(\-c\\eta^\{2\}t\)\\leq\\deltaprovidedt≥C​η−2​log⁡\(2/δ\)t\\geq C\\eta^\{\-2\}\\log\(2/\\delta\), which is implied by the hypothesis whenC1C\_\{1\}is large\.

*Step 2 \(martingale Gram increments\)\.*Define the increments

Δi:=Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤−𝔼​\[Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤\|ℱi−1\]\.\\Delta\_\{i\}:=I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\-\\mathbb\{E\}\\bigl\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\,\\big\|\\,\\mathcal\{F\}\_\{i\-1\}\\bigr\]\.By the tower property and \([7](https://arxiv.org/html/2606.31184#S6.E7)\),

𝔼​\[Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤∣ℱi−1\]=𝔼X∼PX​\[eiB​\(X\)𝟏​\{w=1\}​\(1−eiB​\(X\)\)𝟏​\{w=0\}​ϕJ​\(X\)​ϕJ​\(X\)⊤\],\\mathbb\{E\}\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]=\\mathbb\{E\}\_\{X\\sim P\_\{X\}\}\\\!\\bigl\[e\_\{i\}^\{B\}\(X\)^\{\\mathbf\{1\}\\\{w=1\\\}\}\(1\-e\_\{i\}^\{B\}\(X\)\)^\{\\mathbf\{1\}\\\{w=0\\\}\}\\phi\_\{J\}\(X\)\\phi\_\{J\}\(X\)^\{\\top\}\\bigr\],and by clipping,

η​ΓJ⪯𝔼​\[Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤∣ℱi−1\]⪯\(1−η\)​ΓJ\.\\eta\\,\\Gamma\_\{J\}\\preceq\\mathbb\{E\}\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]\\preceq\(1\-\\eta\)\\,\\Gamma\_\{J\}\.\(10\)The incrementsΔi\\Delta\_\{i\}are MDS w\.r\.t\.ℱi\\mathcal\{F\}\_\{i\}\. Their operator norm is bounded a\.s\. by

∥Δi∥op≤2∥ϕJ\(Xi\)∥22≤2κJ=:RJ\.\\\|\\Delta\_\{i\}\\\|\_\{\\rm op\}\\leq 2\\\|\\phi\_\{J\}\(X\_\{i\}\)\\\|\_\{2\}^\{2\}\\leq 2\\kappa J=:R\_\{J\}\.Their predictable matrix variance satisfies, using‖ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤‖op≤κ​J\\\|\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\\|\_\{\\rm op\}\\leq\\kappa J,

𝔼​\[Δi2∣ℱi−1\]\\displaystyle\\mathbb\{E\}\[\\Delta\_\{i\}^\{2\}\\mid\\mathcal\{F\}\_\{i\-1\}\]⪯𝔼​\[Ii​\(w\)​‖ϕJ​\(Xi\)‖22​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤∣ℱi−1\]\\displaystyle\\preceq\\mathbb\{E\}\[I\_\{i\}\(w\)\\\|\\phi\_\{J\}\(X\_\{i\}\)\\\|\_\{2\}^\{2\}\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]⪯κ​J​𝔼​\[Ii​\(w\)​ϕJ​\(Xi\)​ϕJ​\(Xi\)⊤∣ℱi−1\]⪯κ​J​\(1−η\)​ΓJ\.\\displaystyle\\preceq\\kappa J\\,\\mathbb\{E\}\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]\\preceq\\kappa J\(1\-\\eta\)\\Gamma\_\{J\}\.Hence the cumulative predictable variance is bounded as‖Wt‖op≤κ​J​\(1−η\)​t​CX\.\\\|W\_\{t\}\\\|\_\{\\rm op\}\\leq\\kappa J\(1\-\\eta\)\\,t\\,C\_\{X\}\.

*Step 3 \(matrix Freedman\)\.*Apply Lemma[8](https://arxiv.org/html/2606.31184#Thmtheorem8)to∑i≤tΔi\\sum\_\{i\\leq t\}\\Delta\_\{i\}:

Pr⁡\(‖∑i≤tΔi‖op\>u\)≤2​J​exp⁡\(−u2/2κ​J​\(1−η\)​t​CX\+RJ​u/3\)\.\\Pr\\\!\\left\(\\Bigl\\\|\\sum\_\{i\\leq t\}\\Delta\_\{i\}\\Bigr\\\|\_\{\\rm op\}\>u\\right\)\\leq 2J\\exp\\\!\\left\(\-\\frac\{u^\{2\}/2\}\{\\kappa J\(1\-\\eta\)tC\_\{X\}\+R\_\{J\}u/3\}\\right\)\.Choosingu=η​cX​t/4u=\\eta c\_\{X\}t/4\(a constant fraction of the conditional mean\) andt≥C1​J​log⁡\(J/δ\)/ηt\\geq C\_\{1\}J\\log\(J/\\delta\)/\\etaforC1C\_\{1\}large enough, this probability is at mostδ\\delta\.

*Step 4 \(combine\)\.*On the intersection of the Step 1 and Step 3 events \(which has probability≥1−2​δ\\geq 1\-2\\delta\),

∥∑i≤tIi\(w\)ϕJ\(Xi\)ϕJ\(Xi\)⊤−∑i≤t𝔼\[Ii\(w\)ϕJ\(Xi\)ϕJ\(Xi\)⊤∣ℱi−1\]∥op≤ηcXt/4,\\Bigl\\\|\\sum\_\{i\\leq t\}I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\-\\sum\_\{i\\leq t\}\\mathbb\{E\}\[I\_\{i\}\(w\)\\phi\_\{J\}\(X\_\{i\}\)\\phi\_\{J\}\(X\_\{i\}\)^\{\\top\}\\mid\\mathcal\{F\}\_\{i\-1\}\]\\Bigr\\\|\_\{\\rm op\}\\leq\\eta c\_\{X\}t/4,andNt,w≥η​t/2N\_\{t,w\}\\geq\\eta t/2\. Dividing the displayed bound byNt,wN\_\{t,w\}and combining with \([10](https://arxiv.org/html/2606.31184#S7.E10)\) yields \([9](https://arxiv.org/html/2606.31184#S7.E9)\)\.□\\square

### 7\.5Uniform Gram event

###### Lemma 16\(Uniform Gram event\)

FixA\>0A\>0and letδn:=n−A/\(2​Jmax,n​\(log⁡n\)2\)\\delta\_\{n\}:=n^\{\-A\}/\(2J\_\{\\max,n\}\(\\log n\)^\{2\}\)\. Define the event

𝒢n:=⋂w∈\{0,1\}⋂J≤Jmax,n⋂t:η​t≥C1​J​log⁡\(J/δn\)\{\([9](https://arxiv.org/html/2606.31184#S7.E9)\)holds\}\.\\mathcal\{G\}\_\{n\}:=\\bigcap\_\{w\\in\\\{0,1\\\}\}\\bigcap\_\{J\\leq J\_\{\\max,n\}\}\\bigcap\_\{t:\\eta t\\geq C\_\{1\}J\\log\(J/\\delta\_\{n\}\)\}\\\!\\\!\\\!\\bigl\\\{\\eqref\{eq:gram\-adaptive\-concentration\}\\text\{ holds\}\\bigr\\\}\.ThenPr⁡\(𝒢nc\)≤n−A\\Pr\(\\mathcal\{G\}\_\{n\}^\{c\}\)\\leq n^\{\-A\}fornnlarge enough\. Moreover, on𝒢n\\mathcal\{G\}\_\{n\}and for any active dimensiondN,k≤Jmax,nd\_\{N,k\}\\leq J\_\{\\max,n\}satisfyingNt,w≥C1​dN,k​log⁡\(dN,k\)​log⁡\(n\)N\_\{t,w\}\\geq C\_\{1\}d\_\{N,k\}\\log\(d\_\{N,k\}\)\\log\(n\)andρdN,k−1≤Nt,w\\rho\_\{d\_\{N,k\}\}^\{\-1\}\\leq N\_\{t,w\}, the restricted Hessian

HN,k=PN,k​\(Γ^t,w,Jmax,n\+σ¯2Nt,w​Λk−1\)​PN,kH\_\{N,k\}=P\_\{N,k\}\\\!\\left\(\\widehat\{\\Gamma\}\_\{t,w,J\_\{\\max,n\}\}\+\\tfrac\{\\bar\{\\sigma\}^\{2\}\}\{N\_\{t,w\}\}\\Lambda\_\{k\}^\{\-1\}\\right\)\\\!P\_\{N,k\}satisfiesm​I⪯HN,k⪯M​ImI\\preceq H\_\{N,k\}\\preceq MIon the active subspace, withm=cX​η/2m=c\_\{X\}\\eta/2andM=2​\(1−η\)​CX\+σ¯2M=2\(1\-\\eta\)C\_\{X\}\+\\bar\{\\sigma\}^\{2\}\.

Proof\.The event is a union of at most2​Jmax,n⋅tmax2J\_\{\\max,n\}\\cdot t\_\{\\max\}events from Lemma[14](https://arxiv.org/html/2606.31184#Thmtheorem14), each of probability≤2​δn\\leq 2\\delta\_\{n\}\. The choice ofδn\\delta\_\{n\}makes the total deviation at mostn−An^\{\-A\}after discretizingtton a logarithmic grid \(the bound is monotone inttbetween grid points up to a factor1\+o​\(1\)1\+o\(1\)\)\. The Hessian bound follows from \([9](https://arxiv.org/html/2606.31184#S7.E9)\) together withσ¯2​ρdN,k−1/Nt,w≤σ¯2\\bar\{\\sigma\}^\{2\}\\rho\_\{d\_\{N,k\}\}^\{\-1\}/N\_\{t,w\}\\leq\\bar\{\\sigma\}^\{2\}on the active subspace andρj−1≥0\\rho\_\{j\}^\{\-1\}\\geq 0\.□\\square

##### Reading the dynamic\-mask conditions\.

Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)explains the two conditions placed on the dynamic active dimension in the main text: the first conditionNt,w≳J​log⁡JN\_\{t,w\}\\gtrsim J\\log Jguarantees the empirical covariance is well\-conditioned on the active subspace; the second conditionρJ,k−1≲Nt,w\\rho\_\{J,k\}^\{\-1\}\\lesssim N\_\{t,w\}ensures the rescaled prior precisionN−1​Λk−1N^\{\-1\}\\Lambda\_\{k\}^\{\-1\}does not blow up the upper Hessian eigenvalue\. Together they give a constant condition number\.

## 8Proof of Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)

##### Goal\.

Given an adaptive arm\-wwsubsample of sizeN=Nt,wN=N\_\{t,w\}and a Gaussian\-series prior with smoothnessβ\\betamatched to the truthm⋆∈ℋβ​\(R\)m^\{\\star\}\\in\\mathcal\{H\}^\{\\beta\}\(R\), prove that the \(quasi\-\)posterior puts asymptotically all of its mass on a ball of radiusM​εNM\\varepsilon\_\{N\}aroundm⋆m^\{\\star\}inL2​\(PX\)L\_\{2\}\(P\_\{X\}\), with

εN=N−β/\(2​β\+d\)​\(log⁡N\)A\.\\varepsilon\_\{N\}=N^\{\-\\beta/\(2\\beta\+d\)\}\\,\(\\log N\)^\{A\}\.\(11\)Apply the result to four targets\(w,h\)∈\{0,1\}×\{μ,q\}\(w,h\)\\in\\\{0,1\\\}\\times\\\{\\mu,q\\\}to recover Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)\.

##### Strategy \(Ghosal–Ghosh–van der Vaart template\)\.

Posterior contraction at rateεN\\varepsilon\_\{N\}follows from three ingredients:

1. 1\.*Prior small\-ball\.*Π​\(‖f−m⋆‖L2​\(PX\)≤εN\)≥e−C​N​εN2\\Pi\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq\\varepsilon\_\{N\}\)\\geq e^\{\-CN\\varepsilon\_\{N\}^\{2\}\}— the prior must not be exponentially thin near the truth\.
2. 2\.*Sieve and entropy\.*A model spaceℱn\\mathcal\{F\}\_\{n\}withΠ​\(ℱnc\)≤e−\(C\+4\)​N​εN2\\Pi\(\\mathcal\{F\}\_\{n\}^\{c\}\)\\leq e^\{\-\(C\+4\)N\\varepsilon\_\{N\}^\{2\}\}and metric entropy≤N​εN2\\leq N\\varepsilon\_\{N\}^\{2\}at scaleεN\\varepsilon\_\{N\}\.
3. 3\.*Tests\.*Exponentially powerful tests separatingm⋆m^\{\\star\}from every alternative inℱn\\mathcal\{F\}\_\{n\}at distance≥M​εN\\geq M\\varepsilon\_\{N\}\.

Combining \(I\)–\(III\) via Bayes’ rule yieldsΠ​\(‖f−m⋆‖\>M​εN∣𝒟N\)≤e−N​εN2→0\\Pi\(\\\|f\-m^\{\\star\}\\\|\>M\\varepsilon\_\{N\}\\mid\\mathcal\{D\}\_\{N\}\)\\leq e^\{\-N\\varepsilon\_\{N\}^\{2\}\}\\to 0\.

##### Where the rateεN=N−β/\(2​β\+d\)\\varepsilon\_\{N\}=N^\{\-\\beta/\(2\\beta\+d\)\}comes from \(bias–variance heuristic\)\.

WithJJactive coefficients andNNsamples, an OLS fit incurs a*stochastic error*of orderJ/N\\sqrt\{J/N\}on each coefficient \(variance\); truncating toJJcoefficients incurs a*truncation bias*∑j\>J\(θj⋆\)2≲J−2​β/d\\sum\_\{j\>J\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\lesssim J^\{\-2\\beta/d\}\. The two pieces together give

MSE​\(J\)≍J−2​β/d⏟bias2\+J/N⏟variance\.\\mathrm\{MSE\}\(J\)\\;\\asymp\\;\\underbrace\{J^\{\-2\\beta/d\}\}\_\{\\text\{bias\}^\{2\}\}\+\\underbrace\{J/N\}\_\{\\text\{variance\}\}\.\(12\)Setting the derivative to zero gives the optimalJN≍Nd/\(2​β\+d\)J\_\{N\}\\asymp N^\{d/\(2\\beta\+d\)\}and the optimalMSE≍N−2​β/\(2​β\+d\)\\mathrm\{MSE\}\\asymp N^\{\-2\\beta/\(2\\beta\+d\)\}\. Thelog⁡N\\log Nin \([11](https://arxiv.org/html/2606.31184#S8.E11)\) is the price of the union bound in the test step \(III\)\.

##### Why the prior does not dominate\.

The smoothness prior puts varianceρj​\(β\)≍j−1−2​β/d\\rho\_\{j\}\(\\beta\)\\asymp j^\{\-1\-2\\beta/d\}on coefficientjj; equivalently, it contributes precisionN−1​ρj−1≍j1\+2​β/d/NN^\{\-1\}\\rho\_\{j\}^\{\-1\}\\asymp j^\{1\+2\\beta/d\}/Nto the effective posterior precision \(since the data contributesNNon each active coordinate\)\. Atj=JNj=J\_\{N\},N−1​ρJN−1≍JN1\+2​β/d/N≍1N^\{\-1\}\\rho\_\{J\_\{N\}\}^\{\-1\}\\asymp J\_\{N\}^\{1\+2\\beta/d\}/N\\asymp 1— comparable to, not larger than, the data information\. PastJNJ\_\{N\}the prior dominates and shrinks the coefficient to zero; beforeJNJ\_\{N\}the data dominates and we get a1/N1/Nposterior variance\. The ”active dimension”JNJ\_\{N\}is exactly the cutoff where the two information sources balance, validating \([12](https://arxiv.org/html/2606.31184#S8.E12)\) as the right MSE expression\.

### 8\.1Setup

Fix armwwand a moment target\. LetN=Nt,wN=N\_\{t,w\}and let𝒟N=\{\(Xi,Zi\):i≤t,Wi=w\}\\mathcal\{D\}\_\{N\}=\\\{\(X\_\{i\},Z\_\{i\}\):i\\leq t,\\,W\_\{i\}=w\\\}be the arm\-wwsubsample\. By the predictable filtration \([7](https://arxiv.org/html/2606.31184#S6.E7)\), conditional on\(ℱi−1,Xi,Wi=w\)\(\\mathcal\{F\}\_\{i\-1\},X\_\{i\},W\_\{i\}=w\), the noiseξi:=Zi−m⋆​\(Xi\)\\xi\_\{i\}:=Z\_\{i\}\-m^\{\\star\}\(X\_\{i\}\)has𝔼​\[ξi∣ℱi−1,Xi,Wi=w\]=0\\mathbb\{E\}\[\\xi\_\{i\}\\mid\\mathcal\{F\}\_\{i\-1\},X\_\{i\},W\_\{i\}=w\]=0and is sub\-exponential with parameterτ\\tauuniform ini,wi,w\(Assumption[2](https://arxiv.org/html/2606.31184#Thmassumption2)\)\.

##### Quasi\-likelihood and pseudo\-posterior\.

The teacher uses the Gaussian quasi\-likelihoodLN​\(f\)∝exp⁡\{−\(2​σ¯2\)−1​∑i:Wi=w\(Zi−f​\(Xi\)\)2\}\.L\_\{N\}\(f\)\\propto\\exp\\bigl\\\{\-\(2\\bar\{\\sigma\}^\{2\}\)^\{\-1\}\\sum\_\{i:W\_\{i\}=w\}\(Z\_\{i\}\-f\(X\_\{i\}\)\)^\{2\}\\bigr\\\}\.We do*not*assume the noise is Gaussian; the squared\-loss likelihood is treated as a quasi\-likelihood whose population minimizer ism⋆m^\{\\star\}\. The contraction rate is verified by a likelihood\-ratio argument that uses sub\-exponential noise concentration \(Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)\) instead of Gaussian likelihood ratios\.

##### Prior\.

For fixed smoothnessβ∈\[βmin,βmax\]\\beta\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]matched to the target, the Gaussian\-series prior isf=∑j≥1θj​ψjf=\\sum\_\{j\\geq 1\}\\theta\_\{j\}\\psi\_\{j\}withθj∼N​\(0,ρj​\(β\)\)\\theta\_\{j\}\\sim N\(0,\\rho\_\{j\}\(\\beta\)\),ρj​\(β\)=j−1−2​β/d\\rho\_\{j\}\(\\beta\)=j^\{\-1\-2\\beta/d\}up to constants \(Section[2](https://arxiv.org/html/2606.31184#S2)\)\.

##### Target rate\.

Define

JN:=⌈\(N/log⁡N\)d/\(2​β\+d\)⌉,εN:=N−β/\(2​β\+d\)​\(log⁡N\)A,J\_\{N\}:=\\bigl\\lceil\(N/\\log N\)^\{d/\(2\\beta\+d\)\}\\bigr\\rceil,\\qquad\\varepsilon\_\{N\}:=N^\{\-\\beta/\(2\\beta\+d\)\}\(\\log N\)^\{A\},\(13\)for a constantA\>0A\>0to be chosen large enough \(depending only on structural constants\)\. NoteJN​log⁡N≍N​εN2J\_\{N\}\\log N\\asymp N\\varepsilon\_\{N\}^\{2\}\.

### 8\.2Step 1: Truncation bias

*Idea\.*The smoothness assumption forces high\-frequency coefficientsθj⋆\\theta\_\{j\}^\{\\star\}to decay polynomially\. Truncating the basis atJNJ\_\{N\}therefore loses only a power ofJNJ\_\{N\}inL2L\_\{2\}, and our choice ofJNJ\_\{N\}matches that loss to the target rateεN2\\varepsilon\_\{N\}^\{2\}\.

LetmJ⋆​\(x\)=∑j≤JNθj⋆​ψj​\(x\)m\_\{J\}^\{\\star\}\(x\)=\\sum\_\{j\\leq J\_\{N\}\}\\theta\_\{j\}^\{\\star\}\\psi\_\{j\}\(x\)be the projection ofm⋆m^\{\\star\}onto the firstJNJ\_\{N\}basis functions\. Bounded density gives‖f‖L2​\(PX\)2≤CX​‖f‖Lebesgue2\\\|f\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq C\_\{X\}\\\|f\\\|\_\{\\rm Lebesgue\}^\{2\}for anyf∈L2f\\in L\_\{2\}, and the cosine basis is Lebesgue\-orthonormal, so

‖m⋆−mJ⋆‖L2​\(PX\)2≤CX​∑j\>JN\(θj⋆\)2\.\\\|m^\{\\star\}\-m\_\{J\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq C\_\{X\}\\sum\_\{j\>J\_\{N\}\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\.\(14\)The Sobolev norm∑jj2​β/d​\(θj⋆\)2≤R2\\sum\_\{j\}j^\{2\\beta/d\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\leq R^\{2\}controls the high\-frequency tail by pulling out the smallest weight on the right tail:

∑j\>JN\(θj⋆\)2≤JN−2​β/d​∑j\>JNj2​β/d​\(θj⋆\)2≤R2​JN−2​β/d\.\\sum\_\{j\>J\_\{N\}\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\;\\leq\\;J\_\{N\}^\{\-2\\beta/d\}\\\!\\sum\_\{j\>J\_\{N\}\}\\\!j^\{2\\beta/d\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\;\\leq\\;R^\{2\}\\,J\_\{N\}^\{\-2\\beta/d\}\.\(15\)WithJN≍\(N/log⁡N\)d/\(2​β\+d\)J\_\{N\}\\asymp\(N/\\log N\)^\{d/\(2\\beta\+d\)\}, the right\-hand side is≍εN2\\asymp\\varepsilon\_\{N\}^\{2\}, so‖m⋆−mJ⋆‖L2​\(PX\)≲εN\\\|m^\{\\star\}\-m\_\{J\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\lesssim\\varepsilon\_\{N\}\. This is the*bias*\-half of the bias–variance tradeoff \([12](https://arxiv.org/html/2606.31184#S8.E12)\)\.

### 8\.3Step 2: Prior small\-ball

*Idea\.*We needΠ​\(‖f−m⋆‖L2​\(PX\)≤c​εN\)≥e−C​N​εN2\\Pi\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq c\\varepsilon\_\{N\}\)\\geq e^\{\-CN\\varepsilon\_\{N\}^\{2\}\}\. Decomposef−m⋆f\-m^\{\\star\}into a low\-frequency part \(firstJNJ\_\{N\}coordinates\) and a high\-frequency tail\. For the low\-frequency part, we need the Gaussian prior to put adequate mass on a small ball aroundθ1:JN⋆\\theta^\{\\star\}\_\{1:J\_\{N\}\}— this follows from Anderson’s inequality and is determined by \(i\) how spread the prior is on each coordinate \(JN​log⁡NJ\_\{N\}\\log Nterms\) and \(ii\) how farθ⋆\\theta^\{\\star\}is from zero on each coordinate \(the∑\(θj⋆\)2/ρj​\(β\)\\sum\(\\theta\_\{j\}^\{\\star\}\)^\{2\}/\\rho\_\{j\}\(\\beta\)“signal\-to\-prior” term\)\. The Sobolev assumption controls \(ii\) so that the sum is at mostJN​R2≍N​εN2J\_\{N\}R^\{2\}\\asymp N\\varepsilon\_\{N\}^\{2\}— the same order as \(i\)\. The high\-frequency tail is small in expectation because the prior variances are summable pastJNJ\_\{N\}, so a Markov bound gives constant probability\.

We bound belowΠβ​\(‖f−m⋆‖L2​\(PX\)≤c​εN\)\\Pi\_\{\\beta\}\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq c\\varepsilon\_\{N\}\)\.

*Low\-frequency block\.*Fora∈ℝJNa\\in\\mathbb\{R\}^\{J\_\{N\}\}with∑j≤JN\(aj−θj⋆\)2≤c​εN2\\sum\_\{j\\leq J\_\{N\}\}\(a\_\{j\}\-\\theta\_\{j\}^\{\\star\}\)^\{2\}\\leq c\\varepsilon\_\{N\}^\{2\}, the correspondingga=∑j≤JNaj​ψjg\_\{a\}=\\sum\_\{j\\leq J\_\{N\}\}a\_\{j\}\\psi\_\{j\}satisfies‖ga−mJ⋆‖L2​\(PX\)≤CX​c​εN\.\\\|g\_\{a\}\-m\_\{J\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq\\sqrt\{C\_\{X\}\}\\,\\sqrt\{c\}\\,\\varepsilon\_\{N\}\.The Gaussian small\-ball lower bound for a product Gaussian measure \(Anderson’s inequality, see\(van der Vaart and van Zanten,[2008](https://arxiv.org/html/2606.31184#bib.bib26), Lemma 5\.3\)\) gives

−log⁡Πβ​\(∑j≤JN\(θj−θj⋆\)2≤c​εN2\)≤C1​\(JN​log⁡JNc​εN2\+∑j≤JN\(θj⋆\)2ρj​\(β\)\)\.\-\\log\\Pi\_\{\\beta\}\\\!\\left\(\\sum\_\{j\\leq J\_\{N\}\}\(\\theta\_\{j\}\-\\theta\_\{j\}^\{\\star\}\)^\{2\}\\leq c\\varepsilon\_\{N\}^\{2\}\\right\)\\leq C\_\{1\}\\\!\\left\(J\_\{N\}\\log\\frac\{J\_\{N\}\}\{c\\varepsilon\_\{N\}^\{2\}\}\+\\sum\_\{j\\leq J\_\{N\}\}\\frac\{\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\}\{\\rho\_\{j\}\(\\beta\)\}\\right\)\.\(16\)Forj≤JNj\\leq J\_\{N\},ρj​\(β\)−1=j1\+2​β/d≤JN​j2​β/d\\rho\_\{j\}\(\\beta\)^\{\-1\}=j^\{1\+2\\beta/d\}\\leq J\_\{N\}j^\{2\\beta/d\}, so

∑j≤JN\(θj⋆\)2ρj​\(β\)≤JN​∑j≤JNj2​β/d​\(θj⋆\)2≤JN​R2\.\\sum\_\{j\\leq J\_\{N\}\}\\frac\{\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\}\{\\rho\_\{j\}\(\\beta\)\}\\leq J\_\{N\}\\sum\_\{j\\leq J\_\{N\}\}j^\{2\\beta/d\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\leq J\_\{N\}R^\{2\}\.Combining with \([16](https://arxiv.org/html/2606.31184#S8.E16)\) andJN​log⁡\(JN/εN2\)≤C​JN​log⁡N≤C​N​εN2J\_\{N\}\\log\(J\_\{N\}/\\varepsilon\_\{N\}^\{2\}\)\\leq CJ\_\{N\}\\log N\\leq CN\\varepsilon\_\{N\}^\{2\}forAAlarge,

−log⁡Πβ​\(∑j≤JN\(θj−θj⋆\)2≤c​εN2\)≤C​N​εN2\.\-\\log\\Pi\_\{\\beta\}\\\!\\left\(\\sum\_\{j\\leq J\_\{N\}\}\(\\theta\_\{j\}\-\\theta\_\{j\}^\{\\star\}\)^\{2\}\\leq c\\varepsilon\_\{N\}^\{2\}\\right\)\\leq CN\\varepsilon\_\{N\}^\{2\}\.\(17\)
*High\-frequency block\.*For the tail∑j\>JNθj2\\sum\_\{j\>J\_\{N\}\}\\theta\_\{j\}^\{2\},

𝔼β​∑j\>JNθj2=∑j\>JNρj​\(β\)≍∑j\>JNj−1−2​β/d≍JN−2​β/d≲εN2\.\\mathbb\{E\}\_\{\\beta\}\\sum\_\{j\>J\_\{N\}\}\\theta\_\{j\}^\{2\}=\\sum\_\{j\>J\_\{N\}\}\\rho\_\{j\}\(\\beta\)\\asymp\\sum\_\{j\>J\_\{N\}\}j^\{\-1\-2\\beta/d\}\\asymp J\_\{N\}^\{\-2\\beta/d\}\\lesssim\\varepsilon\_\{N\}^\{2\}\.Markov’s inequality givesΠβ​\(∑j\>JNθj2≤C​εN2\)≥1/2\.\\Pi\_\{\\beta\}\(\\sum\_\{j\>J\_\{N\}\}\\theta\_\{j\}^\{2\}\\leq C\\varepsilon\_\{N\}^\{2\}\)\\geq 1/2\.Combining the two blocks \(independence under the Gaussian\-series prior\),

Πβ​\(‖f−m⋆‖L2​\(PX\)≤C2​εN\)≥12​exp⁡\(−C​N​εN2\)\.\\Pi\_\{\\beta\}\\\!\\left\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq C\_\{2\}\\varepsilon\_\{N\}\\right\)\\geq\\tfrac\{1\}\{2\}\\exp\(\-CN\\varepsilon\_\{N\}^\{2\}\)\.\(18\)

### 8\.4Step 3: Sieve

*Idea\.*The full prior assigns nontrivial mass to functions with arbitrarily large coefficients; we cannot cover the entire support by finitely many balls\. We carve out a sieveℱn\\mathcal\{F\}\_\{n\}small enough to have entropyO​\(N​εN2\)O\(N\\varepsilon\_\{N\}^\{2\}\)— exactly the budget allowed by the exponential test errors — yet large enough that the prior tail outside is exponentially small\. The two bounds \(Π​\(ℱnc\)≤e−\(C\+4\)​N​εN2\\Pi\(\\mathcal\{F\}\_\{n\}^\{c\}\)\\leq e^\{\-\(C\+4\)N\\varepsilon\_\{N\}^\{2\}\}and entropy≤N​εN2\\leq N\\varepsilon\_\{N\}^\{2\}\) together with the test errors of Step 4 suffice for posterior concentration\.

Define the sieve

ℱn:=\{f=∑j≥1aj​ψj:∑j≤JNaj2≤Rn2,∑j\>JNaj2≤εN2,‖f‖∞≤Bn\},\\mathcal\{F\}\_\{n\}:=\\Bigl\\\{f=\\sum\_\{j\\geq 1\}a\_\{j\}\\psi\_\{j\}:\\sum\_\{j\\leq J\_\{N\}\}a\_\{j\}^\{2\}\\leq R\_\{n\}^\{2\},\\,\\sum\_\{j\>J\_\{N\}\}a\_\{j\}^\{2\}\\leq\\varepsilon\_\{N\}^\{2\},\\,\\\|f\\\|\_\{\\infty\}\\leq B\_\{n\}\\Bigr\\\},withRn=NAR\_\{n\}=N^\{A\}andBn=κ​JN​RnB\_\{n\}=\\sqrt\{\\kappa J\_\{N\}\}R\_\{n\}to absorb basis\-envelopeL∞L\_\{\\infty\}bounds\. We verify two properties\.

*Prior tail outside the sieve\.*Πβ​\(ℱnc\)≤Πβ​\(∑j≤JNaj2\>Rn2\)\+Πβ​\(∑j\>JNaj2\>εN2\)≤exp⁡\(−C​Rn2\)\+exp⁡\(−C​N​εN2\)\\Pi\_\{\\beta\}\(\\mathcal\{F\}\_\{n\}^\{c\}\)\\leq\\Pi\_\{\\beta\}\(\\sum\_\{j\\leq J\_\{N\}\}a\_\{j\}^\{2\}\>R\_\{n\}^\{2\}\)\+\\Pi\_\{\\beta\}\(\\sum\_\{j\>J\_\{N\}\}a\_\{j\}^\{2\}\>\\varepsilon\_\{N\}^\{2\}\)\\leq\\exp\(\-CR\_\{n\}^\{2\}\)\+\\exp\(\-CN\\varepsilon\_\{N\}^\{2\}\)by Gaussian tail and Markov on the truncated sum\. WithRn2≫N​εN2R\_\{n\}^\{2\}\\gg N\\varepsilon\_\{N\}^\{2\}, this is at mostexp⁡\(−\(C\+4\)​N​εN2\)\\exp\(\-\(C\+4\)N\\varepsilon\_\{N\}^\{2\}\)\.

*Entropy\.*TheL2​\(PX\)L\_\{2\}\(P\_\{X\}\)covering entropy ofℱn\\mathcal\{F\}\_\{n\}at scaleε\\varepsilonsatisfieslog⁡N​\(ε,ℱn,L2​\(PX\)\)≤JN​log⁡\(C​Rn/ε\)\+log⁡\(Jmax,nJN\)≲JN​log⁡N≲N​εN2,\\log N\(\\varepsilon,\\mathcal\{F\}\_\{n\},L\_\{2\}\(P\_\{X\}\)\)\\leq J\_\{N\}\\log\(CR\_\{n\}/\\varepsilon\)\+\\log\\\!\\binom\{J\_\{\\max,n\}\}\{J\_\{N\}\}\\lesssim J\_\{N\}\\log N\\lesssim N\\varepsilon\_\{N\}^\{2\},since the high\-frequency block contributes onlyε\\varepsilon\-balls of fixed radius and the low\-frequency block is aJNJ\_\{N\}\-dimensional Euclidean ball\.

### 8\.5Step 4: Tests

*Idea\.*A testTfT\_\{f\}acceptsm⋆m^\{\\star\}when the empirical squared residual1N​∑\(Zi−f\)2\\frac\{1\}\{N\}\\sum\(Z\_\{i\}\-f\)^\{2\}is too large compared to1N​∑\(Zi−m⋆\)2\\frac\{1\}\{N\}\\sum\(Z\_\{i\}\-m^\{\\star\}\)^\{2\}\. By straightforward algebra the difference of these two empirical sums equals‖f−m⋆‖N2\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}minus an inner product between the noiseξi\\xi\_\{i\}andf−m⋆f\-m^\{\\star\}\. The first term is≍ρ2\\asymp\\rho^\{2\}whenffisρ\\rho\-separated fromm⋆m^\{\\star\}, by Gram concentration \(Lemma[14](https://arxiv.org/html/2606.31184#Thmtheorem14)\); the noise inner product is mean\-zero and sub\-exponential, so a Bernstein\-MDS bound shows it cannot exceedρ2/4\\rho^\{2\}/4except with probabilitye−c​N​ρ2e^\{\-cN\\rho^\{2\}\}\. Hence the test makes both kinds of errors at levele−c​N​ρ2e^\{\-cN\\rho^\{2\}\}, and aρ\\rho\-cover of the sieve costseN​εN2e^\{N\\varepsilon\_\{N\}^\{2\}\}tests by entropy — affordable whenMMinρ=M​εN\\rho=M\\varepsilon\_\{N\}is large\.

We construct tests for the alternative\{f∈ℱn:‖f−m⋆‖L2​\(PX\)\>M​εN\}\\\{f\\in\\mathcal\{F\}\_\{n\}:\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\varepsilon\_\{N\}\\\}via Lemma[9](https://arxiv.org/html/2606.31184#Thmtheorem9)\. Fix oneffin this set and define the test statistic

L^N​\(f\):=1N​∑i:Wi=w\[\(Zi−f​\(Xi\)\)2−\(Zi−m⋆​\(Xi\)\)2\]\.\\widehat\{L\}\_\{N\}\(f\):=\\frac\{1\}\{N\}\\sum\_\{i:W\_\{i\}=w\}\\bigl\[\(Z\_\{i\}\-f\(X\_\{i\}\)\)^\{2\}\-\(Z\_\{i\}\-m^\{\\star\}\(X\_\{i\}\)\)^\{2\}\\bigr\]\.Decompose:

L^N​\(f\)=‖f−m⋆‖N2−2N​∑i:Wi=wξi​\(f​\(Xi\)−m⋆​\(Xi\)\)\.\\widehat\{L\}\_\{N\}\(f\)=\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\frac\{2\}\{N\}\\sum\_\{i:W\_\{i\}=w\}\\xi\_\{i\}\\bigl\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\\bigr\)\.\(19\)
*Empirical\-vs\-population norm\.*On the Gram event𝒢n\\mathcal\{G\}\_\{n\}\(Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)\), the restriction of∥⋅∥N\\\|\\cdot\\\|\_\{N\}to theJmax,nJ\_\{\\max,n\}\-dimensional active class satisfies12​‖f−m⋆‖L2​\(PX\)2≤‖f−m⋆‖N2≤32​‖f−m⋆‖L2​\(PX\)2,\\tfrac\{1\}\{2\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\\leq\\tfrac\{3\}\{2\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\},uniformly inf∈ℱnf\\in\\mathcal\{F\}\_\{n\}providedN≥C​Jmax,n​log⁡nN\\geq CJ\_\{\\max,n\}\\log n\. Hence

‖f−m⋆‖N2≥12​‖f−m⋆‖L2​\(PX\)2\.\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\\geq\\tfrac\{1\}\{2\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\.\(20\)
*Noise inner product\.*The sequenceDi:=Ii​\(w\)​ξi​\(f​\(Xi\)−m⋆​\(Xi\)\)D\_\{i\}:=I\_\{i\}\(w\)\\xi\_\{i\}\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\),1≤i≤t1\\leq i\\leq t, is an MDS w\.r\.t\.ℱiW\\mathcal\{F\}\_\{i\}^\{W\}because𝔼​\[ξi∣ℱi−1,Xi,Wi=w\]=0\\mathbb\{E\}\[\\xi\_\{i\}\\mid\\mathcal\{F\}\_\{i\-1\},X\_\{i\},W\_\{i\}=w\]=0\. Its predictable variance is

𝔼​\[Di2∣ℱi−1\]=𝔼​\[Ii​\(w\)​\(f​\(Xi\)−m⋆​\(Xi\)\)2​𝔼​\[ξi2∣ℱi−1,Xi,Wi=w\]\|ℱi−1\]≤τ2​‖f−m⋆‖PX2,\\mathbb\{E\}\[D\_\{i\}^\{2\}\\mid\\mathcal\{F\}\_\{i\-1\}\]=\\mathbb\{E\}\\\!\\bigl\[I\_\{i\}\(w\)\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\)^\{2\}\\mathbb\{E\}\[\\xi\_\{i\}^\{2\}\\mid\\mathcal\{F\}\_\{i\-1\},X\_\{i\},W\_\{i\}=w\]\\,\\big\|\\,\\mathcal\{F\}\_\{i\-1\}\\bigr\]\\leq\\tau^\{2\}\\,\\\|f\-m^\{\\star\}\\\|\_\{P\_\{X\}\}^\{2\},since𝔼​\[ξi2∣ℱi−1,Xi,Wi=w\]≤τ2\\mathbb\{E\}\[\\xi\_\{i\}^\{2\}\\mid\\mathcal\{F\}\_\{i\-1\},X\_\{i\},W\_\{i\}=w\]\\leq\\tau^\{2\}uniformly\. EachDiD\_\{i\}is sub\-exponential with‖Di‖ψ1∣ℱi−1≤τ​Bn\\\|D\_\{i\}\\\|\_\{\\psi\_\{1\}\\mid\\mathcal\{F\}\_\{i\-1\}\}\\leq\\tau B\_\{n\}\(boundedness off−m⋆f\-m^\{\\star\}on the sieve\)\. Apply Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)withu=c​N​ρ2u=cN\\rho^\{2\},Vt≤N​τ2​ρ2V\_\{t\}\\leq N\\tau^\{2\}\\rho^\{2\},K=τ​BnK=\\tau B\_\{n\}:

Pr⁡\(\|1N​∑i:Wi=wξi​\(f​\(Xi\)−m⋆​\(Xi\)\)\|\>ρ2/4\)≤2​exp⁡\(−c​min⁡\{N​ρ2τ2,N​ρτ​Bn\}\)\.\\Pr\\\!\\left\(\\Bigl\|\\tfrac\{1\}\{N\}\\sum\_\{i:W\_\{i\}=w\}\\xi\_\{i\}\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\)\\Bigr\|\>\\rho^\{2\}/4\\right\)\\leq 2\\exp\\\!\\left\(\-c\\min\\\!\\Bigl\\\{\\tfrac\{N\\rho^\{2\}\}\{\\tau^\{2\}\},\\tfrac\{N\\rho\}\{\\tau B\_\{n\}\}\\Bigr\\\}\\right\)\.\(21\)Setρ=‖f−m⋆‖L2​\(PX\)\>M​εN\\rho=\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\varepsilon\_\{N\}\. SinceBn=poly​\(N\)B\_\{n\}=\\mathrm\{poly\}\(N\)andN​ρ≥N​M​εNN\\rho\\geq NM\\varepsilon\_\{N\}, the second term in the minimum dominates only whenM​εNM\\varepsilon\_\{N\}is too small; choosingAAinεN\\varepsilon\_\{N\}large enough so thatN​M​εN/\(τ​Bn\)≥c​N​εN2NM\\varepsilon\_\{N\}/\(\\tau B\_\{n\}\)\\geq cN\\varepsilon\_\{N\}^\{2\}, the bound reduces toexp⁡\(−c​N​ρ2/τ2\)\\exp\(\-cN\\rho^\{2\}/\\tau^\{2\}\)\.

*Test definition and errors\.*DefineTf:=𝟏​\{L^N​\(f\)≤−ρ2/4\}T\_\{f\}:=\\mathbf\{1\}\\\{\\widehat\{L\}\_\{N\}\(f\)\\leq\-\\rho^\{2\}/4\\\}\.

*Type\-I error underm⋆m^\{\\star\}\.*On𝒢n∩\{\([21](https://arxiv.org/html/2606.31184#S8.E21)\)holds\}\\mathcal\{G\}\_\{n\}\\cap\\\{\\eqref\{eq:noise\-bernstein\}\\text\{ holds\}\\\}, combining \([19](https://arxiv.org/html/2606.31184#S8.E19)\), \([20](https://arxiv.org/html/2606.31184#S8.E20)\), and the Bernstein bound,

L^N​\(f\)≥‖f−m⋆‖N2−ρ24≥12​ρ2−ρ24=ρ24\.\\widehat\{L\}\_\{N\}\(f\)\\geq\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\tfrac\{\\rho^\{2\}\}\{4\}\\geq\\tfrac\{1\}\{2\}\\rho^\{2\}\-\\tfrac\{\\rho^\{2\}\}\{4\}=\\tfrac\{\\rho^\{2\}\}\{4\}\.HenceTf=0T\_\{f\}=0on this event\. The complement has probability≤3​exp⁡\(−c​N​ρ2/τ2\)\\leq 3\\exp\(\-cN\\rho^\{2\}/\\tau^\{2\}\)\.

*Type\-II error under anyggwith‖g−m⋆‖L2​\(PX\)\>ρ\\\|g\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>\\rho\.*Symmetric argument: undergg,L^N​\(f\)\\widehat\{L\}\_\{N\}\(f\)is𝔼g\\mathbb\{E\}\_\{g\}\-mean‖f−g‖N2−‖m⋆−g‖N2\\\|f\-g\\\|\_\{N\}^\{2\}\-\\\|m^\{\\star\}\-g\\\|\_\{N\}^\{2\}\(a constant offset\) shifted by a similar sub\-exponential MDS\. The same Bernstein bound showsL^N​\(f\)≤−ρ2/4\\widehat\{L\}\_\{N\}\(f\)\\leq\-\\rho^\{2\}/4with high probability when‖f−g‖<ρ/2<‖m⋆−g‖/2\\\|f\-g\\\|<\\rho/2<\\\|m^\{\\star\}\-g\\\|/2, in particular wheng=fg=fitself\. The detailed algebra is identical to the type\-I argument and uses the empirical\-norm bound onf−gf\-gand ong−m⋆g\-m^\{\\star\}\. The error is again≤3​exp⁡\(−c​N​ρ2/τ2\)\\leq 3\\exp\(\-cN\\rho^\{2\}/\\tau^\{2\}\)\.

*Cover and union\.*Takeρ=M​εN\\rho=M\\varepsilon\_\{N\}and let𝒩ρ\\mathcal\{N\}\_\{\\rho\}be aρ/4\\rho/4\-cover ofℱn∩\{‖f−m⋆‖L2​\(PX\)\>M​εN\}\\mathcal\{F\}\_\{n\}\\cap\\\{\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\varepsilon\_\{N\}\\\}\. By Step 3,log⁡\|𝒩ρ\|≲N​εN2\\log\|\\mathcal\{N\}\_\{\\rho\}\|\\lesssim N\\varepsilon\_\{N\}^\{2\}\. DefineφN=maxf∈𝒩ρ⁡Tf\\varphi\_\{N\}=\\max\_\{f\\in\\mathcal\{N\}\_\{\\rho\}\}T\_\{f\}\. Union bounds give

𝔼m⋆​φN≤3​\|𝒩ρ\|​e−c​M2​N​εN2,supf:‖f−m⋆‖L2​\(PX\)\>2​M​εN𝔼f​\(1−φN\)≤3​e−c​M2​N​εN2\.\\mathbb\{E\}\_\{m^\{\\star\}\}\\varphi\_\{N\}\\leq 3\|\\mathcal\{N\}\_\{\\rho\}\|e^\{\-cM^\{2\}N\\varepsilon\_\{N\}^\{2\}\},\\qquad\\sup\_\{f:\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>2M\\varepsilon\_\{N\}\}\\mathbb\{E\}\_\{f\}\(1\-\\varphi\_\{N\}\)\\leq 3e^\{\-cM^\{2\}N\\varepsilon\_\{N\}^\{2\}\}\.\(22\)WithMMlarge enough so thatc​M2−C≥1cM^\{2\}\-C\\geq 1\(whereCCis the entropy constant\), both bounds are≤e−N​εN2\\leq e^\{\-N\\varepsilon\_\{N\}^\{2\}\}\.

### 8\.6Step 5: Posterior denominator

*Idea\.*Bayes’ rule writes the posterior as a ratioΠ\(⋅∣𝒟N\)=NN\(⋅\)/DN\\Pi\(\\cdot\\mid\\mathcal\{D\}\_\{N\}\)=N\_\{N\}\(\\cdot\)/D\_\{N\}\. The numerator is small for alternatives by the tests of Step 4; for the contraction inequality to bite we also need the denominatorDND\_\{N\}not to be too small\. Restricting the integral definingDND\_\{N\}to the small\-ball\{‖f−m⋆‖≤c​εN\}\\\{\\\|f\-m^\{\\star\}\\\|\\leq c\\varepsilon\_\{N\}\\\}and noting that on this ball the log\-likelihood ratioℓN​\(f\)−ℓN​\(m⋆\)\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)is at mostO​\(N​εN2\)O\(N\\varepsilon\_\{N\}^\{2\}\)in absolute value \(small empirical bias, controlled noise inner product\) givesDN≥e−C​N​εN2⋅Π​\(small ball\)≥e−\(C\+C\)​N​εN2D\_\{N\}\\geq e^\{\-CN\\varepsilon\_\{N\}^\{2\}\}\\cdot\\Pi\(\\text\{small ball\}\)\\geq e^\{\-\(C\+C\)N\\varepsilon\_\{N\}^\{2\}\}by Step 2\.

LetℓN​\(f\):=−\(2​σ¯2\)−1​∑i:Wi=w\(Zi−f​\(Xi\)\)2\\ell\_\{N\}\(f\):=\-\(2\\bar\{\\sigma\}^\{2\}\)^\{\-1\}\\sum\_\{i:W\_\{i\}=w\}\(Z\_\{i\}\-f\(X\_\{i\}\)\)^\{2\}be the quasi log\-likelihood\. Define

DN:=∫eℓN​\(f\)−ℓN​\(m⋆\)​𝑑Πβ​\(f\)\.D\_\{N\}:=\\int e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi\_\{\\beta\}\(f\)\.We bound belowDND\_\{N\}on𝒢n\\mathcal\{G\}\_\{n\}\.

*Restriction to the small ball\.*Forffwith‖f−m⋆‖L2​\(PX\)≤c​εN\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq c\\varepsilon\_\{N\}, the squared\-error gap is

ℓN​\(f\)−ℓN​\(m⋆\)=−N2​σ¯2​‖f−m⋆‖N2\+1σ¯2​∑i:Wi=wξi​\(f​\(Xi\)−m⋆​\(Xi\)\)\.\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)=\-\\frac\{N\}\{2\\bar\{\\sigma\}^\{2\}\}\\,\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\+\\frac\{1\}\{\\bar\{\\sigma\}^\{2\}\}\\sum\_\{i:W\_\{i\}=w\}\\xi\_\{i\}\\bigl\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\\bigr\)\.\(23\)On𝒢n\\mathcal\{G\}\_\{n\},‖f−m⋆‖N2≤32​‖f−m⋆‖L2​\(PX\)2≤32​c2​εN2\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\\leq\\tfrac\{3\}\{2\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq\\tfrac\{3\}\{2\}c^\{2\}\\varepsilon\_\{N\}^\{2\}\. Apply Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)with the same parameters as in Step 4 to the noise inner product, restricted to the small\-ball cover at scalec​εN/4c\\varepsilon\_\{N\}/4of sizeexp⁡\(N​εN2\)\\exp\(N\\varepsilon\_\{N\}^\{2\}\): with probability≥1−exp⁡\(−C​N​εN2\)\\geq 1\-\\exp\(\-CN\\varepsilon\_\{N\}^\{2\}\), uniformly,\|∑ξi​\(f​\(Xi\)−m⋆​\(Xi\)\)\|/N≤c​εN2​τ\.\\bigl\|\\sum\\xi\_\{i\}\(f\(X\_\{i\}\)\-m^\{\\star\}\(X\_\{i\}\)\)\\bigr\|/N\\leq c\\varepsilon\_\{N\}^\{2\}\\tau\.HenceℓN​\(f\)−ℓN​\(m⋆\)≥−C​N​εN2\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\\geq\-CN\\varepsilon\_\{N\}^\{2\}uniformly over the small ball\. Combining with \([18](https://arxiv.org/html/2606.31184#S8.E18)\),

DN≥e−C​N​εN2​Πβ​\(‖f−m⋆‖L2​\(PX\)≤c​εN\)≥e−\(C\+C\)​N​εN2\.D\_\{N\}\\geq e^\{\-CN\\varepsilon\_\{N\}^\{2\}\}\\,\\Pi\_\{\\beta\}\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq c\\varepsilon\_\{N\}\)\\geq e^\{\-\(C\+C\)N\\varepsilon\_\{N\}^\{2\}\}\.\(24\)

### 8\.7Step 6: Posterior numerator

LetBNc:=\{f:‖f−m⋆‖L2​\(PX\)\>2​M​εN\}B\_\{N\}^\{c\}:=\\\{f:\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>2M\\varepsilon\_\{N\}\\\}\. We decompose along the sieveℱn\\mathcal\{F\}\_\{n\}from Step 3:

Πβ​\(BNc∣𝒟N\)≤φN⏟type I error\+\(1−φN\)​∫BNc∩ℱneℓN​\(f\)−ℓN​\(m⋆\)​𝑑Πβ​\(f\)DN⏟tested alternatives\+∫ℱnceℓN​\(f\)−ℓN​\(m⋆\)​𝑑Πβ​\(f\)DN⏟prior tail outside sieve\.\\Pi\_\{\\beta\}\(B\_\{N\}^\{c\}\\mid\\mathcal\{D\}\_\{N\}\)\\leq\\underbrace\{\\varphi\_\{N\}\}\_\{\\text\{type I error\}\}\+\\underbrace\{\\frac\{\(1\-\\varphi\_\{N\}\)\\int\_\{B\_\{N\}^\{c\}\\cap\\mathcal\{F\}\_\{n\}\}e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi\_\{\\beta\}\(f\)\}\{D\_\{N\}\}\}\_\{\\text\{tested alternatives\}\}\+\\underbrace\{\\frac\{\\int\_\{\\mathcal\{F\}\_\{n\}^\{c\}\}e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi\_\{\\beta\}\(f\)\}\{D\_\{N\}\}\}\_\{\\text\{prior tail outside sieve\}\}\.\(25\)The third term is bounded by Markov on the likelihood ratio:𝔼m⋆​∫ℱnceℓN​\(f\)−ℓN​\(m⋆\)​𝑑Πβ​\(f\)≤Πβ​\(ℱnc\)≤e−\(C\+4\)​N​εN2\\mathbb\{E\}\_\{m^\{\\star\}\}\\\!\\int\_\{\\mathcal\{F\}\_\{n\}^\{c\}\}e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi\_\{\\beta\}\(f\)\\leq\\Pi\_\{\\beta\}\(\\mathcal\{F\}\_\{n\}^\{c\}\)\\leq e^\{\-\(C\+4\)N\\varepsilon\_\{N\}^\{2\}\}, which divided byDN≥e−\(C\+2\)​N​εN2D\_\{N\}\\geq e^\{\-\(C\+2\)N\\varepsilon\_\{N\}^\{2\}\}from Step 5 gives≤e−2​N​εN2\\leq e^\{\-2N\\varepsilon\_\{N\}^\{2\}\}\.

For the second term in \([25](https://arxiv.org/html/2606.31184#S8.E25)\), by Fubini and \([22](https://arxiv.org/html/2606.31184#S8.E22)\),

𝔼m⋆​∫BNc∩ℱn\(1−φN\)​eℓN​\(f\)−ℓN​\(m⋆\)​𝑑Πβ​\(f\)=∫BNc∩ℱn𝔼f​\[1−φN\]​𝑑Πβ​\(f\)≤e−c​M2​N​εN2,\\mathbb\{E\}\_\{m^\{\\star\}\}\\\!\\\!\\int\_\{B\_\{N\}^\{c\}\\cap\\mathcal\{F\}\_\{n\}\}\\\!\(1\-\\varphi\_\{N\}\)\\,e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi\_\{\\beta\}\(f\)\\;=\\;\\int\_\{B\_\{N\}^\{c\}\\cap\\mathcal\{F\}\_\{n\}\}\\\!\\mathbb\{E\}\_\{f\}\[1\-\\varphi\_\{N\}\]\\,d\\Pi\_\{\\beta\}\(f\)\\;\\leq\\;e^\{\-cM^\{2\}N\\varepsilon\_\{N\}^\{2\}\},which divided byDN≥e−\(C\+2\)​N​εN2D\_\{N\}\\geq e^\{\-\(C\+2\)N\\varepsilon\_\{N\}^\{2\}\}gives≤e−\(c​M2−C−2\)​N​εN2\\leq e^\{\-\(cM^\{2\}\-C\-2\)N\\varepsilon\_\{N\}^\{2\}\}\. The first termφN\\varphi\_\{N\}is itself≤e−c​M2​N​εN2\\leq e^\{\-cM^\{2\}N\\varepsilon\_\{N\}^\{2\}\}by \([22](https://arxiv.org/html/2606.31184#S8.E22)\)\.

### 8\.8Step 7: Conclusion

Combine: withMMchosen so thatc​M2−C−2≥1cM^\{2\}\-C\-2\\geq 1and on the intersection of the Gram event𝒢n\\mathcal\{G\}\_\{n\}and the denominator event,

Πβ​\(‖f−m⋆‖L2​\(PX\)\>2​M​εN∣𝒟N\)≤𝔼m⋆​φN\+e−N​εN2\+e−2​N​εN2→0\.\\Pi\_\{\\beta\}\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>2M\\varepsilon\_\{N\}\\mid\\mathcal\{D\}\_\{N\}\)\\leq\\mathbb\{E\}\_\{m^\{\\star\}\}\\varphi\_\{N\}\+e^\{\-N\\varepsilon\_\{N\}^\{2\}\}\+e^\{\-2N\\varepsilon\_\{N\}^\{2\}\}\\to 0\.This proves Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)with rateεN=N−β/\(2​β\+d\)​\(log⁡N\)A\\varepsilon\_\{N\}=N^\{\-\\beta/\(2\\beta\+d\)\}\(\\log N\)^\{A\}\.

### 8\.9Step 8: Specialization to four targets

Apply Steps 1–7 to the four cases:

1. 1\.w=0w=0,h=μh=\\mu:Zi=YiZ\_\{i\}=Y\_\{i\},m⋆=μ0⋆m^\{\\star\}=\\mu\_\{0\}^\{\\star\},β=β0,μ⋆\\beta=\\beta\_\{0,\\mu\}^\{\\star\},N=Nt,0N=N\_\{t,0\}\.
2. 2\.w=0w=0,h=qh=q:Zi=Yi2Z\_\{i\}=Y\_\{i\}^\{2\},m⋆=q0⋆m^\{\\star\}=q\_\{0\}^\{\\star\},β=β0,q⋆\\beta=\\beta\_\{0,q\}^\{\\star\},N=Nt,0N=N\_\{t,0\}\.
3. 3\.w=1w=1,h=μh=\\mu:Zi=YiZ\_\{i\}=Y\_\{i\},m⋆=μ1⋆m^\{\\star\}=\\mu\_\{1\}^\{\\star\},β=β1,μ⋆\\beta=\\beta\_\{1,\\mu\}^\{\\star\},N=Nt,1N=N\_\{t,1\}\.
4. 4\.w=1w=1,h=qh=q:Zi=Yi2Z\_\{i\}=Y\_\{i\}^\{2\},m⋆=q1⋆m^\{\\star\}=q\_\{1\}^\{\\star\},β=β1,q⋆\\beta=\\beta\_\{1,q\}^\{\\star\},N=Nt,1N=N\_\{t,1\}\.

The conditional sub\-exponential property carries through: by Assumption[2](https://arxiv.org/html/2606.31184#Thmassumption2),Yi2−qw⋆​\(Xi\)Y\_\{i\}^\{2\}\-q\_\{w\}^\{\\star\}\(X\_\{i\}\)is sub\-exponential under the predictable filtration, with norm controlled by the fourth momentCqC\_\{q\}\. The Bayesian Gram event𝒢n\\mathcal\{G\}\_\{n\}is shared across all four applications\. Each rate becomesεw,h,t=Nt,w−βw,h⋆/\(2​βw,h⋆\+d\)​\(log⁡Nt,w\)A,\\varepsilon\_\{w,h,t\}=N\_\{t,w\}^\{\-\\beta\_\{w,h\}^\{\\star\}/\(2\\beta\_\{w,h\}^\{\\star\}\+d\)\}\(\\log N\_\{t,w\}\)^\{A\},and the contraction statements in Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)follow\.□\\square

## 9Proof of Proposition[2](https://arxiv.org/html/2606.31184#Thmtheorem2)

##### Goal\.

Show that the posterior\-Neyman propensityetBe\_\{t\}^\{\\rm B\}converges inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)to the oracle Neyman propensitye⋆e^\{\\star\}\.

##### Strategy\.

Two short steps:

1. 1\.*Variance contraction\.*The Bayesian residual\-variance beliefv¯w,t=𝔼Πt​\[Π\[cV,CV\]​\(qw−μw2\)\]\\bar\{v\}\_\{w,t\}=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\(q\_\{w\}\-\\mu\_\{w\}^\{2\}\)\]contracts tovw⋆v\_\{w\}^\{\\star\}inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)\. This is essentially the chain rule applied to the contractions ofμw\\mu\_\{w\}andqwq\_\{w\}from Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1), with care for the conditional Jensen gap caused by the clip and the square\.
2. 2\.*Lipschitz Neyman map\.*The propensity is a smooth function of\(v0,v1\)\(v\_\{0\},v\_\{1\}\)on the bounded box\[cV,CV\]2\[c\_\{V\},C\_\{V\}\]^\{2\}\. So once the inputs are close, the outputs are close\.

The clipclip\[η,1−η\]\\mathrm\{clip\}\_\{\[\\eta,1\-\\eta\]\}is non\-expansive, so the proof ignores it\.

### 9\.1Step 1: posterior variance belief contracts

Recall the variance projectionvw​\(x\)=Π\[cV,CV\]​\{qw​\(x\)−μw​\(x\)2\}v\_\{w\}\(x\)=\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\\\!\\bigl\\\{q\_\{w\}\(x\)\-\\mu\_\{w\}\(x\)^\{2\}\\bigr\\\}and the posterior variance beliefv¯w,t​\(x\)=𝔼Πt​\[vw​\(x\)∣ℋt\]\.\\bar\{v\}\_\{w,t\}\(x\)=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[v\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]\.

###### Lemma 17\(Variance posterior contraction\)

Under the conditions of Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1),‖v¯w,t−vw⋆‖L2​\(PX\)→𝑃0\.\\\|\\bar\{v\}\_\{w,t\}\-v\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\xrightarrow\{P\}0\.

Proof\.Letμ~w​\(x\)=𝔼Πt​\[μw​\(x\)∣ℋt\]\\widetilde\{\\mu\}\_\{w\}\(x\)=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]andq~w​\(x\)=𝔼Πt​\[qw​\(x\)∣ℋt\]\\widetilde\{q\}\_\{w\}\(x\)=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[q\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]denote the posterior means\. Define the \(un\-clipped\) posterior variance beliefv~w​\(x\):=q~w​\(x\)−μ~w​\(x\)2\\widetilde\{v\}\_\{w\}\(x\):=\\widetilde\{q\}\_\{w\}\(x\)\-\\widetilde\{\\mu\}\_\{w\}\(x\)^\{2\}\. We bound‖v¯w,t−vw⋆‖L2​\(PX\)\\\|\\bar\{v\}\_\{w,t\}\-v\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}in two steps\.

*\(a\) Uncentered approximation\.*Add and subtractv~w−\(μ~w2−𝔼Πt​\[μw2\]\)\\widetilde\{v\}\_\{w\}\-\(\\widetilde\{\\mu\}\_\{w\}^\{2\}\-\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}^\{2\}\]\)insidev¯w,t\\bar\{v\}\_\{w,t\}:

v¯w,t​\(x\)\\displaystyle\\bar\{v\}\_\{w,t\}\(x\)=𝔼Πt​\[Π\[cV,CV\]​\(qw​\(x\)−μw​\(x\)2\)∣ℋt\]\\displaystyle=\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\(q\_\{w\}\(x\)\-\\mu\_\{w\}\(x\)^\{2\}\)\\mid\\mathcal\{H\}\_\{t\}\]=Π\[cV,CV\]​\(q~w​\(x\)−𝔼Πt​\[μw​\(x\)2∣ℋt\]\)\+Rt​\(x\),\\displaystyle=\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\\bigl\(\\widetilde\{q\}\_\{w\}\(x\)\-\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)^\{2\}\\mid\\mathcal\{H\}\_\{t\}\]\\bigr\)\+\\mathrm\{R\}\_\{t\}\(x\),where the residualRt​\(x\)\\mathrm\{R\}\_\{t\}\(x\)accounts for the conditional Jensen gap between the posterior expectation of the projection and the projection of the posterior expectation\. The clip is 1\-Lipschitz, so by Jensen’s inequality\|Rt​\(x\)\|≤𝔼Πt​\[\|qw​\(x\)−μw​\(x\)2−q~w​\(x\)\+𝔼Πt​μw2\|∣ℋt\]\|\\mathrm\{R\}\_\{t\}\(x\)\|\\leq\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\|q\_\{w\}\(x\)\-\\mu\_\{w\}\(x\)^\{2\}\-\\widetilde\{q\}\_\{w\}\(x\)\+\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\\mu\_\{w\}^\{2\}\|\\mid\\mathcal\{H\}\_\{t\}\]\. This residual is dominated by the conditional standard deviationsSDΠt​\(qw​\(x\)\)\+SDΠt​\(μw2​\(x\)\)\\mathrm\{SD\}\_\{\\Pi\_\{t\}\}\(q\_\{w\}\(x\)\)\+\\mathrm\{SD\}\_\{\\Pi\_\{t\}\}\(\\mu\_\{w\}^\{2\}\(x\)\), which both vanish inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)by Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)\(since posterior contraction implies posterior variance contraction at the same rate up to a constant factor; this is the standard ”posterior second\-moment contraction” corollary of small\-ball prior mass\)\.

*\(b\) Plug\-in error\.*By the bias–variance decomposition of the posterior squared\-mean,𝔼Πt​\[μw​\(x\)2∣ℋt\]=μ~w​\(x\)2\+VarΠt​\[μw​\(x\)∣ℋt\]\.\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)^\{2\}\\mid\\mathcal\{H\}\_\{t\}\]=\\widetilde\{\\mu\}\_\{w\}\(x\)^\{2\}\+\\mathrm\{Var\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]\.Hence

q~w​\(x\)−𝔼Πt​\[μw​\(x\)2∣ℋt\]=q~w​\(x\)−μ~w​\(x\)2−VarΠt​\[μw​\(x\)∣ℋt\]\.\\widetilde\{q\}\_\{w\}\(x\)\-\\mathbb\{E\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)^\{2\}\\mid\\mathcal\{H\}\_\{t\}\]=\\widetilde\{q\}\_\{w\}\(x\)\-\\widetilde\{\\mu\}\_\{w\}\(x\)^\{2\}\-\\mathrm\{Var\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]\.So

v¯w,t​\(x\)\\displaystyle\\bar\{v\}\_\{w,t\}\(x\)=Π\[cV,CV\]​\(q~w​\(x\)−μ~w​\(x\)2−VarΠt​\[μw​\(x\)∣ℋt\]\)\+Rt​\(x\)\.\\displaystyle=\\Pi\_\{\[c\_\{V\},C\_\{V\}\]\}\\\!\\bigl\(\\widetilde\{q\}\_\{w\}\(x\)\-\\widetilde\{\\mu\}\_\{w\}\(x\)^\{2\}\-\\mathrm\{Var\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]\\bigr\)\+\\mathrm\{R\}\_\{t\}\(x\)\.The clip is 1\-Lipschitz, hence

\|v¯w,t​\(x\)−vw⋆​\(x\)\|\\displaystyle\|\\bar\{v\}\_\{w,t\}\(x\)\-v\_\{w\}^\{\\star\}\(x\)\|≤\|q~w​\(x\)−qw⋆​\(x\)\|\+\|μ~w​\(x\)2−μw⋆​\(x\)2\|\\displaystyle\\leq\|\\widetilde\{q\}\_\{w\}\(x\)\-q\_\{w\}^\{\\star\}\(x\)\|\+\|\\widetilde\{\\mu\}\_\{w\}\(x\)^\{2\}\-\\mu\_\{w\}^\{\\star\}\(x\)^\{2\}\|\+VarΠt​\[μw​\(x\)∣ℋt\]\+\|Rt​\(x\)\|\.\\displaystyle\\quad\+\\mathrm\{Var\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\(x\)\\mid\\mathcal\{H\}\_\{t\}\]\+\|\\mathrm\{R\}\_\{t\}\(x\)\|\.
*\(c\) Bound the four terms inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)\.*

- •‖q~w−qw⋆‖L2​\(PX\)→0\\\|\\widetilde\{q\}\_\{w\}\-q\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\to 0by Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)applied toqq\.
- •μ~w2−μw⋆2=\(μ~w−μw⋆\)​\(μ~w\+μw⋆\)\\widetilde\{\\mu\}\_\{w\}^\{2\}\-\\mu\_\{w\}^\{\\star 2\}=\(\\widetilde\{\\mu\}\_\{w\}\-\\mu\_\{w\}^\{\\star\}\)\(\\widetilde\{\\mu\}\_\{w\}\+\\mu\_\{w\}^\{\\star\}\)\. On the high\-probability sieve from Appendix[8](https://arxiv.org/html/2606.31184#S8),\|μ~w\|≤Bμ\|\\widetilde\{\\mu\}\_\{w\}\|\\leq B\_\{\\mu\}uniformly, and\|μw⋆\|≤Bμ\|\\mu\_\{w\}^\{\\star\}\|\\leq B\_\{\\mu\}by Assumption[1](https://arxiv.org/html/2606.31184#Thmassumption1)\(the fourth\-moment bound implies a uniformL∞L\_\{\\infty\}bound up to a vanishing event\), so\|μ~w2−μw⋆2\|≤2​Bμ​\|μ~w−μw⋆\|\.\|\\widetilde\{\\mu\}\_\{w\}^\{2\}\-\\mu\_\{w\}^\{\\star 2\}\|\\leq 2B\_\{\\mu\}\|\\widetilde\{\\mu\}\_\{w\}\-\\mu\_\{w\}^\{\\star\}\|\.Thus‖μ~w2−μw⋆2‖L2​\(PX\)≤2​Bμ​‖μ~w−μw⋆‖L2​\(PX\)→0\\\|\\widetilde\{\\mu\}\_\{w\}^\{2\}\-\\mu\_\{w\}^\{\\star 2\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq 2B\_\{\\mu\}\\\|\\widetilde\{\\mu\}\_\{w\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\to 0\.
- •VarΠt​\[μw∣ℋt\]→0\\mathrm\{Var\}\_\{\\Pi\_\{t\}\}\[\\mu\_\{w\}\\mid\\mathcal\{H\}\_\{t\}\]\\to 0inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)because posterior contraction implies posterior variance contraction at rateεμ,w,t2\\varepsilon\_\{\\mu,w,t\}^\{2\}\.
- •\|Rt\|→0\|\\mathrm\{R\}\_\{t\}\|\\to 0inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)for the same reason\.

Combining the four bounds gives Lemma[17](https://arxiv.org/html/2606.31184#Thmtheorem17)\.□\\square

### 9\.2Step 2: Lipschitz Neyman map

###### Lemma 18\(Lipschitz Neyman map\)

DefineΦ:\[cV,CV\]2→\[ηΦ,1−ηΦ\]\\Phi:\[c\_\{V\},C\_\{V\}\]^\{2\}\\to\[\\eta\_\{\\Phi\},1\-\\eta\_\{\\Phi\}\]byΦ​\(v0,v1\)=v1v0\+v1\.\\Phi\(v\_\{0\},v\_\{1\}\)=\\frac\{\\sqrt\{v\_\{1\}\}\}\{\\sqrt\{v\_\{0\}\}\+\\sqrt\{v\_\{1\}\}\}\.Then for all\(v0,v1\),\(v0′,v1′\)∈\[cV,CV\]2\(v\_\{0\},v\_\{1\}\),\(v\_\{0\}^\{\\prime\},v\_\{1\}^\{\\prime\}\)\\in\[c\_\{V\},C\_\{V\}\]^\{2\},

\|Φ​\(v0,v1\)−Φ​\(v0′,v1′\)\|≤LNey​\(\|v0−v0′\|\+\|v1−v1′\|\),LNey=14​cV\+CV4​cV3/2​cV≤CcV3/2\.\|\\Phi\(v\_\{0\},v\_\{1\}\)\-\\Phi\(v\_\{0\}^\{\\prime\},v\_\{1\}^\{\\prime\}\)\|\\leq L\_\{\\rm Ney\}\\bigl\(\|v\_\{0\}\-v\_\{0\}^\{\\prime\}\|\+\|v\_\{1\}\-v\_\{1\}^\{\\prime\}\|\\bigr\),\\quad L\_\{\\rm Ney\}=\\frac\{1\}\{4\\sqrt\{c\_\{V\}\}\}\+\\frac\{\\sqrt\{C\_\{V\}\}\}\{4\\,c\_\{V\}^\{3/2\}\\sqrt\{c\_\{V\}\}\}\\leq\\frac\{C\}\{c\_\{V\}^\{3/2\}\}\.Moreover the clipclip\[η,1−η\]\\mathrm\{clip\}\_\{\[\\eta,1\-\\eta\]\}is 1\-Lipschitz, soΦclipped=clip∘Φ\\Phi^\{\\rm clipped\}=\\mathrm\{clip\}\\circ\\Phiis alsoLNeyL\_\{\\rm Ney\}\-Lipschitz on\[cV,CV\]2\[c\_\{V\},C\_\{V\}\]^\{2\}\.

Proof\.Φ\\Phiis differentiable on\[cV,CV\]2\[c\_\{V\},C\_\{V\}\]^\{2\}\. Compute

∂Φ∂v0=−v12​v0​\(v0\+v1\)2,∂Φ∂v1=v02​v1​\(v0\+v1\)2\.\\frac\{\\partial\\Phi\}\{\\partial v\_\{0\}\}=\-\\frac\{\\sqrt\{v\_\{1\}\}\}\{2\\sqrt\{v\_\{0\}\}\\,\(\\sqrt\{v\_\{0\}\}\+\\sqrt\{v\_\{1\}\}\)^\{2\}\},\\qquad\\frac\{\\partial\\Phi\}\{\\partial v\_\{1\}\}=\\frac\{\\sqrt\{v\_\{0\}\}\}\{2\\sqrt\{v\_\{1\}\}\\,\(\\sqrt\{v\_\{0\}\}\+\\sqrt\{v\_\{1\}\}\)^\{2\}\}\.Both are bounded in absolute value byCV/\(4​cV​\(cV\)⋅cV\)≤LNey\\sqrt\{C\_\{V\}\}/\(4c\_\{V\}\(\\sqrt\{c\_\{V\}\}\)\\cdot\\sqrt\{c\_\{V\}\}\)\\leq L\_\{\\rm Ney\}on\[cV,CV\]2\[c\_\{V\},C\_\{V\}\]^\{2\}\. Mean\-value theorem gives the Lipschitz bound\.□\\square

### 9\.3Step 3: design convergence

By definition,etB​\(x\)=clip\[η,1−η\]​Φ​\(v¯0,t−1​\(x\),v¯1,t−1​\(x\)\)e\_\{t\}^\{\\rm B\}\(x\)=\\mathrm\{clip\}\_\{\[\\eta,1\-\\eta\]\}\\Phi\(\\bar\{v\}\_\{0,t\-1\}\(x\),\\bar\{v\}\_\{1,t\-1\}\(x\)\)ande⋆​\(x\)=Φ​\(v0⋆​\(x\),v1⋆​\(x\)\)e^\{\\star\}\(x\)=\\Phi\(v\_\{0\}^\{\\star\}\(x\),v\_\{1\}^\{\\star\}\(x\)\)\. By Lemma[18](https://arxiv.org/html/2606.31184#Thmtheorem18),

\|etB​\(x\)−e⋆​\(x\)\|≤LNey​\(\|v¯0,t−1​\(x\)−v0⋆​\(x\)\|\+\|v¯1,t−1​\(x\)−v1⋆​\(x\)\|\)\.\|e\_\{t\}^\{\\rm B\}\(x\)\-e^\{\\star\}\(x\)\|\\leq L\_\{\\rm Ney\}\\Bigl\(\|\\bar\{v\}\_\{0,t\-1\}\(x\)\-v\_\{0\}^\{\\star\}\(x\)\|\+\|\\bar\{v\}\_\{1,t\-1\}\(x\)\-v\_\{1\}^\{\\star\}\(x\)\|\\Bigr\)\.Squaring and integrating againstPXP\_\{X\},

‖etB−e⋆‖L2​\(PX\)2≤2​LNey2​∑w∈\{0,1\}‖v¯w,t−1−vw⋆‖L2​\(PX\)2,\\\|e\_\{t\}^\{\\rm B\}\-e^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq 2L\_\{\\rm Ney\}^\{2\}\\sum\_\{w\\in\\\{0,1\\\}\}\\\|\\bar\{v\}\_\{w,t\-1\}\-v\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\},which goes to zero by Lemma[17](https://arxiv.org/html/2606.31184#Thmtheorem17)\.□\\square

## 10Proof of Theorem[3](https://arxiv.org/html/2606.31184#Thmtheorem3)

The decisive feature of the design analyzed here is that the propensityet​\(⋅\)e\_\{t\}\(\\cdot\)is*known and predictable*——ete\_\{t\}is the actual Bernoulli parameter used to sampleWtW\_\{t\}\. Under this design, AIPW with predictable nuisance estimates is per\-round unbiased and the second\-order nuisance–propensity cross term that usually drives theop​\(n−1/2\)o\_\{p\}\(n^\{\-1/2\}\)double\-robust rate disappears entirely\. As a consequence, mereL2​\(PX\)L\_\{2\}\(P\_\{X\}\)consistency of the posterior mean \(Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)\) is enough; noβmin\>d/2\\beta\_\{\\min\}\>d/2assumption is needed\. We follow the AIPW asymptotic normality argument of\(Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17)\)in the adaptive\-design setting, specialized to our predictable Bayesian\-teacher posterior means and known clipped propensities\.

### 10\.1Setup

The AIPW estimator is

τ^AIPW=1n​∑t=1nψt​\(μ^0,t−1,μ^1,t−1,et\),\\widehat\{\\tau\}\_\{\\rm AIPW\}=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\psi\_\{t\}\(\\widehat\{\\mu\}\_\{0,t\-1\},\\widehat\{\\mu\}\_\{1,t\-1\},e\_\{t\}\),\(26\)ψt​\(μ0,μ1,e\):=μ1​\(Xt\)−μ0​\(Xt\)\+Wt​\(Yt−μ1​\(Xt\)\)e−\(1−Wt\)​\(Yt−μ0​\(Xt\)\)1−e,\\psi\_\{t\}\(\\mu\_\{0\},\\mu\_\{1\},e\):=\\mu\_\{1\}\(X\_\{t\}\)\-\\mu\_\{0\}\(X\_\{t\}\)\+\\frac\{W\_\{t\}\(Y\_\{t\}\-\\mu\_\{1\}\(X\_\{t\}\)\)\}\{e\}\-\\frac\{\(1\-W\_\{t\}\)\(Y\_\{t\}\-\\mu\_\{0\}\(X\_\{t\}\)\)\}\{1\-e\},whereμ^w,t−1\\widehat\{\\mu\}\_\{w,t\-1\}is the posterior mean computed fromℱt−1\\mathcal\{F\}\_\{t\-1\}\(the running predictable posterior\)\. By constructionμ^w,t−1\\widehat\{\\mu\}\_\{w,t\-1\}andete\_\{t\}areℱtX\\mathcal\{F\}\_\{t\}^\{X\}\-measurable\. For batch inference one may equivalently useμ^w\\widehat\{\\mu\}\_\{w\}trained on the full trajectory and recover the same conclusion via cross\-fitting; see\(Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17)\)for details\.

The oracle predictable score is

ψtpred:=\(μ1⋆−μ0⋆\)​\(Xt\)−τ⋆\+Wt−etet​\(1−et\)​\[\(1−et\)​\(Yt−μ1⋆​\(Xt\)\)−et​\(Yt−μ0⋆​\(Xt\)\)\]\.\\psi\_\{t\}^\{\\rm pred\}:=\(\\mu\_\{1\}^\{\\star\}\-\\mu\_\{0\}^\{\\star\}\)\(X\_\{t\}\)\-\\tau^\{\\star\}\+\\frac\{W\_\{t\}\-e\_\{t\}\}\{e\_\{t\}\(1\-e\_\{t\}\)\}\\bigl\[\(1\-e\_\{t\}\)\(Y\_\{t\}\-\\mu\_\{1\}^\{\\star\}\(X\_\{t\}\)\)\-e\_\{t\}\(Y\_\{t\}\-\\mu\_\{0\}^\{\\star\}\(X\_\{t\}\)\)\\bigr\]\.\(27\)
###### Lemma 20\(Predictable score is MDS\)

\(ψtpred\)t≥1\(\\psi\_\{t\}^\{\\rm pred\}\)\_\{t\\geq 1\}is a martingale\-difference sequence w\.r\.t\. the filtration \([7](https://arxiv.org/html/2606.31184#S6.E7)\):𝔼​\[ψtpred∣ℱt\]=0\\mathbb\{E\}\[\\psi\_\{t\}^\{\\rm pred\}\\mid\\mathcal\{F\}\_\{t\}\]=0\. Conditional second moment:

𝔼​\[\(ψtpred\)2∣ℱtX\]=g​\(Xt,et\),g​\(x,e\)=\(μ1⋆−μ0⋆−τ⋆\)2​\(x\)\+v1⋆​\(x\)e\+v0⋆​\(x\)1−e\.\\mathbb\{E\}\[\(\\psi\_\{t\}^\{\\rm pred\}\)^\{2\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]=g\(X\_\{t\},e\_\{t\}\),\\quad g\(x,e\)=\(\\mu\_\{1\}^\{\\star\}\-\\mu\_\{0\}^\{\\star\}\-\\tau^\{\\star\}\)^\{2\}\(x\)\+\\frac\{v\_\{1\}^\{\\star\}\(x\)\}\{e\}\+\\frac\{v\_\{0\}^\{\\star\}\(x\)\}\{1\-e\}\.\(28\)

Proof\.The bias term\(μ1⋆−μ0⋆\)​\(Xt\)−τ⋆\(\\mu\_\{1\}^\{\\star\}\-\\mu\_\{0\}^\{\\star\}\)\(X\_\{t\}\)\-\\tau^\{\\star\}hasPXP\_\{X\}\-mean zero sinceXt∼PXX\_\{t\}\\sim P\_\{X\}independent ofℱt\\mathcal\{F\}\_\{t\}\. The IPW correction has predictable mean zero because𝔼​\[Wt∣ℱtX\]=et\\mathbb\{E\}\[W\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]=e\_\{t\}, and an explicitWtW\_\{t\}\-conditioning gives \([28](https://arxiv.org/html/2606.31184#S10.E28)\) \(cf\.\(Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17), Proposition 1\)\)\.□\\square

### 10\.2Decomposition: predictable MDS plus a vanishing nuisance MDS

###### Proposition 21\(AIPW expansion\)

n​\(τ^AIPW−τ⋆\)=Mnpred\+Rn\(nuis\),\\sqrt\{n\}\(\\widehat\{\\tau\}\_\{\\rm AIPW\}\-\\tau^\{\\star\}\)=M\_\{n\}^\{\\rm pred\}\+R\_\{n\}^\{\\rm\(nuis\)\},where

Mnpred\\displaystyle M\_\{n\}^\{\\rm pred\}=1n​∑t=1nψtpred,\\displaystyle=\\frac\{1\}\{\\sqrt\{n\}\}\\sum\_\{t=1\}^\{n\}\\psi\_\{t\}^\{\\rm pred\},Rn\(nuis\)\\displaystyle R\_\{n\}^\{\\rm\(nuis\)\}=1n​∑t=1n\[\(μ^1,t−1−μ1⋆\)​\(Xt\)​\(1−Wt/et\)−\(μ^0,t−1−μ0⋆\)​\(Xt\)​\(1−\(1−Wt\)/\(1−et\)\)\]\.\\displaystyle=\\frac\{1\}\{\\sqrt\{n\}\}\\sum\_\{t=1\}^\{n\}\\Bigl\[\(\\widehat\{\\mu\}\_\{1,t\-1\}\-\\mu\_\{1\}^\{\\star\}\)\(X\_\{t\}\)\\bigl\(1\-W\_\{t\}/e\_\{t\}\\bigr\)\-\(\\widehat\{\\mu\}\_\{0,t\-1\}\-\\mu\_\{0\}^\{\\star\}\)\(X\_\{t\}\)\\bigl\(1\-\(1\-W\_\{t\}\)/\(1\-e\_\{t\}\)\\bigr\)\\Bigr\]\.Both summands are MDS w\.r\.t\.\(ℱtW\)\(\\mathcal\{F\}\_\{t\}^\{W\}\): each centered factor1−Wt/et1\-W\_\{t\}/e\_\{t\},1−\(1−Wt\)/\(1−et\)1\-\(1\-W\_\{t\}\)/\(1\-e\_\{t\}\)has𝔼\[⋅∣ℱtX\]=0\\mathbb\{E\}\[\\cdot\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]=0, andμ^w,t−1\\widehat\{\\mu\}\_\{w,t\-1\}isℱtX\\mathcal\{F\}\_\{t\}^\{X\}\-measurable\.

Proof\.Add and subtractψt​\(μ0⋆,μ1⋆,et\)=ψtpred\+τ⋆\\psi\_\{t\}\(\\mu\_\{0\}^\{\\star\},\\mu\_\{1\}^\{\\star\},e\_\{t\}\)=\\psi\_\{t\}^\{\\rm pred\}\+\\tau^\{\\star\}inside \([26](https://arxiv.org/html/2606.31184#S10.E26)\)\. The remainder is exactlyRn\(nuis\)R\_\{n\}^\{\\rm\(nuis\)\}\. Predictable mean\-zero of the centered IPW factor follows from𝔼​\[Wt∣ℱtX\]=et\\mathbb\{E\}\[W\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]=e\_\{t\}\.□\\square

###### Lemma 22\(Nuisance term vanishes under mereL2L\_\{2\}consistency\)

Suppose𝔼​‖μ^w,t−1−μw⋆‖L2​\(PX\)2→0\\mathbb\{E\}\\\|\\widehat\{\\mu\}\_\{w,t\-1\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\to 0ast→∞t\\to\\infty\. ThenRn\(nuis\)=oP​\(1\)R\_\{n\}^\{\\rm\(nuis\)\}=o\_\{P\}\(1\)\.

Proof\.By orthogonality of the MDS terms,

𝔼​\[\(Rn\(nuis\)\)2\]=1n​∑t=1n𝔼​\[\(μ^w,t−1−μw⋆\)2​\(Xt\)⋅Var​\(1−Wt/et∣ℱtX\)\]≤Cη2⋅1n​∑t=1n𝔼​‖μ^w,t−1−μw⋆‖L2​\(PX\)2,\\mathbb\{E\}\[\(R\_\{n\}^\{\\rm\(nuis\)\}\)^\{2\}\]=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\\\!\\left\[\(\\widehat\{\\mu\}\_\{w,t\-1\}\-\\mu\_\{w\}^\{\\star\}\)^\{2\}\(X\_\{t\}\)\\cdot\\mathrm\{Var\}\(1\-W\_\{t\}/e\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\)\\right\]\\leq\\frac\{C\}\{\\eta^\{2\}\}\\cdot\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\\\|\\widehat\{\\mu\}\_\{w,t\-1\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\},sinceVar​\(Wt/et∣ℱtX\)=\(1−et\)/et≤1/η−1\\mathrm\{Var\}\(W\_\{t\}/e\_\{t\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\)=\(1\-e\_\{t\}\)/e\_\{t\}\\leq 1/\\eta\-1\. Cesàro averaging on the per\-round MSE — which goes to zero under Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)— yields𝔼​\[\(Rn\(nuis\)\)2\]→0\\mathbb\{E\}\[\(R\_\{n\}^\{\\rm\(nuis\)\}\)^\{2\}\]\\to 0, henceRn\(nuis\)=oP​\(1\)R\_\{n\}^\{\\rm\(nuis\)\}=o\_\{P\}\(1\)\.□\\square

##### Why no rate condition is needed\.

The standard double machine learning bound\|Rn\|≤n​‖μ^w−μw⋆‖​‖e^−e⋆‖\|R\_\{n\}\|\\leq\\sqrt\{n\}\\\|\\widehat\{\\mu\}\_\{w\}\-\\mu\_\{w\}^\{\\star\}\\\|\\,\\\|\\widehat\{e\}\-e^\{\\star\}\\\|forces a product rate becausee^\\widehat\{e\}is estimated from data\. Hereete\_\{t\}is the*actual*Bernoulli parameter — there is nothing to estimate\. The IPW correction with the trueete\_\{t\}is per\-round unbiased givenℱtX\\mathcal\{F\}\_\{t\}^\{X\}, so the nuisance term collapses to a single MDS factor, and itsL2L\_\{2\}size is governed by𝔼​‖μ^w,t−1−μw⋆‖2\\mathbb\{E\}\\\|\\widehat\{\\mu\}\_\{w,t\-1\}\-\\mu\_\{w\}^\{\\star\}\\\|^\{2\}alone\. This is precisely the gain analyzed by\(Li et al\.,[2024](https://arxiv.org/html/2606.31184#bib.bib17)\)for adaptive designs with known propensities\.

### 10\.3Triangular\-array martingale CLT forMnpredM\_\{n\}^\{\\rm pred\}

We invoke Lemma[11](https://arxiv.org/html/2606.31184#Thmtheorem11):

- •*Conditional variance\.*By Proposition[2](https://arxiv.org/html/2606.31184#Thmtheorem2)and dominated convergence \(usingg​\(x,⋅\)g\(x,\\cdot\)continuous and bounded on\[η,1−η\]\[\\eta,1\-\\eta\]\),1n​∑t=1ng​\(Xt,et\)→𝑃V⋆:=𝔼PX​\[g​\(X,e⋆​\(X\)\)\]\.\\tfrac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}g\(X\_\{t\},e\_\{t\}\)\\xrightarrow\{P\}V^\{\\star\}:=\\mathbb\{E\}\_\{P\_\{X\}\}\[g\(X,e^\{\\star\}\(X\)\)\]\.
- •*Conditional Lindeberg\.*Forδ\>0\\delta\>0,𝔼​\[\(ψtpred\)4∣ℱtX\]≤C/η4\\mathbb\{E\}\[\(\\psi\_\{t\}^\{\\rm pred\}\)^\{4\}\\mid\\mathcal\{F\}\_\{t\}^\{X\}\]\\leq C/\\eta^\{4\}by Assumption[1](https://arxiv.org/html/2606.31184#Thmassumption1)\(fourth\-moment bound\)\. Cauchy–Schwarz gives the Lindeberg sum≤n−1​δ−2​C/η4→0\\leq n^\{\-1\}\\delta^\{\-2\}C/\\eta^\{4\}\\to 0\.

HenceMnpred↝N​\(0,V⋆\)M\_\{n\}^\{\\rm pred\}\\rightsquigarrow N\(0,V^\{\\star\}\)stably\.

### 10\.4IdentifyingV⋆V^\{\\star\}as the Hahn bound under Neyman

###### Lemma 23\(Variance under Neyman\)

Withe⋆​\(x\)=v1⋆​\(x\)/\(v0⋆​\(x\)\+v1⋆​\(x\)\)e^\{\\star\}\(x\)=\\sqrt\{v\_\{1\}^\{\\star\}\(x\)\}/\(\\sqrt\{v\_\{0\}^\{\\star\}\(x\)\}\+\\sqrt\{v\_\{1\}^\{\\star\}\(x\)\}\),

V⋆=𝔼PX​\[\(v0⋆​\(X\)\+v1⋆​\(X\)\)2\+\(μ1⋆−μ0⋆−τ⋆\)2​\(X\)\],V^\{\\star\}=\\mathbb\{E\}\_\{P\_\{X\}\}\\\!\\bigl\[\(\\sqrt\{v\_\{0\}^\{\\star\}\(X\)\}\+\\sqrt\{v\_\{1\}^\{\\star\}\(X\)\}\)^\{2\}\+\(\\mu\_\{1\}^\{\\star\}\-\\mu\_\{0\}^\{\\star\}\-\\tau^\{\\star\}\)^\{2\}\(X\)\\bigr\],which equals the Hahn\(Hahn,[1998](https://arxiv.org/html/2606.31184#bib.bib10)\)semiparametric efficiency bound for ATE estimation under unconfounded sampling, minimized over feasible propensitiese∈\[η,1−η\]e\\in\[\\eta,1\-\\eta\]at the Neyman value\.

Proof\.Substitutee=e⋆e=e^\{\\star\}and1−e=v0⋆/\(v0⋆\+v1⋆\)1\-e=\\sqrt\{v\_\{0\}^\{\\star\}\}/\(\\sqrt\{v\_\{0\}^\{\\star\}\}\+\\sqrt\{v\_\{1\}^\{\\star\}\}\)into \([28](https://arxiv.org/html/2606.31184#S10.E28)\):v1⋆/e⋆\+v0⋆/\(1−e⋆\)=v1⋆​\(v0⋆\+v1⋆\)\+v0⋆​\(v0⋆\+v1⋆\)=\(v0⋆\+v1⋆\)2\.v\_\{1\}^\{\\star\}/e^\{\\star\}\+v\_\{0\}^\{\\star\}/\(1\-e^\{\\star\}\)=\\sqrt\{v\_\{1\}^\{\\star\}\}\(\\sqrt\{v\_\{0\}^\{\\star\}\}\+\\sqrt\{v\_\{1\}^\{\\star\}\}\)\+\\sqrt\{v\_\{0\}^\{\\star\}\}\(\\sqrt\{v\_\{0\}^\{\\star\}\}\+\\sqrt\{v\_\{1\}^\{\\star\}\}\)=\(\\sqrt\{v\_\{0\}^\{\\star\}\}\+\\sqrt\{v\_\{1\}^\{\\star\}\}\)^\{2\}\.Hahn’s efficient\-influence\-function variance is𝔼​\[v1⋆/e\+v0⋆/\(1−e\)\+\(μ1⋆−μ0⋆−τ⋆\)2\]\\mathbb\{E\}\[v\_\{1\}^\{\\star\}/e\+v\_\{0\}^\{\\star\}/\(1\-e\)\+\(\\mu\_\{1\}^\{\\star\}\-\\mu\_\{0\}^\{\\star\}\-\\tau^\{\\star\}\)^\{2\}\], and by Cauchy\-Schwarza2/e\+b2/\(1−e\)≥\(a\+b\)2a^\{2\}/e\+b^\{2\}/\(1\-e\)\\geq\(a\+b\)^\{2\}with equality ate=a/\(a\+b\)e=a/\(a\+b\), so the minimum is attained at the Neyman propensity\.□\\square

### 10\.5Conclusion

By Proposition[21](https://arxiv.org/html/2606.31184#Thmtheorem21),n​\(τ^AIPW−τ⋆\)=Mnpred\+Rn\(nuis\)\\sqrt\{n\}\(\\widehat\{\\tau\}\_\{\\rm AIPW\}\-\\tau^\{\\star\}\)=M\_\{n\}^\{\\rm pred\}\+R\_\{n\}^\{\\rm\(nuis\)\}\. Lemma[22](https://arxiv.org/html/2606.31184#Thmtheorem22)givesRn\(nuis\)=oP​\(1\)R\_\{n\}^\{\\rm\(nuis\)\}=o\_\{P\}\(1\)from Proposition[1](https://arxiv.org/html/2606.31184#Thmtheorem1)\. The MCLT in Section[10\.3](https://arxiv.org/html/2606.31184#S10.SS3)givesMnpred↝N​\(0,V⋆\)M\_\{n\}^\{\\rm pred\}\\rightsquigarrow N\(0,V^\{\\star\}\)\. Slutsky and Lemma[23](https://arxiv.org/html/2606.31184#Thmtheorem23)yieldn​\(τ^AIPW−τ⋆\)↝N​\(0,V⋆\),\\sqrt\{n\}\(\\widehat\{\\tau\}\_\{\\rm AIPW\}\-\\tau^\{\\star\}\)\\rightsquigarrow N\(0,V^\{\\star\}\),withV⋆V^\{\\star\}the Hahn semiparametric efficiency bound under Neyman allocation\.□\\square

## 11Proof of Theorem[4](https://arxiv.org/html/2606.31184#Thmtheorem4)

##### Goal\.

The Bayesian teacher does not know the true smoothnessβ⋆\\beta^\{\\star\}\. We show that placing a hyperpriorΠβ\\Pi\_\{\\beta\}on\[βmin,βmax\]\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]makes the resulting hierarchical posterior contract at the oracle rateεNad=N−β⋆/\(2​β⋆\+d\)​\(log⁡N\)A\\varepsilon\_\{N\}^\{\\rm ad\}=N^\{\-\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\(\\log N\)^\{A\}, up to logarithmic factors\. The Bayesian model selects the right complexity scale on its own\.

##### Why hierarchical Bayes adapts: shape of the marginal likelihood\.

For each smoothnessβ\\beta, the marginal likelihoodDN\(β\)=∫eℓN​\(f\)​𝑑Π0\(β\)​\(f\)D\_\{N\}^\{\(\\beta\)\}=\\int e^\{\\ell\_\{N\}\(f\)\}\\,d\\Pi\_\{0\}^\{\(\\beta\)\}\(f\)aggregates the likelihood across allffdrawn from the conditional prior\. Heuristically:

- •Forβ≈β⋆\\beta\\approx\\beta^\{\\star\}: the prior mass concentrates on functions matching the true smoothness; the data fits well;DN\(β\)D\_\{N\}^\{\(\\beta\)\}is large, of orderexp⁡\(−c⋆​N​\(εNad\)2\)\\exp\(\-c\_\{\\star\}N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)\.
- •Forβ≫β⋆\\beta\\gg\\beta^\{\\star\}\(oversmoothed\): the prior puts almost all mass on functions smoother thanm⋆m^\{\\star\}; the truncation bias is huge and the squared\-error is large;DN\(β\)D\_\{N\}^\{\(\\beta\)\}decays exponentially faster thanDN\(β⋆\)D\_\{N\}^\{\(\\beta^\{\\star\}\)\}\.
- •Forβ≪β⋆\\beta\\ll\\beta^\{\\star\}\(undersmoothed\): the prior is too rough, the variance of the posterior mean is large, and the squared\-error term again dominates;DN\(β\)D\_\{N\}^\{\(\\beta\)\}also decays faster\.

The transition between these regimes is sharp: a smoothness window of widthO​\(1/log⁡N\)O\(1/\\log N\)aroundβ⋆\\beta^\{\\star\}already gets withineO​\(1\)e^\{O\(1\)\}of the optimum\. That window is what we call the*near\-oracle slab*INI\_\{N\}\. OutsideINI\_\{N\}the posterior weight onβ\\betais exponentially suppressed; inside, the conditional posteriorΠ\(⋅∣β,𝒟N\)\\Pi\(\\cdot\\mid\\beta,\\mathcal\{D\}\_\{N\}\)contracts at the oracle rate\.

##### Strategy\.

Given the goal of boundingΠ​\(‖f−m⋆‖\>M​εNad∣𝒟N\)\\Pi\(\\\|f\-m^\{\\star\}\\\|\>M\\varepsilon\_\{N\}^\{\\rm ad\}\\mid\\mathcal\{D\}\_\{N\}\):

1. 1\.*Slab\.*DefineIN⊂\[βmin,βmax\]I\_\{N\}\\subset\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]of width1/log⁡N1/\\log Naroundβ⋆\\beta^\{\\star\}\. By Lipschitzness ofr​\(β\)=β/\(2​β\+d\)r\(\\beta\)=\\beta/\(2\\beta\+d\), everyβ∈IN\\beta\\in I\_\{N\}induces a contraction rate withineO​\(1\)⋅εNade^\{O\(1\)\}\\cdot\\varepsilon\_\{N\}^\{\\rm ad\}of the oracle\. In particular the prior small\-ball at scaleεNad\\varepsilon\_\{N\}^\{\\rm ad\}is uniform acrossINI\_\{N\}\.
2. 2\.*Hierarchical denominator\.*Restricting the marginal likelihood integral toβ∈IN\\beta\\in I\_\{N\}givesDN≥Πβ​\(IN\)⋅e−C​N​\(εNad\)2≥\(log⁡N\)−1​e−C​N​\(εNad\)2D\_\{N\}\\geq\\Pi\_\{\\beta\}\(I\_\{N\}\)\\cdot e^\{\-CN\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\\geq\(\\log N\)^\{\-1\}e^\{\-CN\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\. The\(log⁡N\)−1\(\\log N\)^\{\-1\}is what we pay for adaptation\.
3. 3\.*Hierarchical numerator\.*Use a single sieve covering the*union*of per\-β\\betasieves from Appendix[8\.4](https://arxiv.org/html/2606.31184#S8.SS4)\. Its entropy is stillO​\(N​\(εNad\)2\)O\(N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)because the largest per\-β\\betaeffective dimension is atβmin\\beta\_\{\\min\}\. Tests \+ Bernstein\-MDS as in Appendix[8\.5](https://arxiv.org/html/2606.31184#S8.SS5)bound the numerator bye−c​M2​N​\(εNad\)2e^\{\-cM^\{2\}N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\.

Combining \(II\) and \(III\) gives the contraction at rateεNad\\varepsilon\_\{N\}^\{\\rm ad\}\.

We extend the fixed\-smoothness argument of Appendix[8](https://arxiv.org/html/2606.31184#S8)to the hierarchical\-Bayes setting where the smoothnessβ\\betais unknown and given a hyperpriorΠβ\\Pi\_\{\\beta\}on\[βmin,βmax\]\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]with bounded density\. The proof follows the standard “near\-oracle smoothness slab \+ denominator \+ tests” template, with care for the predictable filtration and the sub\-exponential quasi\-likelihood used by the Bayesian teacher\.

### 11\.1Setup and adaptive target rate

Recall the hierarchical posteriord​Πt​\(f,β\)∝eℓN​\(f\)​d​Π0\(β\)​\(f\)​d​Πβ​\(β\),d\\Pi\_\{t\}\(f,\\beta\)\\propto e^\{\\ell\_\{N\}\(f\)\}\\,d\\Pi^\{\(\\beta\)\}\_\{0\}\(f\)\\,d\\Pi\_\{\\beta\}\(\\beta\),withd​Π0\(β\)​\(f\)d\\Pi^\{\(\\beta\)\}\_\{0\}\(f\)the conditional Gaussian\-series prior at smoothnessβ\\beta\. As before, setN=Nt,wN=N\_\{t,w\}\.

Define the adaptive target rate

εNad:=N−β⋆/\(2​β⋆\+d\)​\(log⁡N\)A,JN⋆:=⌈\(N/log⁡N\)d/\(2​β⋆\+d\)⌉,\\varepsilon\_\{N\}^\{\\rm ad\}:=N^\{\-\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\(\\log N\)^\{A\},\\qquad J\_\{N\}^\{\\star\}:=\\lceil\(N/\\log N\)^\{d/\(2\\beta^\{\\star\}\+d\)\}\\rceil,\(29\)whereβ⋆∈\[βmin,βmax\]\\beta^\{\\star\}\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]is the true smoothness ofm⋆m^\{\\star\}, andA\>0A\>0is a structural constant chosen large enough\.

### 11\.2Step 1: near\-oracle slab

###### Lemma 25\(Lipschitz exponent\)

The mapr:β↦β/\(2​β\+d\)r:\\beta\\mapsto\\beta/\(2\\beta\+d\)is Lipschitz on\[βmin,βmax\]\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]with constantLr=d/\(2​βmin\+d\)2L\_\{r\}=d/\(2\\beta\_\{\\min\}\+d\)^\{2\}\. In particular, for anyβ,β′\\beta,\\beta^\{\\prime\}in this interval,N\|r​\(β\)−r​\(β′\)\|≤eLr​\|β−β′\|​log⁡N\.N^\{\|r\(\\beta\)\-r\(\\beta^\{\\prime\}\)\|\}\\leq e^\{L\_\{r\}\|\\beta\-\\beta^\{\\prime\}\|\\log N\}\.

Proof\.Direct fromr′​\(β\)=d/\(2​β\+d\)2r^\{\\prime\}\(\\beta\)=d/\(2\\beta\+d\)^\{2\}\.□\\square

Define the near\-oracle slab

IN:=\{\[β⋆−ΔN,β⋆\]if​β⋆−ΔN\>βmin,\[βmin,βmin\+ΔN\]if​β⋆≤βmin\+ΔN,ΔN=c∗log⁡N,I\_\{N\}:=\\begin\{cases\}\[\\beta^\{\\star\}\-\\Delta\_\{N\},\\beta^\{\\star\}\]&\\text\{if \}\\beta^\{\\star\}\-\\Delta\_\{N\}\>\\beta\_\{\\min\},\\\\ \[\\beta\_\{\\min\},\\beta\_\{\\min\}\+\\Delta\_\{N\}\]&\\text\{if \}\\beta^\{\\star\}\\leq\\beta\_\{\\min\}\+\\Delta\_\{N\},\\end\{cases\}\\qquad\\Delta\_\{N\}=\\frac\{c\_\{\*\}\}\{\\log N\},\(30\)for a small constantc∗c\_\{\*\}chosen below\. By Lemma[25](https://arxiv.org/html/2606.31184#Thmtheorem25)andΔN​log⁡N=c∗=O​\(1\)\\Delta\_\{N\}\\log N=c\_\{\*\}=O\(1\), everyβ∈IN\\beta\\in I\_\{N\}satisfiesNr​\(β\)≍Nr​\(β⋆\)N^\{r\(\\beta\)\}\\asymp N^\{r\(\\beta^\{\\star\}\)\}, i\.e\., the rate is preserved up to constants\. SinceΠβ\\Pi\_\{\\beta\}has density bounded below bycβ\>0c\_\{\\beta\}\>0,

Πβ​\(IN\)≥cβ​ΔN≍1/log⁡N\.\\Pi\_\{\\beta\}\(I\_\{N\}\)\\geq c\_\{\\beta\}\\Delta\_\{N\}\\asymp 1/\\log N\.\(31\)

### 11\.3Step 2: uniform prior small\-ball overINI\_\{N\}

###### Lemma 26\(Uniform prior small\-ball\)

Forβ∈IN\\beta\\in I\_\{N\}andJ=JN⋆J=J\_\{N\}^\{\\star\}from \([29](https://arxiv.org/html/2606.31184#S11.E29)\),

Π0\(β\)​\(‖f−m⋆‖L2​\(PX\)≤C​εNad\)≥exp⁡\(−C​N​\(εNad\)2\),\\Pi^\{\(\\beta\)\}\_\{0\}\\\!\\left\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq C\\varepsilon\_\{N\}^\{\\rm ad\}\\right\)\\geq\\exp\(\-CN\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\),with constants depending only on structural parameters\.

Proof\.Re\-run Step 2 of Appendix[8\.3](https://arxiv.org/html/2606.31184#S8.SS3)withβ∈IN\\beta\\in I\_\{N\}\. The bias bound∑j\>J\(θj⋆\)2≲J−2​β⋆/d≲\(εNad\)2\\sum\_\{j\>J\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\lesssim J^\{\-2\\beta^\{\\star\}/d\}\\lesssim\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}usesβ⋆\\beta^\{\\star\}, notβ\\beta, and so is unchanged\.

The low\-frequency block requires∑j≤J\(θj⋆\)2/ρj​\(β\)≲N​\(εNad\)2\.\\sum\_\{j\\leq J\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}/\\rho\_\{j\}\(\\beta\)\\lesssim N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\.Usingρj​\(β\)−1≍j1\+2​β/d=j⋅j2​β⋆/d⋅j−2​\(β⋆−β\)/d\\rho\_\{j\}\(\\beta\)^\{\-1\}\\asymp j^\{1\+2\\beta/d\}=j\\cdot j^\{2\\beta^\{\\star\}/d\}\\cdot j^\{\-2\(\\beta^\{\\star\}\-\\beta\)/d\}andj≤J≤nj\\leq J\\leq n, solog⁡j≤log⁡n\\log j\\leq\\log n:

j−2​\(β⋆−β\)/d≤e2​\|β⋆−β\|​log⁡n/d≤e2​c∗​log⁡n/\(d​log⁡N\)=O​\(1\)j^\{\-2\(\\beta^\{\\star\}\-\\beta\)/d\}\\leq e^\{2\|\\beta^\{\\star\}\-\\beta\|\\log n/d\}\\leq e^\{2c\_\{\*\}\\log n/\(d\\log N\)\}=O\(1\)forc∗c\_\{\*\}small\. Henceρj​\(β\)−1≲J​j2​β⋆/d\\rho\_\{j\}\(\\beta\)^\{\-1\}\\lesssim Jj^\{2\\beta^\{\\star\}/d\}uniformly inβ∈IN\\beta\\in I\_\{N\}and∑j≤J\(θj⋆\)2/ρj​\(β\)≲J​∑j2​β⋆/d​\(θj⋆\)2≲J​R2\.\\sum\_\{j\\leq J\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}/\\rho\_\{j\}\(\\beta\)\\lesssim J\\sum j^\{2\\beta^\{\\star\}/d\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\lesssim JR^\{2\}\.The high\-frequency tail∑j\>Jρj​\(β\)≲J−2​β/d\\sum\_\{j\>J\}\\rho\_\{j\}\(\\beta\)\\lesssim J^\{\-2\\beta/d\}also matchesJ−2​β⋆/d≲\(εNad\)2J^\{\-2\\beta^\{\\star\}/d\}\\lesssim\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}uniformly inβ∈IN\\beta\\in I\_\{N\}by the same argument applied withβ⋆→β\\beta^\{\\star\}\\to\\beta\. The rest of the small\-ball calculation goes through unchanged\.□\\square

### 11\.4Step 3: hierarchical denominator

###### Lemma 27\(Hierarchical denominator lower bound\)

On the Gram event𝒢n\\mathcal\{G\}\_\{n\}\(Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)\), with high probability,

DN:=∫eℓN​\(f\)−ℓN​\(m⋆\)​𝑑Π0\(β\)​\(f\)​𝑑Πβ​\(β\)≥exp⁡\(−CD​N​\(εNad\)2\)⋅clog⁡N\.D\_\{N\}:=\\int e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}\\,d\\Pi^\{\(\\beta\)\}\_\{0\}\(f\)\\,d\\Pi\_\{\\beta\}\(\\beta\)\\geq\\exp\(\-C\_\{D\}N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)\\cdot\\frac\{c\}\{\\log N\}\.

Proof\.By restriction to the slabINI\_\{N\},

DN≥∫INΠ0\(β\)​\(‖f−m⋆‖≤C​εNad\)⋅e−C​N​\(εNad\)2​𝑑Πβ​\(β\),D\_\{N\}\\geq\\int\_\{I\_\{N\}\}\\Pi^\{\(\\beta\)\}\_\{0\}\\bigl\(\\\|f\-m^\{\\star\}\\\|\\leq C\\varepsilon\_\{N\}^\{\\rm ad\}\\bigr\)\\cdot e^\{\-CN\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\\,d\\Pi\_\{\\beta\}\(\\beta\),where the inner factor is the uniform small\-ball lower bound on the log\-likelihood within the small ball, established as in Step 5 of Appendix[8\.6](https://arxiv.org/html/2606.31184#S8.SS6), and which is in turn≥exp⁡\(−C​N​\(εNad\)2\)\\geq\\exp\(\-CN\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)uniformly inβ∈IN\\beta\\in I\_\{N\}by Lemma[26](https://arxiv.org/html/2606.31184#Thmtheorem26)\. Combining with \([31](https://arxiv.org/html/2606.31184#S11.E31)\),DN≥cβlog⁡N​exp⁡\(−\(C\+C\)​N​\(εNad\)2\)\.D\_\{N\}\\geq\\frac\{c\_\{\\beta\}\}\{\\log N\}\\exp\(\-\(C\+C\)N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)\.The extralog⁡log⁡N\\log\\log NfromΠβ​\(IN\)\\Pi\_\{\\beta\}\(I\_\{N\}\)is absorbed into theO~\\widetilde\{O\}\-rate\.□\\square

### 11\.5Step 4: hierarchical numerator

Define the joint sieve

ℱnhier:=⋃β∈\[βmin,βmax\]ℱn\(β\),\\mathcal\{F\}\_\{n\}^\{\\rm hier\}:=\\bigcup\_\{\\beta\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]\}\\mathcal\{F\}\_\{n\}^\{\(\\beta\)\},whereℱn\(β\)\\mathcal\{F\}\_\{n\}^\{\(\\beta\)\}is the per\-β\\betasieve from Appendix[8\.4](https://arxiv.org/html/2606.31184#S8.SS4)with target rateεNad\\varepsilon\_\{N\}^\{\\rm ad\}\. Since the per\-β\\betaeffective dimension is at mostJn≍nd/\(2​βmin\+d\),J\_\{n\}\\asymp n^\{d/\(2\\beta\_\{\\min\}\+d\)\},which depends only logarithmically onβ\\beta, the entropy ofℱnhier\\mathcal\{F\}\_\{n\}^\{\\rm hier\}at scaleρ\\rhosatisfieslog⁡N​\(ρ,ℱnhier,L2​\(PX\)\)≲Jn​log⁡\(C/ρ\)≲N​\(εNad\)2\.\\log N\(\\rho,\\mathcal\{F\}\_\{n\}^\{\\rm hier\},L\_\{2\}\(P\_\{X\}\)\)\\lesssim J\_\{n\}\\log\(C/\\rho\)\\lesssim N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\.The tail outside the sieve is also exponentially small, summed over a fineβ\\beta\-grid of cardinality\|βmax−βmin\|⋅log⁡N≤C​log⁡N\|\\beta\_\{\\max\}\-\\beta\_\{\\min\}\|\\cdot\\log N\\leq C\\log N\.

The same Le Cam–Birgé tests as in Step 4 of Appendix[8\.5](https://arxiv.org/html/2606.31184#S8.SS5)apply to alternatives inℱnhier\\mathcal\{F\}\_\{n\}^\{\\rm hier\}separated byM​εNadM\\varepsilon\_\{N\}^\{\\rm ad\}\. Bernstein for sub\-exponential MDS \(Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)\) and matrix Freedman Gram concentration \(Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)\) give the same exponential errors as in the fixed\-smoothness case, withρ2\\rho^\{2\}replaced by\(εNad\)2\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\.

### 11\.6Step 5: hierarchical contraction

Combining Lemma[27](https://arxiv.org/html/2606.31184#Thmtheorem27)with the hierarchical numerator bound, on the intersection of the Gram event and the denominator event,

Π​\(‖f−m⋆‖L2​\(PX\)\>M​εNad∣𝒟N\)≤e−c​M2​N​\(εNad\)2Πβ​\(IN\)​e−CD​N​\(εNad\)2≤\(log⁡N\)​e−\(c​M2−CD\)​N​\(εNad\)2→0,\\Pi\(\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\>M\\varepsilon\_\{N\}^\{\\rm ad\}\\mid\\mathcal\{D\}\_\{N\}\)\\leq\\frac\{e^\{\-cM^\{2\}N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\}\{\\Pi\_\{\\beta\}\(I\_\{N\}\)e^\{\-C\_\{D\}N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\}\\leq\(\\log N\)\\,e^\{\-\(cM^\{2\}\-C\_\{D\}\)N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\}\\to 0,providedMMis chosen so thatc​M2−CD−1≥1cM^\{2\}\-C\_\{D\}\-1\\geq 1\. The factor oflog⁡N\\log Ncontributes at most an additiveO​\(log⁡log⁡N\)O\(\\log\\log N\)to the rate exponent and is absorbed intoO~\\widetilde\{O\}\. This proves Theorem[4](https://arxiv.org/html/2606.31184#Thmtheorem4)for the generic regression target\.

### 11\.7Step 6: adaptive contraction of the smoothness posterior

The above also implies a contraction statement for the smoothness posterior: the marginalΠ​\(β∣𝒟N\)\\Pi\(\\beta\\mid\\mathcal\{D\}\_\{N\}\)is asymptotically supported on smoothness valuesβ\\betawith rateβ/\(2​β\+d\)≥β⋆/\(2​β⋆\+d\)−c/log⁡N\\beta/\(2\\beta\+d\)\\geq\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\-c/\\log N\. Indeed, everyβ\\betaviolating this gives a denominator decay strictly faster than the rate off∈INf\\in I\_\{N\}, while the slabINI\_\{N\}supplies the denominator\. Equivalently, the posterior overβ\\betaconcentrates on smoothness levels whose induced contraction rate is within logarithmic factors of the oracle rate\.

### 11\.8Step 7: Pseudo\-Bayes generality

The argument above does not require the noise to be Gaussian\. The quasi\-likelihood\-ratioeℓN​\(f\)−ℓN​\(m⋆\)e^\{\\ell\_\{N\}\(f\)\-\\ell\_\{N\}\(m^\{\\star\}\)\}is bounded below the exact\-likelihood ratio for sub\-exponential noise, with exponential correctionexp⁡\(O​\(N​\(εNad\)2\)\)\\exp\(O\(N\(\\varepsilon\_\{N\}^\{\\rm ad\}\)^\{2\}\)\)from the Bernstein\-MDS bounds\. This correction is absorbed into the constantCDC\_\{D\}in Lemma[27](https://arxiv.org/html/2606.31184#Thmtheorem27), so the same conclusion holds\.

### 11\.9Step 8: Specialization to four targets

Apply Steps 1–7 to the four cases as in Appendix[8\.9](https://arxiv.org/html/2606.31184#S8.SS9)\. The resulting contraction rate isεw,h,tad=Nt,w−βw,h⋆/\(2​βw,h⋆\+d\)​\(log⁡Nt,w\)A\\varepsilon^\{\\rm ad\}\_\{w,h,t\}=N\_\{t,w\}^\{\-\\beta\_\{w,h\}^\{\\star\}/\(2\\beta\_\{w,h\}^\{\\star\}\+d\)\}\(\\log N\_\{t,w\}\)^\{A\}inL2​\(PX\)L\_\{2\}\(P\_\{X\}\)for each\(w,h\)∈\{0,1\}×\{μ,q\}\(w,h\)\\in\\\{0,1\\\}\\times\\\{\\mu,q\\\}\.□\\square

### Smoothness\-adaptive design consistency

Combining Theorem[4](https://arxiv.org/html/2606.31184#Thmtheorem4)with the proof of Proposition[2](https://arxiv.org/html/2606.31184#Thmtheorem2)verbatim \(substitutingεw,h,tad\\varepsilon^\{\\rm ad\}\_\{w,h,t\}forεw,h,t\\varepsilon\_\{w,h,t\}in Appendix[9](https://arxiv.org/html/2606.31184#S9)\) yields‖etHB−e⋆‖L2​\(PX\)→𝑃0,\\\|e\_\{t\}^\{\\rm HB\}\-e^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\xrightarrow\{P\}0,whereetHBe\_\{t\}^\{\\rm HB\}is the posterior Neyman propensity formed under the hierarchical smoothness posterior\. Theorem[3](https://arxiv.org/html/2606.31184#Thmtheorem3)continues to hold under the hierarchical design with the rate condition replaced by the adaptive analogueβw,h⋆/\(2​βw,h⋆\+d\)\\beta^\{\\star\}\_\{w,h\}/\(2\\beta^\{\\star\}\_\{w,h\}\+d\)\.

## 12Transformer construction and proof of Theorem[5](https://arxiv.org/html/2606.31184#Thmtheorem5)

##### Goal\.

Show that a ReLU\-attention transformer of widthO~​\(Jn\)\\widetilde\{O\}\(J\_\{n\}\)and depthO​\(log⁡n\)O\(\\log n\)contains a parameter setting whose output approximates the smoothness\-adaptive Bayesian moment estimator at the oracle rateO~​\(N−β⋆/\(2​β⋆\+d\)\)\\widetilde\{O\}\(N^\{\-\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)for every arm and every moment target\. This is what populates the trainable hypothesis class of Appendix[13](https://arxiv.org/html/2606.31184#S13)with a low\-risk comparator\.

##### Strategy: split into computation \+ statistics\.

The proof has two logically independent parts:

1. 1\.*Computational realization*\(Sections[12\.1](https://arxiv.org/html/2606.31184#S12.SS1)–[12\.6](https://arxiv.org/html/2606.31184#S12.SS6)\)\. For each smoothness expertkk, the transformer exactly executes finite\-rank masked ridge regression\. We map each piece of the Bayesian update to a transformer primitive: - •*Sample averages*\(cN=N−1​∑iϕi​Zic\_\{N\}=N^\{\-1\}\\sum\_\{i\}\\phi\_\{i\}Z\_\{i\},GN​a=N−1​∑iϕi​\(ϕi⊤​a\)G\_\{N\}a=N^\{\-1\}\\sum\_\{i\}\\phi\_\{i\}\(\\phi\_\{i\}^\{\\top\}a\),zN\(2\)=N−1​∑iZi2z\_\{N\}^\{\(2\)\}=N^\{\-1\}\\sum\_\{i\}Z\_\{i\}^\{2\}, dot products\)↔\\leftrightarrownormalized linear attention with coordinate\-selectorQ,K,VQ,K,Vmatrices\. Attention is the natural “compute a per\-history average” operation\. - •*Coordinate masking and dynamic active dimension*↔\\leftrightarrowReLU FFN\. The indicator𝟏​\{N≥Tj,k\}\\mathbf\{1\}\\\{N\\geq T\_\{j,k\}\\\}is realized exactly byσ​\(N−T\+1\)−σ​\(N−T\)\\sigma\(N\-T\+1\)\-\\sigma\(N\-T\)on integerNN, and the masked productp⋅ap\\cdot ais realized exactly on a bounded domain \(\([33](https://arxiv.org/html/2606.31184#S12.E33)\)\)\. - •*Diagonal prior shrinkage*↔\\leftrightarrowtokenwise affine FFN\. - •*Projected gradient descent*on the masked ridge objective↔\\leftrightarrowchain ofL=O​\(log⁡n\)L=O\(\\log n\)attention/FFN blocks\. Each iterate computes the gradient by attention, takes a step, and re\-projects via the mask; geometric convergence follows from aO​\(1\)O\(1\)condition number on the active subspace \(Appendix[7](https://arxiv.org/html/2606.31184#S7)\)\. - •*Penalized\-evidence MoE gate*↔\\leftrightarrowReLU softmax with scoreSk=RSSk\+Priork\+A​dk​log⁡n/NS\_\{k\}=\\mathrm\{RSS\}\_\{k\}\+\\mathrm\{Prior\}\_\{k\}\+A\\,d\_\{k\}\\log n/N\. The total error is the sum of finite\-depth GD error \(n−An^\{\-A\}\), embedding error if features are computed from raw covariates \(controllable by encoder size\), and zero score/readout error\.
2. 2\.*Statistical \(penalized\-MoE\) adaptation*\(Section[12\.7](https://arxiv.org/html/2606.31184#S12.SS7)\)\. Even though the gate uses a*computable*penalized\-likelihood approximation rather than the exact log marginal likelihood, the resulting weights still concentrate on the near\-oracle expert\. Two pieces: - •*Bad\-expert lower bound\.*Any expert with riskRk\+penk≫rN2R\_\{k\}\+\\mathrm\{pen\}\_\{k\}\\gg r\_\{N\}^\{2\}has scoreSk≥Sk⋆\+Ω​\(N​rN2\)S\_\{k\}\\geq S\_\{k^\{\\star\}\}\+\\Omega\(Nr\_\{N\}^\{2\}\)\. - •*Near\-oracle upper bound\.*The expert closest toβ⋆\\beta^\{\\star\}on the grid has scoreSk⋆≤O~​\(rN2\)S\_\{k^\{\\star\}\}\\leq\\widetilde\{O\}\(r\_\{N\}^\{2\}\)\. The exponential weightingωk∝πk​exp⁡\(−ηg​N​Sk\)\\omega\_\{k\}\\propto\\pi\_\{k\}\\exp\(\-\\eta\_\{g\}NS\_\{k\}\)then suppresses bad experts at ratee−Ω​\(N​rN2\)e^\{\-\\Omega\(Nr\_\{N\}^\{2\}\)\}, and convexity of squared norm bounds the mixture risk by the weighted sum of individual risks\.

##### Why the penaltyA​dk​log⁡n/NA\\,d\_\{k\}\\log n/Nis the right complexity term\.

The exact log marginal likelihood for ridge contains a log\-determinant of the posterior covariance, which on the active subspace is approximatelydk​log⁡Nd\_\{k\}\\log N\(onelog⁡N\\log Nfor each active coordinate’s posterior precision, which scales linearly inNN\)\. SoPriork\+A​dk​log⁡n/N\\mathrm\{Prior\}\_\{k\}\+A\\,d\_\{k\}\\log n/Nis the computable approximation to the exact−N2​RSSk\+12​log​det\-\\frac\{N\}\{2\}\\mathrm\{RSS\}\_\{k\}\+\\frac\{1\}\{2\}\\log\\detscore, up to constants\. This is why Lemma[32](https://arxiv.org/html/2606.31184#Thmtheorem32)below recovers the true risk plus complexity penaltyRk\+penkR\_\{k\}\+\\mathrm\{pen\}\_\{k\}\.

We work with a single generic regression targetZ=m⋆​\(X\)\+ξZ=m^\{\\star\}\(X\)\+\\xi,m⋆∈ℋβ⋆​\(R\)m^\{\\star\}\\in\\mathcal\{H\}^\{\\beta^\{\\star\}\}\(R\)\. The specialization to\(w,h\)∈\{0,1\}×\{μ,q\}\(w,h\)\\in\\\{0,1\\\}\\times\\\{\\mu,q\\\}uses the same arm\-specific sample𝒟N\\mathcal\{D\}\_\{N\}and the same predictable filtration as in Appendices[8](https://arxiv.org/html/2606.31184#S8)–[11](https://arxiv.org/html/2606.31184#S11)\.

### 12\.1Common basis, finite truncation, and active dimensions

SetJn=⌈nd/\(2​βmin\+d\)⌉J\_\{n\}=\\lceil n^\{d/\(2\\beta\_\{\\min\}\+d\)\}\\rceilandϕn​\(x\)=\(ψ1​\(x\),…,ψJn​\(x\)\)⊤\\phi\_\{n\}\(x\)=\(\\psi\_\{1\}\(x\),\\ldots,\\psi\_\{J\_\{n\}\}\(x\)\)^\{\\top\}\. Each smoothness expertk∈\{1,…,Kn\}k\\in\\\{1,\\ldots,K\_\{n\}\\\}corresponds to a smoothness valueβk∈\[βmin,βmax\]\\beta\_\{k\}\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\]; the grid is chosen so that mesh≤c/log⁡n\\leq c/\\log n, henceKn=O​\(log⁡n\)K\_\{n\}=O\(\\log n\)\. Expertkkuses the diagonal prior precisionΛk−1=diag​\{ρ1​\(βk\)−1,…,ρJn​\(βk\)−1\}\\Lambda\_\{k\}^\{\-1\}=\\mathrm\{diag\}\\\{\\rho\_\{1\}\(\\beta\_\{k\}\)^\{\-1\},\\ldots,\\rho\_\{J\_\{n\}\}\(\\beta\_\{k\}\)^\{\-1\}\\\}\.

The dynamic active dimension for expertkkat sample sizeNNisdN,k≤Jnd\_\{N,k\}\\leq J\_\{n\}, defined as the largestJ≤JnJ\\leq J\_\{n\}such thatN≥C1​J​log⁡J,ρJ​\(βk\)−1≤N\.N\\geq C\_\{1\}J\\log J,\\qquad\\rho\_\{J\}\(\\beta\_\{k\}\)^\{\-1\}\\leq N\.The mask isPN,k=diag​\{pj,k​\(N\)\}P\_\{N,k\}=\\mathrm\{diag\}\\\{p\_\{j,k\}\(N\)\\\},pj,k​\(N\)=𝟏​\{j≤dN,k\}p\_\{j,k\}\(N\)=\\mathbf\{1\}\\\{j\\leq d\_\{N,k\}\\\}, or equivalently𝟏​\{N≥Tj,k\}\\mathbf\{1\}\\\{N\\geq T\_\{j,k\}\\\}for precomputed thresholds\.

##### Truncation bias\.

As in Appendix[8\.2](https://arxiv.org/html/2606.31184#S8.SS2),‖m⋆−mJn⋆‖L2​\(PX\)2≲Jn−2​β⋆/d\.\\\|m^\{\\star\}\-m^\{\\star\}\_\{J\_\{n\}\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\lesssim J\_\{n\}^\{\-2\\beta^\{\\star\}/d\}\.SinceJn≍nd/\(2​βmin\+d\)J\_\{n\}\\asymp n^\{d/\(2\\beta\_\{\\min\}\+d\)\}andβ⋆≥βmin\\beta^\{\\star\}\\geq\\beta\_\{\\min\}, this is no worse than the roughest statistical rate and is absorbed\.

### 12\.2Token structure

For an arm\-filtered history𝒟N=\{\(Xi,Zi\)\}i=1N\\mathcal\{D\}\_\{N\}=\\\{\(X\_\{i\},Z\_\{i\}\)\\\}\_\{i=1\}^\{N\}:

Hi\\displaystyle H\_\{i\}=\[ϕn​\(Xi\),Zi,Zi2,i/N,N/n,1/N,log⁡n/N,history\]\\displaystyle=\[\\phi\_\{n\}\(X\_\{i\}\),\\,Z\_\{i\},\\,Z\_\{i\}^\{2\},\\,i/N,\\,N/n,\\,1/N,\\,\\log n/N,\\,\\mathrm\{history\}\]\\quad\(history token\)Q\\displaystyle Q=\[ϕn​\(x\),N/n,1/N,query\]\\displaystyle=\[\\phi\_\{n\}\(x\),\\,N/n,\\,1/N,\\,\\mathrm\{query\}\]\\quad\(query token\)Ek\(ℓ\)\\displaystyle E\_\{k\}^\{\(\\ell\)\}=\[βk,1/N,ϕn​\(x\),ak\(ℓ\),pk​\(N\),cN,zN\(2\),expert\]\\displaystyle=\[\\beta\_\{k\},\\,1/N,\\,\\phi\_\{n\}\(x\),\\,a\_\{k\}^\{\(\\ell\)\},\\,p\_\{k\}\(N\),\\,c\_\{N\},\\,z\_\{N\}^\{\(2\)\},\\,\\mathrm\{expert\}\]\\quad\(expert register\)\.\\displaystyle\\text\{\(expert register\)\}\.The deterministic prefix\-level features1/N1/N,log⁡n/N\\log n/N,i/Ni/N,N/nN/nremove the need to approximate reciprocal maps in attention and avoid all per\-step division operations\.

### 12\.3Statistic\-attention computes the sufficient statistics

###### Lemma 28\(Exactness of statistic attention\)

On the active subspace and for exact features, normalized linear attentionA​\(H;Q,K,V\)i=1N​∑j\(Q​hi\)⊤​\(K​hj\)​V​hjA\(H;Q,K,V\)\_\{i\}=\\frac\{1\}\{N\}\\sum\_\{j\}\(Qh\_\{i\}\)^\{\\top\}\(Kh\_\{j\}\)\\,Vh\_\{j\}implements each of the following sufficient statistics exactly with selector matrices of operator norm at most one:

1. 1\.cN=N−1​∑iϕi​Zic\_\{N\}=N^\{\-1\}\\sum\_\{i\}\\phi\_\{i\}Z\_\{i\};
2. 2\.zN\(2\)=N−1​∑iZi2z\_\{N\}^\{\(2\)\}=N^\{\-1\}\\sum\_\{i\}Z\_\{i\}^\{2\};
3. 3\.GN​ak\(ℓ\)=N−1​∑iϕi​\(ϕi⊤​ak\(ℓ\)\)G\_\{N\}a\_\{k\}^\{\(\\ell\)\}=N^\{\-1\}\\sum\_\{i\}\\phi\_\{i\}\(\\phi\_\{i\}^\{\\top\}a\_\{k\}^\{\(\\ell\)\}\);
4. 4\.the dot productsak⊤​cNa\_\{k\}^\{\\top\}c\_\{N\}andak⊤​GN​aka\_\{k\}^\{\\top\}G\_\{N\}a\_\{k\}\.

Signed inner products are realized by two ReLU\-attention heads usingσ​\(z\)−σ​\(−z\)=z\\sigma\(z\)\-\\sigma\(\-z\)=z\.

Proof\.For \(i\) takeQ=𝟏Q=\\mathbf\{1\}\(selecting the constant coordinate\),KKselectingZiZ\_\{i\},VVselectingϕi\\phi\_\{i\}\. ThenN−1​∑i\(Q​hi\)⊤​\(K​hj\)​V​hj=N−1​∑iZi​ϕiN^\{\-1\}\\sum\_\{i\}\(Qh\_\{i\}\)^\{\\top\}\(Kh\_\{j\}\)Vh\_\{j\}=N^\{\-1\}\\sum\_\{i\}Z\_\{i\}\\phi\_\{i\}\. For \(ii\) takeK=Zi,V=ZiK=Z\_\{i\},V=Z\_\{i\}\. For \(iii\) takeQQselectingak\(ℓ\)a\_\{k\}^\{\(\\ell\)\},KKselectingϕi\\phi\_\{i\},VVselectingϕi\\phi\_\{i\}\. The dot products in \(iv\) are analogous\. All selectors are coordinate projections of operator norm one\.□\\square

##### Embedding error\.

If the input contains exact featuresϕn​\(X\)\\phi\_\{n\}\(X\),δemb=0\\delta\_\{\\rm emb\}=0\. If raw covariates are used, a pointwise ReLU encoderEθ:ℝd→ℝJnE\_\{\\theta\}:\\mathbb\{R\}^\{d\}\\to\\mathbb\{R\}^\{J\_\{n\}\}achievessupx‖Eθ​\(x\)−ϕn​\(x\)‖2≤ϵemb\\sup\_\{x\}\\\|E\_\{\\theta\}\(x\)\-\\phi\_\{n\}\(x\)\\\|\_\{2\}\\leq\\epsilon\_\{\\rm emb\}at parameter costO~​\(Jn2\+d/\(2​s\)​ϵemb−d/s\)\\widetilde\{O\}\(J\_\{n\}^\{2\+d/\(2s\)\}\\epsilon\_\{\\rm emb\}^\{\-d/s\}\)forCsC^\{s\}basis functions\(Yarotsky,[2017](https://arxiv.org/html/2606.31184#bib.bib29)\)\. By the same algebra as in \[TCM §3\.3\], the propagation of embedding error through the ridge update is controlled byJn​ϵemb=o​\(εnad\),\\sqrt\{J\_\{n\}\}\\epsilon\_\{\\rm emb\}=o\(\\varepsilon\_\{n\}^\{\\rm ad\}\),which is the only constraint onϵemb\\epsilon\_\{\\rm emb\}\.

### 12\.4ReLU FFN realization of the dynamic mask

The active\-coordinate indicator𝟏​\{N≥Tj,k\}\\mathbf\{1\}\\\{N\\geq T\_\{j,k\}\\\}for integerNNis exact:

pj,k​\(N\)=σ​\(N−Tj,k\+1\)−σ​\(N−Tj,k\)\.p\_\{j,k\}\(N\)=\\sigma\(N\-T\_\{j,k\}\+1\)\-\\sigma\(N\-T\_\{j,k\}\)\.\(32\)The coordinate projectionaj↦pj,k​aja\_\{j\}\\mapsto p\_\{j,k\}\\,a\_\{j\}is exact on a bounded domain\|aj\|≤Ba\|a\_\{j\}\|\\leq B\_\{a\}:

p​a=σ​\{a−Ba​\(1−p\)\}−σ​\{−a−Ba​\(1−p\)\},p\\,a=\\sigma\\\{a\-B\_\{a\}\(1\-p\)\\\}\-\\sigma\\\{\-a\-B\_\{a\}\(1\-p\)\\\},\(33\)which equalsaawhenp=1p=1and0whenp=0p=0\. Henceδmask=0\\delta\_\{\\rm mask\}=0\.

### 12\.5Projected GD: one block, geometric convergence

For expertkkat sample sizeNN, the masked ridge objective is

Qk​\(a\)=12​N​∑i=1N\(Zi−ϕi⊤​a\)2\+σ¯22​N​a⊤​Λk−1​a,a=PN,k​a\.Q\_\{k\}\(a\)=\\frac\{1\}\{2N\}\\sum\_\{i=1\}^\{N\}\(Z\_\{i\}\-\\phi\_\{i\}^\{\\top\}a\)^\{2\}\+\\frac\{\\bar\{\\sigma\}^\{2\}\}\{2N\}a^\{\\top\}\\Lambda\_\{k\}^\{\-1\}a,\\quad a=P\_\{N,k\}a\.Its gradient is∇Qk​\(a\)=GN​a\+\(σ¯2/N\)​Λk−1​a−cN\\nabla Q\_\{k\}\(a\)=G\_\{N\}a\+\(\\bar\{\\sigma\}^\{2\}/N\)\\Lambda\_\{k\}^\{\-1\}a\-c\_\{N\}\. One block executes

a\(ℓ\+1\)=PN,k​\[a\(ℓ\)−η​\(GN​a\(ℓ\)\+\(σ¯2/N\)​Λk−1​a\(ℓ\)−cN\)\]\.a^\{\(\\ell\+1\)\}=P\_\{N,k\}\\bigl\[a^\{\(\\ell\)\}\-\\eta\(G\_\{N\}a^\{\(\\ell\)\}\+\(\\bar\{\\sigma\}^\{2\}/N\)\\Lambda\_\{k\}^\{\-1\}a^\{\(\\ell\)\}\-c\_\{N\}\)\\bigr\]\.\(34\)Each piece is computed by Lemma[28](https://arxiv.org/html/2606.31184#Thmtheorem28)\(cross\-token via attention\) and the ReLU FFN \(within\-token diagonal map and projection, \([33](https://arxiv.org/html/2606.31184#S12.E33)\)\)\. One attention\-FFN block implements one projected GD step exactly\.

###### Lemma 29\(Geometric GD convergence\)

On the Gram event𝒢n\\mathcal\{G\}\_\{n\}\(Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)\), the restricted HessianHN,k=PN,k​\(GN\+\(σ¯2/N\)​Λk−1\)​PN,kH\_\{N,k\}=P\_\{N,k\}\(G\_\{N\}\+\(\\bar\{\\sigma\}^\{2\}/N\)\\Lambda\_\{k\}^\{\-1\}\)P\_\{N,k\}satisfiesm​I⪯HN,k⪯M​ImI\\preceq H\_\{N,k\}\\preceq MIon the active subspace withm=cX​η/2,M=2​\(1−η\)​CX\+σ¯2m=c\_\{X\}\\eta/2,\\ M=2\(1\-\\eta\)C\_\{X\}\+\\bar\{\\sigma\}^\{2\}\. With step sizeη=1/M\\eta=1/M, the projected GD iterates satisfy‖ak\(ℓ\)−akridge‖HN,k≤qℓ​‖ak\(0\)−akridge‖HN,k,\\\|a\_\{k\}^\{\(\\ell\)\}\-a\_\{k\}^\{\\rm ridge\}\\\|\_\{H\_\{N,k\}\}\\leq q^\{\\ell\}\\\|a\_\{k\}^\{\(0\)\}\-a\_\{k\}^\{\\rm ridge\}\\\|\_\{H\_\{N,k\}\},q=1−m/M<1q=1\-m/M<1\. ChoosingL=⌈c​log⁡n/log⁡\(1/q\)⌉=O​\(log⁡n\)L=\\lceil c\\log n/\\log\(1/q\)\\rceil=O\(\\log n\)blocks yieldsδGD:=‖ak\(L\)−akridge‖L2​\(PX\)=O​\(n−A\)\\delta\_\{\\rm GD\}:=\\\|a\_\{k\}^\{\(L\)\}\-a\_\{k\}^\{\\rm ridge\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=O\(n^\{\-A\}\)for any prescribed constantA\>0A\>0\.

Proof\.The Hessian condition is from Lemma[16](https://arxiv.org/html/2606.31184#Thmtheorem16)\. Projected gradient descent on a strongly convex objective withη∈\(0,2/M\)\\eta\\in\(0,2/M\)contracts in theHN,kH\_\{N,k\}norm at rateqq\(Nesterov et al\.,[2018](https://arxiv.org/html/2606.31184#bib.bib19)\)\.□\\square

### 12\.6Penalized\-evidence MoE gate

The implementable MoE gate uses the score

Sk=RSSk\+Priork\+A​dk​log⁡nN,S\_\{k\}=\\mathrm\{RSS\}\_\{k\}\+\\mathrm\{Prior\}\_\{k\}\+A\\frac\{d\_\{k\}\\log n\}\{N\},\(35\)withRSSk=N−1​∑i\(Zi−ϕi⊤​ak\)2\\mathrm\{RSS\}\_\{k\}=N^\{\-1\}\\sum\_\{i\}\(Z\_\{i\}\-\\phi\_\{i\}^\{\\top\}a\_\{k\}\)^\{2\},Priork=\(σ¯2/N\)​ak⊤​Λk−1​ak\\mathrm\{Prior\}\_\{k\}=\(\\bar\{\\sigma\}^\{2\}/N\)a\_\{k\}^\{\\top\}\\Lambda\_\{k\}^\{\-1\}a\_\{k\},dk=dN,kd\_\{k\}=d\_\{N,k\}, andA\>0A\>0a constant chosen below\. The MoE weight is

ωk=πk​exp⁡\(−ηg​N​Sk\)∑rπr​exp⁡\(−ηg​N​Sr\),\\omega\_\{k\}=\\frac\{\\pi\_\{k\}\\exp\(\-\\eta\_\{g\}NS\_\{k\}\)\}\{\\sum\_\{r\}\\pi\_\{r\}\\exp\(\-\\eta\_\{g\}NS\_\{r\}\)\},withπk≍1/Kn\\pi\_\{k\}\\asymp 1/K\_\{n\}andηg\\eta\_\{g\}a fixed positive constant\. The output ism^mix​\(x\)=∑kωk​m^k​\(x\)\\widehat\{m\}\_\{\\rm mix\}\(x\)=\\sum\_\{k\}\\omega\_\{k\}\\widehat\{m\}\_\{k\}\(x\)wherem^k​\(x\)=ϕx⊤​ak\(L\)\\widehat\{m\}\_\{k\}\(x\)=\\phi\_\{x\}^\{\\top\}a\_\{k\}^\{\(L\)\}\.

##### Computability\.

Each piece ofSkS\_\{k\}is exact under exact features:

- •RSSk=zN\(2\)−2​ak⊤​cN\+ak⊤​GN​ak\\mathrm\{RSS\}\_\{k\}=z\_\{N\}^\{\(2\)\}\-2a\_\{k\}^\{\\top\}c\_\{N\}\+a\_\{k\}^\{\\top\}G\_\{N\}a\_\{k\}is a linear combination of attention\-computable quantities;
- •Priork=\(σ¯2/N\)​ak⊤​Λk−1​ak\\mathrm\{Prior\}\_\{k\}=\(\\bar\{\\sigma\}^\{2\}/N\)a\_\{k\}^\{\\top\}\\Lambda\_\{k\}^\{\-1\}a\_\{k\}uses the diagonal prior FFN and a dot\-product attention head with selector1/N1/N;
- •the active dimensiondk=∑jpj,k​\(N\)d\_\{k\}=\\sum\_\{j\}p\_\{j,k\}\(N\)is a fixed linear sum of mask coordinates, andA​dk​log⁡n/NA\\,d\_\{k\}\\log n/Nuses the prefix\-level scalarlog⁡n/N\\log n/N\.

The exponential weights are realized by a piecewise\-linear ReLU softmax approximation to errorεsm=n−A\\varepsilon\_\{\\rm sm\}=n^\{\-A\}, with parameter costPsm=nO​\(1\)P\_\{\\rm sm\}=n^\{O\(1\)\}\(see \[GPE §3\.7\]\)\.

### 12\.7Penalized\-MoE oracle inequality

We prove that the implementable gate gives an oracle\-rate mixture risk\.

#### 12\.7\.1Smoothness grid and near\-oracle expert

###### Lemma 30\(Near\-oracle expert\)

For anyβ⋆∈\[βmin,βmax\]\\beta^\{\\star\}\\in\[\\beta\_\{\\min\},\\beta\_\{\\max\}\], there existsk⋆∈\{1,…,Kn\}k^\{\\star\}\\in\\\{1,\\ldots,K\_\{n\}\\\}with0≤β⋆−βk⋆≤c/log⁡n0\\leq\\beta^\{\\star\}\-\\beta\_\{k^\{\\star\}\}\\leq c/\\log n\. The active dimension satisfiesdN,k⋆≍Nd/\(2​βk⋆\+d\)/log⁡n≍Nd/\(2​β⋆\+d\)/log⁡nd\_\{N,k^\{\\star\}\}\\asymp N^\{d/\(2\\beta\_\{k^\{\\star\}\}\+d\)\}/\\log n\\asymp N^\{d/\(2\\beta^\{\\star\}\+d\)\}/\\log n, up to constants and logarithmic factors\.

Proof\.Direct from the smoothness grid construction and Lemma[25](https://arxiv.org/html/2606.31184#Thmtheorem25)\.□\\square

#### 12\.7\.2Uniform concentration eventℰN\\mathcal\{E\}\_\{N\}

We establish the key uniform concentration event for the ridge spaces\.

###### Lemma 31\(Uniform concentration over expert classes\)

With probability≥1−n−A\\geq 1\-n^\{\-A\}, for every expertkkand everyf∈ℱk:=\{fa=a⊤​ϕn:a∈ℝJn,PN,k​a=a\}f\\in\\mathcal\{F\}\_\{k\}:=\\\{f\_\{a\}=a^\{\\top\}\\phi\_\{n\}:a\\in\\mathbb\{R\}^\{J\_\{n\}\},P\_\{N,k\}a=a\\\},

\|‖f−m⋆‖N2−‖f−m⋆‖L2​\(PX\)2\|\\displaystyle\\bigl\|\\\|f\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\bigr\|≤18​‖f−m⋆‖L2​\(PX\)2\+C∗​dk​log⁡nN,\\displaystyle\\leq\\tfrac\{1\}\{8\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\+C\_\{\*\}\\frac\{d\_\{k\}\\log n\}\{N\},\(36\)\|2N​∑i=1Nξi​\(f−m⋆\)​\(Xi\)\|\\displaystyle\\Bigl\|\\frac\{2\}\{N\}\\sum\_\{i=1\}^\{N\}\\xi\_\{i\}\(f\-m^\{\\star\}\)\(X\_\{i\}\)\\Bigr\|≤18​‖f−m⋆‖L2​\(PX\)2\+C∗​dk​log⁡nN\.\\displaystyle\\leq\\tfrac\{1\}\{8\}\\\|f\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\+C\_\{\*\}\\frac\{d\_\{k\}\\log n\}\{N\}\.\(37\)

Proof\.Coverℱk\\mathcal\{F\}\_\{k\}by anL2​\(PX\)L\_\{2\}\(P\_\{X\}\)\-net at scaleρ\>0\\rho\>0:log⁡N​\(ρ,ℱk,L2​\(PX\)\)≲dk​log⁡\(C/ρ\)\\log N\(\\rho,\\mathcal\{F\}\_\{k\},L\_\{2\}\(P\_\{X\}\)\)\\lesssim d\_\{k\}\\log\(C/\\rho\)\.

*Empirical\-population norm gap\.*For each cover elementf0f\_\{0\},‖f0−m⋆‖N2−‖f0−m⋆‖L2​\(PX\)2\\\|f\_\{0\}\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\\|f\_\{0\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}is an iid average overXiX\_\{i\}of a bounded random variable \(bounded bysupx\|f0−m⋆\|2≤Bn2\\sup\_\{x\}\|f\_\{0\}\-m^\{\\star\}\|^\{2\}\\leq B\_\{n\}^\{2\}\)\. Bernstein for iid sums \(or Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)\) gives\|‖f0−m⋆‖N2−‖f0−m⋆‖PX2\|≲Bn​ρ/N\|\\\|f\_\{0\}\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\\|f\_\{0\}\-m^\{\\star\}\\\|\_\{P\_\{X\}\}^\{2\}\|\\lesssim B\_\{n\}\\rho/\\sqrt\{N\}with probability1−2​exp⁡\(−c​N​ρ2/Bn2\)1\-2\\exp\(\-cN\\rho^\{2\}/B\_\{n\}^\{2\}\)\.

*Peeling\.*Take a geometric peeling withrj=2j​ρ0r\_\{j\}=2^\{j\}\\rho\_\{0\},ρ0=N−1/2​dk​log⁡n\\rho\_\{0\}=N^\{\-1/2\}\\sqrt\{d\_\{k\}\\log n\}, and union bound over the per\-shell covers\. The displayed inequality holds at each shell withC∗C\_\{\*\}proportional to the union\-bound constant; the additivedk​log⁡n/Nd\_\{k\}\\log n/Nterm comes from the smallest shellr0r\_\{0\}and the slack18​ρ2\\frac\{1\}\{8\}\\rho^\{2\}comes from interpolation between shells\.

*Noise inner product\.*Apply Lemma[6](https://arxiv.org/html/2606.31184#Thmtheorem6)toDi=ξi​\(f0−m⋆\)​\(Xi\)D\_\{i\}=\\xi\_\{i\}\(f\_\{0\}\-m^\{\\star\}\)\(X\_\{i\}\)— an MDS w\.r\.t\.ℱiX\\mathcal\{F\}\_\{i\}^\{X\}with predictable variance≤τ2​‖f0−m⋆‖N2\\leq\\tau^\{2\}\\\|f\_\{0\}\-m^\{\\star\}\\\|\_\{N\}^\{2\}and envelopeτ​Bn\\tau B\_\{n\}— combined with the same peeling\. The result is \([37](https://arxiv.org/html/2606.31184#S12.E37)\)\.

*Union over experts\.*Union bound overKn=O​\(log⁡n\)K\_\{n\}=O\(\\log n\)experts adds at most alog⁡log⁡n\\log\\log nfactor\. The total deviation is≤n−A\\leq n^\{\-A\}forAAlarge\.□\\square

#### 12\.7\.3Lower bound onSkS\_\{k\}for “bad” experts

###### Lemma 32\(Bad\-expert score lower bound\)

ChooseA≥8​C∗\+1A\\geq 8C\_\{\*\}\+1in the score \([35](https://arxiv.org/html/2606.31184#S12.E35)\)\. Then onℰN∩𝒢n\\mathcal\{E\}\_\{N\}\\cap\\mathcal\{G\}\_\{n\}, for every expertkk,Sk−ξ¯N2≥c0​\(Rk\+penk\),S\_\{k\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\geq c\_\{0\}\(R\_\{k\}\+\\mathrm\{pen\}\_\{k\}\),withc0=1/4c\_\{0\}=1/4andpenk=dk​log⁡n/N\\mathrm\{pen\}\_\{k\}=d\_\{k\}\\log n/N, whereRk=‖f^k−m⋆‖L2​\(PX\)2R\_\{k\}=\\\|\\widehat\{f\}\_\{k\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\.

Proof\.ExpandRSSk=zN\(2\)−2​a^k⊤​cN\+a^k⊤​GN​a^k\\mathrm\{RSS\}\_\{k\}=z\_\{N\}^\{\(2\)\}\-2\\widehat\{a\}\_\{k\}^\{\\top\}c\_\{N\}\+\\widehat\{a\}\_\{k\}^\{\\top\}G\_\{N\}\\widehat\{a\}\_\{k\}and use the residual\-bias identity forf=f^kf=\\widehat\{f\}\_\{k\}:

RSSk−ξ¯N2=‖f^k−m⋆‖N2−2N​∑ξi​\(f^k−m⋆\)​\(Xi\)\.\\mathrm\{RSS\}\_\{k\}\-\\bar\{\\xi\}\_\{N\}^\{2\}=\\\|\\widehat\{f\}\_\{k\}\-m^\{\\star\}\\\|\_\{N\}^\{2\}\-\\frac\{2\}\{N\}\\sum\\xi\_\{i\}\(\\widehat\{f\}\_\{k\}\-m^\{\\star\}\)\(X\_\{i\}\)\.Apply Lemma[31](https://arxiv.org/html/2606.31184#Thmtheorem31): onℰN\\mathcal\{E\}\_\{N\},

‖f^k−m⋆‖N2≥78​Rk−C∗​penk,−2N​∑ξi​\(f^k−m⋆\)​\(Xi\)≥−18​Rk−C∗​penk\.\\\|\\widehat\{f\}\_\{k\}\-m^\{\\star\}\\\|\_\{N\}^\{2\}\\geq\\tfrac\{7\}\{8\}R\_\{k\}\-C\_\{\*\}\\mathrm\{pen\}\_\{k\},\\quad\-\\frac\{2\}\{N\}\\sum\\xi\_\{i\}\(\\widehat\{f\}\_\{k\}\-m^\{\\star\}\)\(X\_\{i\}\)\\geq\-\\tfrac\{1\}\{8\}R\_\{k\}\-C\_\{\*\}\\mathrm\{pen\}\_\{k\}\.HenceRSSk−ξ¯N2≥34​Rk−2​C∗​penk\\mathrm\{RSS\}\_\{k\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\geq\\tfrac\{3\}\{4\}R\_\{k\}\-2C\_\{\*\}\\mathrm\{pen\}\_\{k\}\. The prior energy is nonnegative, and addingA​penkA\\,\\mathrm\{pen\}\_\{k\}givesSk−ξ¯N2≥34​Rk\+\(A−2​C∗\)​penk≥14​\(Rk\+penk\)S\_\{k\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\geq\\tfrac\{3\}\{4\}R\_\{k\}\+\(A\-2C\_\{\*\}\)\\mathrm\{pen\}\_\{k\}\\geq\\tfrac\{1\}\{4\}\(R\_\{k\}\+\\mathrm\{pen\}\_\{k\}\)whenA≥8​C∗\+1A\\geq 8C\_\{\*\}\+1\.□\\square

#### 12\.7\.4Upper bound onSk⋆S\_\{k^\{\\star\}\}

###### Lemma 33\(Near\-oracle score upper bound\)

OnℰN∩𝒢n\\mathcal\{E\}\_\{N\}\\cap\\mathcal\{G\}\_\{n\},Sk⋆−ξ¯N2≤C1​dk⋆−2​β⋆/d\+C2​penk⋆=O~​\(N−2​β⋆/\(2​β⋆\+d\)\)\.S\_\{k^\{\\star\}\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\leq C\_\{1\}d\_\{k^\{\\star\}\}^\{\-2\\beta^\{\\star\}/d\}\+C\_\{2\}\\,\\mathrm\{pen\}\_\{k^\{\\star\}\}=\\widetilde\{O\}\(N^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)\.

Proof\.Letfk⋆∘=∑j≤dk⋆θj⋆​ψjf\_\{k^\{\\star\}\}^\{\\circ\}=\\sum\_\{j\\leq d\_\{k^\{\\star\}\}\}\\theta\_\{j\}^\{\\star\}\\psi\_\{j\}be the projection ofm⋆m^\{\\star\}onto the active space ofk⋆k^\{\\star\}\. Sincef^k⋆\\widehat\{f\}\_\{k^\{\\star\}\}minimizesRSS\+Prior\\mathrm\{RSS\}\+\\mathrm\{Prior\}overℱk⋆\\mathcal\{F\}\_\{k^\{\\star\}\},RSSk⋆\+Priork⋆≤RSS​\(fk⋆∘\)\+Prior​\(fk⋆∘\)\.\\mathrm\{RSS\}\_\{k^\{\\star\}\}\+\\mathrm\{Prior\}\_\{k^\{\\star\}\}\\leq\\mathrm\{RSS\}\(f\_\{k^\{\\star\}\}^\{\\circ\}\)\+\\mathrm\{Prior\}\(f\_\{k^\{\\star\}\}^\{\\circ\}\)\.

*RSS bound forfk⋆∘f\_\{k^\{\\star\}\}^\{\\circ\}\.*By the residual\-bias identity and Lemma[31](https://arxiv.org/html/2606.31184#Thmtheorem31),RSS​\(fk⋆∘\)−ξ¯N2≤98​‖fk⋆∘−m⋆‖L2​\(PX\)2\+C∗​penk⋆≤C​dk⋆−2​β⋆/d\+C∗​penk⋆,\\mathrm\{RSS\}\(f\_\{k^\{\\star\}\}^\{\\circ\}\)\-\\bar\{\\xi\}\_\{N\}^\{2\}\\leq\\tfrac\{9\}\{8\}\\\|f\_\{k^\{\\star\}\}^\{\\circ\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\+C\_\{\*\}\\mathrm\{pen\}\_\{k^\{\\star\}\}\\leq Cd\_\{k^\{\\star\}\}^\{\-2\\beta^\{\\star\}/d\}\+C\_\{\*\}\\mathrm\{pen\}\_\{k^\{\\star\}\},using the truncation bias bound from Appendix[12\.1](https://arxiv.org/html/2606.31184#S12.SS1)\.

*Prior energy bound\.*Prior​\(fk⋆∘\)=\(σ¯2/N\)​∑j≤dk⋆ρj​\(βk⋆\)−1​\(θj⋆\)2\.\\mathrm\{Prior\}\(f\_\{k^\{\\star\}\}^\{\\circ\}\)=\(\\bar\{\\sigma\}^\{2\}/N\)\\sum\_\{j\\leq d\_\{k^\{\\star\}\}\}\\rho\_\{j\}\(\\beta\_\{k^\{\\star\}\}\)^\{\-1\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\.Usingρj​\(βk⋆\)−1≍j1\+2​βk⋆/d≤dk⋆​j2​β⋆/d\\rho\_\{j\}\(\\beta\_\{k^\{\\star\}\}\)^\{\-1\}\\asymp j^\{1\+2\\beta\_\{k^\{\\star\}\}/d\}\\leq d\_\{k^\{\\star\}\}j^\{2\\beta^\{\\star\}/d\}\(sinceβk⋆≤β⋆\\beta\_\{k^\{\\star\}\}\\leq\\beta^\{\\star\}up to1/log⁡n1/\\log n\),Prior​\(fk⋆∘\)≲\(dk⋆/N\)​∑j2​β⋆/d​\(θj⋆\)2≲dk⋆/N≤penk⋆\.\\mathrm\{Prior\}\(f\_\{k^\{\\star\}\}^\{\\circ\}\)\\lesssim\(d\_\{k^\{\\star\}\}/N\)\\sum j^\{2\\beta^\{\\star\}/d\}\(\\theta\_\{j\}^\{\\star\}\)^\{2\}\\lesssim d\_\{k^\{\\star\}\}/N\\leq\\mathrm\{pen\}\_\{k^\{\\star\}\}\.

*Combine\.*Sk⋆−ξ¯N2≤C​dk⋆−2​β⋆/d\+\(A\+C∗\+1\)​penk⋆\.S\_\{k^\{\\star\}\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\leq Cd\_\{k^\{\\star\}\}^\{\-2\\beta^\{\\star\}/d\}\+\(A\+C\_\{\*\}\+1\)\\mathrm\{pen\}\_\{k^\{\\star\}\}\.Withdk⋆≍Nd/\(2​β⋆\+d\)/log⁡nd\_\{k^\{\\star\}\}\\asymp N^\{d/\(2\\beta^\{\\star\}\+d\)\}/\\log n, both terms areO~​\(N−2​β⋆/\(2​β⋆\+d\)\)\\widetilde\{O\}\(N^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)\.□\\square

#### 12\.7\.5Bad\-expert weight decay

LetrN2=O~​\(N−2​β⋆/\(2​β⋆\+d\)\)r\_\{N\}^\{2\}=\\widetilde\{O\}\(N^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)be the oracle squared rate\. DefineℬM=\{k:Rk\+penk\>M​rN2\}\.\\mathcal\{B\}\_\{M\}=\\\{k:R\_\{k\}\+\\mathrm\{pen\}\_\{k\}\>Mr\_\{N\}^\{2\}\\\}\.Fork∈ℬMk\\in\\mathcal\{B\}\_\{M\}, Lemma[32](https://arxiv.org/html/2606.31184#Thmtheorem32)givesSk−ξ¯N2≥c0​M​rN2S\_\{k\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\geq c\_\{0\}Mr\_\{N\}^\{2\}, whileSk⋆−ξ¯N2≤C0​rN2S\_\{k^\{\\star\}\}\-\\bar\{\\xi\}\_\{N\}^\{2\}\\leq C\_\{0\}r\_\{N\}^\{2\}\(Lemma[33](https://arxiv.org/html/2606.31184#Thmtheorem33)\)\. Hence the score gap isSk−Sk⋆≥\(c0​M−C0\)​rN2≥c1​M​rN2S\_\{k\}\-S\_\{k^\{\\star\}\}\\geq\(c\_\{0\}M\-C\_\{0\}\)r\_\{N\}^\{2\}\\geq c\_\{1\}Mr\_\{N\}^\{2\}forMMlarge\. The weight ratioωk/ωk⋆=\(πk/πk⋆\)​exp⁡\(−ηg​N​\(Sk−Sk⋆\)\)≤C​exp⁡\(−ηg​c1​M​N​rN2\)\.\\omega\_\{k\}/\\omega\_\{k^\{\\star\}\}=\(\\pi\_\{k\}/\\pi\_\{k^\{\\star\}\}\)\\exp\(\-\\eta\_\{g\}N\(S\_\{k\}\-S\_\{k^\{\\star\}\}\)\)\\leq C\\exp\(\-\\eta\_\{g\}c\_\{1\}MNr\_\{N\}^\{2\}\)\.Summing over the at mostKn=O​\(log⁡n\)K\_\{n\}=O\(\\log n\)bad experts and usingN​rN2=O~​\(Nd/\(2​β⋆\+d\)\)→∞Nr\_\{N\}^\{2\}=\\widetilde\{O\}\(N^\{d/\(2\\beta^\{\\star\}\+d\)\}\)\\to\\infty,

∑k∈ℬMωk≤Kn​exp⁡\(−ηg​c1​M​N​rN2\)→0\\sum\_\{k\\in\\mathcal\{B\}\_\{M\}\}\\omega\_\{k\}\\leq K\_\{n\}\\exp\(\-\\eta\_\{g\}c\_\{1\}MNr\_\{N\}^\{2\}\)\\to 0\(38\)exponentially fast\.

#### 12\.7\.6Mixture risk bound

###### Theorem 34\(Mixture oracle inequality\)

OnℰN∩𝒢n\\mathcal\{E\}\_\{N\}\\cap\\mathcal\{G\}\_\{n\},‖f^mix−m⋆‖L2​\(PX\)2≤∑kωk​Rk≤M​rN2\+oP​\(rN2\)\.\\\|\\widehat\{f\}\_\{\\rm mix\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}^\{2\}\\leq\\sum\_\{k\}\\omega\_\{k\}R\_\{k\}\\leq Mr\_\{N\}^\{2\}\+o\_\{P\}\(r\_\{N\}^\{2\}\)\.

Proof\.Convexity of squared norm gives the first inequality\. Decompose∑kωk​Rk=∑k∈𝒢Mωk​Rk\+∑k∈ℬMωk​Rk\\sum\_\{k\}\\omega\_\{k\}R\_\{k\}=\\sum\_\{k\\in\\mathcal\{G\}\_\{M\}\}\\omega\_\{k\}R\_\{k\}\+\\sum\_\{k\\in\\mathcal\{B\}\_\{M\}\}\\omega\_\{k\}R\_\{k\}\. Fork∈𝒢Mk\\in\\mathcal\{G\}\_\{M\},Rk≤M​rN2R\_\{k\}\\leq Mr\_\{N\}^\{2\}by definition\. On a high\-probability event,RkR\_\{k\}is uniformly bounded by a constantCRC\_\{R\}\(by the boundedness off^k\\widehat\{f\}\_\{k\}on the sieve\)\. Hence∑ℬMωk​Rk≤CR​∑ℬMωk=o​\(rN2\)\\sum\_\{\\mathcal\{B\}\_\{M\}\}\\omega\_\{k\}R\_\{k\}\\leq C\_\{R\}\\sum\_\{\\mathcal\{B\}\_\{M\}\}\\omega\_\{k\}=o\(r\_\{N\}^\{2\}\)by \([38](https://arxiv.org/html/2606.31184#S12.E38)\)\.□\\square

### 12\.8Combining computational and statistical errors

LetmTFm\_\{\\rm TF\}denote the actual transformer output andf^mix\\widehat\{f\}\_\{\\rm mix\}the ideal MoE estimator\. The error decomposition is

‖mTF−m⋆‖L2​\(PX\)≤δemb\+δGD\+δscore\+δsm\+‖f^mix−m⋆‖L2​\(PX\)\.\\\|m\_\{\\rm TF\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq\\delta\_\{\\rm emb\}\+\\delta\_\{\\rm GD\}\+\\delta\_\{\\rm score\}\+\\delta\_\{\\rm sm\}\+\\\|\\widehat\{f\}\_\{\\rm mix\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\.Under the construction:

- •δemb=0\\delta\_\{\\rm emb\}=0for exact features; orδemb=o​\(εnad\)\\delta\_\{\\rm emb\}=o\(\\varepsilon\_\{n\}^\{\\rm ad\}\)for ReLU encoder withJn​ϵemb=o​\(εnad\)\\sqrt\{J\_\{n\}\}\\epsilon\_\{\\rm emb\}=o\(\\varepsilon\_\{n\}^\{\\rm ad\}\);
- •δGD=O​\(n−A\)\\delta\_\{\\rm GD\}=O\(n^\{\-A\}\)by Lemma[29](https://arxiv.org/html/2606.31184#Thmtheorem29);
- •δscore=0\\delta\_\{\\rm score\}=0by Lemma[28](https://arxiv.org/html/2606.31184#Thmtheorem28)and exact mask realization \([33](https://arxiv.org/html/2606.31184#S12.E33)\);
- •δsm=Bm​εsm=o​\(εnad\)\\delta\_\{\\rm sm\}=B\_\{m\}\\varepsilon\_\{\\rm sm\}=o\(\\varepsilon\_\{n\}^\{\\rm ad\}\)by ReLU softmax approximation;
- •‖f^mix−m⋆‖L2​\(PX\)=O~p​\(N−β⋆/\(2​β⋆\+d\)\)\\\|\\widehat\{f\}\_\{\\rm mix\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=\\widetilde\{O\}\_\{p\}\(N^\{\-\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)by Theorem[34](https://arxiv.org/html/2606.31184#Thmtheorem34)\.

Hence‖mTF−m⋆‖L2​\(PX\)=O~p​\(N−β⋆/\(2​β⋆\+d\)\)\.\\\|m\_\{\\rm TF\}\-m^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=\\widetilde\{O\}\_\{p\}\(N^\{\-\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)\.

### 12\.9Specialization to four targets

Apply the result to each\(w,h\)∈\{0,1\}×\{μ,q\}\(w,h\)\\in\\\{0,1\\\}\\times\\\{\\mu,q\\\}with arm\-specific sampleN=Nt,wN=N\_\{t,w\}\. The conditional sub\-exponential noise condition \(Assumption[2](https://arxiv.org/html/2606.31184#Thmassumption2)\) is identical to the one used in Lemma[31](https://arxiv.org/html/2606.31184#Thmtheorem31), so the four rate statements‖μ^w,t−μw⋆‖L2​\(PX\)=O~p​\(Nt,w−βw,μ⋆/\(2​βw,μ⋆\+d\)\),\\\|\\widehat\{\\mu\}\_\{w,t\}\-\\mu\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=\\widetilde\{O\}\_\{p\}\(N\_\{t,w\}^\{\-\\beta^\{\\star\}\_\{w,\\mu\}/\(2\\beta^\{\\star\}\_\{w,\\mu\}\+d\)\}\),‖q^w,t−qw⋆‖L2​\(PX\)=O~p​\(Nt,w−βw,q⋆/\(2​βw,q⋆\+d\)\),\\\|\\widehat\{q\}\_\{w,t\}\-q\_\{w\}^\{\\star\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}=\\widetilde\{O\}\_\{p\}\(N\_\{t,w\}^\{\-\\beta^\{\\star\}\_\{w,q\}/\(2\\beta^\{\\star\}\_\{w,q\}\+d\)\}\),all follow\.

### 12\.10Neyman head

Composing\(μ^0,μ^1,q^0,q^1\)↦eNey\(\\widehat\{\\mu\}\_\{0\},\\widehat\{\\mu\}\_\{1\},\\widehat\{q\}\_\{0\},\\widehat\{q\}\_\{1\}\)\\mapsto e^\{\\rm Ney\}in Lemma[18](https://arxiv.org/html/2606.31184#Thmtheorem18)requires only Lipschitz operations on a bounded domain: square, clip, square root, ratio, clip\. A ReLU FFN piecewise\-linearly approximates each to accuracyεNey=n−A\\varepsilon\_\{\\rm Ney\}=n^\{\-A\}with parameter costPNey=nO​\(1\)P\_\{\\rm Ney\}=n^\{O\(1\)\}\(see \[GPE §6\.1\]\)\. The propagatedL2L\_\{2\}error is‖e^t−etB‖L2​\(PX\)≤LNey⋅O​\(εad\)\+εNey=o​\(εad\)\\\|\\widehat\{e\}\_\{t\}\-e\_\{t\}^\{\\rm B\}\\\|\_\{L\_\{2\}\(P\_\{X\}\)\}\\leq L\_\{\\rm Ney\}\\cdot O\(\\varepsilon^\{\\rm ad\}\)\+\\varepsilon\_\{\\rm Ney\}=o\(\\varepsilon^\{\\rm ad\}\)\. This completes the construction underlying Theorem[5](https://arxiv.org/html/2606.31184#Thmtheorem5)\.□\\square

## 13ERM oracle inequality for the pretrained experimenter

##### Goal\.

Show that the parameterθ^\\widehat\{\\theta\}minimizing the empirical pretraining risk overNpreN\_\{\\rm pre\}independent trajectories satisfies the high\-probability bound

Rest​\(θ^\)≤infθ∈𝒯nRest​\(θ\)\+C​O~​\(Pn\)\+log⁡\(1/δ\)Npre\.R\_\{\\rm est\}\(\\widehat\{\\theta\}\)\\leq\\inf\_\{\\theta\\in\\mathcal\{T\}\_\{n\}\}R\_\{\\rm est\}\(\\theta\)\+C\\sqrt\{\\frac\{\\widetilde\{O\}\(P\_\{n\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.\(39\)This is the bound informally stated in Section 3\.2\. We do not compose with any decision\-risk transfer; the appendix stops at this oracle inequality\.

##### Strategy: textbook covering \+ Lipschitz net\.

The proof is the standard learning\-theory recipe:

1. 1\.*Pointwise concentration\.*For any fixedθ\\theta, the empirical risk is an iid average overNpreN\_\{\\rm pre\}trajectories of a bounded loss; Hoeffding gives sub\-Gaussian deviations\.
2. 2\.*Cover the parameter space\.*The norm\-bounded ball of transformer parameters has anε\\varepsilon\-net of size\(pn​Bn/ε\)pn\(p\_\{n\}B\_\{n\}/\\varepsilon\)^\{p\_\{n\}\}\. Union bound across the net gives uniform concentration on the net\.
3. 3\.*Lipschitz extension\.*Push concentration from the net to the entire parameter ball using the parameter Lipschitz constantLnet,nL\_\{\{\\rm net\},n\}of the loss\. Chooseε\\varepsilonto balance the two sources of error\.
4. 4\.*Comparator\.*The constructive transformer of Appendix[12](https://arxiv.org/html/2606.31184#S12)lies inside the trainable class, so the in\-class minimum risk is bounded by its approximation errorεapp,n2=O~​\(n−2​β⋆/\(2​β⋆\+d\)\)\\varepsilon\_\{\\rm app,n\}^\{2\}=\\widetilde\{O\}\(n^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)\.

##### Two subtle points the proof emphasizes\.

- •*Trajectory\- vs\. token\-level independence\.*Tokens within a single trajectory are dependent \(the design is adaptive:WtW\_\{t\}usesℱt\\mathcal\{F\}\_\{t\}\)\. The trajectory\-level loss1n​∑t=1n\[⋯\]2\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\[\\cdots\]^\{2\}*averages*over tokens within a trajectory, then we average across theNpreN\_\{\\rm pre\}*independent*trajectories\. Hoeffding’s iid concentration applies at the trajectory level, so the rate is1/Npre1/\\sqrt\{N\_\{\\rm pre\}\}, not1/n​Npre1/\\sqrt\{nN\_\{\\rm pre\}\}\. Treating tokens as iid would be wrong by a factor ofn\\sqrt\{n\}\.
- •*Polynomial Lipschitz constant\.*The entries of attention and FFN scale with the norm envelopeBnB\_\{n\}to high powers; without careful book\-keeping the Lipschitz constant could blow up super\-polynomially innn\. Section[13\.3](https://arxiv.org/html/2606.31184#S13.SS3)verifies it islog⁡Lnet,n=O~​\(1\)\\log L\_\{\{\\rm net\},n\}=\\widetilde\{O\}\(1\), which is what makes the finalO~​\(⋅\)\\widetilde\{O\}\(\\cdot\)inside the square root tame\.

This appendix proves the empirical\-risk\-minimization concentration bound quoted in Section[3](https://arxiv.org/html/2606.31184#S3): with high probability the pretrained transformer attains risk withinO~​\(Pn/Npre\)\\widetilde\{O\}\(\\sqrt\{P\_\{n\}/N\_\{\\rm pre\}\}\)of the in\-class minimum\. The argument is a covering\-number Hoeffding bound, with the two crucial points emphasized above\.

### 13\.1Theoretical class

##### Architecture\.

The theoretical class𝒯nest​\(Bn,Rn\)\\mathcal\{T\}\_\{n\}^\{\\rm est\}\(B\_\{n\},R\_\{n\}\)consists of encoder\-only residual transformers with:

- •Token widthDnD\_\{n\}, FFN hidden widthDn′D^\{\\prime\}\_\{n\},MnM\_\{n\}heads per layer,LnL\_\{n\}layers\.
- •Normalized ReLU attentionAi​\(H;Q,K,V\)=S−1​∑jσ​\(\(Q​hi\)⊤​\(K​hj\)\)​V​hjA\_\{i\}\(H;Q,K,V\)=S^\{\-1\}\\sum\_\{j\}\\sigma\(\(Qh\_\{i\}\)^\{\\top\}\(Kh\_\{j\}\)\)Vh\_\{j\}, with signed linear attention realized asσ​\(z\)−σ​\(−z\)\\sigma\(z\)\-\\sigma\(\-z\)via two heads\.
- •ReLU FFNFFN​\(h\)=W2​σ​\(W1​h\)\\mathrm\{FFN\}\(h\)=W\_\{2\}\\sigma\(W\_\{1\}h\)\.
- •Tokenwise Euclidean clipping to radiusRnR\_\{n\}after each sublayer \(1\-Lipschitz\)\.
- •Operator\-norm constraints‖Q‖op,‖K‖op,‖V‖op,‖W1‖op,‖W2‖op,‖Wout‖op≤Bn\\\|Q\\\|\_\{\\rm op\},\\\|K\\\|\_\{\\rm op\},\\\|V\\\|\_\{\\rm op\},\\\|W\_\{1\}\\\|\_\{\\rm op\},\\\|W\_\{2\}\\\|\_\{\\rm op\},\\\|W\_\{\\rm out\}\\\|\_\{\\rm op\}\\leq B\_\{n\}\.
- •2​Kn2K\_\{n\}\-dimensional readout:KnK\_\{n\}expert predictionsm^θ,k​\(x\)\\widehat\{m\}\_\{\\theta,k\}\(x\)andKnK\_\{n\}evidence logitsℓ^θ,k\\widehat\{\\ell\}\_\{\\theta,k\}\.
- •Mixture weightsω~=Softmax~​\(ℓ^\)\\widetilde\{\\omega\}=\\widetilde\{\\mathrm\{Softmax\}\}\(\\widehat\{\\ell\}\)via piecewise\-linear ReLU softmax approximation with parameter countPsmP\_\{\\rm sm\}\.

The design class𝒯npol​\(Bn,Rn\)\\mathcal\{T\}\_\{n\}^\{\\rm pol\}\(B\_\{n\},R\_\{n\}\)adds a ReLU FFN Neyman head with parameter countPNeyP\_\{\\rm Ney\}\.

##### Parameter dimension\.

pnest=Penc\+O​\{Ln​\(Mn​Dn2\+Dn​Dn′\)\}\+O​\(Kn​Dn\)\+Psm\.p\_\{n\}^\{\\rm est\}=P\_\{\\rm enc\}\+O\\\{L\_\{n\}\(M\_\{n\}D\_\{n\}^\{2\}\+D\_\{n\}D^\{\\prime\}\_\{n\}\)\\\}\+O\(K\_\{n\}D\_\{n\}\)\+P\_\{\\rm sm\}\.\(40\)WithDn≍JnD\_\{n\}\\asymp J\_\{n\},Dn′=r​DnD^\{\\prime\}\_\{n\}=rD\_\{n\},Mn=O​\(1\)M\_\{n\}=O\(1\),Ln=Kn=O​\(log⁡n\)L\_\{n\}=K\_\{n\}=O\(\\log n\),pnest=Penc\+O~​\(Jn2\)\+Psmp\_\{n\}^\{\\rm est\}=P\_\{\\rm enc\}\+\\widetilde\{O\}\(J\_\{n\}^\{2\}\)\+P\_\{\\rm sm\}\.

### 13\.2Constructive comparator

###### Lemma 35\(Constructive comparator membership\)

The constructive smoothness\-adaptive estimation transformer of Appendix[12](https://arxiv.org/html/2606.31184#S12)lies in𝒯nest​\(Bn,Rn\)\\mathcal\{T\}\_\{n\}^\{\\rm est\}\(B\_\{n\},R\_\{n\}\)withlog⁡Bn\+log⁡Rn=O~​\(1\)\.\\log B\_\{n\}\+\\log R\_\{n\}=\\widetilde\{O\}\(1\)\.Moreover, its expected estimation risk satisfiesRest​\(θcons\)≤εapp,n2=O~​\(n−2​β⋆/\(2​β⋆\+d\)\)\.R\_\{\\rm est\}\(\\theta\_\{\\rm cons\}\)\\leq\\varepsilon\_\{\\rm app,n\}^\{2\}=\\widetilde\{O\}\(n^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\)\.

GPE §3\.1–3\.7\.verifies that each component \(selector attention, mask FFN, diagonal prior FFN, GD FFN, evidence FFN, ReLU softmax, expert combination\) has matrix and bias norm bounded bypoly​\(n,Jn,Kn,BZ,n\)\\mathrm\{poly\}\(n,J\_\{n\},K\_\{n\},B\_\{Z,n\}\), solog⁡Bn=O~​\(1\)\\log B\_\{n\}=\\widetilde\{O\}\(1\)\. Hidden states are clipped at radiusRn=poly​\(n,Jn,Kn,BZ,n\)R\_\{n\}=\\mathrm\{poly\}\(n,J\_\{n\},K\_\{n\},B\_\{Z,n\}\), solog⁡Rn=O~​\(1\)\\log R\_\{n\}=\\widetilde\{O\}\(1\)\. The risk bound is from Theorem[5](https://arxiv.org/html/2606.31184#Thmtheorem5)applied to the constructive parameter setting\.□\\square

### 13\.3Parameter Lipschitz bound

###### Lemma 36\(Per\-head parameter Lipschitz\)

For one normalized ReLU attention head with hidden states‖H‖∞,2≤Rn\\\|H\\\|\_\{\\infty,2\}\\leq R\_\{n\}and matrices in operator norm≤Bn\\leq B\_\{n\}:‖Ai​\(H;θ\)−Ai​\(H;θ′\)‖2≤C​Bn2​Rn3​\(‖Δ​Q‖F\+‖Δ​K‖F\+‖Δ​V‖F\)\.\\\|A\_\{i\}\(H;\\theta\)\-A\_\{i\}\(H;\\theta^\{\\prime\}\)\\\|\_\{2\}\\leq CB\_\{n\}^\{2\}R\_\{n\}^\{3\}\(\\\|\\Delta Q\\\|\_\{F\}\+\\\|\\Delta K\\\|\_\{F\}\+\\\|\\Delta V\\\|\_\{F\}\)\.

Proof\.Δ​V\\Delta Vpiece:\|si​j\|=\|σ​\(\(Q​hi\)⊤​\(K​hj\)\)\|≤Bn2​Rn2\|s\_\{ij\}\|=\|\\sigma\(\(Qh\_\{i\}\)^\{\\top\}\(Kh\_\{j\}\)\)\|\\leq B\_\{n\}^\{2\}R\_\{n\}^\{2\}, so‖S−1​∑si​j​Δ​V​hj‖≤Bn2​Rn2⋅Rn​‖Δ​V‖op\\\|S^\{\-1\}\\sum s\_\{ij\}\\Delta Vh\_\{j\}\\\|\\leq B\_\{n\}^\{2\}R\_\{n\}^\{2\}\\cdot R\_\{n\}\\\|\\Delta V\\\|\_\{\\rm op\}\.Δ​Q\\Delta Qpiece: ReLU is 1\-Lipschitz,\|si​j−si​j′\|≤Bn​Rn2​‖Δ​Q‖op\|s\_\{ij\}\-s^\{\\prime\}\_\{ij\}\|\\leq B\_\{n\}R\_\{n\}^\{2\}\\\|\\Delta Q\\\|\_\{\\rm op\}, multiplied by‖V​hj‖≤Bn​Rn\\\|Vh\_\{j\}\\\|\\leq B\_\{n\}R\_\{n\}givesBn2​Rn3​‖Δ​Q‖opB\_\{n\}^\{2\}R\_\{n\}^\{3\}\\\|\\Delta Q\\\|\_\{\\rm op\}\.Δ​K\\Delta Kanalogous\. Summing the three pieces gives the displayed bound\.□\\square

###### Lemma 37\(Hidden\-state Lipschitz of one block\)

WithAn=\(1\+C​Mn​Bn3​Rn2\)​\(1\+Bn2\)A\_\{n\}=\(1\+CM\_\{n\}B\_\{n\}^\{3\}R\_\{n\}^\{2\}\)\(1\+B\_\{n\}^\{2\}\), one residual attention\-FFN block has hidden\-state Lipschitz constant at mostAnA\_\{n\}\.

Proof\.For attention,\|si​j−si​j′\|≤2​Bn2​Rn​Δ\|s\_\{ij\}\-s^\{\\prime\}\_\{ij\}\|\\leq 2B\_\{n\}^\{2\}R\_\{n\}\\Delta, so the head output differs by at mostC​Bn3​Rn2​ΔCB\_\{n\}^\{3\}R\_\{n\}^\{2\}\\Delta\. Sum overMnM\_\{n\}heads, add identity shortcut and 1\-Lipschitz clip: factor1\+C​Mn​Bn3​Rn21\+CM\_\{n\}B\_\{n\}^\{3\}R\_\{n\}^\{2\}\. For FFN: residual plusσ\\sigma1\-Lipschitz gives1\+Bn21\+B\_\{n\}^\{2\}\. Compose\.□\\square

###### Lemma 38\(Network parameter Lipschitz\)

There is a constantLnet,nL\_\{\{\\rm net\},n\}withlog⁡Lnet,n=O~​\(1\)\\log L\_\{\{\\rm net\},n\}=\\widetilde\{O\}\(1\)such that for every input sequenceTT,\|Fθ​\(T\)−Fθ′​\(T\)\|≤Lnet,n​‖θ−θ′‖2\.\|F\_\{\\theta\}\(T\)\-F\_\{\\theta^\{\\prime\}\}\(T\)\|\\leq L\_\{\{\\rm net\},n\}\\\|\\theta\-\\theta^\{\\prime\}\\\|\_\{2\}\.

Proof\.By induction on layers using the recurrenceΔℓ≤An​Δℓ−1\+Cblock,n​‖θℓ−θℓ′‖2\\Delta\_\{\\ell\}\\leq A\_\{n\}\\Delta\_\{\\ell\-1\}\+C\_\{\\rm block,n\}\\\|\\theta\_\{\\ell\}\-\\theta^\{\\prime\}\_\{\\ell\}\\\|\_\{2\}, yieldingΔLn≤AnLn​\(Cenc,n\+Ln​Cblock,n\)​‖θ−θ′‖2\\Delta\_\{L\_\{n\}\}\\leq A\_\{n\}^\{L\_\{n\}\}\(C\_\{\\rm enc,n\}\+\\sqrt\{L\_\{n\}\}C\_\{\\rm block,n\}\)\\\|\\theta\-\\theta^\{\\prime\}\\\|\_\{2\}\. SubstitutingLn=O​\(log⁡n\)L\_\{n\}=O\(\\log n\),log⁡An=O~​\(1\)\\log A\_\{n\}=\\widetilde\{O\}\(1\),log⁡Cblock,n=O~​\(1\)\\log C\_\{\\rm block,n\}=\\widetilde\{O\}\(1\)giveslog⁡Lnet,n=O~​\(1\)\\log L\_\{\{\\rm net\},n\}=\\widetilde\{O\}\(1\)\. The readout \(linear map \+ softmax \+ expert combination\) adds polynomial factors that are absorbed\.□\\square

###### Corollary 39\(Loss parameter Lipschitz\)

With clipped outputs/labels in\[−By,By\]\[\-B\_\{y\},B\_\{y\}\],\|ℓθ−ℓθ′\|≤Lℓ,nest​‖θ−θ′‖2,Lℓ,nest=4​By​Lnet,n\.\|\\ell\_\{\\theta\}\-\\ell\_\{\\theta^\{\\prime\}\}\|\\leq L\_\{\\ell,n\}^\{\\rm est\}\\\|\\theta\-\\theta^\{\\prime\}\\\|\_\{2\},\\quad L\_\{\\ell,n\}^\{\\rm est\}=4B\_\{y\}L\_\{\{\\rm net\},n\}\.For the BCE policy loss with clipped propensities in\[η,1−η\]\[\\eta,1\-\\eta\],Lℓ,npol=η−1​Lnet,npolL\_\{\\ell,n\}^\{\\rm pol\}=\\eta^\{\-1\}L\_\{\{\\rm net\},n\}^\{\\rm pol\}\.

### 13\.4Trajectory\-level independence

The pretraining data areNpreN\_\{\\rm pre\}independent trajectories\{𝒵i\}i=1Npre\\\{\\mathcal\{Z\}\_\{i\}\\\}\_\{i=1\}^\{N\_\{\\rm pre\}\}, each of lengthnn\. Tokens within a trajectory are dependent \(the design is adaptive\)\. The trajectory\-level loss

ℓest​\(θ;𝒵\)=1n​∑t=1n𝔼x∼PX​\[Fθ​\(𝒟t,x\)−mtB​\(x\)\]2\\ell\_\{\\rm est\}\(\\theta;\\mathcal\{Z\}\)=\\frac\{1\}\{n\}\\sum\_\{t=1\}^\{n\}\\mathbb\{E\}\_\{x\\sim P\_\{X\}\}\[F\_\{\\theta\}\(\\mathcal\{D\}\_\{t\},x\)\-m^\{B\}\_\{t\}\(x\)\]^\{2\}is bounded by4By2=:Bℓ,est4B\_\{y\}^\{2\}=:B\_\{\\ell,\{\\rm est\}\}\. Define the population/empirical risksRest​\(θ\)=𝔼​\[ℓest​\(θ;𝒵\)\],R^est​\(θ\)=Npre−1​∑iℓest​\(θ;𝒵i\)\.R\_\{\\rm est\}\(\\theta\)=\\mathbb\{E\}\[\\ell\_\{\\rm est\}\(\\theta;\\mathcal\{Z\}\)\],\\quad\\widehat\{R\}\_\{\\rm est\}\(\\theta\)=N\_\{\\rm pre\}^\{\-1\}\\sum\_\{i\}\\ell\_\{\\rm est\}\(\\theta;\\mathcal\{Z\}\_\{i\}\)\.The independent sample size for generalization isNpreN\_\{\\rm pre\}, notn​NprenN\_\{\\rm pre\}\.

### 13\.5Covering uniform deviation

###### Lemma 40\(Pointwise Hoeffding\)

For any fixedθ\\theta,Pr⁡\(\|R^est​\(θ\)−Rest​\(θ\)\|\>u\)≤2​e−2​Npre​u2/Bℓ,est2\.\\Pr\(\|\\widehat\{R\}\_\{\\rm est\}\(\\theta\)\-R\_\{\\rm est\}\(\\theta\)\|\>u\)\\leq 2e^\{\-2N\_\{\\rm pre\}u^\{2\}/B\_\{\\ell,\{\\rm est\}\}^\{2\}\}\.

###### Lemma 41\(Net size\)

The parameter ballΘn:=\{θ:‖θ‖2≤C​Bn​pnest\}\\Theta\_\{n\}:=\\\{\\theta:\\\|\\theta\\\|\_\{2\}\\leq CB\_\{n\}\\sqrt\{p\_\{n\}^\{\\rm est\}\}\\\}admits anε\\varepsilon\-net𝒩ε\\mathcal\{N\}\_\{\\varepsilon\}withlog⁡\|𝒩ε\|≤pnest​log⁡\(C​Bn​pnest/ε\)\.\\log\|\\mathcal\{N\}\_\{\\varepsilon\}\|\\leq p\_\{n\}^\{\\rm est\}\\log\(CB\_\{n\}\\sqrt\{p\_\{n\}^\{\\rm est\}\}/\\varepsilon\)\.

Combining the two by union bound:

maxθ∈𝒩ε⁡\|R^est​\(θ\)−Rest​\(θ\)\|≤C​Bℓ,est​pnest​log⁡\(C​Bn​pnest/ε\)\+log⁡\(1/δ\)Npre\\max\_\{\\theta\\in\\mathcal\{N\}\_\{\\varepsilon\}\}\|\\widehat\{R\}\_\{\\rm est\}\(\\theta\)\-R\_\{\\rm est\}\(\\theta\)\|\\leq CB\_\{\\ell,\{\\rm est\}\}\\sqrt\{\\frac\{p\_\{n\}^\{\\rm est\}\\log\(CB\_\{n\}\\sqrt\{p\_\{n\}^\{\\rm est\}\}/\\varepsilon\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\(41\)with probability at least1−δ1\-\\delta\.

For an arbitraryθ∈Θn\\theta\\in\\Theta\_\{n\}pickθ♯∈𝒩ε\\theta^\{\\sharp\}\\in\\mathcal\{N\}\_\{\\varepsilon\}with‖θ−θ♯‖2≤ε\\\|\\theta\-\\theta^\{\\sharp\}\\\|\_\{2\}\\leq\\varepsilon\. By Corollary[39](https://arxiv.org/html/2606.31184#Thmtheorem39),\|Rest​\(θ\)−Rest​\(θ♯\)\|\+\|R^est​\(θ\)−R^est​\(θ♯\)\|≤2​Lℓ,nest​ε\.\|R\_\{\\rm est\}\(\\theta\)\-R\_\{\\rm est\}\(\\theta^\{\\sharp\}\)\|\+\|\\widehat\{R\}\_\{\\rm est\}\(\\theta\)\-\\widehat\{R\}\_\{\\rm est\}\(\\theta^\{\\sharp\}\)\|\\leq 2L\_\{\\ell,n\}^\{\\rm est\}\\varepsilon\.Chooseε=Bℓ,est/\(Lℓ,nest​Npre\)\\varepsilon=B\_\{\\ell,\{\\rm est\}\}/\(L\_\{\\ell,n\}^\{\\rm est\}\\sqrt\{N\_\{\\rm pre\}\}\)to balance the two error sources\. Substitutinglog⁡Bn\+log⁡Lℓ,nest=O~​\(1\)\\log B\_\{n\}\+\\log L\_\{\\ell,n\}^\{\\rm est\}=\\widetilde\{O\}\(1\)yields

supθ∈Θn\|R^est​\(θ\)−Rest​\(θ\)\|≤C​Bℓ,est​O~​\(pnest\)\+log⁡\(1/δ\)Npre\.\\sup\_\{\\theta\\in\\Theta\_\{n\}\}\|\\widehat\{R\}\_\{\\rm est\}\(\\theta\)\-R\_\{\\rm est\}\(\\theta\)\|\\leq CB\_\{\\ell,\{\\rm est\}\}\\sqrt\{\\frac\{\\widetilde\{O\}\(p\_\{n\}^\{\\rm est\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.\(42\)

### 13\.6Final ERM bound

Letθ^est∈arg⁡minθ∈Θn⁡R^est​\(θ\)\\widehat\{\\theta\}\_\{\\rm est\}\\in\\arg\\min\_\{\\theta\\in\\Theta\_\{n\}\}\\widehat\{R\}\_\{\\rm est\}\(\\theta\)\. Standard ERM:Rest​\(θ^est\)≤infθ∈ΘnRest​\(θ\)\+2​supθ\|R^−R\|\.R\_\{\\rm est\}\(\\widehat\{\\theta\}\_\{\\rm est\}\)\\leq\\inf\_\{\\theta\\in\\Theta\_\{n\}\}R\_\{\\rm est\}\(\\theta\)\+2\\sup\_\{\\theta\}\|\\widehat\{R\}\-R\|\.By Lemma[35](https://arxiv.org/html/2606.31184#Thmtheorem35),infθ∈ΘnRest​\(θ\)≤εapp,n2\\inf\_\{\\theta\\in\\Theta\_\{n\}\}R\_\{\\rm est\}\(\\theta\)\\leq\\varepsilon\_\{\\rm app,n\}^\{2\}\. Combining with \([42](https://arxiv.org/html/2606.31184#S13.E42)\),

Rest​\(θ^est\)≤εapp,n2\+C​Bℓ,est​O~​\(pnest\)\+log⁡\(1/δ\)Npre\.R\_\{\\rm est\}\(\\widehat\{\\theta\}\_\{\\rm est\}\)\\leq\\varepsilon\_\{\\rm app,n\}^\{2\}\+CB\_\{\\ell,\{\\rm est\}\}\\sqrt\{\\frac\{\\widetilde\{O\}\(p\_\{n\}^\{\\rm est\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.\(43\)Substitutingpnest=Penc\+O~​\(Jn2\)\+Psmp\_\{n\}^\{\\rm est\}=P\_\{\\rm enc\}\+\\widetilde\{O\}\(J\_\{n\}^\{2\}\)\+P\_\{\\rm sm\}gives the bound stated in §3\.2\.

### 13\.7Design ERM analogue

For the design transformer with BCE loss and clipped propensity in\[η,1−η\]\[\\eta,1\-\\eta\],Bℓ,pol=log⁡\(1/η\)B\_\{\\ell,\{\\rm pol\}\}=\\log\(1/\\eta\)andLℓ,npol≤η−1​Lnet,npolL\_\{\\ell,n\}^\{\\rm pol\}\\leq\\eta^\{\-1\}L\_\{\{\\rm net\},n\}^\{\\rm pol\}, where the parameter dimension adds the Neyman head:pnpol=pnest\+PNey\.p\_\{n\}^\{\\rm pol\}=p\_\{n\}^\{\\rm est\}\+P\_\{\\rm Ney\}\.The same covering argument gives

Rpol​\(θ^pol\)≤infθ∈𝒯npolRpol​\(θ\)\+C​log⁡\(1/η\)​O~​\(pnpol\)\+log⁡\(1/δ\)Npre\.R\_\{\\rm pol\}\(\\widehat\{\\theta\}\_\{\\rm pol\}\)\\leq\\inf\_\{\\theta\\in\\mathcal\{T\}\_\{n\}^\{\\rm pol\}\}R\_\{\\rm pol\}\(\\theta\)\+C\\log\(1/\\eta\)\\sqrt\{\\frac\{\\widetilde\{O\}\(p\_\{n\}^\{\\rm pol\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.\(44\)

### 13\.8Summary

The estimation ERM bound \([43](https://arxiv.org/html/2606.31184#S13.E43)\) simplifies, under the structural choicesDn≍JnD\_\{n\}\\asymp J\_\{n\},Ln=Kn=O​\(log⁡n\)L\_\{n\}=K\_\{n\}=O\(\\log n\),Dn′=r​DnD^\{\\prime\}\_\{n\}=rD\_\{n\},Mn=O​\(1\)M\_\{n\}=O\(1\), to

Rest​\(θ^est\)≤εapp,n2\+C​Bℓ,est​O~​\(Penc\+Jn2\+Psm\)\+log⁡\(1/δ\)Npre\.R\_\{\\rm est\}\(\\widehat\{\\theta\}\_\{\\rm est\}\)\\leq\\varepsilon\_\{\\rm app,n\}^\{2\}\+CB\_\{\\ell,\{\\rm est\}\}\\sqrt\{\\frac\{\\widetilde\{O\}\(P\_\{\\rm enc\}\+J\_\{n\}^\{2\}\+P\_\{\\rm sm\}\)\+\\log\(1/\\delta\)\}\{N\_\{\\rm pre\}\}\}\.The constructive transformer of Appendix[12](https://arxiv.org/html/2606.31184#S12)lies inΘn\\Theta\_\{n\}and approximates the Bayesian teacher with risk at mostεapp,n2=O~p​\(n−2​β⋆/\(2​β⋆\+d\)\)\\varepsilon\_\{\\rm app,n\}^\{2\}=\\widetilde\{O\}\_\{p\}\(n^\{\-2\\beta^\{\\star\}/\(2\\beta^\{\\star\}\+d\)\}\), so the ERM bound delivers an algorithm\-imitation guarantee at the same rate up to theO~​\(Pn/Npre\)\\widetilde\{O\}\(\\sqrt\{P\_\{n\}/N\_\{\\rm pre\}\}\)statistical penalty\.□\\square

## References

- Armstrong \[2022\]Timothy B Armstrong\.Asymptotic efficiency bounds for a class of experimental designs\.*arXiv preprint arXiv:2205\.02726*, 2022\.
- Bai et al\. \[2023\]Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei\.Transformers as statisticians: Provable in\-context learning with in\-context algorithm selection\.*arXiv preprint arXiv:2306\.04637*, 2023\.
- Bartlett et al\. \[2017\]Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky\.Spectrally\-normalized margin bounds for neural networks\.*Advances in neural information processing systems*, 30, 2017\.
- Berry et al\. \[2010\]Scott M Berry, Bradley P Carlin, J Jack Lee, and Peter Muller\.*Bayesian adaptive methods for clinical trials*\.CRC press, 2010\.
- Boucheron et al\. \[2003\]Stéphane Boucheron, Gábor Lugosi, and Pascal Massart\.Concentration inequalities using the entropy method\.*The Annals of Probability*, 31\(3\):1583–1614, 2003\.
- Chaloner and Verdinelli \[1995\]Kathryn Chaloner and Isabella Verdinelli\.Bayesian experimental design: A review\.*Statistical science*, pages 273–304, 1995\.
- Cook et al\. \[2024\]Thomas Cook, Alan Mishler, and Aaditya Ramdas\.Semiparametric efficient inference in adaptive experiments\.In*Causal Learning and Reasoning*, pages 1033–1064\. PMLR, 2024\.
- Dai et al\. \[2023\]Jessica Dai, Paula Gradu, and Christopher Harshaw\.Clip\-ogd: An experimental design for adaptive neyman allocation in sequential experiments\.*arXiv preprint arXiv:2305\.17187*, 2023\.
- Geer \[2000\]Sara A Geer\.*Empirical Processes in M\-estimation*, volume 6\.Cambridge university press, 2000\.
- Hahn \[1998\]Jinyong Hahn\.On the role of the propensity score in efficient semiparametric estimation of average treatment effects\.*Econometrica*, 66\(2\):315–331, 1998\.
- Hall and Heyde \[2014\]Peter Hall and Christopher C Heyde\.*Martingale limit theory and its application*\.Academic press, 2014\.
- Hu and Rosenberger \[2006\]Feifang Hu and William F Rosenberger\.*The theory of response\-adaptive randomization in clinical trials*\.John Wiley & Sons, 2006\.
- Hu et al\. \[2025\]Jerry Yao\-Chieh Hu, Hude Liu, Jennifer Yuntong Zhang, and Han Liu\.In\-context algorithm emulation in fixed\-weight transformers\.*arXiv preprint arXiv:2508\.17550*, 2025\.
- Huang et al\. \[2025\]Yu Huang, Zixin Wen, Aarti Singh, Yuejie Chi, and Yuxin Chen\.Transformers provably learn chain\-of\-thought reasoning with length generalization\.*arXiv preprint arXiv:2511\.07378*, 2025\.
- Kato et al\. \[2020\]Masahiro Kato, Takuya Ishihara, Junya Honda, and Yusuke Narita\.Efficient adaptive experimental design for average treatment effect estimation\.*arXiv preprint arXiv:2002\.05308*, 2020\.
- Lee et al\. \[2023\]Jonathan Lee, Annie Xie, Aldo Pacchiano, Yash Chandak, Chelsea Finn, Ofir Nachum, and Emma Brunskill\.Supervised pretraining can learn in\-context reinforcement learning\.*Advances in Neural Information Processing Systems*, 36:43057–43083, 2023\.
- Li et al\. \[2024\]Jiachun Li, David Simchi\-Levi, and Yunxiao Zhao\.Optimal adaptive experimental design for estimating treatment effect\.*arXiv preprint arXiv:2410\.05552*, 2024\.
- Lin et al\. \[2023\]Licong Lin, Yu Bai, and Song Mei\.Transformers as decision makers: Provable in\-context reinforcement learning via supervised pretraining\.*arXiv preprint arXiv:2310\.08566*, 2023\.
- Nesterov et al\. \[2018\]Yurii Nesterov et al\.*Lectures on convex optimization*, volume 137\.Springer, 2018\.
- Shen and Ghosal \[2015\]Weining Shen and Subhashis Ghosal\.Adaptive bayesian procedures using random series priors\.*Scandinavian Journal of Statistics*, 42\(4\):1194–1213, 2015\.
- Szabó et al\. \[2015\]Botond Szabó, Aad W Van Der Vaart, and JH Van Zanten\.Frequentist coverage of adaptive nonparametric bayesian credible sets\.2015\.
- Tian et al\. \[2021\]Yunsheng Tian, Mina Konakovic Lukovic, Michael Foshey, Timothy Erps, Beichen Li, and Wojciech Matusik\.Autooed: Automated optimal experimental design platform with data\-and time\-efficient multi\-objective optimization\.2021\.
- Trauger and Tewari \[2024\]Jacob Trauger and Ambuj Tewari\.Sequence length independent norm\-based generalization bounds for transformers\.In*International Conference on Artificial Intelligence and Statistics*, pages 1405–1413\. PMLR, 2024\.
- Tropp \[2011\]Joel Tropp\.Freedman’s inequality for matrix martingales\.2011\.
- Tropp \[2015\]Joel A Tropp\.An introduction to matrix concentration inequalities\.*Foundations and trends® in machine learning*, 8\(1\-2\):1–230, 2015\.
- van der Vaart and van Zanten \[2008\]Aad van der Vaart and Harry van Zanten\.Rates of contraction of posterior distributions based on Gaussian process priors\.*The Annals of Statistics*, 36\(3\):1435–1463, 2008\.
- van der Vaart and van Zanten \[2009\]Aad van der Vaart and Harry van Zanten\.Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth\.*The Annals of Statistics*, 37\(5B\):2655–2675, 2009\.
- van Waaij and van Zanten \[2017\]Jan van Waaij and Harry van Zanten\.Full adaptation to smoothness using randomly truncated series priors with gaussian coefficients and inverse gamma scaling\.*Statistics & Probability Letters*, 123:93–99, 2017\.
- Yarotsky \[2017\]Dmitry Yarotsky\.Error bounds for approximations with deep relu networks\.*Neural networks*, 94:103–114, 2017\.
- Zhao \[2023\]Jinglong Zhao\.Adaptive neyman allocation\.2023\.