Evidence-Aligned Local Composition of Discrete Experts for Sequence Restoration

# Evidence-Aligned Local Composition of Discrete Expertsfor Sequence Restoration
Source: [https://arxiv.org/html/2609.05801](https://arxiv.org/html/2609.05801)
Mohammad PanahazariAffiliation:Department of Electrical and Computer Engineering, Tufts University, Medford, MassachusettsUsman A\. KhanAffiliation:Boston College, Chestnut Hill, Massachusetts E\-mail: mohammad\.panahazari@tufts\.edu, usman\.khan@bc\.edu, shuchin@ece\.tufts\.eduShuchin AeronAffiliation:Department of Electrical and Computer Engineering, Tufts University, Medford, Massachusetts

###### Abstract

A document modeled as a discrete sequence of tokens can be thought of as being generated from a composition of texts from different domains; a README file, for example, moves between prose, code, and configuration\. When such a document is corrupted and only frozen domain experts are available, restoring it requires deciding both what is missing and which expert to trust at each position, at test time and without region labels or a trained router\. We introduce evidence\-aligned local composition, which infers a soft, position\-wise weighting over the experts from the marginal evidence of the corrupted observation under a given corruption model, estimating the evidence from the experts’ own denoising losses and smoothing the weights across positions\. Because the weighting is soft, it recovers a mixture when the true composition is mixed and concentrates on one expert when that suffices\. Across a categorical simulator, byte\-level experts, and experts fine\-tuned from a1\.31\.3B discrete flow\-matching model, the inferred weights track the true regions at0\.850\.85field accuracy on naturally mixed scientific documents, and at0\.980\.98on constructed mixtures whose regions are lexically disjoint\. Restoration improves over a single global weight when the experts are genuinely distinct and reduces to it when they converge, tracking a measure of expert separation\. Code is available at[https://github\.com/panahazari/evidence\-aligned\-local\-composition](https://github.com/panahazari/evidence-aligned-local-composition)\.

## 1Introduction

Real documents mix several kinds of content\. A software README interleaves prose, code, and configuration; a scientific article interleaves prose and mathematics; a data\-processing script embeds queries in a host language\. When such a document is damaged, by imperfect optical character recognition, a lossy channel, or aggressive filtering during data curation, restoring it means recovering the missing symbols\. We call this channel a corruption channel, or forward model, and we assume access to it, expressed as the likelihood of the corrupted output conditioned on the true input\.

Strong generative priors for each kind of content already exist as frozen, separately trained models, yet no single one serves a whole document well: the best expert changes from position to position, and the regions carry no labels\. We study how to restore such a document at test time by composing frozen domain experts, inferring at each position both what symbol is missing and which combination or composition of experts to trust, without position labels or a trained model, i\.e\. a router for composition\. We do not assume a single expert per position\. The object we infer is a composition of experts at each position; routing to one expert is the special case in which the composition concentrates on a single vertex of the simplex\.

This poses two coupled problems: the missing content must be inferred, but also the composition of the experts that should guide each position\. Weighted products of experts, or equivalently weighted sums of their log probabilities, provide a natural way to compose models\[[11](https://arxiv.org/html/2609.05801#bib.bib10),[19](https://arxiv.org/html/2609.05801#bib.bib12),[6](https://arxiv.org/html/2609.05801#bib.bib8)\]\. However, most existing approaches assign one weight to each expert for the entire sequence\[[22](https://arxiv.org/html/2609.05801#bib.bib15)\], which works when expert competence is roughly uniform but not when the preferred composition changes across positions\. Local positional weights offer the needed flexibility but are usually supplied by supervision or a trained router, while the test\-time mixtures that need no training fix one weight for the whole sequence, chosen from the input’s own likelihood\[[22](https://arxiv.org/html/2609.05801#bib.bib15),[41](https://arxiv.org/html/2609.05801#bib.bib5)\]; Table[1](https://arxiv.org/html/2609.05801#S1.T1)places these lines along the three axes that define our setting\. Our question is different: Can a soft field of expert weights be inferred at test time from the corrupted observation and the given corruption channel, without region labels or router training?

We approach this question through marginal evidence: the position\-dependent weights parameterize a composed prior, and the likelihood of the observation determines how they should change\[[21](https://arxiv.org/html/2609.05801#bib.bib14)\], connecting to evidence optimization and model selection for continuous\-diffusion priors\[[38](https://arxiv.org/html/2609.05801#bib.bib4),[39](https://arxiv.org/html/2609.05801#bib.bib2)\]but making the weights local and discrete\. The resulting update compares each expert’s fit to the observation with its fit under the current composition, a posterior\-minus\-prior difference we make precise in Section[4](https://arxiv.org/html/2609.05801#S4)\. Because the exact update is not directly computable, we approximate it, which motivates the name*evidence\-aligned local composition*\.

Table 1:Methods along our three axes and their data domain\.*Local*: the weights vary by position within one input, rather than one weighting being applied to the whole input\.*Test\-time*: the weights are set at inference from the input itself, with no router or weight predictor trained beforehand\.*Corruption\-aware*: inferred from the corrupted observation under a given channel\. wPoE optimizes a single scalar at test time from one data point, so it is test\-time but not local\.#### Main Contributions\.

- •Local composition from corrupted\-observation evidence\.We define a normalized sequence energy model with a simplex\-valued expert weight at every position and infer it at test time by an exponentiated\-gradient update on the marginal evidence, smoothed across positions\. The evidence gradient compares posterior and prior expectations of the same local expert energy\.
- •Denoising losses as local energies\.We score candidate sequences with the experts’ denoising losses\. For mask\-source experts, the schedule\-weighted population loss upper\-bounds sequence negative log likelihood; an unrestricted Bayes\-optimal denoiser recovers the associated any\-order conditional decomposition on supported contexts\. The deployed unweighted score remains a surrogate\.
- •Field recovery without supervision\.Through extensive experiments, we show that, without region labels, the inferred field closely matches the true regions and improves restoration over an independently optimized global weight on constructed mixtures; on natural documents the gain tracks how distinct the experts are, improving restoration over the global family on scientific text mixing prose and mathematics and tying when the experts converge\.

## 2Related Work

#### Composing and fusing experts\.

Composing models by multiplying densities, or summing log scores, underlies products of experts\[[11](https://arxiv.org/html/2609.05801#bib.bib10)\]and compositional or superposed diffusion\[[19](https://arxiv.org/html/2609.05801#bib.bib12),[6](https://arxiv.org/html/2609.05801#bib.bib8),[7](https://arxiv.org/html/2609.05801#bib.bib27),[32](https://arxiv.org/html/2609.05801#bib.bib28)\]; frozen language models are similarly fused or merged without training, as in PackLLM’s perplexity\-optimized global mixture\[[22](https://arxiv.org/html/2609.05801#bib.bib15)\], DEMix’s parameter\-free domain posterior\[[10](https://arxiv.org/html/2609.05801#bib.bib7)\], wPoE’s single test\-time scalar for compression\[[41](https://arxiv.org/html/2609.05801#bib.bib5)\], and output\- or parameter\-level ensembling and merging\[[37](https://arxiv.org/html/2609.05801#bib.bib31),[16](https://arxiv.org/html/2609.05801#bib.bib32),[34](https://arxiv.org/html/2609.05801#bib.bib33)\]\. All of these assign one weighting to the whole input\. Position\-dependent weights are otherwise supplied by supervision or, in concurrent work, produced by factorized composition of discrete diffusion experts from confidence margins\[[14](https://arxiv.org/html/2609.05801#bib.bib3)\]\. In contrast, our weight varies with position and is inferred at test time from the corrupted observation and a given channel, with no router training or region labels\.

#### Discrete diffusion priors and inverse problems\.

Our experts are discrete diffusion and flow\-matching models\[[1](https://arxiv.org/html/2609.05801#bib.bib1),[2](https://arxiv.org/html/2609.05801#bib.bib6),[20](https://arxiv.org/html/2609.05801#bib.bib13),[29](https://arxiv.org/html/2609.05801#bib.bib17),[31](https://arxiv.org/html/2609.05801#bib.bib18),[8](https://arxiv.org/html/2609.05801#bib.bib9),[18](https://arxiv.org/html/2609.05801#bib.bib20)\]that now scale to billions of parameters\[[24](https://arxiv.org/html/2609.05801#bib.bib23),[9](https://arxiv.org/html/2609.05801#bib.bib22),[25](https://arxiv.org/html/2609.05801#bib.bib21),[40](https://arxiv.org/html/2609.05801#bib.bib24)\]; for mask\-source absorbing diffusion, the schedule\-weighted denoising objective upper\-bounds sequence negative log likelihood and a Bayes\-optimal denoiser recovers clean\-data conditionals and their expected any\-order autoregressive decomposition\[[27](https://arxiv.org/html/2609.05801#bib.bib16),[36](https://arxiv.org/html/2609.05801#bib.bib19),[12](https://arxiv.org/html/2609.05801#bib.bib11)\]\. Restoring an observation under such a prior is the subject of diffusion inverse problems, in the continuous setting\[[3](https://arxiv.org/html/2609.05801#bib.bib29),[17](https://arxiv.org/html/2609.05801#bib.bib30),[38](https://arxiv.org/html/2609.05801#bib.bib4)\]and, more recently, with guidance tailored to discrete state spaces\[[26](https://arxiv.org/html/2609.05801#bib.bib25),[30](https://arxiv.org/html/2609.05801#bib.bib26)\], while marginal\-evidence estimation also drives model selection\[[39](https://arxiv.org/html/2609.05801#bib.bib2),[21](https://arxiv.org/html/2609.05801#bib.bib14)\]\. Our corruption channel plays the role of the forward operator, but the quantity we infer is a local composition field over frozen experts rather than a single guided reconstruction\.

## 3Problem Formulation

#### Problem setting\.

Let𝒱\\mathcal\{V\}be a finite vocabulary andx=\(x1,…,xL\)∈𝒱Lx=\(x\_\{1\},\\dots,x\_\{L\}\)\\in\\mathcal\{V\}^\{L\}a clean sequence, whereℓ∈\[1:L\]\\ell\\in\[1:L\]is the position index\. The goal is to recover x from y\. In order to do this, We are givenmmgenerative expertsq1,…,qmq\_\{1\},\\dots,q\_\{m\}, each defining a distribution over𝒱L\\mathcal\{V\}^\{L\}\. These experts are contextual: the per\-position conditionalqi​\(xℓ∣x∖ℓ\)q\_\{i\}\(x\_\{\\ell\}\\mid x\_\{\\setminus\\ell\}\)over𝒱\\mathcal\{V\}depends on the other positionsx∖ℓx\_\{\\setminus\\ell\}, as in the discrete flow\-matching models we use\. A position\-factorized expert, whose conditional ignoresx∖ℓx\_\{\\setminus\\ell\}, is the special case we use for the local pool in Eq\. \([1](https://arxiv.org/html/2609.05801#S3.E1)\)\. The experts remain frozen during composition, and their training data are not available\. We observe onlyyy, produced fromxxby a given corruption channelpc​\(y∣x\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)\. In the*mask channel*, each token is independently replaced with probabilityrrby a symbol\[𝙼𝙰𝚂𝙺\]∉𝒱\\mathtt\{\[MASK\]\}\\notin\\mathcal\{V\}; in the*replacement channel*, it is replaced by a uniform draw from𝒱\\mathcal\{V\}\. Both channels factorize across positions, sopc​\(y∣x\)=∏ℓpc​\(yℓ∣xℓ\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)=\\prod\_\{\\ell\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid x\_\{\\ell\}\)\. We denote the local likelihood of candidate tokenvvbycℓ​\(v\)=pc​\(yℓ∣xℓ=v\)c\_\{\\ell\}\(v\)=p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid x\_\{\\ell\}=v\)\.

#### Local composition\.

We attach a simplex\-valued weight to every position\. Specifically, the*composition field*isλ1:L=\(λ1,…,λL\)\\lambda\_\{1:L\}=\(\\lambda\_\{1\},\\dots,\\lambda\_\{L\}\)withλℓ∈Δm=\{λ∈ℝ\+m:∑iλi=1\}\\lambda\_\{\\ell\}\\in\\Delta\_\{m\}=\\\{\\lambda\\in\\mathbb\{R\}\_\{\+\}^\{m\}:\\sum\_\{i\}\\lambda\_\{i\}=1\\\}\. If expertiiassigns the categorical distributionπi,ℓ\\pi\_\{i,\\ell\}to the token at positionℓ\\ell, their local log\-linear pool is

πλℓ,ℓ​\(v\)=exp⁡\(∑iλℓ,i​log⁡πi,ℓ​\(v\)\)∑u∈𝒱exp⁡\(∑iλℓ,i​log⁡πi,ℓ​\(u\)\)\.\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)=\\frac\{\\exp\\big\(\\textstyle\\sum\_\{i\}\\lambda\_\{\\ell,i\}\\log\\pi\_\{i,\\ell\}\(v\)\\big\)\}\{\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\\big\(\\sum\_\{i\}\\lambda\_\{\\ell,i\}\\log\\pi\_\{i,\\ell\}\(u\)\\big\)\}\.\(1\)A vertexλℓ=ei\\lambda\_\{\\ell\}=e\_\{i\}\(theii\-th standard basis vector ofℝm\\mathbb\{R\}^\{m\}\) routes the position to the singleii\-th expert, whereas an interior point composes several experts\. We say a sequence has a*region structure*when its positions split into contiguous blocks, each generated by a single expert; the field is then near a vertex inside a block and mixed near a boundary\. Region labels are never given to the method\. Where they exist, we use them only to evaluate the inferred field \(Section[5](https://arxiv.org/html/2609.05801#S5)\)\. The local pool of Eq\. \([1](https://arxiv.org/html/2609.05801#S3.E1)\) composes the experts one position at a time and is well defined only when the experts are position\-factorized\. Our experts are contextual, so a token’s conditional depends on the rest of the sequence, and the product of their local conditionals need not be a valid joint distribution\. We therefore, compose the experts at the level of the whole sequence, and recover the local pool as the position\-factorized special case\.

#### Sequence\-level composition\.

The local pool in \([1](https://arxiv.org/html/2609.05801#S3.E1)\) is sufficient for position\-factorized experts\. The contextual experts here are the sameq1,…,qmq\_\{1\},\\dots,q\_\{m\}defined above; there is no separate set of models\. For these experts, however, multiplying their position\-wise conditionals need not produce a normalized joint distribution\. We therefore define the sequence model directly\. Letϕi,ℓ​\(x\)∈ℝ\\phi\_\{i,\\ell\}\(x\)\\in\\mathbb\{R\}denote the energy that expertiiassigns to positionℓ\\ellwithin sequencexx; Section[4](https://arxiv.org/html/2609.05801#S4)later constructs its practical surrogate\. The resulting normalized log\-linear prior is

Qλ​\(x\)\\displaystyle Q\_\{\\lambda\}\(x\)=1Z⁡\(λ\)​exp⁡\(∑ℓ=1L∑i=1mλℓ,i​ϕi,ℓ​\(x\)\),\\displaystyle=\\tfrac\{1\}\{Z\(\\lambda\)\}\\exp\\Big\(\\textstyle\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{i=1\}^\{m\}\\lambda\_\{\\ell,i\}\\,\\phi\_\{i,\\ell\}\(x\)\\Big\),\(2\)Z⁡\(λ\)\\displaystyle Z\(\\lambda\)=∑x′∈𝒱Lexp⁡\(∑ℓ,iλℓ,i​ϕi,ℓ​\(x′\)\)\.\\displaystyle=\\textstyle\\sum\_\{x^\{\\prime\}\\in\\mathcal\{V\}^\{L\}\}\\exp\\Big\(\\sum\_\{\\ell,i\}\\lambda\_\{\\ell,i\}\\,\\phi\_\{i,\\ell\}\(x^\{\\prime\}\)\\Big\)\.Its marginal likelihood for the observation ispλ​\(y\)=∑xpc​\(y∣x\)​Qλ​\(x\)p\_\{\\lambda\}\(y\)=\\sum\_\{x\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)\. When all experts are position\-factorized andϕi,ℓ​\(x\)=log⁡πi,ℓ​\(xℓ\)\\phi\_\{i,\\ell\}\(x\)=\\log\\pi\_\{i,\\ell\}\(x\_\{\\ell\}\),QλQ\_\{\\lambda\}factorizes and reduces to the local construction in \([1](https://arxiv.org/html/2609.05801#S3.E1)\)\. For contextual experts, the evidence under \([2](https://arxiv.org/html/2609.05801#S3.E2)\) remains our target, although the practical sampler is not known to draw fromQλQ\_\{\\lambda\}\. Section[4](https://arxiv.org/html/2609.05801#S4)makes this approximation explicit\.

#### Objective\.

Given the composed prior \([2](https://arxiv.org/html/2609.05801#S3.E2)\), the field is inferred from the corrupted observation alone\. Define the log marginal likelihood of the observation, with the unknown clean sequence integrated out, and the total variation of the field along the sequence:

L\(λ;y\)=log∑x∈𝒱Lpc\(y∣x\)Qλ\(x\),Ω\(λ\)=∑ℓ=2L∥λℓ−λℓ−1∥1,λ^1:L∈arg​maxλ1:L∈ΔmLF\(λ;y\),F\(λ;y\)=L\(λ;y\)−τΩ\(λ\),τ≥0\.\\begin\{gathered\}L\(\\lambda;y\)=\\log\\textstyle\\sum\_\{x\\in\\mathcal\{V\}^\{L\}\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)\\,Q\_\{\\lambda\}\(x\),\\quad\\Omega\(\\lambda\)=\\textstyle\\sum\_\{\\ell=2\}^\{L\}\\\|\\lambda\_\{\\ell\}\-\\lambda\_\{\\ell\-1\}\\\|\_\{1\},\\\\ \\hat\{\\lambda\}\_\{1:L\}\\in\\operatorname\*\{arg\\,max\}\_\{\\lambda\_\{1:L\}\\in\\Delta\_\{m\}^\{L\}\}F\(\\lambda;y\),\\quad F\(\\lambda;y\)=L\(\\lambda;y\)\-\\tau\\,\\Omega\(\\lambda\),\\ \\ \\tau\\geq 0\.\\end\{gathered\}\(3\)The regularizer admits a prior reading\. A Laplace density on the incrementsλℓ−λℓ−1\\lambda\_\{\\ell\}\-\\lambda\_\{\\ell\-1\}with scaleτ−1\\tau^\{\-1\}haslog⁡p⁡\(λ\)=−τ​Ω​\(λ\)\\log p\(\\lambda\)=\-\\tau\\Omega\(\\lambda\)up to an additive constant, soFFis the log posteriorlog⁡p⁡\(y∣λ\)\+log⁡p⁡\(λ\)\\log p\(y\\mid\\lambda\)\+\\log p\(\\lambda\)\. Penalizing the increments inℓ1\\ell\_\{1\}makes the maximizer piecewise constant, matching regions that persist and then change at a boundary\. Atτ=0\\tau=0the objective reduces to the evidence\.

WhenNNcorrupted observations share one field, as in the simulator of Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1), we averageL⁡\(λ,y\(n\)\)L\(\\lambda;y^\{\(n\)\}\)overnn; every real\-data experiment has one observation per document\.

For position\-factorized experts and a memoryless channel \(one that acts on each position independently,pc​\(y∣x\)=∏ℓpc​\(yℓ∣xℓ\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)=\\prod\_\{\\ell\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid x\_\{\\ell\}\), as with the mask and replacement channels above\), the evidence term separates across positions; this is proved as Proposition[2](https://arxiv.org/html/2609.05801#Thmproposition2)in Appendix[A\.2](https://arxiv.org/html/2609.05801#A1.SS2)\. OnlyΩ\\Omegacouples adjacent positions, so atτ=0\\tau=0the objective decouples and eachλℓ\\lambda\_\{\\ell\}is optimized independently\. This is the case used in the controlled simulator\. Writingcn,ℓ​\(v\)=pc​\(yℓ\(n\)∣xℓ=v\)c\_\{n,\\ell\}\(v\)=p\_\{\\mathrm\{c\}\}\(y^\{\(n\)\}\_\{\\ell\}\\mid x\_\{\\ell\}=v\), the contribution of positionℓ\\ellto theNN\-observation average becomes

Lℓ\(λℓ\)=1N∑n=1Nlog∑v∈𝒱cn,ℓ\(v\)πλℓ,ℓ\(v\),λ^ℓ∈arg​maxλℓ∈Δm⁡Lℓ​\(λℓ\)\.\\begin\{gathered\}L\_\{\\ell\}\(\\lambda\_\{\\ell\}\)=\\tfrac\{1\}\{N\}\\textstyle\\sum\_\{n=1\}^\{N\}\\log\\sum\_\{v\\in\\mathcal\{V\}\}c\_\{n,\\ell\}\(v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\),\\\\ \\hat\{\\lambda\}\_\{\\ell\}\\in\\operatorname\*\{arg\\,max\}\_\{\\lambda\_\{\\ell\}\\in\\Delta\_\{m\}\}L\_\{\\ell\}\(\\lambda\_\{\\ell\}\)\.\\end\{gathered\}\(4\)We use this tractable case for controlled validation\. For contextual experts, the sequence\-level objective \([3](https://arxiv.org/html/2609.05801#S3.E3)\) remains the target\.

#### Expert requirements\.

Each expertqiq\_\{i\}must expose two operations for our method; we do not introduce new models here\. A discrete flow\-matching model, equivalently a mask\-source discrete diffusion model, generates a sequence from a fully masked state by iteratively revealing tokens along a time schedule: at each step a denoiser predicts the clean token at every masked position from the revealed context, trained by a schedule\-weighted denoising cross\-entropy whose population value upper\-bounds sequence negative log likelihood\[[1](https://arxiv.org/html/2609.05801#bib.bib1),[2](https://arxiv.org/html/2609.05801#bib.bib6),[20](https://arxiv.org/html/2609.05801#bib.bib13),[8](https://arxiv.org/html/2609.05801#bib.bib9),[29](https://arxiv.org/html/2609.05801#bib.bib17),[31](https://arxiv.org/html/2609.05801#bib.bib18)\]\. Unlike autoregressive models the denoiser is non\-causal, conditioning on both sides of a masked position, which makes these models natural priors for restoration; we use them as our experts\. Given a partially masked contextzzand timett, each expert returns logitshi​\(z,t\)ℓ,vh\_\{i\}\(z,t\)\_\{\\ell,v\}for the clean token at each position, with softmaxpθi​\(xℓ=v∣z,t\)p\_\{\\theta\_\{i\}\}\(x\_\{\\ell\}=v\\mid z,t\)\. It must also evaluate its denoising loss on a candidate clean sequence\. Beyond these operations, the method is agnostic to architecture and model size\.

## 4Method

Proofs of the statements in this section are collected in Appendix[A](https://arxiv.org/html/2609.05801#A1), apart from the short argument for Lemma[1](https://arxiv.org/html/2609.05801#Thmlemma1), which we keep in place because it is what makes the update a minorize\-maximize step\.

### 4\.1The evidence gradient

The update direction is the gradient with respect toλ\\lambdaof the sequence\-level evidence \([3](https://arxiv.org/html/2609.05801#S3.E3)\)\. The simplex constraintλℓ∈Δm\\lambda\_\{\\ell\}\\in\\Delta\_\{m\}is not imposed on the gradient itself; it is maintained by the multiplicative exponentiated\-gradient update in Eq\. \([10](https://arxiv.org/html/2609.05801#S4.E10)\), which renormalizes eachλℓ\\lambda\_\{\\ell\}back onto the simplex after every step\. The deployed sampler is not known to draw fromQλQ\_\{\\lambda\}, so the result below describes the exact target rather than the practical estimator\.

###### Proposition 1\(Sequence\-level evidence gradient\)\.

LetQλQ\_\{\\lambda\}be the normalized composed prior \([2](https://arxiv.org/html/2609.05801#S3.E2)\) with finite energiesϕi,ℓ\\phi\_\{i,\\ell\}, letpc​\(y∣x\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)be a channel with∑xpc​\(y∣x\)​Qλ​\(x\)\>0\\sum\_\{x\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)\>0, and letQλ​\(x∣y\)∝pc​\(y∣x\)​Qλ​\(x\)Q\_\{\\lambda\}\(x\\mid y\)\\propto p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)be the induced posterior\. ThenL⁡\(λ,y\)=log⁡pλ​\(y\)L\(\\lambda;y\)=\\log p\_\{\\lambda\}\(y\)is differentiable inλ\\lambdaon the relative interior ofΔmL\\Delta\_\{m\}^\{L\}and, for every positionℓ\\elland expertii,

∂L⁡\(λ,y\)∂λℓ,i=\\displaystyle\\frac\{\\partial L\(\\lambda;y\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=\{\}𝔼x∼Qλ\(⋅∣y\)\[ϕi,ℓ\(x\)\]\\displaystyle\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\(\\cdot\\mid y\)\}\\big\[\\phi\_\{i,\\ell\}\(x\)\\big\]\(5\)−𝔼x∼Qλ​\[ϕi,ℓ​\(x\)\]\.\\displaystyle\-\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\}\\big\[\\phi\_\{i,\\ell\}\(x\)\\big\]\.

Appendix[A\.1](https://arxiv.org/html/2609.05801#A1.SS1)gives the proof\. The gradient has a simple interpretation: it compares the same local expert energy under sequences consistent with the observation and under sequences drawn from the composed prior\. A coordinate increases relative to the others when its expert has a larger posterior\-minus\-prior energy gap\. We call this difference the*evidence correction*; Section[5\.4](https://arxiv.org/html/2609.05801#S5.SS4)compares it with local heuristics that omit the prior term\. Expert\-specific additive offsets cancel in the subtraction, but relative scales do not, so the correction alone does not calibrate the experts\. At a masked position the local likelihood is flat, so in a factorized model the evidence gradient vanishes there \(Proposition[3](https://arxiv.org/html/2609.05801#Thmproposition3), proved in Appendix[A\.2](https://arxiv.org/html/2609.05801#A1.SS2)\); evidence then reaches the position only through contextual dependence in the sequence model or through smoothing from neighbors, the two mechanisms our single\-observation real\-data experiments rely on\.

### 4\.2Denoising losses as local expert energies

The exact gradient requires local expert energies\. Recall from Eq\. \([2](https://arxiv.org/html/2609.05801#S3.E2)\) that the local expert energyϕi,ℓ​\(x\)∈ℝ\\phi\_\{i,\\ell\}\(x\)\\in\\mathbb\{R\}is the contribution of positionℓ\\ellto the log score expertiiassigns to the whole sequencexx, so that expertii’s total log score is∑ℓϕi,ℓ​\(x\)\\sum\_\{\\ell\}\\phi\_\{i,\\ell\}\(x\)up to the normalizer; a higherϕi,ℓ​\(x\)\\phi\_\{i,\\ell\}\(x\)means expertiifinds the token at positionℓ\\ellmore compatible with its context\. Flow\-matching models do not expose normalized sequence densities directly, but their denoising losses provide a tractable surrogate\. Letpt\(⋅∣x\)p\_\{t\}\(\\cdot\\mid x\)denote the mask\-source path that reveals each position of candidate sequencexxindependently with probabilityκ⁡\(t\)\\kappa\(t\)at timettand masks it otherwise, whereκ\\kappais an increasing schedule withκ⁡\(0\)=0\\kappa\(0\)=0andκ⁡\(1\)=1\\kappa\(1\)=1\. The schedule\-weighted population denoising loss is a variational upper bound on sequence negative log likelihood\[[1](https://arxiv.org/html/2609.05801#bib.bib1),[2](https://arxiv.org/html/2609.05801#bib.bib6),[29](https://arxiv.org/html/2609.05801#bib.bib17),[31](https://arxiv.org/html/2609.05801#bib.bib18)\]\. For an unrestricted Bayes\-optimal denoiser, it recovers clean\-data conditionals on contexts with positiveqiq\_\{i\}\-probability\. Their order\-averaged negative log scores areℰi,ℓ⋆​\(x\)=𝔼σ​\[−log⁡qi​\(xℓ∣xPσ​\(ℓ\)\)\]\\mathcal\{E\}^\{\\star\}\_\{i,\\ell\}\(x\)=\\mathbb\{E\}\_\{\\sigma\}\[\-\\log q\_\{i\}\(x\_\{\\ell\}\\mid x\_\{P\_\{\\sigma\}\(\\ell\)\}\)\], whereσ\\sigmais a uniformly random order andPσ​\(ℓ\)P\_\{\\sigma\}\(\\ell\)contains the positions precedingℓ\\ell\[[36](https://arxiv.org/html/2609.05801#bib.bib19),[12](https://arxiv.org/html/2609.05801#bib.bib11),[27](https://arxiv.org/html/2609.05801#bib.bib16)\]\. By the chain rule these scores sum to−log⁡qi​\(x\)\-\\log q\_\{i\}\(x\)\. For a factorized expert,ℰi,ℓ⋆\\mathcal\{E\}^\{\\star\}\_\{i,\\ell\}is the token negative log density, whileϕi,ℓ​\(x\)=−ℰi,ℓ⋆​\(x\)\\phi\_\{i,\\ell\}\(x\)=\-\\mathcal\{E\}^\{\\star\}\_\{i,\\ell\}\(x\)is the token log density\. In practice, we approximate this energy usingKKsampled times:

DScore^i,ℓ\(x\)=1K∑k=1K−logsoftmaxv=xℓhi\(xtk,tk\)ℓ,v,s^i,ℓ​\(x\)=−αi​DScore^i,ℓ​\(x\),\\begin\{gathered\}\\widehat\{\\mathrm\{DScore\}\}\_\{i,\\ell\}\(x\)=\\tfrac\{1\}\{K\}\\textstyle\\sum\_\{k=1\}^\{K\}\-\\log\\mathrm\{softmax\}\_\{v=x\_\{\\ell\}\}h\_\{i\}\(x\_\{t\_\{k\}\},t\_\{k\}\)\_\{\\ell,v\},\\\\ \\hat\{s\}\_\{i,\\ell\}\(x\)=\-\\alpha\_\{i\}\\,\\widehat\{\\mathrm\{DScore\}\}\_\{i,\\ell\}\(x\),\\end\{gathered\}\(6\)wheretk∼U⁡\[0,1\]t\_\{k\}\\sim\\mathrm\{U\}\[0,1\]andxtk∼ptk\(⋅∣x\)x\_\{t\_\{k\}\}\\sim p\_\{t\_\{k\}\}\(\\cdot\\mid x\)\. BecauseDScore^i,ℓ\\widehat\{\\mathrm\{DScore\}\}\_\{i,\\ell\}omits the schedule weight, we call it a denoising score rather than a Negative ELBO \(NELBO\)\. We uses^i,ℓ\\hat\{s\}\_\{i,\\ell\}as a surrogate forϕi,ℓ\\phi\_\{i,\\ell\}; in the factorized case, it plays the role of the token log densitylog⁡πi,ℓ\\log\\pi\_\{i,\\ell\}\. The negative sign maps a lower denoising loss to greater compatibility\. Additive expert\-specific offsets cancel from the evidence correction, whereas the scalesαi\\alpha\_\{i\}do not\. We setαi=1\\alpha\_\{i\}=1throughout\. The specialization check measures whether each expert favors its intended domain, but it does not calibrate these relative scales\.

To estimate the two expectations, we compose the logits ashλ​\(z,t\)ℓ,v=∑iλℓ,i​hi​\(z,t\)ℓ,vh\_\{\\lambda\}\(z,t\)\_\{\\ell,v\}=\\sum\_\{i\}\\lambda\_\{\\ell,i\}h\_\{i\}\(z,t\)\_\{\\ell,v\}\. Givenhλh\_\{\\lambda\}and observationyy, the sampler𝒮⁡\(hλ,y\)\\mathcal\{S\}\(h\_\{\\lambda\},y\)returns an unclamped particle set𝒳pr=\{xpr\(b\)\}b=1P\\mathcal\{X\}\_\{\\mathrm\{pr\}\}=\\\{x\_\{\\mathrm\{pr\}\}^\{\(b\)\}\\\}\_\{b=1\}^\{P\}and an observation\-clamped set𝒳po=\{xpo\(b\)\}b=1P\\mathcal\{X\}\_\{\\mathrm\{po\}\}=\\\{x\_\{\\mathrm\{po\}\}^\{\(b\)\}\\\}\_\{b=1\}^\{P\}\. We use their empirical averages for𝔼prior\\mathbb\{E\}\_\{\\mathrm\{prior\}\}and𝔼post\\mathbb\{E\}\_\{\\mathrm\{post\}\}, respectively\. Averaging the surrogate over these two particle sets gives the implemented gradient estimate for∂L⁡\(λ,y\)/∂λℓ,i\\partial L\(\\lambda;y\)/\\partial\\lambda\_\{\\ell,i\},

gℓ,i=\\displaystyle g\_\{\\ell,i\}=\{\}−αi​𝔼post​\[DScore^i,ℓ​\(x\)\]\\displaystyle\-\\alpha\_\{i\}\\,\\mathbb\{E\}\_\{\\mathrm\{post\}\}\\big\[\\widehat\{\\mathrm\{DScore\}\}\_\{i,\\ell\}\(x\)\\big\]\(7\)\+αi​𝔼prior​\[DScore^i,ℓ​\(x\)\]\.\\displaystyle\+\\alpha\_\{i\}\\,\\mathbb\{E\}\_\{\\mathrm\{prior\}\}\\big\[\\widehat\{\\mathrm\{DScore\}\}\_\{i,\\ell\}\(x\)\\big\]\.
#### Relation to the exact objective\.

Equation \([5](https://arxiv.org/html/2609.05801#S4.E5)\) is exact for the normalized model \([2](https://arxiv.org/html/2609.05801#S3.E2)\), whereas \([7](https://arxiv.org/html/2609.05801#S4.E7)\) replaces its energies with finite\-sample denoising scores, omits the schedule weight, scores revealed positions, and uses a composed\-logit sampler not known to targetQλQ\_\{\\lambda\}\. Thus the deployed score is not asserted to be a likelihood bound\. We therefore call the method evidence\-aligned rather than evidence\-maximizing and compare it with the exact objective in Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1)\.

### 4\.3Optimization, smoothing and decoding

The two terms of \([3](https://arxiv.org/html/2609.05801#S3.E3)\) call for different treatment:LLis smooth inλ\\lambdabut not concave, whileΩ\\Omegais convex but nonsmooth\. We therefore take a gradient step onLLand a proximal step onΩ\\Omega\. Throughout,g\(s\)g^\{\(s\)\}denotes the estimate \([7](https://arxiv.org/html/2609.05801#S4.E7)\) of∇λL\\nabla\_\{\\lambda\}Lat the current iterateλ\(s\)\\lambda^\{\(s\)\}; the derivation is stated for the exact gradient, andg\(s\)g^\{\(s\)\}substitutes for it in the deployed algorithm\.

#### The evidence is a difference of convex functions\.

Writec​\(x\)=pc​\(y∣x\)c\(x\)=p\_\{\\mathrm\{c\}\}\(y\\mid x\)and recallΦλ​\(x\)=∑ℓ,iλℓ,i​ϕi,ℓ​\(x\)\\Phi\_\{\\lambda\}\(x\)=\\sum\_\{\\ell,i\}\\lambda\_\{\\ell,i\}\\phi\_\{i,\\ell\}\(x\), which is affine inλ\\lambda\. Splitting the evidence into its two normalizers,

L\(λ;y\)=log∑xc\(x\)eΦλ​\(x\)−log∑xeΦλ​\(x\),L\(\\lambda;y\)=\\log\\textstyle\\sum\_\{x\}c\(x\)e^\{\\Phi\_\{\\lambda\}\(x\)\}\-\\log\\textstyle\\sum\_\{x\}e^\{\\Phi\_\{\\lambda\}\(x\)\},\(8\)exhibits it as a difference of two log\-sum\-exp functions of affine arguments, both convex\. Since𝒱L\\mathcal\{V\}^\{L\}is finite and every summand is positive,LLis smooth; it is not concave, and \([3](https://arxiv.org/html/2609.05801#S3.E3)\) is not a concave program\.

#### A concave surrogate, tight at the current iterate\.

###### Lemma 1\(Minorant\)\.

Fixλ\(s\)\\lambda^\{\(s\)\}in the relative interior ofΔmL\\Delta\_\{m\}^\{L\}and setM\(λ∣λ\(s\)\)=𝔼x∼Qλ\(s\)\(⋅∣y\)\[Φλ\(x\)\]−logZ\(λ\)M\(\\lambda\\mid\\lambda^\{\(s\)\}\)=\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda^\{\(s\)\}\}\(\\cdot\\mid y\)\}\\big\[\\Phi\_\{\\lambda\}\(x\)\\big\]\-\\log Z\(\\lambda\)\. Then*\(i\)*L⁡\(λ,y\)≥M⁡\(λ∣λ\(s\)\)\+κL\(\\lambda;y\)\\geq M\(\\lambda\\mid\\lambda^\{\(s\)\}\)\+\\kappafor a constantκ\\kappaindependent ofλ\\lambda, with equality atλ=λ\(s\)\\lambda=\\lambda^\{\(s\)\};*\(ii\)*M\(⋅∣λ\(s\)\)M\(\\cdot\\mid\\lambda^\{\(s\)\}\)is concave; and*\(iii\)*its gradient atλ\(s\)\\lambda^\{\(s\)\}is the evidence gradient \([5](https://arxiv.org/html/2609.05801#S4.E5)\)\.

###### Proof\.

For*\(i\)*, apply Jensen to the first term of \([8](https://arxiv.org/html/2609.05801#S4.E8)\) using the weightsQλ\(s\)​\(x∣y\)∝c⁡\(x\)​eΦλ\(s\)​\(x\)Q\_\{\\lambda^\{\(s\)\}\}\(x\\mid y\)\\propto c\(x\)e^\{\\Phi\_\{\\lambda^\{\(s\)\}\}\(x\)\}, which are positive and sum to one;κ\\kappacollects their entropy and𝔼⁡\[log⁡c\]\\mathbb\{E\}\[\\log c\]\. Jensen is tight when the weights are proportional to the summand, that is atλ=λ\(s\)\\lambda=\\lambda^\{\(s\)\}\. For*\(ii\)*,𝔼​\[Φλ​\(x\)\]\\mathbb\{E\}\[\\Phi\_\{\\lambda\}\(x\)\]is affine inλ\\lambdaandlog⁡Z\\log Zis convex, so the difference is concave\. For*\(iii\)*, the first term is affine inλ\\lambdawith its expectation taken under the*fixed*lawQλ\(s\)\(⋅∣y\)Q\_\{\\lambda^\{\(s\)\}\}\(\\cdot\\mid y\), so it differentiates to𝔼x∼Qλ\(s\)\(⋅∣y\)\[ϕi,ℓ\(x\)\]\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda^\{\(s\)\}\}\(\\cdot\\mid y\)\}\[\\phi\_\{i,\\ell\}\(x\)\], while∂log⁡Z⁡\(λ\)/∂λℓ,i=𝔼x∼Qλ​\[ϕi,ℓ​\(x\)\]\\partial\\log Z\(\\lambda\)/\\partial\\lambda\_\{\\ell,i\}=\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\}\[\\phi\_\{i,\\ell\}\(x\)\]by the exponential\-family identity used in Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)\. Settingλ=λ\(s\)\\lambda=\\lambda^\{\(s\)\}turns the second expectation into one underQλ\(s\)Q\_\{\\lambda^\{\(s\)\}\}and recovers \([5](https://arxiv.org/html/2609.05801#S4.E5)\)\. ∎

Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)is thus the gradient of a concave surrogate that touchesLLat the current iterate\. Since−τ​Ω\-\\tau\\Omegais concave,M−τ​ΩM\-\\tau\\OmegaminorizesFF, and maximizing it exactly would give monotone ascent,F⁡\(λ\(s\+1\)\)≥M⁡\(λ\(s\+1\)∣λ\(s\)\)−τ​Ω​\(λ\(s\+1\)\)\+κ≥F⁡\(λ\(s\)\)F\(\\lambda^\{\(s\+1\)\}\)\\geq M\(\\lambda^\{\(s\+1\)\}\\mid\\lambda^\{\(s\)\}\)\-\\tau\\Omega\(\\lambda^\{\(s\+1\)\}\)\+\\kappa\\geq F\(\\lambda^\{\(s\)\}\)\. That maximization is intractable, asMMcontainslog⁡Z\\log Z\. We therefore take one composite mirror ascent step on its linearization, with the entropic mirror map on each row:

λ\(s\+1\)=arg​maxλ∈ΔmL\{⟨g\(s\),λ⟩−τΩ\(λ\)−1η∑ℓDKL\(λℓ∥λℓ\(s\)\)\}\.\\lambda^\{\(s\+1\)\}=\\operatorname\*\{arg\\,max\}\_\{\\lambda\\in\\Delta\_\{m\}^\{L\}\}\\Big\\\{\\langle g^\{\(s\)\},\\lambda\\rangle\-\\tau\\Omega\(\\lambda\)\-\\tfrac\{1\}\{\\eta\}\\textstyle\\sum\_\{\\ell\}D\_\{\\mathrm\{KL\}\}\\big\(\\lambda\_\{\\ell\}\\,\\\|\\,\\lambda^\{\(s\)\}\_\{\\ell\}\\big\)\\Big\\\}\.\(9\)The entropic map is the natural geometry on a product of simplices: it keeps every iterate in the relative interior without an explicit projection, and its Bregman divergence is the KL divergence\.

#### Splitting \([9](https://arxiv.org/html/2609.05801#S4.E9)\) into two exactly solved steps\.

Equation \([9](https://arxiv.org/html/2609.05801#S4.E9)\) has no closed form, becauseΩ\\Omegacouples adjacent rows\. DroppingΩ\\Omegaleaves a problem that does: each row of theτ=0\\tau=0case of \([9](https://arxiv.org/html/2609.05801#S4.E9)\) is solved exactly by

λ~ℓ,i\(s\+1\)=λℓ,i\(s\)​exp⁡\(η​gℓ,i\(s\)\)∑jλℓ,j\(s\)​exp⁡\(η​gℓ,j\(s\)\)\.\\tilde\{\\lambda\}^\{\(s\+1\)\}\_\{\\ell,i\}=\\frac\{\\lambda^\{\(s\)\}\_\{\\ell,i\}\\exp\\\!\\big\(\\eta\\,g^\{\(s\)\}\_\{\\ell,i\}\\big\)\}\{\\sum\_\{j\}\\lambda^\{\(s\)\}\_\{\\ell,j\}\\exp\\\!\\big\(\\eta\\,g^\{\(s\)\}\_\{\\ell,j\}\\big\)\}\.\(10\)The normalization in \([10](https://arxiv.org/html/2609.05801#S4.E10)\) is the Bregman projection onto the simplex: dividing by the row sum is the exact solution ofarg​minλℓ∈ΔmDKL\(λℓ∥λℓ\(s\)eη​gℓ\(s\)\)\\operatorname\*\{arg\\,min\}\_\{\\lambda\_\{\\ell\}\\in\\Delta\_\{m\}\}D\_\{\\mathrm\{KL\}\}\\big\(\\lambda\_\{\\ell\}\\,\\\|\\,\\lambda^\{\(s\)\}\_\{\\ell\}e^\{\\eta g^\{\(s\)\}\_\{\\ell\}\}\\big\)\. Equation \([10](https://arxiv.org/html/2609.05801#S4.E10)\) is therefore a single mirror ascent step, not a gradient step followed by a separate renormalization\.

The penalty is then applied as a proximal step from that point,

λ\(s\+1\)=arg​minλ1:L∈ΔmL\{12∑ℓ∥λℓ−λ~ℓ\(s\+1\)∥22\+τΩ\(λ\)\},\\lambda^\{\(s\+1\)\}=\\operatorname\*\{arg\\,min\}\_\{\\lambda\_\{1:L\}\\in\\Delta\_\{m\}^\{L\}\}\\Big\\\{\\tfrac\{1\}\{2\}\\sum\_\{\\ell\}\\big\\\|\\lambda\_\{\\ell\}\-\\tilde\{\\lambda\}^\{\(s\+1\)\}\_\{\\ell\}\\big\\\|\_\{2\}^\{2\}\+\\tau\\Omega\(\\lambda\)\\Big\\\},\(11\)the stepSmoothτ\\mathrm\{Smooth\}\_\{\\tau\}of Algorithm[1](https://arxiv.org/html/2609.05801#alg1), which returns a piecewise\-constant field\. The simplex constraint sits inside the argmin, so \([11](https://arxiv.org/html/2609.05801#S4.E11)\) is the proximal map ofτ​Ω\\tau\\Omegatogether with the constraint indicator, not ofτ​Ω\\tau\\Omegaalone\.

The order of the two steps is determined by scale\. The quadratic term in \([11](https://arxiv.org/html/2609.05801#S4.E11)\) is not scale invariant, and the unnormalized vectorλℓ\(s\)​eη​gℓ\(s\)\\lambda^\{\(s\)\}\_\{\\ell\}e^\{\\eta g^\{\(s\)\}\_\{\\ell\}\}has total mass varying withg\(s\)g^\{\(s\)\}\. Applying \([11](https://arxiv.org/html/2609.05801#S4.E11)\) before \([10](https://arxiv.org/html/2609.05801#S4.E10)\) would therefore let the effective strength ofτ\\taudrift with the gradient magnitude across iterations\. Normalizing first fixes the scale, and it makesΩ\\Omegameasure variation in composition rather than in magnitude\.

For two experts,λℓ=\(aℓ,1−aℓ\)\\lambda\_\{\\ell\}=\(a\_\{\\ell\},1\-a\_\{\\ell\}\)reduces \([11](https://arxiv.org/html/2609.05801#S4.E11)\) tomina⁡12​∑ℓ\(aℓ−a~ℓ\)2\+τ​∑ℓ≥2\|aℓ−aℓ−1\|\\min\_\{a\}\\tfrac\{1\}\{2\}\\sum\_\{\\ell\}\(a\_\{\\ell\}\-\\tilde\{a\}\_\{\\ell\}\)^\{2\}\+\\tau\\sum\_\{\\ell\\geq 2\}\|a\_\{\\ell\}\-a\_\{\\ell\-1\}\|up to an overall factor of two, the scalar fused lasso\[[35](https://arxiv.org/html/2609.05801#bib.bib40)\], which Condat’s algorithm solves exactly in linear time\[[5](https://arxiv.org/html/2609.05801#bib.bib41)\]\. Its solution cannot leave the range of its input, so it stays in\[0,1\]\[0,1\], the constraint is slack, and \([11](https://arxiv.org/html/2609.05801#S4.E11)\) is solved exactly; this covers every two\-expert study in the paper\. For three or more experts we apply the same map per column and renormalize, which does not maximize \([11](https://arxiv.org/html/2609.05801#S4.E11)\), since the proximal map of a sum is not the composition of the proximal maps\.

#### Scope of the derivation\.

Each step solves a stated problem exactly: \([10](https://arxiv.org/html/2609.05801#S4.E10)\) is theτ=0\\tau=0case of \([9](https://arxiv.org/html/2609.05801#S4.E9)\), and \([11](https://arxiv.org/html/2609.05801#S4.E11)\) is solved exactly for two experts\. Their composition, however, is not \([9](https://arxiv.org/html/2609.05801#S4.E9)\): the forward step is taken in the KL geometry and the backward step in the Euclidean one, and a composition of proximal maps is not in general the proximal map of the sum\. With the nonconcavity of \([8](https://arxiv.org/html/2609.05801#S4.E8)\), this places the procedure as a splitting with a structural prior, for which we claim no convergence\. The experiments on naturally mixed documents instead use a low\-pass filter, convolving each column with a uniform kernel and blending with the unsmoothed field; it attenuates variation uniformly rather than preserving edges, and itsτ\\tauis a blend weight, not comparable with \([11](https://arxiv.org/html/2609.05801#S4.E11)\)\. The constructed studies use the proximal step at every scale, the naturally mixed document studies use the filter, and the simulator uses neither; Appendix[B\.4](https://arxiv.org/html/2609.05801#A2.SS4)gives each experiment’s form andτ\\tau\.

Once the field has been inferred, we restore each masked token by selecting the per\-position mode of the composed logits,

x^ℓ=arg​maxv∈𝒱∑iλ^ℓ,ihi\(z,tdec\)ℓ,v,ℓ∈M,\\hat\{x\}\_\{\\ell\}=\\operatorname\*\{arg\\,max\}\_\{v\\in\\mathcal\{V\}\}\\textstyle\\sum\_\{i\}\\hat\{\\lambda\}\_\{\\ell,i\}\\,h\_\{i\}\(z,t\_\{\\mathrm\{dec\}\}\)\_\{\\ell,v\},\\quad\\ell\\in M,\(12\)whereMMis the masked set,zzcontains the visible tokens, and all visible positions are copied unchanged\. The decoder fills masked positions independently and therefore returns per\-position modes rather than the joint posterior mode ofQλQ\_\{\\lambda\}\. At byte scale, decoding is deterministic given the field\. The pretrained decoder also samples a uniform\-source initialization, fixed once per document and shared across methods to pair decoding randomness; the iterative\-decoder ablation preserves the ordering\. Algorithm[1](https://arxiv.org/html/2609.05801#alg1)summarizes the procedure using the scores^\\hat\{s\}of \([6](https://arxiv.org/html/2609.05801#S4.E6)\)\.

Algorithm 1Evidence\-aligned local composition1:masked

yywith masked set

MM, experts

\{hi\}i=1m\\\{h\_\{i\}\\\}\_\{i=1\}^\{m\}, sampler

𝒮\\mathcal\{S\}
2:particles

PP, score samples

KK, iterations

RR, step

η\\eta
3:smoother

Smoothτ\\mathrm\{Smooth\}\_\{\\tau\}, decoder time

tdect\_\{\\mathrm\{dec\}\}
4:

λℓ,i\(0\)←1/m\\lambda^\{\(0\)\}\_\{\\ell,i\}\\leftarrow 1/mfor all

ℓ,i\\ell,i
5:for

s=0,…,R−1s=0,\\dots,R\-1do

6:

\(𝒳pr,𝒳po\)←𝒮⁡\(hλ\(s\),y\)\(\\mathcal\{X\}\_\{\\mathrm\{pr\}\},\\mathcal\{X\}\_\{\\mathrm\{po\}\}\)\\leftarrow\\mathcal\{S\}\(h\_\{\\lambda^\{\(s\)\}\},y\)
7:

g\(s\)←g^\{\(s\)\}\\leftarrowgradient estimate \([7](https://arxiv.org/html/2609.05801#S4.E7)\) from

𝒳pr,𝒳po\\mathcal\{X\}\_\{\\mathrm\{pr\}\},\\mathcal\{X\}\_\{\\mathrm\{po\}\}
8:

λ~ℓ,i\(s\+1\)∝λℓ,i\(s\)​exp⁡\(η​gℓ,i\(s\)\)\\tilde\{\\lambda\}^\{\(s\+1\)\}\_\{\\ell,i\}\\propto\\lambda^\{\(s\)\}\_\{\\ell,i\}\\exp\(\\eta g^\{\(s\)\}\_\{\\ell,i\}\), renormalize over

ii
9:

λ\(s\+1\)←Smoothτ​\(λ~\(s\+1\)\)\\lambda^\{\(s\+1\)\}\\leftarrow\\mathrm\{Smooth\}\_\{\\tau\}\\big\(\\tilde\{\\lambda\}^\{\(s\+1\)\}\\big\)
10:endfor

11:

λ^1:L←λ\(R\)1:L\\hat\{\\lambda\}\_\{1:L\}\\leftarrow\\lambda^\{\(R\)\}\_\{1:L\}; initialize decoder state

zzfrom

yy
12:

x^ℓ←\\hat\{x\}\_\{\\ell\}\\leftarrowdecode \([12](https://arxiv.org/html/2609.05801#S4.E12)\) for

ℓ∈M\\ell\\in M;

x^ℓ←yℓ\\hat\{x\}\_\{\\ell\}\\leftarrow y\_\{\\ell\}otherwise

13:return

λ^1:L,x^\\hat\{\\lambda\}\_\{1:L\},\\hat\{x\}

## 5Experiments

Our experiments address two questions: can the evidence\-aligned update recover a known composition field, and does the recovered field improve restoration? We begin with an exact categorical simulator, where field recovery can be measured directly, then turn to learned byte\-level experts and to experts adapted from a1\.31\.3B discrete flow\-matching model, and close with naturally mixed documents, whose regions are not constructed for the experiment\.

#### Protocol\.

Within an experiment all methods share the frozen experts, corrupted input, channel, decoder, and source initialization; only the composition weights vary\. Comparisons are paired over the stated unit, aggregating by source document when a document contributes several windows, and we report95%95\\%bootstrap intervals, two\-sided sign\-flip permutationpp\-values, and Cohen’sdzd\_\{z\}\. Unless noted, the channel is masking\.

#### Metrics\.

We report restoration quality and field quality separately\.*Restoration accuracy*is the fraction of corrupted positions recovered exactly\. For the field we use two measures, according to what is known\. When the true fieldλ⋆\\lambda^\{\\star\}is available, as in the simulator of Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1), we report the*field mean absolute error*1L​m​∑ℓ,i\|λ^ℓ,i−λℓ,i⋆\|\\frac\{1\}\{Lm\}\\sum\_\{\\ell,i\}\|\\hat\{\\lambda\}\_\{\\ell,i\}\-\\lambda^\{\\star\}\_\{\\ell,i\}\|\. When only region labels are available, which is every learned\-expert setting, we report*field accuracy*, the fraction of positions at whicharg​maxi⁡λ^ℓ,i\\operatorname\*\{arg\\,max\}\_\{i\}\\hat\{\\lambda\}\_\{\\ell,i\}equals the region label\. Field accuracy is meaningful only for a position\-varying field: a constant field has the same argmax everywhere, and we mark those entries⋅\\cdot\. Region labels enter the metric only, never the method, the hard label router excepted, which is privileged for that reason\.

#### Baselines\.

We compare local evidence with each individual expert, equal weights, an independently optimized global weight that enforcesλℓ=λ\\lambda\_\{\\ell\}=\\lambdaunder the same optimization, reported at the byte,1\.31\.3B and arXiv scales, a best\-single\-expert selector given by the vertex with the lowest own\-posterior denoising score, shuffled\-field controls, and a hard router that receives the true region labels\. The router is privileged but not an upper bound, since a soft composition can mix experts near region boundaries\. Section[5\.4](https://arxiv.org/html/2609.05801#S5.SS4)adds three local routers that omit the evidence correction, and we adapt three published test\-time fusion methods to corrupted windows: PackLLM in its softmax and greedy variants\[[22](https://arxiv.org/html/2609.05801#bib.bib15)\], a DEMix\-style per\-position posterior router\[[10](https://arxiv.org/html/2609.05801#bib.bib7)\], and a marginal\-evidence selector\[[39](https://arxiv.org/html/2609.05801#bib.bib2)\]\. Each scores the visible tokens with the experts’ own denoising losses and feeds the same decoder\. Appendices[B\.6](https://arxiv.org/html/2609.05801#A2.SS6)and[C\.10](https://arxiv.org/html/2609.05801#A3.SS10)detail every baseline and the full statistical protocol\.

### 5\.1Recovering a known composition field

We begin where the composition field is known exactly\. The categorical simulator uses vocabulary size1212, length4848, and two experts, with three segments \(two routed, one mixed\); we draw a clean sequence from the composed prior and corrupt it with the replacement channel at rate0\.40\.4\. Optimizing the exact evidence \([4](https://arxiv.org/html/2609.05801#S3.E4)\) recovers the field at mean absolute error0\.0130\.013, versus0\.3330\.333for equal weights and0\.5000\.500for a single expert \(Figure[1](https://arxiv.org/html/2609.05801#S5.F1)\); replacing the exact densities with two trained flow\-matching experts and the denoising surrogate \([6](https://arxiv.org/html/2609.05801#S4.E6)\) raises the error only to0\.0310\.031, and the surrogate energy correlates0\.9740\.974/0\.9710\.971with the exact log likelihood\. Under the exact posterior decoder the recovered and ground\-truth fields both reach accuracy0\.6520\.652, above equal weights \(0\.6340\.634\) and either expert \(0\.3820\.382; Table[2](https://arxiv.org/html/2609.05801#S5.T2)\)\. A broader sweep shows recovery improving with more observations and greater separation and degrading with corruption\.

Table 2:Field recovery and reconstruction in the categorical simulator \(exact posterior decoder\)\. Recon\.: masked\-token accuracy\.Figure 1:Recovering a known field from corrupted observations alone: two routed segments \(positions00to3131\) and a fused segment \(3232to4747\)\.
### 5\.2Byte\-level experts

Table 3:Three byte\-level experts,256256windows from160160documents, mask channel, one\-step decoder, equal\-document estimand\. Hard label router uses true labels \(privileged, not an upper bound\)\.Now we consider learned byte\-level experts, first pairing prose and code on masked software\-documentation windows\. On a single corruption seed, local evidence reaches0\.5340\.534, above the hard label router \(0\.5140\.514\), best single expert \(0\.5070\.507\), global weight \(0\.4780\.478\), and equal weights \(0\.4400\.440\); the adapted fusion baselines fall in the same band \(0\.5080\.508PackLLM\-sim,0\.5040\.504PackLLM\-opt,0\.5070\.507marginal,0\.4920\.492DEMix\)\. Averaged over three corruption seeds the same ordering holds within0\.0030\.003per method \(Table S5\)\. The largest gain is in code regions, where accuracy rises from0\.4100\.410\(equal\) to0\.5770\.577, likely because the soft field composes experts near a boundary rather than committing to one region\. We then add a configuration expert for the main three\-expert setting; each window spans at least two regimes with a minimum length so no region dominates\. Across256256windows from160160documents, local evidence reaches document\-weighted accuracy0\.6600\.660\(Table[3](https://arxiv.org/html/2609.05801#S5.T3)\), indistinguishable from the hard label router \(p=0\.50p=0\.50\) and above the best single expert, the independently optimized global weight, equal weights, every individual expert, and the adapted literature baselines \(allp<10−3p<10^\{\-3\}\)\. Shuffling the field within a document costs0\.1190\.119and across documents0\.1810\.181, so restoration depends on where the weights are placed, not only their average; the field recovers the three regimes at macro one\-versus\-rest AUROC0\.9930\.993, and an iterative decoder preserves the ordering\.

Figure[2](https://arxiv.org/html/2609.05801#S5.F2)shows representative windows and a failure case chosen by prespecified rules: in the two median windows the field approaches each region’s expert in the interior and turns within a few positions of the boundary, while in the lowest\-performing window for the harder pair it identifies the prose and code interiors but briefly weights the absent configuration expert near their transition\. Errors thus concentrate at region boundaries\.

Figure 2:Recovered fields for constructed windows\. Ribbon: true regions; curves: weightsλℓ\\lambda\_\{\\ell\}inferred from the masked window alone\.
### 5\.3Pretrained experts at scale

We then ask whether the same behavior persists when the experts are adapted from a large pretrained model\. We fine\-tune prose and code experts with rank\-1616LoRA from a shared1\.31\.3B discrete flow\-matching model\. The evaluation uses constructed windows sampled with replacement\. The statistical unit here is the window rather than the source document, since source\-document identities are not tracked in this setting; possible reuse of source material across windows is therefore a limitation\.

#### These results fall outside Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)’s assumptions\.

The denoising\-loss argument of Section[4](https://arxiv.org/html/2609.05801#S4)is stated for the mask\-source path, but this1\.31\.3B base model has a*uniform*source, so at intermediate times a corrupted position holds a random token rather than a mask symbol\. We have no analogue of the identity for that path, so here the per\-position denoising score is a heuristic energy rather than an order\-averaged conditional log score, and the mismatch is uncontrolled\. This applies to every result in this subsection and to both natural\-document settings, which is to say to the results we lead with\. Their high field accuracy shows the surrogate stays informative; it does not validate the identity for a uniform source\. The supplement discusses the uniform\-source mismatch in full\.

Local evidence reaches0\.6530\.653, above the independently optimized global weight \(0\.6110\.611;\+0\.042\+0\.042,95%95\\%CI\[\+0\.039,\+0\.046\]\[\+0\.039,\+0\.046\],p<10−3p<10^\{\-3\},dz=1\.47d\_\{z\}=1\.47\), the best single expert \(0\.5990\.599\), and equal weights \(0\.5990\.599; Table[4](https://arxiv.org/html/2609.05801#S5.T4)\); the adapted PackLLM and marginal\-evidence baselines track the global family, all below local \(p<10−3p<10^\{\-3\}\), and the field agrees with the true regions at0\.9950\.995\. A single expert fine\-tuned on the*union*of both domains, the “train one model on everything” alternative, needs pooled data our setting does not assume yet reaches only0\.6360\.636\(above the global weight, below local;\+0\.017\+0\.017,p<10−3p<10^\{\-3\}\) and falls to0\.6490\.649on the README documents of Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5), below both the global weight and the best single expert\. This union expert receives the same60006000fine\-tuning steps as each specialist while covering twice the domain diversity, so it is undertrained relative to a specialist by construction and the comparison is not compute\-matched; it bounds what one model trained on the pooled domains achieves at a matched step count, not at a matched budget\. The independent global is not detectably different from the field\-average reference \(0\.6110\.611versus0\.6100\.610,p=0\.72p=0\.72\), the privileged hard label router is0\.0020\.002higher than local evidence \(0\.6550\.655,p=0\.01p=0\.01\), and the ordering is stable across mask rates\.

Table 4:Fine\-tuned1\.31\.3B experts,256256windows, mask rate0\.30\.3\. Paired differences vs local evidence\.∗p<0\.05\{\}^\{\*\}p<0\.05,∗∗∗p<10−3\{\}^\{\*\*\*\}p<10^\{\-3\}\.

### 5\.4Local routers without the evidence correction

The preceding comparisons show that position\-dependent weights can help, but they do not establish that the posterior\-minus\-prior correction is responsible\. We therefore compare against three routers that replace the evidence update with a local heuristic: normalized per\-expert scores \(a DEMix\-style responsibility router\[[10](https://arxiv.org/html/2609.05801#bib.bib7)\]\), raw denoising loss without the prior term, and a two\-state HMM over the per\-expert scores\.

In the two pretrained settings, the responsibility, raw\-loss and HMM routers perform at or below equal weights and trail local evidence \(p<10−3p<10^\{\-3\}; Table[5](https://arxiv.org/html/2609.05801#S5.T5)\); at1\.31\.3B the first two land on equal weights to within0\.0010\.001\. They also trail on the arXiv documents of Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5)\(0\.6530\.653,0\.6500\.650and0\.6440\.644against0\.6910\.691; Table[6](https://arxiv.org/html/2609.05801#S5.T6)\)\. Their field accuracy is far lower throughout \(0\.800\.80,0\.810\.81and0\.720\.72against0\.9950\.995at1\.31\.3B;0\.560\.56,0\.560\.56and0\.510\.51against0\.7650\.765on README\)\. Local evidence is thus the only router that combines strong restoration with high field accuracy across settings\.

We caution that this section does not isolate the posterior\-minus\-prior correction on its own\. Each comparison router differs from local evidence in more than one way at once; the raw local\-loss router, for instance, drops both the prior expectation and the imputed\-particle posterior estimate, since the adapted baselines score only visible tokens\. Attributing the gap to the prior term alone would need an ablation that keeps the deployed posterior term and removes only the prior expectation, which we have not run\.

Table 5:Local routers that omit the posterior\-minus\-prior correction, across the three settings; each router cell is masked\-token accuracy / field accuracy\. References are constant fields \(accuracy only\); the global weight is the independent one at byte and1\.31\.3B and the jointly optimized one on README, which are indistinguishable where both were run\. The byte and README columns are document\-weighted; the1\.31\.3B column takes the window as the unit, as declared in Section[5\.3](https://arxiv.org/html/2609.05801#S5.SS3)\. The HMM router was not run at byte scale\.Table 6:Distinct natural experts: arXiv prose andLaTeXmathematics,256256windows from7171documents, mask rate0\.30\.3, document\-clustered\. Field: field accuracy \(⋅\\cdot= constant field\)\. Best in each column is bold, the oracle label router excepted\.∗∗∗p<10−3\{\}^\{\*\*\*\}p<10^\{\-3\}\.
### 5\.5Naturally mixed documents

Constructed mixtures have known boundaries; we now test whether local composition helps on documents whose regions arise naturally\. Scientific articles interleave natural\-language prose withLaTeXmathematics, two lexically distant regimes that co\-occur in every paper\. We fine\-tune prose and mathematics experts from the1\.31\.3B model on collected arXiv sources and label each token by its math environment; the two experts separate strongly, with margins of0\.470\.47and0\.460\.46nats\. Across256256windows from7171documents with document\-clustered statistics, local evidence reaches accuracy0\.6910\.691and outperforms an independently optimized global weight by0\.0100\.010\(p<10−3p<10^\{\-3\},dz=0\.56d\_\{z\}=0\.56\), equal weights by0\.0180\.018, and the best single expert by0\.0220\.022\(Table[6](https://arxiv.org/html/2609.05801#S5.T6)\), while recovering the regions at field accuracy0\.8520\.852\. The adapted global\-fusion baselines match the global weight \(PackLLM\-opt\+0\.012\+0\.012, PackLLM\-sim\+0\.015\+0\.015, marginal\-evidence\+0\.024\+0\.024; allp<10−3p<10^\{\-3\}\), and the local routers of Section[5\.4](https://arxiv.org/html/2609.05801#S5.SS4)trail it further\. Local evidence is therefore the only method here that both improves restoration over the global family and attributes the regions faithfully\.

Whether local composition improves on a global weight in natural documents appears to depend on how distinct the experts are\. When they converge on the evaluation text, as with prose and code experts fine\-tuned on matched README corpora, we detect no restoration gain over a global weight \(0\.7090\.709versus0\.7080\.708\), and the field instead contributes regional attribution \(field accuracy0\.7650\.765\)\. This is consistent with the constructed\-mixture trend in which greater separation yields a larger gain, but it rests on two natural settings, so we report it as an observation rather than as a rule that predicts the regime in advance\.

## 6Conclusion

We introduced evidence\-aligned local composition, a test\-time method for restoring corrupted discrete sequences with frozen experts and no region supervision or trained router: instead of one expert per sequence, it infers a simplex\-valued field from the corrupted observation, whose evidence gradient, for a normalized sequence energy model, is exactly a posterior\-minus\-prior difference of expert energies estimated from the experts’ denoising losses \(the deployed algorithm only approximates this, so we call it evidence\-aligned\)\. The most consistent result is recovery of position\-wise expert preference: on constructed windows the field tracks the known regions and outperforms an independently optimized global weight, and on natural documents the gain tracks expert separation, improving over the global family on distinct prose andLaTeXmathematics and reducing to attribution on converged README prose and code\. Three limitations\. First, the schedule\-weighted likelihood identity requires a Bayes\-optimal mask\-source denoiser on supported contexts, whereas the deployed unweighted score remains a surrogate, the composed\-logit sampler is not known to draw fromQλQ\_\{\\lambda\}, and the1\.31\.3B experts have a uniform source for which we have no analogue of the identity at all\. Second, composition helps when the experts are specialized and distinct, its advantage over a global weight vanishing when they converge\. Third, our comparisons do not isolate the posterior\-minus\-prior correction on its own, since every local router we compare against differs from local evidence in more than one way at once; an ablation that keeps the deployed posterior term and removes only the prior expectation remains open\.

## Acknowledgments

Shuchin Aeron would like to acknowledge support by NSF under Cooperative Agreement PHY\-2019786 and NSF DMS 2309519\.

## References

- \[1\]\(2021\)Structured denoising diffusion models in discrete state\-spaces\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[2\]A\. Campbell, J\. Benton, V\. De Bortoli, T\. Rainforth, G\. Deligiannidis, and A\. Doucet\(2022\)A continuous time framework for discrete denoising models\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[3\]H\. Chung, J\. Kim, M\. T\. McCann, M\. L\. Klasky, and J\. C\. Ye\(2023\)Diffusion posterior sampling for general noisy inverse problems\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[4\]CodeParrot\(2022\)Codeparrot\-clean: a deduplicated python subset of github code\.Note:Hugging Face dataset,[https://huggingface\.co/datasets/codeparrot/codeparrot\-clean](https://huggingface.co/datasets/codeparrot/codeparrot-clean)Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px3.p1.1)\.
- \[5\]L\. Condat\(2013\)A direct algorithm for 1\-D total variation denoising\.IEEE Signal Processing Letters20\(11\),pp\. 1054–1057\.Cited by:[§4\.3](https://arxiv.org/html/2609.05801#S4.SS3.SSS0.Px3.p4.1)\.
- \[6\]Y\. Du, C\. Durkan, R\. Strudel, J\. B\. Tenenbaum, S\. Dieleman, R\. Fergus, J\. Sohl\-Dickstein, A\. Doucet, and W\. Grathwohl\(2023\)Reduce, reuse, recycle: compositional generation with energy\-based diffusion models and MCMC\.InICML,Cited by:[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.2.1),[§1](https://arxiv.org/html/2609.05801#S1.p3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[7\]T\. Garipov, S\. De Peuter, G\. Yang, V\. Garg, S\. Kaski, and T\. Jaakkola\(2023\)Compositional sculpting of iterative generative processes\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[8\]I\. Gat, T\. Remez, N\. Shaul, F\. Kreuk, R\. T\. Q\. Chen, G\. Synnaeve, Y\. Adi, and Y\. Lipman\(2024\)Discrete flow matching\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1)\.
- \[9\]S\. Gong, S\. Agarwal, Y\. Zhang, J\. Ye, L\. Zheng, M\. Li, C\. An, P\. Zhao, W\. Bi, H\. Peng, J\. Han, and L\. Kong\(2025\)Scaling diffusion language models via adaptation from autoregressive models\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[10\]S\. Gururangan, M\. Lewis, A\. Holtzman, N\. A\. Smith, and L\. Zettlemoyer\(2022\)DEMix layers: disentangling domains for modular language modeling\.InNAACL,Cited by:[§C\.10](https://arxiv.org/html/2609.05801#A3.SS10.SSS0.Px3.p3.1),[Table S9](https://arxiv.org/html/2609.05801#A3.T9.5.1.1.1.1.1.4.1),[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.05801#S5.SS0.SSS0.Px3.p1.1),[§5\.4](https://arxiv.org/html/2609.05801#S5.SS4.p1.1),[Table 3](https://arxiv.org/html/2609.05801#S5.T3.5.1.1.1.1.1.1.9.1)\.
- \[11\]G\. E\. Hinton\(2002\)Training products of experts by minimizing contrastive divergence\.Neural Computation14\(8\)\.Cited by:[§1](https://arxiv.org/html/2609.05801#S1.p3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[12\]E\. Hoogeboom, A\. A\. Gritsenko, J\. Bastings, B\. Poole, R\. van den Berg, and T\. Salimans\(2022\)Autoregressive diffusion models\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[13\]Hoskinson Center for Formal Mathematics\(2022\)Proof\-pile: a corpus of mathematical text\.Note:Hugging Face dataset,[https://huggingface\.co/datasets/hoskinson\-center/proof\-pile](https://huggingface.co/datasets/hoskinson-center/proof-pile)Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px5.p1.1)\.
- \[14\]H\. Huang, Y\. Xu, A\. Mandal, and A\. Aspuru\-Guzik\(2026\)From global to factor\-wise expert composition in discrete diffusion models\.External Links:2607\.11758Cited by:[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.6.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]H\. Husain, H\. Wu, T\. Gazit, M\. Allamanis, and M\. Brockschmidt\(2019\)CodeSearchNet challenge: evaluating the state of semantic code search\.arXiv preprint arXiv:1909\.09436\.Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px2.p1.1)\.
- \[16\]D\. Jiang, X\. Ren, and B\. Y\. Lin\(2023\)LLM\-Blender: ensembling large language models with pairwise ranking and generative fusion\.InACL,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[17\]B\. Kawar, M\. Elad, S\. Ermon, and J\. Song\(2022\)Denoising diffusion restoration models\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[18\]Y\. Lipman, R\. T\. Q\. Chen, H\. Ben\-Hamu, M\. Nickel, and M\. Le\(2023\)Flow matching for generative modeling\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[19\]N\. Liu, S\. Li, Y\. Du, A\. Torralba, and J\. B\. Tenenbaum\(2022\)Compositional visual generation with composable diffusion models\.InECCV,Cited by:[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.2.1),[§1](https://arxiv.org/html/2609.05801#S1.p3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[20\]A\. Lou, C\. Meng, and S\. Ermon\(2024\)Discrete diffusion modeling by estimating the ratios of the data distribution\.InICML,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1)\.
- \[21\]D\. J\. C\. MacKay\(1992\)Bayesian interpolation\.Neural Computation4\(3\)\.Cited by:[§1](https://arxiv.org/html/2609.05801#S1.p4.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[22\]C\. Mavromatis, P\. Karypis, and G\. Karypis\(2024\)Pack of LLMs: model fusion at test\-time via perplexity optimization\.External Links:2404\.11531Cited by:[§C\.10](https://arxiv.org/html/2609.05801#A3.SS10.SSS0.Px3.p2.1),[Table S9](https://arxiv.org/html/2609.05801#A3.T9.5.1.1.1.1.1.10.1),[Table S9](https://arxiv.org/html/2609.05801#A3.T9.5.1.1.1.1.1.8.1),[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.4.1),[§1](https://arxiv.org/html/2609.05801#S1.p3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.05801#S5.SS0.SSS0.Px3.p1.1),[Table 3](https://arxiv.org/html/2609.05801#S5.T3.5.1.1.1.1.1.1.4.1),[Table 3](https://arxiv.org/html/2609.05801#S5.T3.5.1.1.1.1.1.1.5.1)\.
- \[23\]S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\(2017\)Pointer sentinel mixture models\.InInternational Conference on Learning Representations,Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px2.p1.1),[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px3.p1.1)\.
- \[24\]S\. Nie, F\. Zhu, C\. Du, T\. Pang, Q\. Liu, G\. Zeng, M\. Lin, and C\. Li\(2025\)Scaling up masked diffusion models on text\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[25\]S\. Nie, F\. Zhu, Z\. You, X\. Zhang, J\. Ou, J\. Hu, J\. Zhou, Y\. Lin, J\.\-R\. Wen, and C\. Li\(2025\)Large language diffusion models\.External Links:2502\.09992Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[26\]H\. Nisonoff, J\. Xiong, S\. Allenspach, and J\. Listgarten\(2025\)Unlocking guidance for discrete state\-space diffusion and flow models\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[27\]J\. Ou, S\. Nie, K\. Xue, F\. Zhu, J\. Sun, Z\. Li, and C\. Li\(2025\)Your absorbing discrete diffusion secretly models the conditional distributions of clean data\.InICLR,Note:arXiv:2406\.03736Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[28\]Riabov, Maksim\(2024\)Github\-top1000\-repos\-markdown: readme and markdown files from the 1000 most\-starred github repositories\.Note:Hugging Face dataset,[https://huggingface\.co/datasets/MRiabov/github\-top1000\-repos\-markdown](https://huggingface.co/datasets/MRiabov/github-top1000-repos-markdown)Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px4.p1.1)\.
- \[29\]S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. Rush, and V\. Kuleshov\(2024\)Simple and effective masked diffusion language models\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[30\]Y\. Schiff, S\. S\. Sahoo, H\. Phung, G\. Wang, S\. Boshar, H\. Dalla\-Torre, B\. P\. Almeida, A\. M\. Rush, T\. Pierrot, and V\. Kuleshov\(2025\)Simple guidance mechanisms for discrete diffusion models\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[31\]J\. Shi, K\. Han, Z\. Wang, A\. Doucet, and M\. K\. Titsias\(2024\)Simplified and generalized masked diffusion for discrete data\.InNeurIPS,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§3](https://arxiv.org/html/2609.05801#S3.SS0.SSS0.Px5.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[32\]M\. Skreta, L\. Atanackovic, A\. J\. Bose, A\. Tong, and K\. Neklyudov\(2025\)The superposition of diffusion models using the Itô density estimator\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[33\]Substratus AI\(2023\)The\-stack\-yaml\-k8s: kubernetes yaml manifests from the stack\.Note:Hugging Face dataset,[https://huggingface\.co/datasets/substratusai/the\-stack\-yaml\-k8s](https://huggingface.co/datasets/substratusai/the-stack-yaml-k8s)Cited by:[§B\.1](https://arxiv.org/html/2609.05801#A2.SS1.SSS0.Px2.p1.1)\.
- \[34\]S\. Sukhbaatar, O\. Golovneva, V\. Sharma, H\. Xu, X\. V\. Lin, B\. Rozière, J\. Kahn, D\. Li, W\.\-t\. Yih, J\. Weston, and X\. Li\(2024\)Branch\-train\-MiX: mixing expert LLMs into a mixture\-of\-experts LLM\.External Links:2403\.07816Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[35\]R\. Tibshirani, M\. Saunders, S\. Rosset, J\. Zhu, and K\. Knight\(2005\)Sparsity and smoothness via the fused lasso\.Journal of the Royal Statistical Society: Series B67\(1\),pp\. 91–108\.Cited by:[§4\.3](https://arxiv.org/html/2609.05801#S4.SS3.SSS0.Px3.p4.1)\.
- \[36\]B\. Uria, I\. Murray, and H\. Larochelle\(2014\)A deep and tractable density estimator\.InICML,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§4\.2](https://arxiv.org/html/2609.05801#S4.SS2.p1.1)\.
- \[37\]F\. Wan, X\. Huang, D\. Cai, X\. Quan, W\. Bi, and S\. Shi\(2024\)Knowledge fusion of large language models\.InICLR,Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.
- \[38\]F\. Wang and K\. L\. Bouman\(2026\)Optimizing diffusion priors in image reconstruction from a single observation\.External Links:2604\.21066Cited by:[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.7.1),[§1](https://arxiv.org/html/2609.05801#S1.p4.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[39\]F\. Wang and K\. L\. Bouman\(2026\)Sample\-efficient evidence estimation of score\-based priors for model selection\.InInternational Conference on Learning Representations \(ICLR\),Note:arXiv:2602\.20549Cited by:[§C\.10](https://arxiv.org/html/2609.05801#A3.SS10.SSS0.Px3.p5.1),[Table S9](https://arxiv.org/html/2609.05801#A3.T9.5.1.1.1.1.1.11.1),[§1](https://arxiv.org/html/2609.05801#S1.p4.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.05801#S5.SS0.SSS0.Px3.p1.1),[Table 3](https://arxiv.org/html/2609.05801#S5.T3.5.1.1.1.1.1.1.7.1)\.
- \[40\]J\. Ye, Z\. Xie, L\. Zheng, J\. Gao, Z\. Wu, X\. Jiang, Z\. Li, and L\. Kong\(2025\)Dream 7B: diffusion large language models\.External Links:2508\.15487Cited by:[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px2.p1.1)\.
- \[41\]Q\. Zhang, M\. Li, Z\. Wang, R\. Liao, and L\. Wang\(2025\)Test\-time steering for lossless text compression via weighted product of experts\.InFindings of the Association for Computational Linguistics: EMNLP,pp\. 2076–2088\.Note:arXiv:2511\.10660Cited by:[Table 1](https://arxiv.org/html/2609.05801#S1.T1.14.1.1.1.1.1.1.5.1),[§1](https://arxiv.org/html/2609.05801#S1.p3.1),[§2](https://arxiv.org/html/2609.05801#S2.SS0.SSS0.Px1.p1.1)\.

Equations, tables and figures introduced in this appendix carry anSprefix and appendix sections are lettered; plain numbers refer to the main text\. Proposition numbering continues from the main text, whose Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)is proved below\.

## Appendix AProofs

Appendix[A\.1](https://arxiv.org/html/2609.05801#A1.SS1)proves Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)of the main text, the sequence\-level evidence gradient that the method rests on\. Appendix[A\.2](https://arxiv.org/html/2609.05801#A1.SS2)proves the position\-factorized special case, which is the form used in the controlled simulator\.

### A\.1The sequence\-level evidence gradient

We first fix notation and state the standing assumptions, then prove the identity in full\.

#### Setting\.

The vocabulary𝒱\\mathcal\{V\}is finite and the lengthLLis fixed, so the state space𝒱L\\mathcal\{V\}^\{L\}is finite\. Each energyϕi,ℓ:𝒱L→ℝ\\phi\_\{i,\\ell\}:\\mathcal\{V\}^\{L\}\\to\\mathbb\{R\}is finite valued\. Forλ∈ℝL×m\\lambda\\in\\mathbb\{R\}^\{L\\times m\}write

Φλ​\(x\)=∑ℓ=1L∑i=1mλℓ,i​ϕi,ℓ​\(x\),Z⁡\(λ\)=∑x′∈𝒱Lexp⁡Φλ​\(x′\),\\begin\{gathered\}\\Phi\_\{\\lambda\}\(x\)=\\textstyle\\sum\_\{\\ell=1\}^\{L\}\\sum\_\{i=1\}^\{m\}\\lambda\_\{\\ell,i\}\\,\\phi\_\{i,\\ell\}\(x\),\\\\ Z\(\\lambda\)=\\textstyle\\sum\_\{x^\{\\prime\}\\in\\mathcal\{V\}^\{L\}\}\\exp\\Phi\_\{\\lambda\}\(x^\{\\prime\}\),\\end\{gathered\}\(S1\)so that the composed prior of \([2](https://arxiv.org/html/2609.05801#S3.E2)\) isQλ​\(x\)=exp⁡Φλ​\(x\)/Z⁡\(λ\)Q\_\{\\lambda\}\(x\)=\\exp\\Phi\_\{\\lambda\}\(x\)/Z\(\\lambda\)\. The channelpc​\(y∣x\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)is arbitrary subject topλ​\(y\)=∑xpc​\(y∣x\)​Qλ​\(x\)\>0p\_\{\\lambda\}\(y\)=\\sum\_\{x\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)\>0, and the posterior isQλ​\(x∣y\)=pc​\(y∣x\)​Qλ​\(x\)/pλ​\(y\)Q\_\{\\lambda\}\(x\\mid y\)=p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)/p\_\{\\lambda\}\(y\)\. No structure on the channel is needed: it need not be memoryless, andϕi,ℓ\\phi\_\{i,\\ell\}may depend on the whole sequence\.

###### Proof of Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)\.

*Step 1: the composed prior is positive and smooth inλ\\lambda\.*EachΦλ​\(x′\)\\Phi\_\{\\lambda\}\(x^\{\\prime\}\)is linear inλ\\lambdawith finite coefficientsϕi,ℓ​\(x′\)\\phi\_\{i,\\ell\}\(x^\{\\prime\}\), soexp⁡Φλ​\(x′\)\\exp\\Phi\_\{\\lambda\}\(x^\{\\prime\}\)is positive and infinitely differentiable inλ\\lambdaon all ofℝL×m\\mathbb\{R\}^\{L\\times m\}\. As𝒱L\\mathcal\{V\}^\{L\}is finite,Z⁡\(λ\)Z\(\\lambda\)is a finite sum of such terms, henceZ⁡\(λ\)\>0Z\(\\lambda\)\>0andZZis smooth\. ThereforeQλ​\(x\)\>0Q\_\{\\lambda\}\(x\)\>0andλ↦Qλ​\(x\)\\lambda\\mapsto Q\_\{\\lambda\}\(x\)is smooth for everyxx\.

*Step 2: the evidence is smooth, so the derivative exists\.*pλ​\(y\)=∑xpc​\(y∣x\)​Qλ​\(x\)p\_\{\\lambda\}\(y\)=\\sum\_\{x\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)Q\_\{\\lambda\}\(x\)is a finite sum of smooth functions ofλ\\lambdaand is therefore smooth\. It is positive by assumption, solog⁡pλ​\(y\)\\log p\_\{\\lambda\}\(y\)is smooth, in particular differentiable on the relative interior ofΔmL\\Delta\_\{m\}^\{L\}, and the posteriorQλ\(⋅∣y\)Q\_\{\\lambda\}\(\\cdot\\mid y\)is a well defined distribution on𝒱L\\mathcal\{V\}^\{L\}\. Every interchange of differentiation and summation below is over a finite index set and so needs no further justification\.

*Step 3: the score of the composed prior\.*Since∂Φλ​\(x\)/∂λℓ,i=ϕi,ℓ​\(x\)\\partial\\Phi\_\{\\lambda\}\(x\)/\\partial\\lambda\_\{\\ell,i\}=\\phi\_\{i,\\ell\}\(x\), differentiatingZZterm by term gives

∂log⁡Z⁡\(λ\)∂λℓ,i\\displaystyle\\frac\{\\partial\\log Z\(\\lambda\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=1Z⁡\(λ\)​∑x′exp⁡\(Φλ​\(x′\)\)​ϕi,ℓ​\(x′\)\\displaystyle=\\frac\{1\}\{Z\(\\lambda\)\}\\textstyle\\sum\_\{x^\{\\prime\}\}\\exp\\big\(\\Phi\_\{\\lambda\}\(x^\{\\prime\}\)\\big\)\\,\\phi\_\{i,\\ell\}\(x^\{\\prime\}\)=𝔼x′∼Qλ​\[ϕi,ℓ​\(x′\)\]\.\\displaystyle=\\mathbb\{E\}\_\{x^\{\\prime\}\\sim Q\_\{\\lambda\}\}\\big\[\\phi\_\{i,\\ell\}\(x^\{\\prime\}\)\\big\]\.Becauselog⁡Qλ​\(x\)=Φλ​\(x\)−log⁡Z⁡\(λ\)\\log Q\_\{\\lambda\}\(x\)=\\Phi\_\{\\lambda\}\(x\)\-\\log Z\(\\lambda\), subtracting yields

∂log⁡Qλ​\(x\)∂λℓ,i=ϕi,ℓ​\(x\)−𝔼x′∼Qλ​\[ϕi,ℓ​\(x′\)\]\.\\frac\{\\partial\\log Q\_\{\\lambda\}\(x\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=\\phi\_\{i,\\ell\}\(x\)\-\\mathbb\{E\}\_\{x^\{\\prime\}\\sim Q\_\{\\lambda\}\}\\big\[\\phi\_\{i,\\ell\}\(x^\{\\prime\}\)\\big\]\.\(S2\)This is the exponential\-family identity: the derivative of the log normalizer is the mean of the sufficient statistic, here the energyϕi,ℓ\\phi\_\{i,\\ell\}\.

*Step 4: differentiate the evidence\.*Differentiatingpλ​\(y\)p\_\{\\lambda\}\(y\)term by term and using∂Qλ=Qλ​∂log⁡Qλ\\partial Q\_\{\\lambda\}=Q\_\{\\lambda\}\\,\\partial\\log Q\_\{\\lambda\}, which is legitimate becauseQλ​\(x\)\>0Q\_\{\\lambda\}\(x\)\>0by Step 1,

∂log⁡pλ​\(y\)∂λℓ,i\\displaystyle\\frac\{\\partial\\log p\_\{\\lambda\}\(y\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=1pλ​\(y\)​∑xpc​\(y∣x\)​∂Qλ​\(x\)∂λℓ,i\\displaystyle=\\frac\{1\}\{p\_\{\\lambda\}\(y\)\}\\textstyle\\sum\_\{x\}p\_\{\\mathrm\{c\}\}\(y\\mid x\)\\,\\frac\{\\partial Q\_\{\\lambda\}\(x\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=∑xpc​\(y∣x\)​Qλ​\(x\)pλ​\(y\)⏟=Qλ​\(x∣y\)\\displaystyle=\\textstyle\\sum\_\{x\}\\underbrace\{\\frac\{p\_\{\\mathrm\{c\}\}\(y\\mid x\)\\,Q\_\{\\lambda\}\(x\)\}\{p\_\{\\lambda\}\(y\)\}\}\_\{=\\;Q\_\{\\lambda\}\(x\\mid y\)\}×∂log⁡Qλ​\(x\)∂λℓ,i\\displaystyle\\times\\,\\frac\{\\partial\\log Q\_\{\\lambda\}\(x\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=𝔼x∼Qλ\(⋅∣y\)\[∂log⁡Qλ​\(x\)∂λℓ,i\]\.\\displaystyle=\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\(\\cdot\\mid y\)\}\\Big\[\\frac\{\\partial\\log Q\_\{\\lambda\}\(x\)\}\{\\partial\\lambda\_\{\\ell,i\}\}\\Big\]\.The bracketed weight is nonnegative and sums to one overxx, so the last line is an expectation under the posterior\.

*Step 5: conclude\.*Substitute \([S2](https://arxiv.org/html/2609.05801#A1.E2)\)\. The prior expectation𝔼x′∼Qλ​\[ϕi,ℓ​\(x′\)\]\\mathbb\{E\}\_\{x^\{\\prime\}\\sim Q\_\{\\lambda\}\}\[\\phi\_\{i,\\ell\}\(x^\{\\prime\}\)\]does not depend onxx, so it passes through the posterior expectation unchanged, leaving

∂log⁡pλ​\(y\)∂λℓ,i\\displaystyle\\frac\{\\partial\\log p\_\{\\lambda\}\(y\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=𝔼x∼Qλ\(⋅∣y\)\[ϕi,ℓ\(x\)\]\\displaystyle=\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\(\\cdot\\mid y\)\}\\big\[\\phi\_\{i,\\ell\}\(x\)\\big\]−𝔼x∼Qλ​\[ϕi,ℓ​\(x\)\],\\displaystyle\-\\mathbb\{E\}\_\{x\\sim Q\_\{\\lambda\}\}\\big\[\\phi\_\{i,\\ell\}\(x\)\\big\],which is \([5](https://arxiv.org/html/2609.05801#S4.E5)\) and proves the proposition\. ∎

### A\.2The factorized special case

Throughout this subsection we fix a positionℓ\\elland suppress it when no ambiguity arises\. Expertiisupplies the categorical distributionπi,ℓ\\pi\_\{i,\\ell\}at positionℓ\\ell, andai​\(v\)=log⁡πi,ℓ​\(v\)a\_\{i\}\(v\)=\\log\\pi\_\{i,\\ell\}\(v\)\. Forλℓ∈Δm\\lambda\_\{\\ell\}\\in\\Delta\_\{m\}, the composed local prior is the normalized log\-linear pool from \([1](https://arxiv.org/html/2609.05801#S3.E1)\):

πλℓ,ℓ​\(v\)=exp⁡Aλℓ​\(v\)Zλℓ,Aλℓ\(v\)=∑j=1mλℓ,jaj\(v\),Zλℓ=∑u∈𝒱expAλℓ\(u\)\.\\begin\{gathered\}\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)=\\frac\{\\exp A\_\{\\lambda\_\{\\ell\}\}\(v\)\}\{Z\_\{\\lambda\_\{\\ell\}\}\},\\\\ A\_\{\\lambda\_\{\\ell\}\}\(v\)=\\textstyle\\sum\_\{j=1\}^\{m\}\\lambda\_\{\\ell,j\}\\,a\_\{j\}\(v\),\\quad Z\_\{\\lambda\_\{\\ell\}\}=\\textstyle\\sum\_\{u\\in\\mathcal\{V\}\}\\exp A\_\{\\lambda\_\{\\ell\}\}\(u\)\.\\end\{gathered\}\(S3\)We assume every expert assigns positive probability to every token, so thatai​\(v\)a\_\{i\}\(v\)is finite for alliiandvv\. This is the only regularity condition required for the local derivative arguments below, and it holds for the softmax parameterization of the experts we use\.

We call the corruption channel*memoryless*when it acts on each position independently, so that it factorizes aspc​\(y∣x\)=∏ℓ=1Lpc​\(yℓ∣xℓ\)p\_\{\\mathrm\{c\}\}\(y\\mid x\)=\\prod\_\{\\ell=1\}^\{L\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid x\_\{\\ell\}\)\. Both channels of Section[3](https://arxiv.org/html/2609.05801#S3)are memoryless: the mask channel replaces each token by\[𝙼𝙰𝚂𝙺\]\\mathtt\{\[MASK\]\}independently with probabilityrr, and the replacement channel replaces each token by an independent uniform draw from𝒱\\mathcal\{V\}\.

For position\-factorized experts the composed prior factorizes in the same way,Qλ​\(x\)=∏ℓπλℓ,ℓ​\(xℓ\)Q\_\{\\lambda\}\(x\)=\\prod\_\{\\ell\}\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(x\_\{\\ell\}\)\. Substituting both factorizations into the marginal likelihood and exchanging the sum over sequences for a product of per\-position sums,

pλ​\(y\)\\displaystyle p\_\{\\lambda\}\(y\)=∑x∈𝒱L∏ℓ=1Lpc​\(yℓ∣xℓ\)​πλℓ,ℓ​\(xℓ\)\\displaystyle=\\sum\_\{x\\in\\mathcal\{V\}^\{L\}\}\\prod\_\{\\ell=1\}^\{L\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid x\_\{\\ell\}\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(x\_\{\\ell\}\)=∏ℓ=1L∑v∈𝒱pc​\(yℓ∣v\)​πλℓ,ℓ​\(v\),\\displaystyle=\\prod\_\{\\ell=1\}^\{L\}\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\),where the second equality is the distributive law: a sum over all sequences of a product of per\-position factors equals the product of the per\-position sums\. Taking logarithms,logpλ\(y\)=∑ℓlog∑v∈𝒱pc\(yℓ∣v\)πλℓ,ℓ\(v\)\\log p\_\{\\lambda\}\(y\)=\\sum\_\{\\ell\}\\log\\sum\_\{v\\in\\mathcal\{V\}\}p\_\{\\mathrm\{c\}\}\(y\_\{\\ell\}\\mid v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)is a sum of per\-position terms, theℓ\\ell\-th of which depends onλ\\lambdaonly throughλℓ\\lambda\_\{\\ell\}\. Averaging overNNshared observationsy\(1\),…,y\(N\)y^\{\(1\)\},\\dots,y^\{\(N\)\}and writingcn,ℓ​\(v\)=pc​\(yℓ\(n\)∣xℓ=v\)c\_\{n,\\ell\}\(v\)=p\_\{\\mathrm\{c\}\}\(y^\{\(n\)\}\_\{\\ell\}\\mid x\_\{\\ell\}=v\)recovers the per\-position objectiveLℓ​\(λℓ\)L\_\{\\ell\}\(\\lambda\_\{\\ell\}\)of \([4](https://arxiv.org/html/2609.05801#S3.E4)\), whose maximization decouples across positions\.

Its gradient can therefore be read at a single position\. Fixℓ\\elland, for observationnn, define the channel posterior over the clean token as

γn,ℓ​\(v\)=cn,ℓ​\(v\)​πλℓ,ℓ​\(v\)∑ucn,ℓ​\(u\)​πλℓ,ℓ​\(u\)\.\\gamma\_\{n,\\ell\}\(v\)=\\frac\{c\_\{n,\\ell\}\(v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\}\{\\sum\_\{u\}c\_\{n,\\ell\}\(u\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(u\)\}\.\(S4\)
###### Proposition 2\(Local evidence gradient\)\.

Forλℓ\\lambda\_\{\\ell\}in the relative interior ofΔm\\Delta\_\{m\}, the objective \([4](https://arxiv.org/html/2609.05801#S3.E4)\) is differentiable and

∂Lℓ​\(λℓ\)∂λℓ,i=\\displaystyle\\frac\{\\partial L\_\{\\ell\}\(\\lambda\_\{\\ell\}\)\}\{\\partial\\lambda\_\{\\ell,i\}\}=\{\}1N​∑n=1N𝔼v∼γn,ℓ​\[ai​\(v\)\]\\displaystyle\\tfrac\{1\}\{N\}\\textstyle\\sum\_\{n=1\}^\{N\}\\mathbb\{E\}\_\{v\\sim\\gamma\_\{n,\\ell\}\}\\big\[a\_\{i\}\(v\)\\big\]\(S5\)−𝔼v∼πλℓ,ℓ​\[ai​\(v\)\]\.\\displaystyle\-\\mathbb\{E\}\_\{v\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\\big\[a\_\{i\}\(v\)\\big\]\.

We first isolate the derivative of the composed prior, which is where the normalizer of \([S3](https://arxiv.org/html/2609.05801#A1.E3)\) enters and which is the source of the prior term in the final expression\.

###### Lemma 2\(Score of the log\-linear pool\)\.

For everyλℓ\\lambda\_\{\\ell\}in the relative interior ofΔm\\Delta\_\{m\}and everyv∈𝒱v\\in\\mathcal\{V\},

∂∂λℓ,i​log⁡πλℓ,ℓ​\(v\)\\displaystyle\\frac\{\\partial\}\{\\partial\\lambda\_\{\\ell,i\}\}\\log\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\(S6\)=ai​\(v\)−𝔼u∼πλℓ,ℓ​\[ai​\(u\)\]\.\\displaystyle=a\_\{i\}\(v\)\-\\mathbb\{E\}\_\{u\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\\big\[a\_\{i\}\(u\)\\big\]\.

###### Proof\.

By \([S3](https://arxiv.org/html/2609.05801#A1.E3)\),log⁡πλℓ,ℓ​\(v\)=Aλℓ​\(v\)−log⁡Zλℓ\\log\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)=A\_\{\\lambda\_\{\\ell\}\}\(v\)\-\\log Z\_\{\\lambda\_\{\\ell\}\}\. The first term is linear inλℓ\\lambda\_\{\\ell\}with∂Aλℓ​\(v\)/∂λℓ,i=ai​\(v\)\\partial A\_\{\\lambda\_\{\\ell\}\}\(v\)/\\partial\\lambda\_\{\\ell,i\}=a\_\{i\}\(v\)\. For the second, the sum definingZλℓZ\_\{\\lambda\_\{\\ell\}\}is finite and each summand is a positive, smooth function ofλℓ\\lambda\_\{\\ell\}, so we may differentiate term by term:

∂∂λℓ,i​log⁡Zλℓ\\displaystyle\\frac\{\\partial\}\{\\partial\\lambda\_\{\\ell,i\}\}\\log Z\_\{\\lambda\_\{\\ell\}\}=1Zλℓ​∑u∈𝒱exp⁡\(Aλℓ​\(u\)\)​ai​\(u\)\\displaystyle=\\frac\{1\}\{Z\_\{\\lambda\_\{\\ell\}\}\}\\textstyle\\sum\_\{u\\in\\mathcal\{V\}\}\\exp\\big\(A\_\{\\lambda\_\{\\ell\}\}\(u\)\\big\)a\_\{i\}\(u\)=∑u∈𝒱πλℓ,ℓ​\(u\)​ai​\(u\)=𝔼u∼πλℓ,ℓ​\[ai​\(u\)\]\.\\displaystyle=\\textstyle\\sum\_\{u\\in\\mathcal\{V\}\}\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(u\)\\,a\_\{i\}\(u\)=\\mathbb\{E\}\_\{u\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\\big\[a\_\{i\}\(u\)\\big\]\.Subtracting gives \([S6](https://arxiv.org/html/2609.05801#A1.E6)\)\. ∎

Equation \([S6](https://arxiv.org/html/2609.05801#A1.E6)\) is the standard exponential\-family identity: the derivative of the log\-normalizer is the mean sufficient statistic under the current pool\. The score is invariant to expert\-specific additive constants inaia\_\{i\}; this invariance does not remove scale differences or state\-dependent surrogate errors\. It also has zero mean under the current pool,𝔼v∼πλℓ,ℓ​\[∂λℓ,ilog⁡πλℓ,ℓ​\(v\)\]=0\\mathbb\{E\}\_\{v\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\[\\partial\_\{\\lambda\_\{\\ell,i\}\}\\log\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\]=0, which is the standard score\-function identity\.

###### Proof of Proposition[2](https://arxiv.org/html/2609.05801#Thmproposition2)\.

Fix an observation indexnnand write

En,ℓ​\(λℓ\)=∑v∈𝒱cn,ℓ​\(v\)​πλℓ,ℓ​\(v\),𝒥n,ℓ=log⁡En,ℓ,E\_\{n,\\ell\}\(\\lambda\_\{\\ell\}\)=\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}c\_\{n,\\ell\}\(v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\),\\quad\\mathcal\{J\}\_\{n,\\ell\}=\\log E\_\{n,\\ell\},so thatLℓ​\(λℓ\)=N−1​∑n=1N𝒥n,ℓ​\(λℓ\)L\_\{\\ell\}\(\\lambda\_\{\\ell\}\)=N^\{\-1\}\\sum\_\{n=1\}^\{N\}\\mathcal\{J\}\_\{n,\\ell\}\(\\lambda\_\{\\ell\}\)by \([4](https://arxiv.org/html/2609.05801#S3.E4)\)\. Since𝒱\\mathcal\{V\}is finite and everyπλℓ,ℓ​\(v\)\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)is positive and smooth in the relative interior of the simplex,En,ℓE\_\{n,\\ell\}is a finite sum of smooth functions\. Althoughcn,ℓc\_\{n,\\ell\}need not have a positive uniform lower bound, at least one token satisfiescn,ℓ​\(v\)\>0c\_\{n,\\ell\}\(v\)\>0\. Full support givesπλℓ,ℓ​\(v\)\>0\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\>0for that token, soEn,ℓ\>0E\_\{n,\\ell\}\>0andlog⁡En,ℓ\\log E\_\{n,\\ell\}is differentiable\.

Using∂π=π​∂log⁡π\\partial\\pi=\\pi\\,\\partial\\log\\pi,

∂𝒥n,ℓ∂λℓ,i\\displaystyle\\frac\{\\partial\\mathcal\{J\}\_\{n,\\ell\}\}\{\\partial\\lambda\_\{\\ell,i\}\}=1En,ℓ​\(λℓ\)​∑v∈𝒱cn,ℓ​\(v\)​∂λℓ,iπλℓ,ℓ​\(v\)\\displaystyle=\\frac\{1\}\{E\_\{n,\\ell\}\(\\lambda\_\{\\ell\}\)\}\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}c\_\{n,\\ell\}\(v\)\\,\\partial\_\{\\lambda\_\{\\ell,i\}\}\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)=∑v∈𝒱cn,ℓ​\(v\)​πλℓ,ℓ​\(v\)En,ℓ​\(λℓ\)⏟=γn,ℓ​\(v\)\\displaystyle=\\textstyle\\sum\_\{v\\in\\mathcal\{V\}\}\\underbrace\{\\frac\{c\_\{n,\\ell\}\(v\)\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\}\{E\_\{n,\\ell\}\(\\lambda\_\{\\ell\}\)\}\}\_\{=\\;\\gamma\_\{n,\\ell\}\(v\)\}×∂λℓ,ilog⁡πλℓ,ℓ​\(v\),\\displaystyle\\times\\,\\partial\_\{\\lambda\_\{\\ell,i\}\}\\log\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\),whereγn,ℓ\\gamma\_\{n,\\ell\}is precisely the channel posterior \([S4](https://arxiv.org/html/2609.05801#A1.E4)\)\. The bracketed factor is a probability distribution on𝒱\\mathcal\{V\}by construction, so the sum is an expectation underγn,ℓ\\gamma\_\{n,\\ell\}\. Substituting Lemma[2](https://arxiv.org/html/2609.05801#Thmlemma2),

∂𝒥n,ℓ∂λℓ,i\\displaystyle\\frac\{\\partial\\mathcal\{J\}\_\{n,\\ell\}\}\{\\partial\\lambda\_\{\\ell,i\}\}=𝔼v∼γn,ℓ​\[ai​\(v\)−𝔼u∼πλℓ,ℓ​\[ai​\(u\)\]\]\\displaystyle=\\mathbb\{E\}\_\{v\\sim\\gamma\_\{n,\\ell\}\}\\Big\[a\_\{i\}\(v\)\-\\mathbb\{E\}\_\{u\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\[a\_\{i\}\(u\)\]\\Big\]=𝔼v∼γn,ℓ​\[ai​\(v\)\]−𝔼u∼πλℓ,ℓ​\[ai​\(u\)\],\\displaystyle=\\mathbb\{E\}\_\{v\\sim\\gamma\_\{n,\\ell\}\}\\big\[a\_\{i\}\(v\)\\big\]\-\\mathbb\{E\}\_\{u\\sim\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\}\\big\[a\_\{i\}\(u\)\\big\],the last step because the inner expectation does not depend onvv\. Averaging overn=1,…,Nn=1,\\dots,Ngives \([S5](https://arxiv.org/html/2609.05801#A1.E5)\)\. ∎

###### Proposition 3\(Degeneracy of the factorized objective under masking\)\.

Suppose positionℓ\\ellis masked in observationnn, so that the channel likelihood is fixed,cn,ℓ​\(v\)=rc\_\{n,\\ell\}\(v\)=rfor allv∈𝒱v\\in\\mathcal\{V\}\. Then observationnncontributes nothing to the gradient \([S5](https://arxiv.org/html/2609.05801#A1.E5)\) at positionℓ\\ell\.

###### Proof\.

Ifcn,ℓ≡rc\_\{n,\\ell\}\\equiv rthenEn,ℓ​\(λℓ\)=r​∑vπλℓ,ℓ​\(v\)=rE\_\{n,\\ell\}\(\\lambda\_\{\\ell\}\)=r\\sum\_\{v\}\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)=r, which is constant inλℓ\\lambda\_\{\\ell\}, and the channel posterior reduces to the prior,γn,ℓ​\(v\)=r​πλℓ,ℓ​\(v\)/r=πλℓ,ℓ​\(v\)\\gamma\_\{n,\\ell\}\(v\)=r\\,\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)/r=\\pi\_\{\\lambda\_\{\\ell\},\\ell\}\(v\)\. The two expectations in \([S5](https://arxiv.org/html/2609.05801#A1.E5)\) are then taken under the same distribution and cancel\. ∎

This formalizes the masking difficulty noted in Section[4](https://arxiv.org/html/2609.05801#S4): in the factorized model, a masked observation contributes no local information about its position\-specific weight\. Another observation in which the position is visible may contribute information\. For contextual experts,λℓ\\lambda\_\{\\ell\}can affect the marginal distribution of visible tokens through cross\-position dependence, so the sequence\-level evidence can remain informative\. The practical estimator also scores imputed posterior particles and smooths weights across positions\.

### A\.3Contextual2×22\\times 2validation

To examine the energy and sampling approximations separately, we use a contextual model small enough to enumerate\. It is a normalized bigram\-energy prior with two experts whose local energies depend on bothxℓ−1x\_\{\\ell\-1\}andxℓx\_\{\\ell\}; unlike a factorized model, it therefore exposes a genuine difference between the surrogate and exact energies\. The short sequence and small vocabulary allow us to computeQλQ\_\{\\lambda\}, its posterior, and its gradient exactly, as well as a mean\-field law that samples positions independently from the composed local conditionals\. Table[S2](https://arxiv.org/html/2609.05801#A1.T2)crosses exact or surrogate energies with exact or mean\-field expectations\.

The exact sequence\-level gradient agrees with a finite\-difference gradient at cosine1\.0001\.000, a numerical implementation check for Proposition[1](https://arxiv.org/html/2609.05801#Thmproposition1)\(Table[S2](https://arxiv.org/html/2609.05801#A1.T2)\)\. Replacing the exact energies with the denoising surrogate, or the exact expectations with mean\-field sampling, perturbs the gradient direction, the energy surrogate more \(cosine0\.650\.65\) than the sampler \(0\.920\.92\); the deployed combination of both reaches cosine0\.620\.62\. This enumerable model isolates the two approximations but does not by itself establish tolerance to them at scale\. The reported spreads are standard deviations across the2020seeds; the two surrogate\-energy configurations vary widely, so their means should be read as averages over seeds whose alignment ranges from weak to near exact\.

#### Field recovery in the contextual model\.

The comparison above is taken at the uniform initialization and measures agreement of the gradient, not whether the objective identifies the field\. We therefore run the recovery update to convergence in the same enumerable model, with the exact gradient, varying the number of corrupted observationsNNthat share one field\. The true field routes the first half of the positions to one expert and the second half to the other, softened off the vertices, so a uniform field sits at mean absolute error0\.4550\.455\. Recovery improves monotonically withNNand closes about three quarters of that distance \(Table[S1](https://arxiv.org/html/2609.05801#A1.T1)\)\. A single observation of length four carries too little information to separate the four configurations of Table[S2](https://arxiv.org/html/2609.05801#A1.T2), which is why we report gradient agreement rather than field error there\. This is the contextual counterpart of the factorized recovery result of Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1)and of its dependence on the number of observations\.

Table S1:Field recovery in the enumerable contextual modelTable S2:Enumerable diagnostic on a contextual bigram\-energy model \(4 tokens, length 4, 256 sequences, 20 seeds\): an implementation check, not evidence that the large\-scale approximation is controlled\. The mean\-field law differs from the exact prior and posterior at total variation0\.380\.38and0\.340\.34\.

## Appendix BExperimental details

This section contains the data construction, expert configurations, optimization settings, decoders, and statistical procedures used in the main experiments\. The implementation, the evaluation\-set construction scripts and the analysis code are at[https://github\.com/panahazari/evidence\-aligned\-local\-composition](https://github.com/panahazari/evidence-aligned-local-composition)\. No corpus is redistributed there; the evaluation sets are rebuilt from the public sources listed below\.

### B\.1Data and corruption

#### Exact simulator \(Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1)\)\.

Vocabulary\|𝒱\|=12\|\\mathcal\{V\}\|=12, lengthL=48L=48,m=2m=2experts\. Each expert is a position\-factorized categorical model with per\-position logits drawn so that the two experts are separated by a logit\-gap parameter, whose default value2\.52\.5corresponds to a measured Kullback\-Leibler divergence of4\.564\.56nats between the experts\. The separation parameter is not itself a divergence; figures report the measured KL\. The ground\-truth field routes positions00to1515to the first expert, positions1616to3131to the second, and fuses both on positions3232to4747\. Clean sequences are drawn from the composed prior and corrupted by the replacement channel at rate0\.40\.4\. We use30003000corrupted sequences for recovery\.

#### Byte\-level experts \(Section[5\.2](https://arxiv.org/html/2609.05801#S5.SS2)\)\.

We use three corpora, each truncated to55M characters: prose from WikiText\-103 \(raw\)\[[23](https://arxiv.org/html/2609.05801#bib.bib34)\], Python code from CodeSearchNet\[[15](https://arxiv.org/html/2609.05801#bib.bib35)\]with comments and docstrings removed by a tokenizer\-based filter, and Kubernetes YAML\[[33](https://arxiv.org/html/2609.05801#bib.bib36)\]with full\-line comments removed\. These filters reduce trivial overlap between domains\. Evaluation windows concatenate segments from the evaluation split of each corpus without artificial fences or boundary markers, so region labels are known by construction\. The three\-expert study uses256256windows ofL=256L=256bytes, each containing at least two regimes with a minimum length for each\. The two\-expert study uses128128prose and code windows\.

#### Pretrained experts \(Section[5\.3](https://arxiv.org/html/2609.05801#S5.SS3)\)\.

For the1\.31\.3B study, evaluation windows ofL=512L=512tokens are built the same way from WikiText\-103 prose\[[23](https://arxiv.org/html/2609.05801#bib.bib34)\]and CodeParrot Python code\[[4](https://arxiv.org/html/2609.05801#bib.bib37)\]held apart from the fine\-tuning data, giving256256windows\. For the77B study we use6464windows in the shared tokenizer of the two models\.

#### Real documents \(Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5)\)\.

The natural evaluation contains256256windows from7575developer\-written README documents drawn from the most\-starred GitHub repositories\[[28](https://arxiv.org/html/2609.05801#bib.bib39)\]\. We tokenize each document as a whole and retain its original prose and code regions and boundaries\. Because a source document may contribute several windows, every statistic in this study resamples documents rather than individual windows\.

#### Scientific documents \(Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5)\)\.

The distinct\-expert natural setting uses arXivLaTeXsources \(the proof\-pile corpus,[13](https://arxiv.org/html/2609.05801#bib.bib38)\)\. We split each document into a prose stream and a mathematics stream by its math environments \(inline $…$, display $$…$$ and\\\[…\\\], andequation/align\-type environments\) and fine\-tune one expert on each stream, holding documents out for evaluation\. Each held\-out document is tokenized whole with the GPT\-2 tokenizer, and each token is labeled prose or mathematics by whether its characters fall inside a math environment; a token straddling a boundary takes the majority label and is excluded from region\-conditional metrics\. We keep windows in which both regions clear a0\.250\.25share, giving256256windows from7171documents at mask rate0\.30\.3, with a0\.500\.50mathematics share and a token\-level region\-switch rate about seven times higher than the README documents\.

#### Channels\.

The mask channel replaces each token independently with probabilityrrby\[𝙼𝙰𝚂𝙺\]\\mathtt\{\[MASK\]\}\. The default isr=0\.3r=0\.3for the pretrained studies andr=0\.2r=0\.2for the byte studies, and Appendix[C\.1](https://arxiv.org/html/2609.05801#A3.SS1)sweeps it\. The replacement channel replaces each token independently with probabilityrrby a uniform draw from the vocabulary; we use it in the exact simulator, where the posterior decoder is exact\.

### B\.2Experts

#### Byte\-level experts\.

Each expert is a discrete flow\-matching model over the byte vocabulary,\|𝒱\|=256\|\\mathcal\{V\}\|=256plus a mask symbol, so257257classes\. All experts in a composition here share an identical architecture, which is convenient but not required \(see the expert interface in Section[3](https://arxiv.org/html/2609.05801#S3)\); what the logit composition needs is a shared vocabulary and output interpretation, not a shared network\. The architecture is a transformer with88layers, model width384384,1212attention heads, dropout0\.10\.1, and contextL=256L=256, about1414M parameters\. Training uses the mask\-source path with a polynomial schedule \(κ\\kappaexponent1\.01\.0,tϵ=10−4t\_\{\\epsilon\}=10^\{\-4\}\), batch size4848,20,00020\{,\}000steps, AdamW with learning rate3×10−43\\times 10^\{\-4\}, weight decay0\.010\.01, and400400warmup steps\. Each expert sees only its own domain\. Final training cross\-entropies are2\.402\.40\(prose\),1\.971\.97\(code\) and1\.831\.83\(configuration\) nats per byte\.

#### Fine\-tuned1\.31\.3B experts\.

Both experts are fine\-tuned from the same1\.31\.3B discrete flow\-matching model with a GPT\-2 vocabulary \(50,25750\{,\}257tokens\)\. We use LoRA with rank1616andα=32\\alpha=32, learning rate10−410^\{\-4\}, batch size1616,60006000steps, mixed precision, and gradient checkpointing on12,00012\{,\}000blocks of512512tokens per domain\. Sharing the base model and fine\-tuning recipe reduces architectural confounding and makes the domain data the intended source of specialization\. Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5)also uses a pair fine\-tuned on matched corpora and the unmodified base model as a generalist\.

#### arXiv prose and mathematics experts\.

The scientific\-document experts of Section[5\.5](https://arxiv.org/html/2609.05801#S5.SS5)are fine\-tuned from the same1\.31\.3B model with the identical LoRA recipe \(60006000steps\), one on the extracted prose stream and one on the mathematics stream\. On the arXiv evaluation regions they pass the specialization check with balanced margins of0\.470\.47\(prose\) and0\.460\.46\(mathematics\) nats, the strongest two\-sided separation among the natural\-document pairs\. The README\-matched pair, by contrast, separates only weakly on its evaluation text, which is why local composition beats a global weight in the scientific setting and ties in the README setting\.

#### Off\-the\-shelf77B experts\.

We useDream\-org/Dream\-v0\-Base\-7Bas the general expert andDream\-org/Dream\-Coder\-v0\-Base\-7Bas the code expert at mask rate0\.60\.6\. These public masked\-diffusion language models share a tokenizer and compatible source and output conventions\. Strictly speaking, they are trained as masked diffusion models rather than under the discrete flow\-matching parameterization used elsewhere in the paper\. The distinction does not affect the method\. Both frameworks expose the same two operations required by the expert interface of Section[3](https://arxiv.org/html/2609.05801#S3): clean\-token logits given a partially masked context and a denoising loss on a candidate sequence\. Moreover, the denoising\-loss identity of Section[4](https://arxiv.org/html/2609.05801#S4)is stated for the mask\-source path and its conclusion does not depend on the schedule, so it covers mask\-source masked diffusion and mask\-source discrete flow matching alike \(Appendix[C\.9](https://arxiv.org/html/2609.05801#A3.SS9)\)\. We therefore refer to all experts as discrete flow\-matching models in the main text and note the training\-objective difference only here\. We do not train or adapt them, and each runs on a separate GPU\. The pair fails the specialization diagnostic in Appendix[B\.3](https://arxiv.org/html/2609.05801#A2.SS3)\. The composed fill is deterministic, so the eight nominal particles are identical and the estimator is a plug\-in rather than a particle average\.

### B\.3The specialization check

The inferred field depends on per\-expert scale, so we use a specialization diagnostic before composition\. We evaluate every expert’s denoising energy on every domain and accept the diagnostic when each expert assigns its lowest energy to its own domain\. For the three byte\-level experts of Section[5\.2](https://arxiv.org/html/2609.05801#S5.SS2), the energies in nats per byte are

with diagonal margins of1\.031\.03,0\.400\.40and0\.510\.51nats\. The byte\-level experts and both1\.31\.3B pairs pass this check\.

#### The77B pair fails the diagnostic\.

For the two off\-the\-shelf models, the measured energies are0\.8880\.888nats per token for Dream on prose and0\.5140\.514on code, compared with1\.0781\.078for Dream\-Coder on prose and0\.3370\.337on code\. Dream\-Coder assigns lower energy to code by0\.740\.74nats, but Dream also assigns lower energy to code by0\.370\.37nats\. The matrix is therefore not diagonally dominant and the diagnostic fails\.

Column\-wise domain discrimination still holds: Dream has lower energy than Dream\-Coder on prose, while Dream\-Coder has lower energy than Dream on code\. Row\-wise own\-domain preference fails because Dream assigns lower energy to code than to prose\. The field nevertheless reaches field accuracy0\.950\.95\. We therefore report the experiment as a stress test outside the diagnostic’s accepted regime\.

Appendix[C\.2](https://arxiv.org/html/2609.05801#A3.SS2)shows the complementary case: a byte\-level pair that fails to specialize also fails to produce a usable field, which is what the check is for\.

### B\.4Composition hyperparameters

Hyperparameters differ across experiments\. Table[S3](https://arxiv.org/html/2609.05801#A2.T3)reports the settings used in each run\. The byte\-level studies useR=30R=30,4848particles,K=4K=4,η=1\\eta=1, andτ=0\.6\\tau=0\.6\. The1\.31\.3B studies useR=12R=12,2424particles,K=2K=2,η=0\.5\\eta=0\.5, andτ=0\.4\\tau=0\.4under the proximal map, orτ=0\.3\\tau=0\.3under the moving average on naturally mixed documents; the77B study usesR=6R=6, eight nominal particles, andτ=0\.2\\tau=0\.2\. The field is initialized uniformly on the simplex and the seed is00\.

Only two settings were searched\. The smoothing strengthτ\\tauwas swept over\{0,0\.2,0\.6\}\\\{0,0\.2,0\.6\\\}at byte scale, over\{0,0\.1,0\.2,0\.4,0\.6,1,2\}\\\{0,0\.1,0\.2,0\.4,0\.6,1,2\\\}at1\.31\.3B and\{0,0\.2,0\.6,1\}\\\{0,0\.2,0\.6,1\\\}at77B under the proximal map \(Appendix[C\.5](https://arxiv.org/html/2609.05801#A3.SS5)\), and the corruption rate over\{0\.1,0\.2,0\.3\}\\\{0\.1,0\.2,0\.3\\\}at byte scale and\{0\.3,0\.5,0\.7\}\\\{0\.3,0\.5,0\.7\\\}at1\.31\.3B \(Appendix[C\.1](https://arxiv.org/html/2609.05801#A3.SS1)\); the rate grid is reported as an ablation rather than used to pick a winner\. The deployedτ\\tauat byte scale was chosen once from the byte sweep, which was run on the same windows we report\. Read literally, that means the byteτ\\tauwas selected on the reported evaluation data, with no calibration subset kept apart, and the byte numbers should be read with that in mind\. The1\.31\.3B and77B protocol below does not have this problem\. At1\.31\.3B and77B it was chosen by a rule fixed before the sweep was run, namely the highest restoration accuracy on a small calibration set held apart from the reported windows, with field accuracy as the tie\-break\. At1\.31\.3B this selectsτ=0\.4\\tau=0\.4from a plateau that is flat to0\.00030\.0003between0\.20\.2and0\.60\.6\. At77B the rule as written selectsτ=0\\tau=0, by0\.00070\.0007over1616windows, but that setting leaves field accuracy at0\.7450\.745against0\.9670\.967atτ=0\.2\\tau=0\.2; we treat the restoration difference as a tie and deployτ=0\.2\\tau=0\.2, which is a deviation from the rule and is the only such deviation in the paper\. The remaining settings \(RR,PP,KK,η\\eta, sampler steps\) were fixed in advance by the compute budget and were not tuned: one value was used per scale throughout, as listed in Table[S3](https://arxiv.org/html/2609.05801#A2.T3)\. No held\-out selection was performed for any hyperparameter\.

The smoothing operator is experiment\-specific\. The reference total\-variation operator is the simplex\-constrained proximal map

λ\(s\+1\)=arg​minλ1:L∈ΔmL\{\\displaystyle\\lambda^\{\(s\+1\)\}=\\operatorname\*\{arg\\,min\}\_\{\\lambda\_\{1:L\}\\in\\Delta\_\{m\}^\{L\}\}\\Big\\\{12​∑ℓ‖λℓ−λ~ℓ\(s\+1\)‖22\\displaystyle\\tfrac\{1\}\{2\}\\textstyle\\sum\_\{\\ell\}\\\|\\lambda\_\{\\ell\}\-\\tilde\{\\lambda\}^\{\(s\+1\)\}\_\{\\ell\}\\\|\_\{2\}^\{2\}\(S7\)\+τ∑ℓ≥2∥λℓ−λℓ−1∥1\}\.\\displaystyle\+\\tau\\textstyle\\sum\_\{\\ell\\geq 2\}\\\|\\lambda\_\{\\ell\}\-\\lambda\_\{\\ell\-1\}\\\|\_\{1\}\\Big\\\}\.For the two\-expert byte\-level study, we apply the exact one\-dimensional fused\-lasso map to the first weight, clip it to\[0,1\]\[0,1\], and set the second weight to its complement; this implements \([S7](https://arxiv.org/html/2609.05801#A2.E7)\)\. For the three\-expert study, we apply the same map independently to each weight column, clip negative values, and renormalize each row to the simplex\. The constructed1\.31\.3B and77B studies use the same two\-expert fused\-lasso map as the byte\-level study, so \([S7](https://arxiv.org/html/2609.05801#A2.E7)\) is the operator behind every constructed result in the paper\. For the studies on naturally mixed documents, we instead smooth each weight column with a moving average, using a centered rectangular kernel of width five whose five taps are all fixed to1/51/5, with zero padding at the two ends\. The new field is\(1−τ\)\(1\-\\tau\)times the current field plusτ\\tautimes its moving average, and each row is then renormalized to the simplex\. Thus,τ\\tauis a total\-variation penalty in the constructed studies and the weight given to the moving average in the naturally mixed document studies\. The two operators do not share one proximal objective, and we make no common convergence claim; within each study the same operator is applied to the evidence field and to every baseline field it is compared against\.

Table S3:Settings used by each experiment; they are*not*shared across scales\.RR: recovery iterations;PP: particles;KK: time samples;η\\eta: step size;τ\\tau: smoothing strength\.†The77B composed fill is deterministic, so its88particles are identical; that estimator is a plug\-in, not a particle average\. The first four rows use the proximal map of \([S7](https://arxiv.org/html/2609.05801#A2.E7)\), whereτ\\tauis a penalty; the two natural\-document rows use the moving\-average blend, whereτ\\tauis a mixing weight\. The two are not comparable\.

### B\.5Decoders

The default decoder is the one\-step reconstruction \([12](https://arxiv.org/html/2609.05801#S4.E12)\) attdec=0\.9t\_\{\\mathrm\{dec\}\}=0\.9: visible tokens are copied and each masked token is filled independently by the argmax of the composed logits\. The iterative decoder revisits positions over several denoising times, using1212steps in the byte studies and66rounds at1\.31\.3B\. All methods use the same decoder and matched source initialization\. This controls decoding variation but does not make the decoder cancel, because decoding is nonlinear in the composition field\. Appendix[C\.4](https://arxiv.org/html/2609.05801#A3.SS4)shows that the method ordering is unchanged\.

### B\.6Protocol and statistics

Every method uses the same experts, corrupted input, channel, decoder, and matched source initialization; only the composition weights differ\. Comparisons are paired over the declared statistical unit\. Byte\-level and natural\-document analyses aggregate and resample by source document\. The1\.31\.3B analysis uses constructed windows and takes the window as the statistical unit, as noted in Section[5\.3](https://arxiv.org/html/2609.05801#S5.SS3)\. We report the mean paired difference from local evidence, its95%95\\%bootstrap confidence interval, a two\-sided sign\-flip permutationpp\-value, and Cohen’sdzd\_\{z\}, each with10,00010\{,\}000resamples and a fixed seed\. Restoration accuracy and field accuracy are as defined in Section[5](https://arxiv.org/html/2609.05801#S5)\.

### B\.7Compute

All experiments run on NVIDIA A100 40GB GPUs\. Training one byte\-level expert takes under an hour on a single GPU\. Fine\-tuning one1\.31\.3B expert takes a few hours on a single GPU\. Composition is the dominant cost at inference: with the settings above, the three\-expert byte\-level run takes about6565seconds per document, dominated by the particle draws and the per\-expert energy evaluations, and the full256256\-window run takes about4\.64\.6GPU\-hours\. The77B study places one expert per GPU\. Costs scale linearly in the number of experts, since each expert is scored independently at each iteration\.

## Appendix CAdditional results

The following experiments test sensitivity to corruption, expert training, decoder choice, and domain granularity, and provide more detailed measurements of the recovered field\.

### C\.1Corruption rate and seeds

Table[S4](https://arxiv.org/html/2609.05801#A3.T4)reports a corruption\-rate sweep\. The byte\-level study averages three corruption seeds; the spread is below0\.0070\.007accuracy for every method\. Its method ordering is unchanged across rates, while the advantage over equal weights narrows from\+0\.101\+0\.101atr=0\.1r=0\.1to\+0\.078\+0\.078atr=0\.3r=0\.3\. The1\.31\.3B sweep also preserves the ordering throughr=0\.7r=0\.7, with a margin near0\.050\.05over equal weights\. However, the1\.31\.3B rates use different evaluation subsets and sample sizes, so this sweep provides descriptive evidence of stable ordering rather than a controlled estimate of the effect of corruption rate\.

Figure S1:Recoverability of the field in the categorical simulator: error falls with more observations and greater expert separation, and rises with corruption\.Table S4:Corruption\-rate sweep\. Byte numbers average three corruption seeds \(128128windows each, two experts\);1\.31\.3B uses256256windows atr=0\.3r=0\.3and128128at the higher rates\.
### C\.2Independently trained expert pairs

We retrained the byte\-level expert pair twice with different seeds and repeated the full evaluation \(Table[S5](https://arxiv.org/html/2609.05801#A3.T5)\)\. The first pair reproduces the main result\. For the second, field accuracy is0\.510\.51, near chance for two experts, and local evidence \(0\.4490\.449\) is close to the best single expert \(0\.4470\.447\)\. This failed composition coincides with a failed specialization diagnostic\. The result supports treating expert specialization as an empirical prerequisite, although it does not isolate specialization as the sole cause\. Two retrainings are also too few to characterize seed stability, and one of the two failing is better read as a warning than as a rate\.

The second pair also puts the hard label router \(0\.3260\.326\) well below equal weights \(0\.4100\.410\)\. This is what the diagnostic predicts rather than an anomaly: the router commits each position to the expert trained on that region’s domain, so it is only privileged when the label tracks competence\. When the pair does not specialize, the label no longer identifies the better expert at that position, and hard routing discards the averaging that equal weights still enjoys\. The same mechanism makes the router an imperfect reference in general, which is why we describe it as privileged but not an upper bound\.

Table S5:Two independently retrained byte\-level expert pairs\. The second pair does not specialize and loses the advantage over the best single expert\.*Read each row against its own baselines, not across rows\.*Both pairs use the same128128documents, decoder and protocol, but they are different models with different absolute skill: Pair 2’s code expert alone reaches0\.4470\.447where Pair 1’s reaches0\.3390\.339, so Pair 2’s higher local\-evidence number reflects stronger experts, not better composition\. The within\-pair margins are the comparable quantity\.
### C\.3A second real domain

We also evaluate a finer\-grained domain pair: Python code versus its comments and docstrings\. These regimes interleave at the line level within a file rather than appearing as long blocks\. Across128128windows, local evidence reaches0\.2270\.227\. We detect no difference from the hard label router \(0\.2270\.227\), while local evidence exceeds the code expert \(0\.2220\.222\), the global weight \(0\.2120\.212\), and equal weights \(0\.1880\.188\)\. Absolute accuracy and margins are lower than in the main study, consistent with the greater vocabulary overlap between code and comments\.

### C\.4Decoder choice

Table[S6](https://arxiv.org/html/2609.05801#A3.T6)repeats the two main studies with an iterative decoder that revisits positions across several denoising times\. The saved iterative outputs support only window\-weighted aggregation, so this is a secondary descriptive analysis rather than the primary document\-weighted byte analysis\. The method ordering is unchanged, although individual accuracies can increase or decrease; at1\.31\.3B the iterative decoder brings local evidence level with the privileged label router, which the one\-step decoder leaves0\.0020\.002ahead\. All methods use the same decoder and matched source initialization, which controls decoding variation but does not remove decoder effects because decoding is nonlinear in the field\.

Table S6:One\-step versus iterative decoding \(window\-weighted\)\. The method ordering is identical under both decoders; the best single expert is the own\-posterior selector \(the code expert at1\.31\.3B\)\.
### C\.5The smoothing prior

We isolate the role of the smoothing prior \(Table[S7](https://arxiv.org/html/2609.05801#A3.T7), Figure[S2](https://arxiv.org/html/2609.05801#A3.F2)\)\. Its effect differs markedly across the two ablations\. At byte scale, removing smoothing lowers local\-evidence accuracy to0\.3480\.348, below the best single expert at0\.4130\.413; settingτ=0\.2\\tau=0\.2raises accuracy to0\.4300\.430, above the reported single\-expert and global baselines\. On the3232\-window calibration subset of the1\.31\.3B study, by contrast, the unsmoothed field already reaches0\.6590\.659, compared with0\.6360\.636for the field\-average global reference\. Raisingτ\\tauto0\.40\.4lifts restoration accuracy to0\.6790\.679and field accuracy from0\.9120\.912to0\.9930\.993\. The response is a plateau rather than a peak: accuracy varies by0\.00030\.0003acrossτ∈\{0\.2,0\.4,0\.6\}\\tau\\in\\\{0\.2,0\.4,0\.6\\\}and decays slowly outside it, to0\.6760\.676atτ=2\\tau=2, so the deployed value is not a sharp choice\. This contrast with the byte scale is consistent with reduced reliance on smoothing when the expert signal is stronger, but two ablations are not sufficient to establish a general scaling law\.

Table S7:Effect of smoothing strengthτ\\tau\. Byte numbers use this ablation’s decoder \(not comparable with Table[3](https://arxiv.org/html/2609.05801#S5.T3)\); the1\.31\.3B numbers use one3232\-window calibration subset, disjoint from the reported windows\. Both scales use the proximal update \([S7](https://arxiv.org/html/2609.05801#A2.E7)\), soτ\\taumeans the same thing in every column\. Note that the deployed byte setting isτ=0\.6\\tau=0\.6, which this sweep does not cover: the byte columns bracket it at00and0\.20\.2but do not contain it, so the deployed configuration does not appear in its own ablation\.Figure S2:Effect of smoothing\. Without it, byte\-level local evidence falls below the best single expert; at1\.31\.3B smoothing mainly improves field accuracy \(Table[S7](https://arxiv.org/html/2609.05801#A3.T7)\)\.
### C\.6Quality of the recovered field

Restoration accuracy does not measure the quality of the inferred composition, and overall field accuracy can hide boundary errors because region interiors contain many more positions than transitions\. We therefore evaluate the three\-expert byte\-level field against the true regions using several metrics, with clustering by source document as in the main byte\-level analysis \(Table[S8](https://arxiv.org/html/2609.05801#A3.T8)\)\. Macro one\-versus\-rest AUROC is0\.9930\.993\(prose0\.9850\.985, code0\.9960\.996, configuration0\.9960\.996\), and class\-wise argmax recall is0\.9240\.924for prose,0\.9950\.995for code, and0\.9980\.998for configuration\.

The field is highly accurate away from transitions: interior accuracy is0\.9870\.987, macro F1 is0\.9720\.972, IoU is0\.9480\.948, and segment F1 is0\.9790\.979\. Performance is lower near transitions\. Accuracy within four positions of a true change is0\.6870\.687, and one\-to\-one boundary F1 at the same tolerance is0\.7200\.720\. Thus, the field identifies region interiors more reliably than exact transition locations, consistent with the qualitative examples in Figure[2](https://arxiv.org/html/2609.05801#S5.F2)\. This suggests boundary\-aware smoothing or decoding as a useful direction\.

The global and equal\-weight references are constant fields and therefore contain no predicted boundaries\. Their boundary F1 is zero, and their field accuracies \(0\.6690\.669and0\.3340\.334\) reflect the prevalence of the region favored by each constant field\. These rows provide context but cannot test boundary recovery\.

Table S8:Attribution metrics for the recovered field, three byte\-level experts,256256windows from160160documents\. Boundaries are within four positions of a regime change; segment F1 uses same\-label IoU≥0\.5\\geq 0\.5\. The constant references predict no boundaries\.
### C\.7Off\-the\-shelf77B stress test

We include one experiment with models that we did not train, treating it as an out\-of\-assumption stress test rather than evidence for the method\. The pair consists of Dream\-v0\-Base\-7B as the general expert and Dream\-Coder\-v0\-Base\-7B as the code expert, evaluated at mask rate0\.60\.6\(Appendix[B\.2](https://arxiv.org/html/2609.05801#A2.SS2)\)\. Three properties limit the study\. First, the pair fails our specialization diagnostic: the general expert scores code0\.370\.37nats better than prose \(Appendix[B\.3](https://arxiv.org/html/2609.05801#A2.SS3)\), so using the uncalibrated choiceαi=1\\alpha\_\{i\}=1is not well supported\. Second, the composed fill is deterministic, making the eight nominal particles identical and reducing the estimator to a single plug\-in evaluation\. Third, the study contains only6464windows\.

Under these conditions, local evidence reaches0\.6050\.605and is not detectably different from the hard label router \(0\.6060\.606,p=0\.90p=0\.90\)\. It exceeds the general model alone by0\.0240\.024, the best single expert by0\.0110\.011, equal weights by0\.0060\.006, and the field\-average global reference by0\.0060\.006, each atp<0\.01p<0\.01\(Table[S9](https://arxiv.org/html/2609.05801#A3.T9)\)\. The adapted baselines fit the same picture: the DEMix router is indistinguishable from local evidence \(0\.6050\.605,p=0\.83p=0\.83\), while PackLLM\-opt \(0\.5970\.597\), PackLLM\-sim \(0\.5930\.593\), and the marginal\-evidence selector \(0\.5920\.592\) trail slightly, consistent with a pair whose specialization is too weak for any weighting scheme to separate strongly\. The margin over a shuffled field is the thinnest in the paper \(\+0\.004\+0\.004,p=0\.04p=0\.04\), so on this pair the positional arrangement of the field carries little of the gain\. The field agrees with the regions at0\.9540\.954\. These numbers apply only to this model pair and do not establish robustness to arbitrary off\-the\-shelf experts; they show that, for two closely related but imperfectly specialized models, the inferred field produces a small gain without reducing restoration accuracy\.

Table S9:Off\-the\-shelf77B stress test \(Dream\-v0\-Base\-7B, Dream\-Coder\-v0\-Base\-7B\),6464windows, mask rate0\.60\.6\. The pair fails the specialization check, so this is a stress test, not evidence for the method\.∗p<0\.05\{\}^\{\*\}p<0\.05,p∗⁣∗<0\.01\{\}^\{\*\*\}p<0\.01,∗∗∗p<10−3\{\}^\{\*\*\*\}p<10^\{\-3\}; unmarked differences are not significant\.

### C\.8A setting where the composition does not matter

We also evaluate the byte\-level study under the replacement channel\. At corruption rates from0\.10\.1to0\.30\.3, every method, including a single expert, achieves accuracy within0\.0150\.015of1−r1\-r\. The one\-step decoder usually copies the observed token because it provides strong evidence about the clean token, so changing the composition weights rarely changes the mode\. This setting therefore does not test the field\. In contrast, the exact simulator in Section[5\.1](https://arxiv.org/html/2609.05801#S5.SS1)uses an exact posterior decoder under the same channel, allowing the prior composition to affect the output\. Evaluating replacement corruption at byte scale requires a decoder that can overturn the observed token\.

### C\.9The uniform\-source mismatch

The denoising\-loss identity of Section[4](https://arxiv.org/html/2609.05801#S4)is stated for the mask\-source path: the intermediate state is the clean sequence with a random subset of positions replaced by\[𝙼𝙰𝚂𝙺\]\\mathtt\{\[MASK\]\}\. The byte\-level experts and the77B models are of this kind\. The1\.31\.3B experts are not: they are fine\-tuned from a base model whose source distribution is*uniform*over the vocabulary, so at intermediate times a corrupted position holds a random token rather than a mask symbol\.

We do not have an analogue of that identity for the uniform\-source path\. The per\-position denoising score is therefore a heuristic energy rather than an order\-averaged conditional log score, and the resulting source mismatch is uncontrolled\. The1\.31\.3B and natural\-document results are empirical evaluations outside the proposition’s assumptions\. Their high field accuracy shows that the surrogate remains informative in these experiments, but it does not validate the mask\-source identity for a uniform source\. Deriving a uniform\-source analogue or fine\-tuning the experts with a mask\-source path would address this gap\.

### C\.10Independent baseline implementations

We distinguish independently optimized baselines from references derived from the local field\.

#### Global weight\.

The independent global baseline solves the same approximate optimization under the constraintλℓ=λ\\lambda\_\{\\ell\}=\\lambdafor allℓ\\ell\. It initializes uniformly and draws its own prior and posterior particles under its current global field at every iteration\. At byte scale it reaches document\-weighted accuracy0\.6180\.618, compared with0\.6600\.660for local evidence, a paired difference of\+0\.042\+0\.042\(p<10−3p<10^\{\-3\},dz=1\.18d\_\{z\}=1\.18\)\. At1\.31\.3B it reaches0\.6110\.611, compared with0\.6530\.653for local evidence \(\+0\.042\+0\.042,95%95\\%CI\[\+0\.039,\+0\.046\]\[\+0\.039,\+0\.046\],p<10−3p<10^\{\-3\},dz=1\.47d\_\{z\}=1\.47, over the256256shared windows\), and is not detectably different from the field\-average reference at0\.6100\.610\(difference\+0\.0004\+0\.0004,p=0\.72p=0\.72\)\. On the arXiv documents it reaches document\-weighted accuracy0\.6820\.682over256256windows from7171documents, against0\.6910\.691for local evidence \(\+0\.010\+0\.010,p<10−3p<10^\{\-3\},dz=0\.56d\_\{z\}=0\.56\), and is not detectably different from the jointly optimized weight \(0\.6820\.682; difference−0\.0006\-0\.0006,95%95\\%CI\[−0\.0029,\+0\.0017\]\[\-0\.0029,\+0\.0017\],p=0\.61p=0\.61\)\. The independently selected best single expert reaches0\.6690\.669there, matching the reported selector\. For the77B and README studies we report the jointly optimized reference in place of this independent baseline, as their tables indicate\.

#### Best single expert\.

At byte scale, each vertex draws its own posterior particles and is scored with its own denoising energy; the selected expert reaches0\.6130\.613\. The1\.31\.3B independent selector uses the same own\-posterior procedure and reaches0\.5990\.599\. These are posterior\-score heuristics rather than marginal\-evidence estimates because they omit the free\-energy and posterior\-entropy terms\. For the77B and natural\-document studies we use the posterior\-score reference based on particles from the local field, as their tables indicate\.

#### Adapted literature baselines\.

Three further baselines adapt published test\-time fusion methods to the corrupted\-window setting\. All three score the corrupted window’s*visible*tokens under each expert with the same denoising surrogate used everywhere else in this paper; corrupted positions carry no target and are excluded\. Every resulting weight field feeds the same decoder as the other methods\.

*PackLLM*\[[22](https://arxiv.org/html/2609.05801#bib.bib15)\]chooses one global weight per window from the input’s own likelihood\. The sim variant setsλi∝exp\(−ℒ¯i/T\)\\lambda\_\{i\}\\propto\\exp\(\-\\bar\{\\mathcal\{L\}\}\_\{i\}/T\), whereℒ¯i\\bar\{\\mathcal\{L\}\}\_\{i\}is expertii’s mean visible\-token score andT=0\.1T=0\.1nats\. The opt variant orders experts by individual fit, then mixes in each next expert through a line search over mixing coefficients that minimizes the composed model’s visible\-token cross\-entropy; candidate mixtures are evaluated under the same weighted\-logit composition the decoder uses, on the same sampled noisings, so the optimized objective matches the deployed operator\. At the77B scale the greedy variant optimizes the log\-probability mixture used by that decoder\.

*DEMix routing*\[[10](https://arxiv.org/html/2609.05801#bib.bib7)\]computes a parameter\-free posterior over experts from the data itself\. Our adaptation sets per\-position weights proportional toexp⁡\(−β​ℒi,ℓ\)\\exp\(\-\\beta\\,\\mathcal\{L\}\_\{i,\\ell\}\)withβ=1\\beta=1on the visible\-token scores; a corrupted position, which has no score of its own, inherits the mean score of the nearest visible positions within an expanding window\. The field then receives the same smoothing as every other local field\.

*HMM routing*is a third local router, reported in Section[5\.4](https://arxiv.org/html/2609.05801#S5.SS4)and in the arXiv table\. It fits a two\-state hidden Markov model over the per\-position per\-expert visible\-token scores, with a symmetric transition prior, and takes the posterior state marginal as the field\. It is the segmentation\-aware member of the router family, and it trails local evidence at every scale we ran it \(0\.5680\.568at1\.31\.3B,0\.6440\.644on arXiv,0\.6840\.684on README\)\. It was not run at byte scale\.

*Marginal\-evidence selection*picks the expert with the lowest mean visible\-token score\. Under the mask channel, the evidence of the corrupted window factorizes into a weight\-independent channel term and the marginal likelihood of the visible tokens, so this selector estimates the single\-expert marginal evidence, in the spirit of evidence\-based model selection for score\-based priors\[[39](https://arxiv.org/html/2609.05801#bib.bib2)\]\. It differs from the own\-posterior selector above, which scores each expert on its own reconstructions; the two selectors are reported side by side\.