Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction

# Forecasting Corpus-Sampling Disagreement in Masked Text Reconstruction
Source: [https://arxiv.org/html/2610.12987](https://arxiv.org/html/2610.12987)
Hare Krishna

University of Texas at Austin

hkrishna\.phy@gmail\.com

††footnotetext:Code:[https://github\.com/learncsai/masked\-diffusion\-model\-text\-reconstruction](https://github.com/learncsai/masked-diffusion-model-text-reconstruction)###### Abstract

If a second text corpus were collected from the same population, how much would a predictor’s outputs change? We study this question for masked\-text reconstruction using a smoothed predictor built from nearby token\-pair counts and evaluated over the full 50,257\-token GPT\-2 vocabulary\. Treating each source document as one observation, we derive a first\-order formula for the expected disagreement between predictors fitted to two independent corpora\. The formula accounts for variable document length, random context counts, and correlations across token offsets, and it can be evaluated from a separate reference corpus without access to either target corpus\. On TinyStories, WikiText\-103, and CNN/DailyMail, growing each corpus from 512 to 2,048 chunks of 64 tokens lowers disagreement by only 5–40%, far short of the 75% implied by1/n1/nscaling\. The forecast tracks this slow decline, which we trace to smoothing of rarely seen contexts\. Forecast accuracy depends less on how these correlations are modeled than on the size of the reference corpus: small references systematically underpredict disagreement\. A source\-split jackknife that uses only reference data reduces this calibration error by 57–59%\. After this correction, a simpler smoothed\-multinomial approximation is most accurate when the reference corpus is smaller than the target corpus, and a nonlinear source bootstrap is most accurate when it is larger\. Finally, small masked diffusion language models trained on the same corpora show that the count\-based forecasts partly identify which inputs are most sensitive to the corpus, but miss the neural models’ much faster decline in disagreement\.

## 1 Introduction and Related Work

A text corpus reflects the particular stories, articles, and other sources that happened to be collected\. A second collection from the same population will contain different examples and token frequencies, so a predictor built from it can change as well\. How large should we expect that change to be? Answering this question would help us decide how much text to collect and judge how strongly a prediction depends on the chosen corpus\. Average test error alone cannot answer it: similarly accurate predictors may disagree on the same input\([Marx et al\., 2020](https://arxiv.org/html/2610.12987#bib.bib28)\), and comparable validation performance can conceal differences in deployment behavior\([D’Amour et al\., 2022](https://arxiv.org/html/2610.12987#bib.bib8)\)\.

We study how corpus sampling affects predictions in a masked\-text reconstruction problem\. Our reconstructor predicts a hidden token by combining conditional token frequencies at nearby positions\. Independent corpora A and B supply separate count tables, while the unigram baseline and combination weights are fixed using shared auxiliary data\. We measure the observed disagreement as the mean squared distance between the two reconstructors’ output vectors on identical masked inputs\. Our forecast targets the expected value of this disagreement across independent draws of A and B at a specified corpus size\. We compute the forecast from an independent reference corpus and assess it against the measured disagreement averaged over multiple A/B pairs\. Because only the count tables differ between the two reconstructors, this comparison isolates variation due to corpus sampling\. Separately, we measure relative MSE improvement over the shared unigram to assess reconstruction accuracy\.

Two features of text make the calculation less straightforward than a standard inverse\-sample\-size argument\. First, token\-pair counts are dependent\. An article can contribute several chunks, overlapping pairs share tokens, and counts at different offsets fluctuate together\. Second, each conditional row is a ratio whose denominator, the number of times the context appears, is itself random\. Our smoothing rule pulls rows with few observations strongly toward the unigram\([Chen and Goodman, 1999](https://arxiv.org/html/2610.12987#bib.bib5)\), so a larger corpus does two things at once\. It reduces sampling noise in the estimated frequencies, and it lets the predictor rely more heavily on those frequencies\. The second effect partly offsets the first, which is why disagreement can fall much more slowly than1/n1/n\.

#### Contributions\.

\(1\)We derive a first\-order, source\-level formula for expected disagreement \(Proposition[1](https://arxiv.org/html/2610.12987#Thmproposition1)\) that accounts for source length, random row totals, and cross\-lag covariance\. It enables forecasts from independent reference data without using the target A/B counts or outputs, and explains how smoothing can produce substantially slower decay than1/n1/nat finite corpus sizes\. We call it the source law throughout the article\.

\(2\)We show that correcting finite\-reference error can matter more than retaining additional covariance detail\. A source\-split jackknife reduces small\-reference error by 57–59% for both the source and smoothed multinomial forecasts\. In our main experiments, the corrected multinomial law outperforms the source law with a small reference, while the corrected nonlinear source bootstrap is most accurate with a larger one\. In simulations with known population values, the multinomial law does better partly because two of its errors cancel\.

\(3\)We test the count\-based forecasts against disagreement between small MDLMs trained on separate corpora\. The forecasts partly identify the inputs on which the MDLMs disagree most\. We also compare the directions in which their token scores change\. Overall token frequencies predict these directions as well as the reconstructor does\. With the number of training steps held fixed, MDLM disagreement falls much faster with corpus size than reconstructor disagreement\. The forecasts therefore help identify inputs sensitive to corpus sampling, but do not fully explain the size or pattern of neural prediction changes\.

#### Related work\.

Existing work studies corpus uncertainty through embedding stability, resampling, and covariance estimates for word representations\([Antoniak and Mimno, 2018](https://arxiv.org/html/2610.12987#bib.bib2);[Gries, 2022](https://arxiv.org/html/2610.12987#bib.bib17);[Vallebueno et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib37)\)\. For word\-frequency comparisons,[Lijffijt et al\. \(2016\)](https://arxiv.org/html/2610.12987#bib.bib26)emphasize treating texts as independent observations while allowing dependence among words within a text\. GloVe\-V\([Vallebueno et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib37)\)instead defines uncertainty over word co\-occurrences rather than over the documents that produce them\. Our multinomial law makes the analogous count\-level assumption, whereas our source law treats complete sources as the sampling unit and retains dependence within each source\. Influence functions measure how single training examples change a model’s predictions\([Koh and Liang, 2017](https://arxiv.org/html/2610.12987#bib.bib24)\)and[Brunet et al\. \(2019\)](https://arxiv.org/html/2610.12987#bib.bib4)trace embedding bias back to individual documents\. The infinitesimal jackknife linearizes a procedure with respect to observation weights\([Jaeckel, 1972](https://arxiv.org/html/2610.12987#bib.bib22);[Efron, 1982](https://arxiv.org/html/2610.12987#bib.bib12);[Giordano et al\., 2019](https://arxiv.org/html/2610.12987#bib.bib15)\)\.[Wager et al\. \(2014\)](https://arxiv.org/html/2610.12987#bib.bib39)use jackknife methods to estimate how much a random forest’s prediction at a fixed input varies across training samples, a target close to ours\.

Datamodels learn how a model’s predictions depend on which examples it is trained on, using outputs from models trained on many subsets of the same dataset\([Ilyas et al\., 2022](https://arxiv.org/html/2610.12987#bib.bib21)\)\. Our analytic forecast does not require training models on many subsets of the corpus: a first\-order approximation gives an explicit variance formula for smoothed lag counts at a fixed chunk budget, and its inputs are estimated from an independent reference corpus\. Treating a complete source as the sampling unit keeps its internal dependence and lets us separate source\-length, row\-normalization, and cross\-lag contributions\. We compare this formula with a smoothed multinomial approximation and a nonlinear source bootstrap\. All three are computed from a finite reference corpus, so a forecast can fail in two ways\. Its approximation to the sampling variation can be too crude, or the reference can estimate the required rows and covariances poorly\.

In our experiments, the second problem is the bigger one\. Both analytic laws \(source law and multinomial\) capture much of the slow, smoothing\-driven decline in disagreement with corpus size, but both underpredict its level when the reference is small\. Part of the gap comes from contexts the reference never sees, which contribute nothing to the forecast even when the new corpora contain them\. We reduce this bias with a source\-split jackknife\([Quenouille, 1956](https://arxiv.org/html/2610.12987#bib.bib33)\), applied the same way to both laws\. After correction, the multinomial law is still the better calibrated of the two\. Exact population calculations explain this: even with no reference error, linearization makes the source formula overshoot the actual disagreement by 3–12% in our controlled experiment\. The multinomial law gives smaller variance estimates that partly offset this overshoot\. As the reference grows and estimation error shrinks, linearization error becomes a larger share of what remains\. This fits our finding that a corrected bootstrap, which refits the smoothed rows on every resample instead of linearizing, gives the best forecasts when the reference is larger than the target\.

We then ask whether these local statistics also explain variation in a trained neural denoiser\. In image diffusion,[Kadkhodaie et al\. \(2024\)](https://arxiv.org/html/2610.12987#bib.bib23)show that denoisers trained on non\-overlapping datasets learn nearly the same score function when the datasets are sufficiently large\.[Zhang et al\. \(2024\)](https://arxiv.org/html/2610.12987#bib.bib42)study consistency of generated outputs across diffusion models under common initial noise and deterministic sampling\.[Wang et al\. \(2026\)](https://arxiv.org/html/2610.12987#bib.bib41)connect image\-diffusion consistency to empirical means and covariances through a Gaussian linear denoiser and random matrix theory\. That Gaussian analysis does not carry over directly to text, where tokens are categorical, corruption is by masking, and the predictor divides by random context counts\. Table[7](https://arxiv.org/html/2610.12987#A3.T7)in Appendix[C\.3](https://arxiv.org/html/2610.12987#A3.SS3)maps out the correspondence\.

For text,[Nguyen \(2024\)](https://arxiv.org/html/2610.12987#bib.bib30)approximates transformer predictions using training\-datann\-gram statistics and relates approximation quality to prediction variability across training runs\. That variability comes from different shuffles of a fixed corpus; our target is variation across independently sampled corpora\. Our matched experiment with small masked diffusion language models \(MDLMs\) compares token\-prediction vectors on identical masked inputs, and tests how far the count\-based predictions carry over in magnitude, input ranking, and output direction\.

## 2 Reconstructor

Our sampling unit is a source, such as a story, WikiText paragraph, or news article, which may contribute several non\-overlappingLL\-token chunks\. Each source belongs to exactly one data role \(Table[1](https://arxiv.org/html/2610.12987#S2.T1)\)\. Auxiliary sources set the shared unigram and reconstruction weights, on which we condition throughout\. Reference sources then estimate sampling variation\. The independent A and B corpora supply the count tables whose disagreement we forecast, and evaluation sources provide their common masked inputs\. This separation allows the forecast to be tested without using A/B outcomes to construct it\.

Table 1:Source\-disjoint data roles\. Auxiliary quantities are shared by A and B and held fixed throughout\.LetX=\(X0,…,XL−1\)X=\(X\_\{0\},\\ldots,X\_\{L\-1\}\)have token IDs in\{1,…,K\}\\\{1,\\ldots,K\\\}\. Each position is hidden with probabilityqq\. The maskMMhas hidden setHH, andZZcontains the visible tokens and mask\. For output vectorsui∈ℝKu\_\{i\}\\in\\mathbb\{R\}^\{K\}, define

‖u‖M2=1max⁡\{1,\|H\|\}​∑i∈H‖ui‖22\.\\\|u\\\|\_\{M\}^\{2\}=\\frac\{1\}\{\\max\\\{1,\|H\|\\\}\}\\sum\_\{i\\in H\}\\\|u\_\{i\}\\\|\_\{2\}^\{2\}\.This norm averages over hidden positions within a chunk, so every chunk receives equal weight\.

For one pair of corpora, define

DA,B;q=𝔼X,M​‖f^A​\(Z\)−f^B​\(Z\)‖M2\.D\_\{A,B;q\}=\\mathbb\{E\}\_\{X,M\}\\\|\\widehat\{f\}\_\{A\}\(Z\)\-\\widehat\{f\}\_\{B\}\(Z\)\\\|\_\{M\}^\{2\}\.\(1\)Averaging also over independent corpus collections gives the population target𝔇nA,nB,q=𝔼𝒟A,𝒟B​DA,B;q\\mathfrak\{D\}\_\{n\_\{A\},n\_\{B\},q\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\_\{A\},\\mathcal\{D\}\_\{B\}\}D\_\{A,B;q\}\. Here size counts number of chunks, and𝔇n,q\\mathfrak\{D\}\_\{n,q\}denotes equal sizesnA=nB=nn\_\{A\}=n\_\{B\}=n\.

#### Reconstruction from lag counts\.

For offsets0<\|δ\|≤R0<\|\\delta\|\\leq R, letSg,δ​\(b,a\)S\_\{g,\\delta\}\(b,a\)count context tokenbbfollowed by targetaaat offsetδ\\deltawithin the chunks of sourcegg\. Both positions must be in the same chunk\. Write

Nn,δ=∑gSg,δ,C^n​\(δ\)=Nn,δn⁡\(L−\|δ\|\)\.N\_\{n,\\delta\}=\\sum\_\{g\}S\_\{g,\\delta\},\\qquad\\widehat\{C\}\_\{n\}\(\\delta\)=\\frac\{N\_\{n,\\delta\}\}\{n\(L\-\|\\delta\|\)\}\.\(2\)These are raw pair frequencies\. With smoothingα\>0\\alpha\>0, the conditional row is

P^δ​\(a∣b\)=Nn,δ​\(b,a\)\+α​paNn,δ​\(b,\+\)\+α\.\\widehat\{P\}\_\{\\delta\}\(a\\mid b\)=\\frac\{N\_\{n,\\delta\}\(b,a\)\+\\alpha p\_\{a\}\}\{N\_\{n,\\delta\}\(b,\+\)\+\\alpha\}\.\(3\)HereNn,δ​\(b,\+\)=∑aNn,δ​\(b,a\)N\_\{n,\\delta\}\(b,\+\)=\\sum\_\{a\}N\_\{n,\\delta\}\(b,a\)\. Unseen contexts returnpp\. Let𝒜i​\(M\)\\mathcal\{A\}\_\{i\}\(M\)contain the offsets with an in\-bounds visible token ati−δi\-\\delta\. The reconstruction method is

f^i\(Z\)=p\+∑δ∈𝒜i​\(M\)wδ,𝒜i​\(M\)\{P^δ\(⋅∣Xi−δ\)−p\}\.\\widehat\{f\}\_\{i\}\(Z\)=p\+\\sum\_\{\\delta\\in\\mathcal\{A\}\_\{i\}\(M\)\}w\_\{\\delta,\\mathcal\{A\}\_\{i\}\(M\)\}\\\{\\widehat\{P\}\_\{\\delta\}\(\\cdot\\mid X\_\{i\-\\delta\}\)\-p\\\}\.\(4\)Regularized least squares on auxiliary sources sets the weights once \(Appendix[B\.4](https://arxiv.org/html/2610.12987#A2.SS4)\), leaving the count tables as the only quantities that vary between A and B\. With no visible neighbor, the predictor returnspp\. For fixed masks and tables, its output is affine in visible one\-hot tokens\. Its dependence on sampled counts is nonlinear because each row is normalized\. Because these scores can be negative, log\-loss is undefined, so we use squared error\.

#### Relative MSE improvement\.

For one\-hot targetseXie\_\{X\_\{i\}\}, define

ℒ⁡\(f\)\\displaystyle\\mathcal\{L\}\(f\)=𝔼X,M​‖f−eX‖M2,\\displaystyle=\\mathbb\{E\}\_\{X,M\}\\\|f\-e\_\{X\}\\\|\_\{M\}^\{2\},\(5\)RelImprove⁡\(f\)\\displaystyle\\operatorname\{RelImprove\}\(f\)=1−ℒ⁡\(f\)ℒ⁡\(p\)\.\\displaystyle=1\-\\frac\{\\mathcal\{L\}\(f\)\}\{\\mathcal\{L\}\(p\)\}\.The denominator is positive in every case we explore, so positive relative improvement means lower MSE than the unigram\. This metric uses the full output vector\. We assess ranking separately with top\-kkaccuracy \(Appendix[D](https://arxiv.org/html/2610.12987#A4)\)\. The unigram ignores context, so by construction it has zero disagreement and zero relative improvement\.

## 3 Disagreement Laws and Reference Correction

A large corpus can contain many rare contexts, for which smoothing strongly affects how predictions vary across corpus samples\. In the multinomial model below, the variance of a smoothed conditional row withNNobservations is proportional toN/\(N\+α\)2N/\(N\+\\alpha\)^\{2\}\. Increasing its count fromN=αN=\\alphatoN=4​αN=4\\alphareduces this variance by only 36%, compared with 75% under inverse\-size scaling\. So the forecast depends on how often each context is seen relative toα\\alpha\. We first derive a source\-level formula that retains dependence among counts, then compare it with a multinomial approximation to examine the roles of smoothing and covariance\.

### 3\.1 Source influence and the population target

Two independent corpora drawn by the same procedure at the same size give identically distributed predictors\. Expanding their squared difference gives the exact identity

𝔇n,q=2​𝔼X,M​𝔼𝒟​‖f^n​\(Z\)−𝔼𝒟​f^n​\(Z\)‖M2\.\\mathfrak\{D\}\_\{n,q\}=2\\mathbb\{E\}\_\{X,M\}\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\|\\widehat\{f\}\_\{n\}\(Z\)\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\widehat\{f\}\_\{n\}\(Z\)\\\|\_\{M\}^\{2\}\.\(6\)The remaining task is to approximate the variance of one predictor\. Appendix[A](https://arxiv.org/html/2610.12987#A1)derives this identity and the first\-order covariance expansion\.

LetTgT\_\{g\}be sourcegg’s chunk contribution before terminal chunk\-budget truncation, and letSg,δS\_\{g,\\delta\}count pairs in that full contribution\. Setτ=𝔼​Tg\\tau=\\mathbb\{E\}T\_\{g\}andsδ=𝔼​Sg,δs\_\{\\delta\}=\\mathbb\{E\}S\_\{g,\\delta\}\. At chunk budgetnn, setGn=n/τG\_\{n\}=n/\\tauandBδ,n=Gn​sδB\_\{\\delta,n\}=G\_\{n\}s\_\{\\delta\}\. The centered source count is

Ag,δ=Sg,δ−\(Tg/τ\)​sδ\.A\_\{g,\\delta\}=S\_\{g,\\delta\}\-\(T\_\{g\}/\\tau\)s\_\{\\delta\}\.\(7\)A longer source supplies more pairs but also uses more of the fixed chunk budget, which is why centering subtracts\(Tg/τ\)​sδ\(T\_\{g\}/\\tau\)s\_\{\\delta\}\. When every source supplies one chunk,Tg=τ=1T\_\{g\}=\\tau=1and this reduces to ordinary mean centering\.

LetPδ,nP\_\{\\delta,n\}be \([3](https://arxiv.org/html/2610.12987#S2.E3)\) evaluated at countsBδ,nB\_\{\\delta,n\}\. We expand around this fixed row, whose value generally differs from𝔼​P^δ\\mathbb\{E\}\\widehat\{P\}\_\{\\delta\}because normalization is nonlinear\. Applying the quotient rule gives

P˙g,δ,n​\(a∣b\)=Ag,δ​\(b,a\)−Pδ,n​\(a∣b\)​Ag,δ​\(b,\+\)Bδ,n​\(b,\+\)\+α\.\\dot\{P\}\_\{g,\\delta,n\}\(a\\mid b\)=\\frac\{A\_\{g,\\delta\}\(b,a\)\-P\_\{\\delta,n\}\(a\\mid b\)A\_\{g,\\delta\}\(b,\+\)\}\{B\_\{\\delta,n\}\(b,\+\)\+\\alpha\}\.\(8\)The subtraction keeps fluctuations in the row total\. The resulting output influence is

hg,n\(i,Z\)=∑δ∈𝒜i​\(M\)wδ,𝒜i​\(M\)P˙g,δ,n\(⋅∣Xi−δ\)\.h\_\{g,n\}\(i,Z\)=\\sum\_\{\\delta\\in\\mathcal\{A\}\_\{i\}\(M\)\}w\_\{\\delta,\\mathcal\{A\}\_\{i\}\(M\)\}\\dot\{P\}\_\{g,\\delta,n\}\(\\cdot\\mid X\_\{i\-\\delta\}\)\.\(9\)The vectorhg,n​\(i,Z\)h\_\{g,n\}\(i,Z\)represents the first\-order change in the reconstructed output at positioniiinduced by sourcegg’s count fluctuation, adjusted for the fixed chunk budget\.

###### Proposition 1\(First\-order corpus reproducibility\)\.

FixK,L,R,qK,L,R,q,α\>0\\alpha\>0, and the shared unigram and auxiliary weights\. Suppose full source contributions are iid, with1≤Tg≤Tmax<∞1\\leq T\_\{g\}\\leq T\_\{\\max\}<\\infty, andsδ​\(b,\+\)\>0s\_\{\\delta\}\(b,\+\)\>0for every row queried by the reconstructor\. For two independent corpora withnnretained chunks and evaluation inputs independent of both,

𝔇n,q=2​Gn​𝔼X,M,g​‖hg,n​\(Z\)‖M2\+o⁡\(n−1\)\\mathfrak\{D\}\_\{n,q\}=2G\_\{n\}\\mathbb\{E\}\_\{X,M,g\}\\\|h\_\{g,n\}\(Z\)\\\|\_\{M\}^\{2\}\+o\(n^\{\-1\}\)\(10\)asn→∞n\\to\\infty\. The leading term isO⁡\(n−1\)O\(n^\{\-1\}\)\.

Appendix[A](https://arxiv.org/html/2610.12987#A1)proves the result, including truncation of the final source to meet the chunk budget\. The finite\-size forecast retains the smoothed denominators in \([8](https://arxiv.org/html/2610.12987#S3.E8)\), so it can decrease more slowly than1/n1/nwhile row counts remain comparable toα\\alpha\.

### 3\.2 Forecast from independent reference data

A reference hasGrefG\_\{\\rm ref\}sources andNrefN\_\{\\rm ref\}chunks\. Replace the source means by their empirical values and useτ^=Nref/Gref\\widehat\{\\tau\}=N\_\{\\rm ref\}/G\_\{\\rm ref\}andG^n=n​Gref/Nref\\widehat\{G\}\_\{n\}=nG\_\{\\rm ref\}/N\_\{\\rm ref\}\. Then

D^n,qan=2​G^nGref−1​∑g=1Gref𝔼X,M​‖h^g,n​\(Z\)‖M2\.\\widehat\{D\}^\{\\rm an\}\_\{n,q\}=\\frac\{2\\widehat\{G\}\_\{n\}\}\{G\_\{\\rm ref\}\-1\}\\sum\_\{g=1\}^\{G\_\{\\rm ref\}\}\\mathbb\{E\}\_\{X,M\}\\\|\\widehat\{h\}\_\{g,n\}\(Z\)\\\|\_\{M\}^\{2\}\.\(11\)The forecast uses only reference and auxiliary quantities on the common evaluation inputs\. The reference size determines how accurately we can estimate source moments and conditional rows, whereas the target sizennsets the expected counts, the strength of smoothing, and the number of sources in each new corpus\. Forecasting anothernntherefore requires recomputing these quantities from the same reference\. The two sizes are free to differ, and Section[4](https://arxiv.org/html/2610.12987#S4)examines how their ratio affects calibration\.

Source bootstrap estimates the same total variance by resampling whole reference sources and rebuilding the nonlinear rows\. The analytic expression instead makes source and lag contributions explicit, without bootstrap Monte Carlo error\. We compare both forecasts in Appendix[B\.2](https://arxiv.org/html/2610.12987#A2.SS2)\.

### 3\.3 The smoothed multinomial approximation

We compare the source forecast with a multinomial model that retains smoothing but fixes row totals\. For contextbbat lagδ\\delta, fixN=Nn,δ​\(b\)N=N\_\{n,\\delta\}\(b\)and letCaC\_\{a\}count targets of typeaa\. AssumeC=\(C1,…,CK\)∼Multinomial⁡\(N,π\)C=\(C\_\{1\},\\ldots,C\_\{K\}\)\\sim\\operatorname\{Multinomial\}\(N,\\pi\), whereπ=πδ\(⋅∣b\)\\pi=\\pi\_\{\\delta\}\(\\cdot\\mid b\)is the conditional target distribution\. The smoothed row\(C\+α​p\)/\(N\+α\)\(C\+\\alpha p\)/\(N\+\\alpha\)has covarianceN⁡\{diag⁡\(π\)−π​π⊤\}/\(N\+α\)2N\\\{\\operatorname\{diag\}\(\\pi\)\-\\pi\\pi^\{\\top\}\\\}/\(N\+\\alpha\)^\{2\}\. For two independent corpora with the same row totals and independent rows across lags, the predicted disagreement at a masked position is

vδ,n​\(b\)\\displaystyle v\_\{\\delta,n\}\(b\)=2​Nn,δ​\(b\)\(Nn,δ​\(b\)\+α\)2\(1−∥πδ\(⋅∣b\)∥22\),\\displaystyle=\\frac\{2N\_\{n,\\delta\}\(b\)\}\{\(N\_\{n,\\delta\}\(b\)\+\\alpha\)^\{2\}\}\(1\-\\\|\\pi\_\{\\delta\}\(\\cdot\\mid b\)\\\|\_\{2\}^\{2\}\),\(12\)dnmulti​\(i,Z\)\\displaystyle d\_\{n\}^\{\\rm multi\}\(i,Z\)=∑δ∈𝒜i​\(M\)wδ,𝒜i​\(M\)2​vδ,n​\(bδ\),\\displaystyle=\\sum\_\{\\delta\\in\\mathcal\{A\}\_\{i\}\(M\)\}w\_\{\\delta,\\mathcal\{A\}\_\{i\}\(M\)\}^\{2\}v\_\{\\delta,n\}\(b\_\{\\delta\}\),wherebδ=Xi−δb\_\{\\delta\}=X\_\{i\-\\delta\}\. Averaging over hidden positions and chunks gives the total forecast\. We estimateπδ\\pi\_\{\\delta\}with reference\-smoothed rows and setNn,δ​\(b\)N\_\{n,\\delta\}\(b\)by scaling reference row totals byn/Nrefn/N\_\{\\rm ref\}\(Appendix[B\.4](https://arxiv.org/html/2610.12987#A2.SS4.SSS0.Px1)\)\.

The formula is exact under this model before substituting reference estimates\. The source law accounts for random row totals and dependence from overlapping pairs, shared sources, and different lags, to first order\. Both include smoothing, so both can predict a decline slower than1/n1/n\. We compare how well each matches the level of disagreement and how it changes with corpus size\.

### 3\.4 Correcting finite\-reference bias

The reference estimates both the conditional rows and their variation, so increasing its size can change the forecast even when A/B size stays fixed\. We test a classical jackknife correction for this estimation error\([Quenouille, 1956](https://arxiv.org/html/2610.12987#bib.bib33)\)\. LetFS​\(n\)F\_\{S\}\(n\)denote a forecast fromSSreference sources\. Split them randomly into two groups of sizesS1,S2S\_\{1\},S\_\{2\}, keeping each source’s retained chunks together, and compute

FJ​\(n\)=2​FS​\(n\)−∑j=12SjS​FSj​\(n\)\.F\_\{J\}\(n\)=2F\_\{S\}\(n\)\-\\sum\_\{j=1\}^\{2\}\\frac\{S\_\{j\}\}\{S\}F\_\{S\_\{j\}\}\(n\)\.\(13\)We use the same target sizennin all three components\. The operation applies to any forecast computed from a reference\. If the mean forecast has biasb/S\+o⁡\(S−1\)b/S\+o\(S^\{\-1\}\)at fixed source count, \([13](https://arxiv.org/html/2610.12987#S3.E13)\) cancels its leading term \(Appendix[A](https://arxiv.org/html/2610.12987#A1.SS0.SSS0.Px8)\)\. This justifies the correction when the bias expansion is regular; whether it still helps with sparse rows and chunk\-budget stopping is an empirical question, which we test below\.

## 4 Calibration and Corpus\-Size Response

We evaluate TinyStories\([Eldan and Li, 2023](https://arxiv.org/html/2610.12987#bib.bib13)\), WikiText\-103\([Merity et al\., 2017](https://arxiv.org/html/2610.12987#bib.bib29)\), and cased CNN/DailyMail 3\.0\.0\([Hermann et al\., 2015](https://arxiv.org/html/2610.12987#bib.bib19);[See et al\., 2017](https://arxiv.org/html/2610.12987#bib.bib36)\)with the GPT\-2 tokenizer\([Radford et al\., 2019](https://arxiv.org/html/2610.12987#bib.bib34)\)\. The shared settings areK=50,257K=50\{,\}257,L=64L=64,R=2R=2,α=5\\alpha=5, and auxiliary ridge0\.010\.01\. The calibration comparisons below useq=0\.5q=0\.5\. On the grid of three corpora, three sizes, three mask rates, and two A/B pairs, the reconstructor improves on the unigram in all 54 combinations, with mean relative MSE improvements of 13\.16%, 5\.19%, and 3\.95% in corpus order\. Shuffling tokens within each chunk removes almost all of this gain \(Appendix[D](https://arxiv.org/html/2610.12987#A4)\), so the gain comes from local context\.

### 4\.1 Disagreement falls more slowly than1/n1/n

On seven independent pairs per corpus, increasing size from 512 to 2,048 chunks reduces disagreement by only 5 to 40%, compared with 75% under1/n1/n\. The source\-law forecast, recomputed at each size, tracks this slow decline more closely than1/n1/nin every corpus\. A separate three\-pair comparison at 4,096 and 8,192 chunks shows that the smoothed multinomial law also reproduces much of the slow decrease \(Appendix[B\.5](https://arxiv.org/html/2610.12987#A2.SS5)\)\. So smoothing, not covariance structure, is the main reason disagreement falls slowly\. The absolute level is harder to predict, since it also depends on random row totals, source dependence, and how well the reference estimates them\.

### 4\.2 Bias correction improves calibration

Our main calibration test uses 20 new A/B pairs and six separate reference banks per corpus\. Each A/B corpus containsn∈\{512,1024,2048,4096\}n\\in\\\{512,1024,2048,4096\\\}chunks, while each reference bank containsm∈\{1024,2048,4096\}m\\in\\\{1024,2048,4096\\\}chunks\. Smaller sizes are nested within each allocation\. No source is shared between reference banks, between A/B corpora, or across data roles\. Appendix[B\.1](https://arxiv.org/html/2610.12987#A2.SS1)gives the protocol and every tested condition\. The primary comparison covers\(m,n\)=\(1024,2048\),\(1024,4096\),\(2048,4096\)\(m,n\)=\(1024,2048\),\(1024,4096\),\(2048,4096\), where the reference is smaller than the target \(A/B\)\. For each method and reference bank, letFFdenote the predicted disagreement andD¯\\bar\{D\}the arithmetic mean of the 20 observed A/B disagreements\. We measure the absolute log error\|log⁡\(F/D¯\)\|\|\\log\(F/\\bar\{D\}\)\|and average equally over banks,\(m,n\)\(m,n\)conditions, and corpora\.

Table 2:Calibration on the twenty\-pair, six\-bank study\. Lower mean absolute log error is better\. The primary subset contains the threem<nm<ncells\. The full grid contains all twelve\(m,n\)\(m,n\)combinations per corpus\. Jackknife rows apply correction at every cell, without them≤nm\\leq nrestriction\. Appendix[B\.2](https://arxiv.org/html/2610.12987#A2.SS2)gives definitions and uncertainty intervals\.The source\-split jackknife reduces the source forecast’s error by 56\.7%, from 0\.469 to 0\.203, bringing it close to the uncorrected multinomial error of 0\.205\. Applying the same correction to multinomial reduces its error by 58\.5% to 0\.085 \(Table[2](https://arxiv.org/html/2610.12987#S4.T2)\)\. The correction helps both laws without changing their order, and the simpler multinomial law remains the better calibrated of the two\. We also evaluate the correction on six fresh A/B pairs and three new reference banks per corpus, saving forecasts before scoring\. Atm=1024,n=2048m=1024,n=2048, the same split jackknife reduces multinomial error from 0\.187 to 0\.077 \(Appendix[B\.3](https://arxiv.org/html/2610.12987#A2.SS3)\)\.

Figure 1:Analytic forecast calibration atn=4096n=4096, with 64 tokens per chunk andq=0\.5q=0\.5\. Each point divides the six\-bank mean forecast by the arithmetic mean disagreement of 20 A/B pairs\. Bars show 95% crossed bootstrap intervals over banks and pairs, conditional on auxiliary and evaluation data\. The dotted line marks exact calibration, and lines connect the three tested reference sizes\. Stars use an uncorrected 8,192\-chunk reference\.Figure[1](https://arxiv.org/html/2610.12987#S4.F1)shows the effect of jackknife correction on the analytic source forecast\. Atm=n=4096m=n=4096, correction moves the ratios from 0\.83–0\.87 to 0\.97–1\.01\. Withm=1024m=1024, the corrected ratios are still only 0\.65–0\.74\. So correction reduces, but does not remove, the shortfall caused by a small reference\. For the analytic source law, we recommend correction only whenm≤nm\\leq n: correcting atm\>nm\>nraises source error from 0\.052 to 0\.095\.

The nonlinear source bootstrap behaves differently\. Atm<nm<n, its corrected error is 0\.175, compared with 0\.085 for corrected multinomial\. Atm\>nm\>n, it improves from 0\.072 to 0\.013 after correction, beating corrected multinomial’s 0\.049\. On the separate confirmation pairs atm=2048,n=1024m=2048,n=1024, the corrected bootstrap again has lower error, 0\.020 versus 0\.049 \(Appendix[B\.3](https://arxiv.org/html/2610.12987#A2.SS3.SSS0.Px1)\)\. We therefore favor corrected multinomial with a small reference and corrected source bootstrap with a larger one\.

An obvious alternative is to scale an uncorrected forecast by a constant, and it works well \(Appendix[B\.3](https://arxiv.org/html/2610.12987#A2.SS3.SSS0.Px2)\)\. We also test whether a fitted scale factor transfers to fresh sources\. We choose a single multiplier to minimize the multinomial forecast’s mean absolute log error over the twenty\-pair calibration grid\. Applying the resulting factor,1\.1181\.118, unchanged to the fresh grid gives error 0\.045, compared with 0\.060 for multinomial jackknife \(Appendix[B\.3](https://arxiv.org/html/2610.12987#A2.SS3.SSS0.Px2)\)\. But fitting this factor needs observed A/B disagreements, while the jackknife needs only the reference bank\.

#### Sensitivity to smoothing\.

The calibration ranking changes with the smoothing strength \(Table[3](https://arxiv.org/html/2610.12987#S4.T3)\)\. Corrected multinomial has lower error atα=1\\alpha=1and 5, but atα=20\\alpha=20corrected source has lower error on TinyStories and WikiText, with the two methods nearly tied on CNN/DailyMail\.

Table 3:Effect ofα\\alphaon calibration for small references \(m<nm<n\)\. Both forecasts use source\-split jackknife correction\. Entries are mean absolute log errors \(↓\\downarrow\) on the same 20 pairs and six reference banks per corpus, with weights fixed at theirα=5\\alpha=5values\. Appendix[E\.3](https://arxiv.org/html/2610.12987#A5.SS3)gives the protocol and interpretation\.

### 4\.3 Where does the remaining error come from?

To find where the remaining shortfall comes from, we use synthetic populations whose mean counts are known exactly \(Appendix[E\.1](https://arxiv.org/html/2610.12987#A5.SS1)\)\. A context that never appears in the reference contributes nothing to the forecast, however often the A/B corpora contain it\. With four chunks per source \(n=1024n=1024,m=128m=128\), these unseen rows and their cross terms account for 32\.6 percentage points of underprediction\. Errors in rows the reference did observe add another 38\.6, so more than half of the shortfall lies in rows that a bias correction can reach\. When we replace every reference estimate with its exact population value \(Appendix[E\.2](https://arxiv.org/html/2610.12987#A5.SS2)\), the multinomial law is still closer to the actual disagreement in all sixα=5\\alpha=5controls\. Linearization makes the source law overshoot, and the multinomial law’s smaller variance happens to cancel part of that\. These are simulations, so real text may show different error sizes\.

## 5 Comparison with Trained MDLMs

We next test whether the count\-based forecasts say anything about neural models trained on the same corpora\. We compare MDLM and reconstructor disagreement in overall size, in which inputs disagree most, and in the direction of the output differences\.

We train MDLMs\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib35)\)with two Transformer blocks of width 96 on three A/B pairs at 512, 1,024, and 2,048 chunks\. A and B share the initialization, training seed, masking streams, and number of updates, so only the training corpus differs\. We score both at 8,000 and 12,000 updates on the common 128 chunks withq=0\.5q=0\.5\. Appendix[C](https://arxiv.org/html/2610.12987#A3)gives the full protocol and seed controls\.

Both models see the same corpora, masked inputs, and vocabulary, and we score both with the squared\-error measures of Section[2](https://arxiv.org/html/2610.12987#S2), applied to the full output vector\. MDLM is trained with its usual likelihood loss\. Two differences remain: It reads every visible position in the 64\-token chunk and outputs probabilities, while the reconstructor reads only the four nearest offsets and outputs affine scores that can be negative\.

#### Magnitude of disagreements\.

From 512 to 2,048 chunks, MDLM disagreement falls by 72\.9%, 81\.5%, and 79\.9% at 12,000 updates\. The reconstructor falls by 38\.9%, 4\.9%, and 11\.4% on the same pairs\. Thus the neural models have a much steeper size response \(Figure[4](https://arxiv.org/html/2610.12987#A3.F4)\), and the ordering of absolute disagreement across corpora also changes at 2,048 chunks \(Table[4](https://arxiv.org/html/2610.12987#S5.T4)\)\.

Table 4:Reconstruction atn=2048n=2048chunks and mask rateq=0\.5q=0\.5for the local reconstructor \(L\) and MDLM after 12,000 updates \(N\)\. For each corpus, both methods are fitted to the A and B corpora of three independent pairs and evaluated on the same 128 masked chunks\.DDis the disagreement between the A and B fits, averaged over the three pairs; lower means more stable\. RelImprove is the MSE improvement over the unigram, averaged over the six fitted models; higher is better\.MDLM improves reconstruction more on TinyStories, but the reconstructor performs better on WikiText and CNN/DailyMail\. The news models are the most stable neural models, yet they improve MSE over the unigram by only 1\.21%\. Training longer increased disagreement in all nine conditions\.

#### Which inputs disagree most\.

Although the magnitudes differ, the count forecasts partly predict which chunks the neural models disagree on most\. Across the 128 evaluation chunks atn=2048n=2048and 12,000 updates, the Spearman correlation between the source\-law forecast and MDLM disagreement averages 0\.48 on TinyStories, 0\.38 on WikiText, and 0\.18 on news\. The multinomial law gives nearly identical values\.

#### Direction of the output differences\.

We next check whether the two models change in the same directions\. We project output differences onto 64 fixed random vocabulary directions\. On half of the evaluation sources, we find the four directions in which the reconstructor’s A/B differences are largest\. On the other half, we measure how much of MDLM’s difference energy falls in them\. The share is 39\.4% on TinyStories, 56\.0% on WikiText, and 52\.2% on news, against 6\.25% for random directions\. But the leading directions of the unigram covariancediag⁡\(p\)−p​p⊤\\operatorname\{diag\}\(p\)\-pp^\{\\top\}capture slightly more \(41\.0%, 58\.9%, and 58\.2%\)\. So token frequencies alone explain the overlap \(Appendix[C\.2](https://arxiv.org/html/2610.12987#A3.SS2)\) and Appendix[D](https://arxiv.org/html/2610.12987#A4)compares top\-kkaccuracy\.

## 6 Discussion and Limitations

Knowing how much a predictor depends on its training corpus helps judge how reproducible its predictions are and how much more data to collect\. For a smoothed count\-based reconstructor, we show that a separate reference sample can forecast disagreement between predictors fitted to new corpora\. Our source\-level formula connects this variation to source length, random row totals, and covariance across lags\. Retaining smoothing also explains why collecting more text can reduce disagreement more slowly than1/n1/nat the corpus sizes studied\.

The experiments show that improving the reference estimates can matter more than refining the covariance model\. A source\-split jackknife reduces small\-reference error by 57–59% for both analytic laws, using only reference data\. Fitted multipliers are competitive, but require earlier observed disagreements\. In our tested settings, the corrected multinomial law performs best when the reference is smaller than the target, while the corrected source bootstrap performs best when it is larger\.

The source formula also explains why the simpler multinomial law can do better\. In our simulations, the multinomial law leaves out some covariance terms, and its smaller forecast partly offsets the overshoot that linearization causes, though the balance changes withα\\alpha\. The bootstrap avoids linearization altogether by refitting the smoothed rows on each resample\.

#### Further applications\.

The approach could extend to other smoothed count statistics, includingnn\-gram models and normalized co\-occurrence or pointwise mutual information estimates\([Chen and Goodman, 1999](https://arxiv.org/html/2610.12987#bib.bib5);[Church and Hanks, 1990](https://arxiv.org/html/2610.12987#bib.bib6)\)\. Their derivatives would propagate source covariance into output variation under suitable positivity and regularity conditions\. A further goal is to use these forecasts to guide corpus collection\. Evaluating candidate sizes could identify the smallest corpus expected to meet a chosen stability target\. Finer size grids and independently chosen targets would allow us to test whether these forecasts can support a concrete decision: how much text to collect for reproducible predictions\.

#### Limitations\.

Larger references need not improve calibration when reference and linearization errors offset\. Proposition[1](https://arxiv.org/html/2610.12987#Thmproposition1)assumes a fixed vocabulary and positive context probabilities, while the jackknife requires a regular bias expansion in the number of reference sources\. Neither argument gives uniform guarantees for sparse rows or growing vocabularies\. The correction also leaves unseen rows unresolved\. Accounting for their uncertainty requires assumptions about conditional outputs and their joint variation, beyond missing mass alone\. Estimating that mass must itself account for sequence dependence\([Pananjady et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib31)\)\.

Our intervals capture variation across reference banks and A/B pairs, conditional on the auxiliary quantities and 128 evaluation chunks\. Six banks limit precision, and WikiText paragraph sampling leaves dependence within articles unmodeled\. The fresh\-source check is small: six A/B pairs and three banks per corpus\.

The MDLM comparison is only a first test of how local corpus statistics relate to neural variation\. It covers small Transformers and corpora, with three A/B pairs and fixed update budgets\. Larger corpora receive fewer passes, so training also affects the measured size response\. Extending the comparison to larger models and datasets, with varied training budgets, would help establish how much neural variation these statistics can explain\. Such tests should measure predictive quality alongside stability and check whether directional predictions improve on the token\-frequency control\.

## AI Use Statement

Generative AI tools, including OpenAI Codex and Anthropic Claude, were used during several stages of this work\. These tools assisted with preprocessing TinyStories, WikiText\-103, and CNN/DailyMail and constructing the text chunks described in the paper\. They were also used in generating code to implement the experiments, preparing tables and figures from recorded experimental outputs\. We have also used these tools in identifying relevant literature and revising parts of the manuscript for clarity and readability\.

The authors take full responsibility for the final manuscript and supplementary materials, including the mathematical claims, experimental results, code, and references\.

## References

- Agresti \(2013\)A\. Agresti\.*Categorical Data Analysis*\. Third edition\. Wiley, 2013\.
- Antoniak and Mimno \(2018\)M\. Antoniak and D\. Mimno\. Evaluating the stability of embedding\-based word similarities\.*Transactions of the Association for Computational Linguistics*, 6:107–119, 2018\.[https://doi\.org/10\.1162/tacl\_a\_00008](https://doi.org/10.1162/tacl_a_00008)\.
- Austin et al\. \(2021\)J\. Austin, D\. D\. Johnson, J\. Ho, D\. Tarlow, and R\. van den Berg\. Structured denoising diffusion models in discrete state\-spaces\. In*Advances in Neural Information Processing Systems*, volume 34, 2021\.[https://arxiv\.org/abs/2107\.03006](https://arxiv.org/abs/2107.03006)\.
- Brunet et al\. \(2019\)M\.\-E\. Brunet, C\. Alkalay\-Houlihan, A\. Anderson, and R\. Zemel\. Understanding the origins of bias in word embeddings\. In*Proceedings of ICML*, PMLR 97:803–811, 2019\.[https://proceedings\.mlr\.press/v97/brunet19a\.html](https://proceedings.mlr.press/v97/brunet19a.html)\.
- Chen and Goodman \(1999\)S\. F\. Chen and J\. Goodman\. An empirical study of smoothing techniques for language modeling\.*Computer Speech & Language*, 13\(4\):359–394, 1999\.[https://doi\.org/10\.1006/csla\.1999\.0128](https://doi.org/10.1006/csla.1999.0128)\.
- Church and Hanks \(1990\)K\. W\. Church and P\. Hanks\. Word association norms, mutual information, and lexicography\.*Computational Linguistics*, 16\(1\):22–29, 1990\.[https://aclanthology\.org/J90\-1003/](https://aclanthology.org/J90-1003/)\.
- Cochran \(1977\)W\. G\. Cochran\.*Sampling Techniques*\. Third edition\. Wiley, 1977\.
- D’Amour et al\. \(2022\)A\. D’Amour, K\. Heller, D\. Moldovan, B\. Adlam, B\. Alipanahi, A\. Beutel, C\. Chen, J\. Deaton, J\. Eisenstein, M\. D\. Hoffman, F\. Hormozdiari, N\. Houlsby, S\. Hou, G\. Jerfel, A\. Karthikesalingam, M\. Lucic, Y\. Ma, C\. McLean, D\. Mincu, A\. Mitani, A\. Montanari, Z\. Nado, V\. Natarajan, C\. Nielson, T\. F\. Osborne, R\. Raman, K\. Ramasamy, R\. Sayres, J\. Schrouff, M\. Seneviratne, S\. Sequeira, H\. Suresh, V\. Veitch, M\. Vladymyrov, X\. Wang, K\. Webster, S\. Yadlowsky, T\. Yun, X\. Zhai, and D\. Sculley\. Underspecification presents challenges for credibility in modern machine learning\.*Journal of Machine Learning Research*, 23\(226\):1–61, 2022\.[https://jmlr\.org/papers/v23/20\-1335\.html](https://jmlr.org/papers/v23/20-1335.html)\.
- Dunning \(1993\)T\. Dunning\. Accurate methods for the statistics of surprise and coincidence\.*Computational Linguistics*, 19\(1\):61–74, 1993\.[https://aclanthology\.org/J93\-1003/](https://aclanthology.org/J93-1003/)\.
- Durbin \(1959\)J\. Durbin\. A note on the application of Quenouille’s method of bias reduction to the estimation of ratios\.*Biometrika*, 46\(3–4\):477–480, 1959\.[https://doi\.org/10\.1093/biomet/46\.3\-4\.477](https://doi.org/10.1093/biomet/46.3-4.477)\.
- Efron \(1979\)B\. Efron\. Bootstrap methods: Another look at the jackknife\.*The Annals of Statistics*, 7\(1\):1–26, 1979\.[https://doi\.org/10\.1214/aos/1176344552](https://doi.org/10.1214/aos/1176344552)\.
- Efron \(1982\)B\. Efron\.*The Jackknife, the Bootstrap and Other Resampling Plans*\. CBMS\-NSF Regional Conference Series in Applied Mathematics, volume 38\. SIAM, 1982\.[https://doi\.org/10\.1137/1\.9781611970319](https://doi.org/10.1137/1.9781611970319)\.
- Eldan and Li \(2023\)R\. Eldan and Y\. Li\. TinyStories: How small can language models be and still speak coherent English?*arXiv:2305\.07759*, 2023\.[https://arxiv\.org/abs/2305\.07759](https://arxiv.org/abs/2305.07759)\.
- Field and Welsh \(2007\)C\. A\. Field and A\. H\. Welsh\. Bootstrapping clustered data\.*Journal of the Royal Statistical Society: Series B \(Statistical Methodology\)*, 69\(3\):369–390, 2007\.[https://doi\.org/10\.1111/j\.1467\-9868\.2007\.00593\.x](https://doi.org/10.1111/j.1467-9868.2007.00593.x)\.
- Giordano et al\. \(2019\)R\. Giordano, W\. Stephenson, R\. Liu, M\. I\. Jordan, and T\. Broderick\. A Swiss Army infinitesimal jackknife\. In*Proceedings of AISTATS*, PMLR 89:1139–1147, 2019\.[https://proceedings\.mlr\.press/v89/giordano19a\.html](https://proceedings.mlr.press/v89/giordano19a.html)\.
- Good \(1953\)I\. J\. Good\. The population frequencies of species and the estimation of population parameters\.*Biometrika*, 40\(3–4\):237–264, 1953\.[https://doi\.org/10\.1093/biomet/40\.3\-4\.237](https://doi.org/10.1093/biomet/40.3-4.237)\.
- Gries \(2022\)S\. Th\. Gries\. Toward more careful corpus statistics: Uncertainty estimates for frequencies, dispersions, association measures, and more\.*Research Methods in Applied Linguistics*, 1\(1\):100002, 2022\.[https://doi\.org/10\.1016/j\.rmal\.2021\.100002](https://doi.org/10.1016/j.rmal.2021.100002)\.
- Hampel \(1974\)F\. R\. Hampel\. The influence curve and its role in robust estimation\.*Journal of the American Statistical Association*, 69\(346\):383–393, 1974\.[https://doi\.org/10\.1080/01621459\.1974\.10482962](https://doi.org/10.1080/01621459.1974.10482962)\.
- Hermann et al\. \(2015\)K\. M\. Hermann, T\. Kočiský, E\. Grefenstette, L\. Espeholt, W\. Kay, M\. Suleyman, and P\. Blunsom\. Teaching machines to read and comprehend\. In*Advances in Neural Information Processing Systems*, volume 28, pages 1693–1701, 2015\.[https://proceedings\.neurips\.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96\-Abstract\.html](https://proceedings.neurips.cc/paper/2015/hash/afdec7005cc9f14302cd0474fd0f3c96-Abstract.html)\.
- Hoeffding \(1963\)W\. Hoeffding\. Probability inequalities for sums of bounded random variables\.*Journal of the American Statistical Association*, 58\(301\):13–30, 1963\.[https://doi\.org/10\.1080/01621459\.1963\.10500830](https://doi.org/10.1080/01621459.1963.10500830)\.
- Ilyas et al\. \(2022\)A\. Ilyas, S\. M\. Park, L\. Engstrom, G\. Leclerc, and A\. Madry\. Datamodels: Understanding predictions with data and data with predictions\. In*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pages 9525–9587, 2022\.[https://proceedings\.mlr\.press/v162/ilyas22a\.html](https://proceedings.mlr.press/v162/ilyas22a.html)\.
- Jaeckel \(1972\)L\. A\. Jaeckel\. The infinitesimal jackknife\. Bell Laboratories Memorandum MM 72\-1215\-11, 1972\.[https://faculty\.washington\.edu/fscholz/Reports/InfinitesimalJackknife\.pdf](https://faculty.washington.edu/fscholz/Reports/InfinitesimalJackknife.pdf)\.
- Kadkhodaie et al\. \(2024\)Z\. Kadkhodaie, F\. Guth, E\. P\. Simoncelli, and S\. Mallat\. Generalization in diffusion models arises from geometry\-adaptive harmonic representations\. In*International Conference on Learning Representations*, 2024\.[https://arxiv\.org/abs/2310\.02557](https://arxiv.org/abs/2310.02557)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. Liang\. Understanding black\-box predictions via influence functions\. In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pages 1885–1894, 2017\.[https://proceedings\.mlr\.press/v70/koh17a\.html](https://proceedings.mlr.press/v70/koh17a.html)\.
- Levy and Goldberg \(2014\)O\. Levy and Y\. Goldberg\. Neural word embedding as implicit matrix factorization\. In*Advances in Neural Information Processing Systems*, volume 27, pages 2177–2185, 2014\.[https://proceedings\.neurips\.cc/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4\-Abstract\.html](https://proceedings.neurips.cc/paper/2014/hash/b78666971ceae55a8e87efb7cbfd9ad4-Abstract.html)\.
- Lijffijt et al\. \(2016\)J\. Lijffijt, T\. Nevalainen, T\. Säily, P\. Papapetrou, K\. Puolamäki, and H\. Mannila\. Significance testing of word frequencies in corpora\.*Digital Scholarship in the Humanities*, 31\(2\):374–397, 2016\.[https://doi\.org/10\.1093/llc/fqu064](https://doi.org/10.1093/llc/fqu064)\.
- Lou et al\. \(2024\)A\. Lou, C\. Meng, and S\. Ermon\. Discrete diffusion modeling by estimating the ratios of the data distribution\. In*Proceedings of ICML*, PMLR 235:32819–32848, 2024\.[https://proceedings\.mlr\.press/v235/lou24a\.html](https://proceedings.mlr.press/v235/lou24a.html)\.
- Marx et al\. \(2020\)C\. Marx, F\. Calmon, and B\. Ustun\. Predictive multiplicity in classification\. In*Proceedings of the 37th International Conference on Machine Learning*, volume 119 of*Proceedings of Machine Learning Research*, pages 6765–6774, 2020\.[https://proceedings\.mlr\.press/v119/marx20a\.html](https://proceedings.mlr.press/v119/marx20a.html)\.
- Merity et al\. \(2017\)S\. Merity, C\. Xiong, J\. Bradbury, and R\. Socher\. Pointer sentinel mixture models\. In*Proceedings of ICLR*, 2017\.[https://openreview\.net/forum?id=Byj72udxe](https://openreview.net/forum?id=Byj72udxe)\.
- Nguyen \(2024\)T\. Nguyen\. Understanding transformers viaNN\-gram statistics\. In*Advances in Neural Information Processing Systems*, volume 37, 2024\.[https://arxiv\.org/abs/2407\.12034](https://arxiv.org/abs/2407.12034)\.
- Pananjady et al\. \(2024\)A\. Pananjady, V\. Muthukumar, and A\. Thangaraj\. Just wing it: Near\-optimal estimation of missing mass in a Markovian sequence\.*Journal of Machine Learning Research*, 25\(312\):1–43, 2024\.[https://jmlr\.org/papers/v25/24\-0511\.html](https://jmlr.org/papers/v25/24-0511.html)\.
- Pennington et al\. \(2014\)J\. Pennington, R\. Socher, and C\. D\. Manning\. GloVe: Global vectors for word representation\. In*Proceedings of EMNLP*, pages 1532–1543, 2014\.[https://doi\.org/10\.3115/v1/D14\-1162](https://doi.org/10.3115/v1/D14-1162)\.
- Quenouille \(1956\)M\. H\. Quenouille\. Notes on bias in estimation\.*Biometrika*, 43\(3–4\):353–360, 1956\.[https://doi\.org/10\.1093/biomet/43\.3\-4\.353](https://doi.org/10.1093/biomet/43.3-4.353)\.
- Radford et al\. \(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. Sutskever\. Language models are unsupervised multitask learners\. OpenAI technical report, 2019\.[https://cdn\.openai\.com/better\-language\-models/language\-models\.pdf](https://cdn.openai.com/better-language-models/language-models.pdf)\.
- Sahoo et al\. \(2024\)S\. S\. Sahoo, M\. Arriola, Y\. Schiff, A\. Gokaslan, E\. Marroquin, J\. T\. Chiu, A\. M\. Rush, and V\. Kuleshov\. Simple and effective masked diffusion language models\. In*Advances in Neural Information Processing Systems*, volume 37, pages 130136–130184, 2024\.[doi:10\.52202/079017\-4135](https://doi.org/10.52202/079017-4135)\.
- See et al\. \(2017\)A\. See, P\. J\. Liu, and C\. D\. Manning\. Get to the point: Summarization with pointer\-generator networks\. In*Proceedings of ACL*, pages 1073–1083, 2017\.[https://doi\.org/10\.18653/v1/P17\-1099](https://doi.org/10.18653/v1/P17-1099)\.
- Vallebueno et al\. \(2024\)A\. Vallebueno, C\. Handan\-Nader, C\. D\. Manning, and D\. E\. Ho\. Statistical uncertainty in word embeddings: GloVe\-V\. In*Proceedings of EMNLP*, pages 9032–9047, 2024\.[https://doi\.org/10\.18653/v1/2024\.emnlp\-main\.510](https://doi.org/10.18653/v1/2024.emnlp-main.510)\.
- van der Vaart \(1998\)A\. W\. van der Vaart\.*Asymptotic Statistics*\. Cambridge University Press, 1998\.[https://doi\.org/10\.1017/CBO9780511802256](https://doi.org/10.1017/CBO9780511802256)\.
- Wager et al\. \(2014\)S\. Wager, T\. Hastie, and B\. Efron\. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife\.*Journal of Machine Learning Research*, 15\(48\):1625–1651, 2014\.[https://jmlr\.org/papers/v15/wager14a\.html](https://jmlr.org/papers/v15/wager14a.html)\.
- Wald \(1944\)A\. Wald\. On cumulative sums of random variables\.*The Annals of Mathematical Statistics*, 15\(3\):283–296, 1944\.[https://doi\.org/10\.1214/aoms/1177731235](https://doi.org/10.1214/aoms/1177731235)\.
- Wang et al\. \(2026\)B\. Wang, J\. A\. Zavatone\-Veth, and C\. Pehlevan\. A random matrix theory perspective on the consistency of diffusion models\. In*Proceedings of ICML*, PMLR 306:132077–132128, 2026\.[https://proceedings\.mlr\.press/v306/wang26kf\.html](https://proceedings.mlr.press/v306/wang26kf.html)\.
- Zhang et al\. \(2024\)H\. Zhang, J\. Zhou, Y\. Lu, M\. Guo, P\. Wang, L\. Shen, and Q\. Qu\. The emergence of reproducibility and consistency in diffusion models\. In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 60558–60590, 2024\.[https://proceedings\.mlr\.press/v235/zhang24cn\.html](https://proceedings.mlr.press/v235/zhang24cn.html)\.

Appendix

## Appendix AProof and Exact Identities

#### Disagreement identity\.

First we fix the auxiliary quantities and an evaluation inputZZ\. LetYAY\_\{A\}andYBY\_\{B\}be the vectors of predictions at the hidden positions, each divided bymax⁡\{1,\|H\|\}\\sqrt\{\\max\\\{1,\|H\|\\\}\}\. Then‖YA−YB‖2\\\|Y\_\{A\}\-Y\_\{B\}\\\|^\{2\}is the squared output difference in \([1](https://arxiv.org/html/2610.12987#S2.E1)\)\. Under this conditioning, the two predictions are independent\. Writing their means asμA\\mu\_\{A\}andμB\\mu\_\{B\}and expanding the square gives

𝔼​‖YA−YB‖2=𝔼​‖YA−μA‖2\+𝔼​‖YB−μB‖2\+‖μA−μB‖2\.\\mathbb\{E\}\\\|Y\_\{A\}\-Y\_\{B\}\\\|^\{2\}=\\mathbb\{E\}\\\|Y\_\{A\}\-\\mu\_\{A\}\\\|^\{2\}\+\\mathbb\{E\}\\\|Y\_\{B\}\-\\mu\_\{B\}\\\|^\{2\}\+\\\|\\mu\_\{A\}\-\\mu\_\{B\}\\\|^\{2\}\.At equal corpus sizes, the two outputs have the same distribution, so their means agree and the two variance terms are equal\. This holds even if the reconstructor is biased\. The last term therefore vanishes, and averaging over evaluation inputs gives \([6](https://arxiv.org/html/2610.12987#S3.E6)\)\. When the corpus sizes differ, the means need not agree; averaging the full expression gives \([15](https://arxiv.org/html/2610.12987#A1.E15)\)\.

#### From count covariance to output covariance\.

We condition on the auxiliary quantities\. Letθ^n\\widehat\{\\theta\}\_\{n\}be the vector containing all lag counts, andΩn=Cov𝒟⁡\(θ^n\)\\Omega\_\{n\}=\\operatorname\{Cov\}\_\{\\mathcal\{D\}\}\(\\widehat\{\\theta\}\_\{n\}\)\. For a fixed evaluation inputZZ, letFZ​\(θ\)F\_\{Z\}\(\\theta\)contain the predictions at hidden positions, divided bymax⁡\{1,\|H\|\}\\sqrt\{\\max\\\{1,\|H\|\\\}\}, when the count vector isθ\\theta\. Define

θn∘=𝔼𝒟​θ^n,Jn​\(Z\)=Dθ​FZ​\(θn∘\)\.\\theta\_\{n\}^\{\\circ\}=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\widehat\{\\theta\}\_\{n\},\\qquad J\_\{n\}\(Z\)=D\_\{\\theta\}F\_\{Z\}\(\\theta\_\{n\}^\{\\circ\}\)\.The first\-order expression is

𝔇n,q≃2​𝔼X,M​tr⁡\{Jn​\(Z\)​Ωn​Jn​\(Z\)⊤\}\.\\mathfrak\{D\}\_\{n,q\}\\simeq 2\\mathbb\{E\}\_\{X,M\}\\operatorname\{tr\}\\\{J\_\{n\}\(Z\)\\Omega\_\{n\}J\_\{n\}\(Z\)^\{\\top\}\\\}\.\(14\)This follows from the multivariate delta method\([van der Vaart, 1998](https://arxiv.org/html/2610.12987#bib.bib38), Chapter 3\)\. Expanding around the mean count vector gives

FZ​\(θ^n\)−FZ​\(θn∘\)≃Jn​\(Z\)​\(θ^n−θn∘\)\.F\_\{Z\}\(\\widehat\{\\theta\}\_\{n\}\)\-F\_\{Z\}\(\\theta\_\{n\}^\{\\circ\}\)\\simeq J\_\{n\}\(Z\)\(\\widehat\{\\theta\}\_\{n\}\-\\theta\_\{n\}^\{\\circ\}\)\.The linear term has covarianceJn​\(Z\)​Ωn​Jn​\(Z\)⊤J\_\{n\}\(Z\)\\Omega\_\{n\}J\_\{n\}\(Z\)^\{\\top\}\. Its expected squared norm is the trace of this covariance, so \([6](https://arxiv.org/html/2610.12987#S3.E6)\) gives \([14](https://arxiv.org/html/2610.12987#A1.E14)\)\. We now derive the count covariance under source sampling and show that the omitted terms areo⁡\(n−1\)o\(n^\{\-1\}\)\.

#### Source counts at a fixed chunk budget\.

Longer sources use more of the chunk budget, leaving room for fewer subsequent sources\. The resulting correction has the residual form familiar from ratio estimation\([Cochran, 1977](https://arxiv.org/html/2610.12987#bib.bib7), Chapter 6\)\.

Let\(Tg,Sg\)\(T\_\{g\},S\_\{g\}\)be iid contributions from complete sources, whereTgT\_\{g\}is the number of chunks andSgS\_\{g\}contains the counts at all offsets\. Assume1≤Tg≤Tmax1\\leq T\_\{g\}\\leq T\_\{\\max\}

τ=𝔼​Tg,s=𝔼​Sg,Gn=nτ,Bn=Gn​s\.\\tau=\\mathbb\{E\}T\_\{g\},\\qquad s=\\mathbb\{E\}S\_\{g\},\\qquad G\_\{n\}=\\frac\{n\}\{\\tau\},\\qquad B\_\{n\}=G\_\{n\}s\.Sampling stops at

N⁡\(n\)=min⁡\{m:∑g=1mTg≥n\}\.N\(n\)=\\min\\left\\\{m:\\sum\_\{g=1\}^\{m\}T\_\{g\}\\geq n\\right\\\}\.We shorten the final source if necessary to obtain exactlynnchunks\. SinceL,R,TmaxL,R,T\_\{\\max\}are fixed, this changes the count vector by a uniformly bounded amount\. With

Ag=Sg−Tgτ​s,A\_\{g\}=S\_\{g\}\-\\frac\{T\_\{g\}\}\{\\tau\}s,we have

θ^n−Bn=∑g=1N⁡\(n\)Ag\+dn,‖dn‖≤C\.\\widehat\{\\theta\}\_\{n\}\-B\_\{n\}=\\sum\_\{g=1\}^\{N\(n\)\}A\_\{g\}\+d\_\{n\},\\qquad\\\|d\_\{n\}\\\|\\leq C\.The residual satisfies𝔼​Ag=0\\mathbb\{E\}A\_\{g\}=0\.Ag/τA\_\{g\}/\\tauis the influence function of the count rates/τs/\\tau\([Hampel, 1974](https://arxiv.org/html/2610.12987#bib.bib18)\)\.

The event\{N\(n\)≥g\}\\\{N\(n\)\\geq g\\\}depends only on sources precedinggg\. Then, the stopped sum of theAgA\_\{g\}has mean zero and covariance

Cov⁡\(∑g=1N⁡\(n\)Ag\)=𝔼​N​\(n\)​𝔼g​\[Ag​Ag⊤\]\.\\operatorname\{Cov\}\\left\(\\sum\_\{g=1\}^\{N\(n\)\}A\_\{g\}\\right\)=\\mathbb\{E\}N\(n\)\\,\\mathbb\{E\}\_\{g\}\[A\_\{g\}A\_\{g\}^\{\\top\}\]\.The bounded overshoot and Wald’s identity\([Wald, 1944](https://arxiv.org/html/2610.12987#bib.bib40)\)give𝔼​N​\(n\)=Gn\+O⁡\(1\)\\mathbb\{E\}N\(n\)=G\_\{n\}\+O\(1\)\. Sincednd\_\{n\}is bounded, its covariance with the stopped sum isO⁡\(n\)O\(\\sqrt\{n\}\)by Cauchy\-Schwarz\. Hence

Ωn=Gn​𝔼g​\[Ag​Ag⊤\]\+o⁡\(n\)\.\\Omega\_\{n\}=G\_\{n\}\\mathbb\{E\}\_\{g\}\[A\_\{g\}A\_\{g\}^\{\\top\}\]\+o\(n\)\.Using one source vector for all offsets includes correlations between lags as well as dependence within each source\.

Taking expectations also givesθn∘=Bn\+O⁡\(1\)\\theta\_\{n\}^\{\\circ\}=B\_\{n\}\+O\(1\)\. Under the positive\-row assumption below, moving the expansion point fromθn∘\\theta\_\{n\}^\{\\circ\}toBnB\_\{n\}changes the derivative byO⁡\(n−2\)O\(n^\{\-2\}\)\. Thus the derivatives evaluated atBδ,n=Gn​sδB\_\{\\delta,n\}=G\_\{n\}s\_\{\\delta\}in \([8](https://arxiv.org/html/2610.12987#S3.E8)\) give the same first\-order result\.

#### Ratio expansion\.

A perturbationϵ​Ag,δ\\epsilon A\_\{g,\\delta\}addsϵ​Ag,δ​\(b,a\)\\epsilon A\_\{g,\\delta\}\(b,a\)to the smoothed numerator andϵ​Ag,δ​\(b,\+\)\\epsilon A\_\{g,\\delta\}\(b,\+\)to its denominator\. Differentiating the ratio atϵ=0\\epsilon=0gives \([8](https://arxiv.org/html/2610.12987#S3.E8)\)\.

We keepα\>0\\alpha\>0,K,L,R,TmaxK,L,R,T\_\{\\max\}, and the auxiliary quantities fixed asnngrows\. Assume that every queried population row has positive mass\. Its denominator atBδ,nB\_\{\\delta,n\}is then proportional tonn\. When the empirical denominator is also bounded below by a positive constant timesnn, the first derivative isO⁡\(n−1\)O\(n^\{\-1\}\)and the Hessian isO⁡\(n−2\)O\(n^\{\-2\}\)throughout the line segment between the population and empirical counts\.

To control small empirical denominators, fix a queried row\(δ,b\)\(\\delta,b\)and write

Yg=Sg,δ​\(b,\+\),0≤Yg≤Cδ=Tmax​\(L−\|δ\|\),μδ​b=𝔼​Yg\>0\.Y\_\{g\}=S\_\{g,\\delta\}\(b,\+\),\\qquad 0\\leq Y\_\{g\}\\leq C\_\{\\delta\}=T\_\{\\max\}\(L\-\|\\delta\|\),\\qquad\\mu\_\{\\delta b\}=\\mathbb\{E\}Y\_\{g\}\>0\.At least

mn=⌈nTmax⌉−1m\_\{n\}=\\left\\lceil\\frac\{n\}\{T\_\{\\max\}\}\\right\\rceil\-1sources are included in full before the final source\. The empirical row total is therefore at least∑g=1mnYg\\sum\_\{g=1\}^\{m\_\{n\}\}Y\_\{g\}\. These firstmnm\_\{n\}variables are iid; their inclusion does not depend on the later stopping index\. Hoeffding’s inequality\([Hoeffding, 1963](https://arxiv.org/html/2610.12987#bib.bib20)\)gives

Pr\{∑g=1mnYg<12mnμδ​b\}≤exp\(−mn​μδ​b22​Cδ2\)\.\\Pr\\left\\\{\\sum\_\{g=1\}^\{m\_\{n\}\}Y\_\{g\}<\\frac\{1\}\{2\}m\_\{n\}\\mu\_\{\\delta b\}\\right\\\}\\leq\\exp\\left\(\-\\frac\{m\_\{n\}\\mu\_\{\\delta b\}^\{2\}\}\{2C\_\{\\delta\}^\{2\}\}\\right\)\.For sufficiently largenn, the empirical denominator is thus bounded below by a positive constant timesnn, except on an event with exponentially small probability\. There are finitely many queried rows becauseK,L,RK,L,Rare fixed, so a union bound gives the same conclusion for all of them\.

The stopped count sum has bounded martingale increments and at mostnnterms\. Its fourth moment is thereforeO⁡\(n2\)O\(n^\{2\}\), and the bounded correctiondnd\_\{n\}gives

𝔼𝒟​‖θ^n−Bn‖4=O⁡\(n2\)\.\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\|\\widehat\{\\theta\}\_\{n\}\-B\_\{n\}\\\|^\{4\}=O\(n^\{2\}\)\.On the event where all queried denominators are sufficiently large, the Taylor remainder is bounded byC​n−2​‖θ^n−Bn‖2Cn^\{\-2\}\\\|\\widehat\{\\theta\}\_\{n\}\-B\_\{n\}\\\|^\{2\}\. On the exceptional event, both the predictions and their linear approximation are bounded: smoothing bounds each conditional row, the weights are fixed, and the count perturbation isO⁡\(n\)O\(n\)\. Together, these facts give anO⁡\(n−2\)O\(n^\{\-2\}\)bound on the expected squared norm of the Taylor remainder\.

The boundary correction contributesDθ​FZ​\(Bn\)​dnD\_\{\\theta\}F\_\{Z\}\(B\_\{n\}\)d\_\{n\}, whose norm isO⁡\(n−1\)O\(n^\{\-1\}\)\. Including this term in the remainder, we obtain

FZ​\(θ^n\)\\displaystyle F\_\{Z\}\(\\widehat\{\\theta\}\_\{n\}\)=FZ​\(Bn\)\+Dθ​FZ​\(Bn\)​∑g=1N⁡\(n\)Ag\+rn​\(Z\),\\displaystyle=F\_\{Z\}\(B\_\{n\}\)\+D\_\{\\theta\}F\_\{Z\}\(B\_\{n\}\)\\sum\_\{g=1\}^\{N\(n\)\}A\_\{g\}\+r\_\{n\}\(Z\),𝔼𝒟​‖rn​\(Z\)‖2\\displaystyle\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\|r\_\{n\}\(Z\)\\\|^\{2\}=O⁡\(n−2\)\.\\displaystyle=O\(n^\{\-2\}\)\.The bound is uniform over evaluation inputs under the stated assumptions\. Its constants depend on the positive population row masses and need not remain bounded as those masses approach zero\.

#### Output variance\.

Applying the row derivative to the reconstruction rule \([4](https://arxiv.org/html/2610.12987#S2.E4)\) giveshg,nh\_\{g,n\}\. Letug,n​\(Z\)u\_\{g,n\}\(Z\)be its vector of hidden outputs with the same normalization asFZF\_\{Z\}\. Then

ug,n​\(Z\)=Dθ​FZ​\(Bn\)​Ag,‖ug,n​\(Z\)‖2=‖hg,n​\(Z\)‖M2\.u\_\{g,n\}\(Z\)=D\_\{\\theta\}F\_\{Z\}\(B\_\{n\}\)A\_\{g\},\\qquad\\\|u\_\{g,n\}\(Z\)\\\|^\{2\}=\\\|h\_\{g,n\}\(Z\)\\\|\_\{M\}^\{2\}\.For fixedZZ, this contribution has mean zero and normO⁡\(n−1\)O\(n^\{\-1\}\)\. The same stopped\-sum calculation gives

Cov𝒟⁡\(∑g=1N⁡\(n\)ug,n​\(Z\)\)\\displaystyle\\operatorname\{Cov\}\_\{\\mathcal\{D\}\}\\left\(\\sum\_\{g=1\}^\{N\(n\)\}u\_\{g,n\}\(Z\)\\right\)=𝔼​N​\(n\)​𝔼g​\[ug,n​\(Z\)​ug,n​\(Z\)⊤\]\\displaystyle=\\mathbb\{E\}N\(n\)\\,\\mathbb\{E\}\_\{g\}\[u\_\{g,n\}\(Z\)u\_\{g,n\}\(Z\)^\{\\top\}\]=Gn​𝔼g​\[ug,n​\(Z\)​ug,n​\(Z\)⊤\]\+O⁡\(n−2\)\.\\displaystyle=G\_\{n\}\\mathbb\{E\}\_\{g\}\[u\_\{g,n\}\(Z\)u\_\{g,n\}\(Z\)^\{\\top\}\]\+O\(n^\{\-2\}\)\.The linear fluctuation has expected squared normO⁡\(n−1\)O\(n^\{\-1\}\), whereas𝔼𝒟​‖rn​\(Z\)‖2=O⁡\(n−2\)\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\|r\_\{n\}\(Z\)\\\|^\{2\}=O\(n^\{\-2\}\)\. Their mixed covariance is thereforeO\(n−3/2\)O\(n^\{\-3/2\}\)by Cauchy\-Schwarz\. Taking the trace, averaging over evaluation inputs, and using \([6](https://arxiv.org/html/2610.12987#S3.E6)\) gives

𝔇n,q=2​Gn​𝔼X,M​𝔼g​‖hg,n​\(Z\)‖M2\+o⁡\(n−1\),\\mathfrak\{D\}\_\{n,q\}=2G\_\{n\}\\mathbb\{E\}\_\{X,M\}\\mathbb\{E\}\_\{g\}\\\|h\_\{g,n\}\(Z\)\\\|\_\{M\}^\{2\}\+o\(n^\{\-1\}\),which proves Proposition[1](https://arxiv.org/html/2610.12987#Thmproposition1)\.

For unequal corpus sizes, the exact identity also contains the difference between the two mean predictions:

𝔇nA,nB,q=VnA,q\+VnB,q\+QnA,nB,q,\\mathfrak\{D\}\_\{n\_\{A\},n\_\{B\},q\}=V\_\{n\_\{A\},q\}\+V\_\{n\_\{B\},q\}\+Q\_\{n\_\{A\},n\_\{B\},q\},\(15\)where

Vn,q=𝔇n,q2,μn​\(Z\)=𝔼𝒟​f^n​\(Z\),V\_\{n,q\}=\\frac\{\\mathfrak\{D\}\_\{n,q\}\}\{2\},\\qquad\\mu\_\{n\}\(Z\)=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\widehat\{f\}\_\{n\}\(Z\),and

QnA,nB,q=𝔼X,M​‖μnA​\(Z\)−μnB​\(Z\)‖M2\.Q\_\{n\_\{A\},n\_\{B\},q\}=\\mathbb\{E\}\_\{X,M\}\\\|\\mu\_\{n\_\{A\}\}\(Z\)\-\\mu\_\{n\_\{B\}\}\(Z\)\\\|\_\{M\}^\{2\}\.Our first\-order prediction adds the two variance estimates:

D^nA,nB,qan=D^nA,qan\+D^nB,qan2\.\\widehat\{D\}^\{\\rm an\}\_\{n\_\{A\},n\_\{B\},q\}=\\frac\{\\widehat\{D\}^\{\\rm an\}\_\{n\_\{A\},q\}\+\\widehat\{D\}^\{\\rm an\}\_\{n\_\{B\},q\}\}\{2\}\.The mean term is lower order under the assumptions above, as we show next\. It is still part of the finite\-size target\. For equal sizes,Qn,n,q=0Q\_\{n,n,q\}=0exactly\.

#### Unequal\-size mean order\.

Letf∞f\_\{\\infty\}be the reconstruction obtained from the population conditional rows without smoothing\. Becauseα\\alphais fixed and the queried row masses are positive, the prediction evaluated atBnB\_\{n\}differs fromf∞f\_\{\\infty\}byO⁡\(n−1\)O\(n^\{\-1\}\)\. The stopped linear term has mean zero, and

‖𝔼𝒟​rn​\(Z\)‖≤\(𝔼𝒟​‖rn​\(Z\)‖2\)1/2=O⁡\(n−1\)\.\\\|\\mathbb\{E\}\_\{\\mathcal\{D\}\}r\_\{n\}\(Z\)\\\|\\leq\\bigl\(\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\|r\_\{n\}\(Z\)\\\|^\{2\}\\bigr\)^\{1/2\}=O\(n^\{\-1\}\)\.It follows that

‖μn​\(Z\)−f∞​\(Z\)‖M=O⁡\(n−1\)\\\|\\mu\_\{n\}\(Z\)\-f\_\{\\infty\}\(Z\)\\\|\_\{M\}=O\(n^\{\-1\}\)uniformly over evaluation inputs\. Hence

QnA,nB,q\\displaystyle Q\_\{n\_\{A\},n\_\{B\},q\}=O⁡\(\(nA−1\+nB−1\)2\)\\displaystyle=O\\bigl\(\(n\_\{A\}^\{\-1\}\+n\_\{B\}^\{\-1\}\)^\{2\}\\bigr\)=o⁡\(nA−1\+nB−1\)\\displaystyle=o\(n\_\{A\}^\{\-1\}\+n\_\{B\}^\{\-1\}\)as both sizes grow\. Thus the mean term is smaller than thenA−1\+nB−1n\_\{A\}^\{\-1\}\+n\_\{B\}^\{\-1\}variance scale\.

#### Multinomial variance\.

Under the fixed\-count model in Section[3](https://arxiv.org/html/2610.12987#S3), a row count vectorCCis the sum ofNNindependent one\-hot draws with meanπ\\pi\. Therefore

Cov⁡\(C\)=N⁡\{diag⁡\(π\)−π​π⊤\}\\operatorname\{Cov\}\(C\)=N\\\{\\operatorname\{diag\}\(\\pi\)\-\\pi\\pi^\{\\top\}\\\}\([Agresti, 2013](https://arxiv.org/html/2610.12987#bib.bib1), Chapter 1\)\. Adding the fixed vectorα​p\\alpha pdoes not change the covariance, while division byN\+αN\+\\alphamultiplies it by\(N\+α\)−2\(N\+\\alpha\)^\{\-2\}\. The difference between two independent copies has twice this covariance, with trace

2​N\(N\+α\)2​\(1−‖π‖2\)\.\\frac\{2N\}\{\(N\+\\alpha\)^\{2\}\}\\bigl\(1\-\\\|\\pi\\\|^\{2\}\\bigr\)\.This gives the row factor in \([12](https://arxiv.org/html/2610.12987#S3.E12)\)\. Under the assumed independence across lags, the cross terms in the squared output difference have zero expectation\. The formula is therefore exact for this model when evaluated at its population quantities\. Substituting reference estimates introduces a separate estimation error\.

#### Source\-split jackknife\.

Fix the target sizennand condition on the auxiliary quantities and evaluation inputs\. Suppose the forecast fromSSiid sources satisfies

𝔼​FS=F∞\+bS\+rS,rS=o⁡\(S−1\)\.\\mathbb\{E\}F\_\{S\}=F\_\{\\infty\}\+\\frac\{b\}\{S\}\+r\_\{S\},\\qquad r\_\{S\}=o\(S^\{\-1\}\)\.Partition the sources independently of their contents into groups of deterministic sizesS1,S2S\_\{1\},S\_\{2\}, where

S1\+S2=S,SjS⟶cj∈\(0,1\)\.S\_\{1\}\+S\_\{2\}=S,\\qquad\\frac\{S\_\{j\}\}\{S\}\\longrightarrow c\_\{j\}\\in\(0,1\)\.LetFSj\(j\)F\_\{S\_\{j\}\}^\{\(j\)\}be the forecast computed from groupjj, and define

FJ=2​FS−∑j=12SjS​FSj\(j\)\.F\_\{J\}=2F\_\{S\}\-\\sum\_\{j=1\}^\{2\}\\frac\{S\_\{j\}\}\{S\}F\_\{S\_\{j\}\}^\{\(j\)\}\.Each group has the marginal distribution of an iid sample of its stated size\. Linearity of expectation therefore gives

𝔼​FJ\\displaystyle\\mathbb\{E\}F\_\{J\}=F∞\+2​bS−∑j=12SjS​bSj\+2​rS−∑j=12SjS​rSj\\displaystyle=F\_\{\\infty\}\+\\frac\{2b\}\{S\}\-\\sum\_\{j=1\}^\{2\}\\frac\{S\_\{j\}\}\{S\}\\frac\{b\}\{S\_\{j\}\}\+2r\_\{S\}\-\\sum\_\{j=1\}^\{2\}\\frac\{S\_\{j\}\}\{S\}r\_\{S\_\{j\}\}=F∞\+o⁡\(S−1\)\.\\displaystyle=F\_\{\\infty\}\+o\(S^\{\-1\}\)\.Both group estimates share data with the full\-bank estimate, but this expectation calculation requires no independence between them\. The weightsSj/SS\_\{j\}/Scancel the leading bias even when the groups differ in size\. Splitting individual chunks would change the sampling unit and would no longer match this iid\-source argument\.

This is the classical bias\-reduction argument of[Quenouille \(1956\)](https://arxiv.org/html/2610.12987#bib.bib33), applied to the reference forecast\. Its use for ratio estimators has a direct precedent in[Durbin \(1959\)](https://arxiv.org/html/2610.12987#bib.bib10)\. HereF∞F\_\{\\infty\}is the limiting reference forecast at the chosen target sizenn\. It can still differ from the exact disagreement of the nonlinear reconstructor\.

The calculation assumes a fixed number of iid sources and a regular bias expansion\. Our reference banks end at a fixed chunk budget, so their source counts are random and their final sources may be shortened\. The calculation above does not by itself establish the same bias order for that sampling rule; we evaluate the correction under that protocol in the experiments\.

A row absent from the whole bank is also absent from both groups, so the correction cannot recover its contribution to the influence variance\. We keep the signs of individual contributions and set only a negative final scalar forecast to zero\. The bias expansion above applies before this final clipping\.

## Appendix BData Protocol and Calibration Evidence

### B\.1 Finite\-reference correction

For each corpus, we use a block of 128,000 distinct texts after the 64,000\-source prefix used in the previous experiments \(see Appendix[B\.5](https://arxiv.org/html/2610.12987#A2.SS5)\)\. From this block, we construct 20 source\-disjoint A/B pairs and six independent reference banks\. Each corpus in a pair contains 4,096 chunks, and each reference bank contains 8,192 chunks\. Smaller sizes are obtained from prefixes\. Source IDs, text hashes, and chunk hashes verify separation between A and B and across data roles\. The auxiliary statistics and 128 evaluation chunks are shared with the matched comparison\.

We writemmfor the reference size andnnfor the size of each corpus in an A/B pair\. Throughout this experiment,q=0\.5q=0\.5,R=2R=2, andα=5\\alpha=5, with a ridge penalty of0\.010\.01\. At each reference size, a random permutation partitions the sources into two groups, with all included chunks from a source assigned to the same group\. We recompute the conditional rows and covariance estimates in each group for the same target sizenn, then combine the forecasts using the source\-count weights in \([13](https://arxiv.org/html/2610.12987#S3.E13)\)\. The correction is evaluated atm∈\{1024,2048,4096\}m\\in\\\{1024,2048,4096\\\}\. The uncorrected forecast atm=8192m=8192provides a comparison with a larger reference bank\.

The correction was checked in three simulation controls atα=5\\alpha=5before the text experiment\. The full source law, its jackknife correction, and the multinomial law were specified before the A/B outcomes were inspected\. The multinomial correction and the additional baselines were examined after the outcomes were available\. Section[B\.3](https://arxiv.org/html/2610.12987#A2.SS3)evaluates the multinomial correction on a disjoint source allocation, with forecasts saved before scoring\.

#### Error measure and uncertainty\.

LetFc,b,m,nhF\_\{c,b,m,n\}^\{h\}denote the forecast from methodhhfor corpuscc, reference bankbb, reference sizemm, and target sizenn\. LetD¯c,n\\bar\{D\}\_\{c,n\}be the mean observed disagreement across the 20 A/B pairs\. For reference banks smaller than the target corpora, we summarize error over

𝒫=\{\(1024,2048\),\(1024,4096\),\(2048,4096\)\}\\mathcal\{P\}=\\\{\(1024,2048\),\(1024,4096\),\(2048,4096\)\\\}as

Eh=13⋅3⋅6​∑c∑\(m,n\)∈𝒫∑b=16\|log⁡Fc,b,m,nhD¯c,n\|\.E^\{h\}=\\frac\{1\}\{3\\cdot 3\\cdot 6\}\\sum\_\{c\}\\sum\_\{\(m,n\)\\in\\mathcal\{P\}\}\\sum\_\{b=1\}^\{6\}\\left\|\\log\\frac\{F\_\{c,b,m,n\}^\{h\}\}\{\\bar\{D\}\_\{c,n\}\}\\right\|\.The three corpora, three size combinations, and six reference banks receive equal weight\.

We estimate uncertainty with 5,000 bootstrap draws\. Within each corpus, a draw resamples six bank indices and twenty pair indices independently\. The same indices are then used for every method and every reference and target size\. We recompute the mean observed disagreements, the errors, and their paired differences in each draw\. This accounts for the dependence between methods and between measurements at nested sizes\. The intervals reflect variation across reference banks and A/B pairs only; the corpora, auxiliary data, evaluation inputs, and masks are held fixed\.

Table[2](https://arxiv.org/html/2610.12987#S4.T2)reports the errors over𝒫\\mathcal\{P\}\. The paired difference between source jackknife and the uncorrected source law is−0\.265​\[−0\.267,−0\.263\]\-0\.265\\ \[\-0\.267,\-0\.263\]\. The difference between source jackknife and uncorrected multinomial is−0\.0017​\[−0\.0052,0\.0020\]\-0\.0017\\ \[\-0\.0052,0\.0020\]\. Applying the correction to the multinomial law reduces error further: source\-jackknife error minus multinomial\-jackknife error is0\.119​\[0\.114,0\.124\]0\.119\\ \[0\.114,0\.124\]\. Figure[2](https://arxiv.org/html/2610.12987#A2.F2)shows the full grid, includingm\>nm\>n, where some corrections overpredict\.

Figure 2:Forecast calibration across reference and target sizes\. Rows correspond to target sizennand columns to corpora; sizes count 64\-token chunks\. Each point is the mean forecast across six reference banks divided by the mean observed disagreement across 20 A/B pairs\. The dotted line marks exact calibration, with values below one indicating underprediction and values above one indicating overprediction\. Error bars are 95% intervals obtained by resampling banks and A/B pairs independently, with auxiliary data, evaluation inputs, and masks fixed\. Curves use reference sizesm∈\{1024,2048,4096\}m\\in\\\{1024,2048,4096\\\}; stars show the uncorrected forecast atm=8192m=8192\.

### B\.2 Bootstrap and calibration baselines

We compare additional forecasts on the same twenty\-pair data\. These comparisons were designed after the A/B outcomes were available\. Before computing these forecasts, we fixed the number of bootstrap resamples, the division of reference sources into two groups, and the multiplier1\.251\.25used to test a uniform 25% increase in the uncorrected source and multinomial forecasts\.

#### Source bootstrap and its correction\.

Following the cluster bootstrap\([Efron, 1979](https://arxiv.org/html/2610.12987#bib.bib11);[Field and Welsh, 2007](https://arxiv.org/html/2610.12987#bib.bib14)\), we sample sources uniformly with replacement until reachingnnchunks, shortening only the final source when necessary\. Each sampled source contributes its chunks and all associated pair counts at every lag\. We rebuild the smoothed conditional rows on each draw and estimate disagreement as twice the unbiased sample output variance on the common evaluation inputs\.

Three independent batches of sixteen replicates give 48 replicates per forecast\. To apply the jackknife, we repeat the procedure on the two source groups used by the analytic corrections and combine the resulting scalar estimates according to \([13](https://arxiv.org/html/2610.12987#S3.E13)\)\. Across 216 forecasts per method, the median Monte Carlo standard errors are 0\.62% for the bootstrap and 1\.19% for its jackknife correction; the maxima are 2\.36% and 4\.17%\. All corrected scalar forecasts are positive\. The intervals below condition on these saved Monte Carlo draws\.

#### Direct disagreement and multiplicative corrections\.

A second approach constructs a reconstructor from each source group and measures their disagreement,DsplitD\_\{\\rm split\}\. If the groups containm1m\_\{1\}andm2m\_\{2\}chunks, output variance proportional to inverse corpus size and a size\-independent mean would imply

Fdirect​\(n\)=Dsplit​hn,h=21/m1\+1/m2\.F\_\{\\rm direct\}\(n\)=D\_\{\\rm split\}\\frac\{h\}\{n\},\\qquad h=\\frac\{2\}\{1/m\_\{1\}\+1/m\_\{2\}\}\.The harmonic meanhhaccounts for unequal group sizes\. Since smoothing can change both the mean and the variance, we also compare the constant forecastDsplitD\_\{\\rm split\}\.

We then ask whether a simple increase in scale can account for the improvement from the jackknife\. For this comparison, we multiply each uncorrected law by 1\.25\. Finally, we consider a conditional correction: apply the jackknife whenm≤nm\\leq nand use the uncorrected forecast whenm\>nm\>n\.

Table[5](https://arxiv.org/html/2610.12987#A2.T5)reports errors for the threem<nm<ncombinations, the full twelve\-combination grid, and the sixm\>nm\>ncombinations\. Restricting the jackknife tom≤nm\\leq nreduces the full\-grid error of the source law from 0\.103 to 0\.081\. The same restriction increases multinomial error from 0\.061 to 0\.071\. So whether to restrict the correction depends on the law\. A fixed multiplier also does well whenm<nm<nbut worse whenm\>nm\>n; Appendix[B\.3](https://arxiv.org/html/2610.12987#A2.SS3)tests whether fitted multipliers transfer\.

Table 5:Mean absolute log prediction error across reference banks, size combinations, and corpora, with 95% bootstrap intervals\. Each forecast is compared with the mean disagreement of the same twenty A/B pairs, and every method uses the same six reference banks per corpus\. Lower values indicate better calibration\. Them<nm<ncolumn averages the three combinations in𝒫\\mathcal\{P\}; the full grid averages all twelve combinations; and them\>nm\>ncolumn averages the six combinations with a larger reference bank than target corpus\. Banks, combinations, and corpora receive equal weight within each column\. The last two rows apply the jackknife only whenm≤nm\\leq nand use the uncorrected law otherwise\. Intervals resample whole banks and A/B pairs jointly across methods and sizes, conditional on auxiliary data, evaluation inputs, masks, and saved Monte Carlo draws\.

### B\.3 Calibration on disjoint source collections

To assess the multinomial correction on different corpus pairs and reference banks, we allocate unused texts from the same 128,000\-text block to six A/B pairs and three reference banks per corpus\. Each A or B corpus and each bank contains 2,048 chunks, with a 1,024\-chunk prefix\. Source IDs, text hashes, and chunk hashes verify separation across roles and from the other source allocations\. The auxiliary quantities and evaluation inputs are unchanged\. This allocation uses 10,638 TinyStories sources, 13,556 WikiText paragraphs, and 15,376 news articles\.

Before allocating these sources, we specified the comparison between multinomial and jackknife\-corrected multinomial forecasts atm=1024,n=2048m=1024,n=2048\. We use the same bank\-level absolute log error and 5,000 bootstrap draws as in the twenty\-pair experiment\. We also report the other three combinations ofm,n∈\{1024,2048\}m,n\\in\\\{1024,2048\\\}and the source\-law forecasts\. The multiplier and bootstrap comparisons were designed after these outcomes were available\.

Table 6:Calibration on six A/B pairs and three reference banks per corpus\. Entries are mean absolute log prediction errors with 95% bootstrap intervals; lower values indicate better calibration\. Sizes count 64\-token chunks\. Them<nm<ncolumn corresponds to\(m,n\)=\(1024,2048\)\(m,n\)=\(1024,2048\), the full grid averages all four combinations ofm,n∈\{1024,2048\}m,n\\in\\\{1024,2048\\\}, and them\>nm\>ncolumn corresponds to\(m,n\)=\(2048,1024\)\(m,n\)=\(2048,1024\)\. Each forecast is compared with the mean disagreement across the six pairs, with banks, size combinations, and corpora weighted equally\. Jackknife corrections are applied at every combination\. The pooled and cross\-corpus multipliers are selected using only the twenty\-pair results\. Intervals condition on auxiliary data, evaluation inputs, masks, and, where applicable, saved Monte Carlo draws\.Atm=1024,n=2048m=1024,n=2048, the multinomial correction reduces mean absolute log error from 0\.187 to 0\.077\. The paired difference, corrected minus uncorrected, is−0\.110​\[−0\.113,−0\.106\]\-0\.110\\ \[\-0\.113,\-0\.106\], a 58\.9% reduction\. The corrected forecast\-to\-observation ratios are 0\.897, 0\.913, and 0\.970 for TinyStories, WikiText\-103, and CNN/DailyMail, respectively\. So the improvement holds on new sources, although the corrected forecasts still run low\. Both experiments draw from the same three corpora\.

#### Bootstrap comparison atm\>nm\>n\.

We evaluate the source bootstrap atm=2048,n=1024m=2048,n=1024using all three banks and six pairs per corpus\. The resampling procedure and source\-split correction follow Section[B\.2](https://arxiv.org/html/2610.12987#A2.SS2), with three batches of sixteen replicates per component and the same source partitions as the analytic corrections\.

The correction reduces bootstrap error from 0\.106 to 0\.020\. Its paired error difference from multinomial jackknife is−0\.030​\[−0\.037,−0\.021\]\-0\.030\\ \[\-0\.037,\-0\.021\], and its difference from the multinomial forecast with the pooled calibration factor is−0\.019​\[−0\.032,−0\.006\]\-0\.019\\ \[\-0\.032,\-0\.006\]\. The corrected bootstrap forecast\-to\-observation ratios are 0\.964, 0\.987, and 1\.001 for TinyStories, WikiText\-103, and CNN/DailyMail, respectively\. Its lower pooled error does not imply an advantage in every corpus: the calibrated multiplier gives closer predictions on TinyStories\.

The median Monte Carlo standard errors are 0\.61% for the bootstrap and 1\.21% for its correction, with maxima of 1\.42% and 2\.40%\. All nine corrected scalar estimates are positive\. The bank\-and\-pair intervals condition on these Monte Carlo draws and on the shared auxiliary and evaluation data\.

The results also suggest combining the two corrections\. On the twenty\-pair grid, using multinomial jackknife whenm≤nm\\leq nand bootstrap jackknife whenm\>nm\>ngives error0\.043​\[0\.040,0\.047\]0\.043\\ \[0\.040,0\.047\]\. This rule was chosen after inspecting those results\. The six\-pair bootstrap comparison evaluates itsm\>nm\>ncomponent at\(m,n\)=\(2048,1024\)\(m,n\)=\(2048,1024\); it does not evaluate the rule across the full grid or establishm=nm=nas a general threshold\.

#### Transfer of multiplicative corrections\.

A fixed multiplier of 1\.25 gives lower multinomial error than the jackknife on them<nm<nsubset of the twenty\-pair experiment\. The same comparison atm=1024,n=2048m=1024,n=2048in the six\-pair experiment gives errors of 0\.041 and 0\.077, with paired difference−0\.036​\[−0\.055,−0\.017\]\-0\.036\\ \[\-0\.055,\-0\.017\]\. Across the four\-combination grid, however, the multiplier gives error 0\.098, compared with 0\.060 for the jackknife\. We therefore test whether a factor chosen on one set of results still works on new sources, corpora, and sizes\.

These comparisons were designed after both experiments had been scored\. Each factor is selected using only the calibration outcomes specified below\. For forecast\-to\-observation ratiosrj=Fj/D¯jr\_\{j\}=F\_\{j\}/\\bar\{D\}\_\{j\}, a constant minimizing mean absolute log error is

a=exp⁡\{−medianj⁡log⁡rj\}\.a=\\exp\\left\\\{\-\\operatorname\{median\}\_\{j\}\\log r\_\{j\}\\right\\\}\.Banks, size combinations, and corpora receive equal weight\. We evaluate a sweep from 0\.85 to 1\.50 in increments of 0\.005\. Figure[3](https://arxiv.org/html/2610.12987#A2.F3)shows the exact optimum for each size combination, computed from the median formula\.

Selecting one factor from the full twenty\-pair grid givesa=1\.118a=1\.118\. Applied unchanged to the six\-pair grid, this factor gives error 0\.045, compared with 0\.060 for the jackknife\. The paired difference is−0\.014​\[−0\.020,−0\.009\]\-0\.014\\ \[\-0\.020,\-0\.009\]\. Selecting the factor using only the twenty\-pairm<nm<nsubset givesa=1\.222a=1\.222and error 0\.030 atm=1024,n=2048m=1024,n=2048in the six\-pair experiment\.

To test transfer across corpora, we select a factor using two corpora from the twenty\-pair experiment and evaluate it on the third corpus in the six\-pair experiment\. The factors for TinyStories, WikiText\-103, and CNN/DailyMail are 1\.086, 1\.116, and 1\.130, respectively\. Their pooled error on the six\-pair grid is 0\.057, with paired difference from the jackknife−0\.003​\[−0\.008,0\.002\]\-0\.003\\ \[\-0\.008,0\.002\]\.

Two further comparisons use the twenty\-pair grid to test transfer across sizes\. We exclude either one target size or one ratiom/nm/nwhen selecting the factor, then evaluate it on the excluded combinations\. Both procedures give error 0\.062, compared with 0\.061 for the jackknife\. The paired differences are0\.001​\[−0\.003,0\.006\]0\.001\\ \[\-0\.003,0\.006\]and0\.001​\[−0\.004,0\.005\]0\.001\\ \[\-0\.004,0\.005\], respectively\.

All intervals use 5,000 resamples of whole reference banks and A/B pairs\. Measurements at nested sizes are carried together, and the factors are selected again within each resample\. The twenty\-pair and six\-pair experiments are resampled independently\. The size\-based comparisons separate the calibration and evaluation combinations, but use shared banks and A/B pairs\.

So a fitted multiplier does about as well as the jackknife, but only when past A/B disagreements are available to fit it; the jackknife needs none\.

Figure 3:Optimal multiplicative correction for the multinomial forecast at each reference–target size combination in the twenty\-pair experiment\. Each factor minimizes mean absolute log error across six reference banks, using the mean observed disagreement of twenty A/B pairs\. Shapes identify target sizenn, and lines connect combinations with the same target size\. Error bars are 95% intervals obtained by resampling banks and A/B pairs and selecting the factor again in each draw\. The dashed line marks the fixed factor 1\.25; the dotted line marks the pooled factor 1\.118 selected across all corpora and size combinations\. The factors shown here are selected using the observations in their respective combinations\. Table[6](https://arxiv.org/html/2610.12987#A2.T6)reports their pooled and cross\-corpus calibration alternatives on disjoint source collections\.

### B\.4 Data allocation and shared quantities

The output vocabulary consists of GPT\-2’s 50,257 token IDs, including EOS; MASK and PAD are excluded\. We use 64\-token chunks and fixR=2R=2,α=5\\alpha=5, and auxiliary ridgeρ=0\.01\\rho=0\.01\. A source is a story in TinyStories, a paragraph in WikiText\-103, or an article in CNN/DailyMail\. Each news article contributes at most two spaced chunks\. Because the cached WikiText token IDs do not mark article boundaries, source grouping accounts for dependence within paragraphs only\.

The 2,048\-chunk reference banks used in the seven\-pair comparison contain 768 stories, 895 paragraphs, and 1,025 articles, respectively\. This comparison includes mask ratesq∈\{0\.2,0\.5,0\.8\}q\\in\\\{0\.2,0\.5,0\.8\\\}\. The three\-pair comparison atn=4096n=4096and81928192uses a separate auxiliary allocation, which also supplies the finite\-reference correction and neural comparisons\. These comparisons share evaluation text and saved masks\.

Within each auxiliary allocation, one split estimatesppand the conditional rowsP^δaux\\widehat\{P\}^\{\\rm aux\}\_\{\\delta\}\. A separate split determines the weights using the centered predictors

uδ,i=P^δaux\(⋅∣Xi−δ\)−pu\_\{\\delta,i\}=\\widehat\{P\}^\{\\rm aux\}\_\{\\delta\}\(\\cdot\\mid X\_\{i\-\\delta\}\)\-pat mask rateq=0\.5q=0\.5, with unavailable predictors set to zero\. We pool inner products between predictors into a Gram matrixGGand their inner products witheXi−pe\_\{X\_\{i\}\}\-pinto a vectorbb\. For a nonempty set𝒜\\mathcal\{A\}of visible offsets, the weights are

w𝒜=\[G𝒜​𝒜\+ρ​max⁡\{tr⁡\(G𝒜​𝒜\)\|𝒜\|,10−12\}​I\]−1​b𝒜\.w\_\{\\mathcal\{A\}\}=\\left\[G\_\{\\mathcal\{A\}\\mathcal\{A\}\}\+\\rho\\max\\left\\\{\\frac\{\\operatorname\{tr\}\(G\_\{\\mathcal\{A\}\\mathcal\{A\}\}\)\}\{\|\\mathcal\{A\}\|\},10^\{\-12\}\\right\\\}I\\right\]^\{\-1\}b\_\{\\mathcal\{A\}\}\.When𝒜\\mathcal\{A\}is empty, the reconstruction ispp\. Within each experiment, the weights are fixed across corpus sizes and mask rates\. Radius, smoothing, and ridge are chosen using development data, independently of the A/B disagreement used for evaluation\.

Each real\-text evaluation uses 128 chunks with one saved Bernoulli mask per chunk at the specified mask rate\. If a mask hides no positions, we hide position zero\. Atq=0\.2q=0\.2, the probability of this event is0\.864<10−60\.8^\{64\}<10^\{\-6\}\. We first average disagreement over hidden positions within each chunk, then average across chunks with equal weight\.

#### Multinomial implementation\.

For a reference bank ofmmchunks and a target corpus size ofnnchunks, we substitute

p~δ​\(a∣b\)=Cδref​\(b,a\)\+α​paCδref​\(b,\+\)\+α,N~δ,n​\(b\)=nm​Cδref​\(b,\+\)\\widetilde\{p\}\_\{\\delta\}\(a\\mid b\)=\\frac\{C^\{\\rm ref\}\_\{\\delta\}\(b,a\)\+\\alpha p\_\{a\}\}\{C^\{\\rm ref\}\_\{\\delta\}\(b,\+\)\+\\alpha\},\\qquad\\widetilde\{N\}\_\{\\delta,n\}\(b\)=\\frac\{n\}\{m\}C^\{\\rm ref\}\_\{\\delta\}\(b,\+\)into \([12](https://arxiv.org/html/2610.12987#S3.E12)\)\. The conditional estimate is fixed asnnchanges\. We compute its collision probability,∑ap~δ​\(a∣b\)2\\sum\_\{a\}\\widetilde\{p\}\_\{\\delta\}\(a\\mid b\)^\{2\}, over the full vocabulary, including tokens absent from the reference row\.

The same\-lag source forecast instead sums squared influence contributions separately for each lag\. This removes cross\-lag covariance terms while accounting for within\-source dependence at each lag, random row totals, and variation in source length\.

### B\.5 Disagreement across corpus and reference sizes

In the seven\-pair comparison, observed disagreement decreases by 5–40% across corpora asnnincreases from 512 to 2,048 chunks\. A1/n1/nlaw would predict a 75% decrease over this range\. The forecast based on a 2,048\-chunk reference underpredicts disagreement by 13–18% atn=2048n=2048\. Atn=4096n=4096, the shortfall increases to 27–33%; this comparison uses seven pairs for TinyStories, five for WikiText\-103, and four for CNN/DailyMail\. Increasing the reference to 6,144 chunks reduces the shortfall atn=4096n=4096to 8–11%, but leads to overprediction atn=512n=512\.

The comparison atn=4096n=4096and81928192uses three A/B pairs per corpus to examine both calibration and the decrease in disagreement with corpus size\. Holding the A/B outcomes fixed, the multinomial forecast is closer to observed disagreement in all six corpus–size conditions at each ofm=2048m=2048and40964096\. Atm=8192m=8192, the full source forecast is closer in four of the six conditions, including all three corpora atn=4096n=4096\. The multinomial forecast nevertheless describes the percentage decrease between the two corpus sizes more accurately\. Reference size can therefore change which method is better calibrated without producing the same ranking for size response\.

A reference\-size comparison uses three independently sampled A/B pairs atn=4096n=4096\. Here we report calibration as the forecast divided by the arithmetic mean of observed disagreement\. Increasingmmfrom 4,096 to 12,288 changes the source forecast ratios from\(0\.845,0\.838,0\.861\)\(0\.845,0\.838,0\.861\)to\(1\.010,1\.012,0\.988\)\(1\.010,1\.012,0\.988\), in the order TinyStories, WikiText\-103, and CNN/DailyMail\. A separate 4,096\-chunk reference bank gives ratios\(0\.846,0\.859,0\.882\)\(0\.846,0\.859,0\.882\)\. Thus, the ratiom/nm/nalone does not determine calibration\.

This sensitivity to reference estimation motivates the finite\-reference correction experiments with multiple independent banks\.

## Appendix CNeural Protocol and Transfer Test

### C\.1 Training and evaluation

For each corpus, we train on three A/B pairs at each sizen∈\{512,1024,2048\}n\\in\\\{512,1024,2048\\\}\. The neural and count\-based comparisons use the same vocabulary, 128 evaluation chunks, and savedq=0\.5q=0\.5masks\. Count forecasts are computed from a 2,048\-chunk reference bank\. Figure[4](https://arxiv.org/html/2610.12987#A3.F4)reports neural disagreement after 8,000 and 12,000 updates\.

Figure 4:Corpus\-size response on the matched task atq=0\.5q=0\.5\. The upper row shows observed mean disagreement for MDLM and the reconstructor, together with reference forecasts for the reconstructor\. Both forecasts use 2,048 reference chunks\. The vertical axis is logarithmic, and bars span the three observed pair means\. The lower row divides each curve by its ownn=512n=512value, with a dotted512/n512/nbenchmark starting at that value\. Sizes count 64\-token chunks, and MDLM curves include both training stages\.The neural model is a bidirectional Transformer with two layers, hidden width 96, four attention heads, feed\-forward width 384, tied input and output embeddings, and no dropout\. It has 5,179,539 parameters\. We train it using the masked\-token objective of MDLM\([Sahoo et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib35)\), within the discrete\-diffusion framework\([Austin et al\., 2021](https://arxiv.org/html/2610.12987#bib.bib3);[Lou et al\., 2024](https://arxiv.org/html/2610.12987#bib.bib27)\)\. We samplet∼Uniform⁡\[0\.05,0\.95\]t\\sim\\operatorname\{Uniform\}\[0\.05,0\.95\]and independently hide each token with probabilitytt\. The time\-conditioned model minimizes

𝔼X,t,M\[−164​t∑i∈Hlogfθ,i\(Xi∣Z,t\)\]\.\\mathbb\{E\}\_\{X,t,M\}\\left\[\-\\frac\{1\}\{64t\}\\sum\_\{i\\in H\}\\log f\_\{\\theta,i\}\(X\_\{i\}\\mid Z,t\)\\right\]\.\(16\)Here,fθ,i​\(Xi∣Z,t\)f\_\{\\theta,i\}\(X\_\{i\}\\mid Z,t\)is the probability that the Transformer with parametersθ\\thetaassigns to the true tokenXiX\_\{i\}at positionii, given the masked inputZZand diffusion timett\. Training uses the MDLM loss over the specified interval of masking probabilities\. The output softmax covers the 50,257\-token vocabulary, excluding MASK and PAD\. A training example with no hidden positions contributes zero loss\.

For each corpus, a model trained on auxiliary data for 4,000 updates provides the common initialization for the A/B runs\. Both models begin with newly initialized AdamW optimizer states\. Within each pair, they use identical minibatch indices, diffusion times, training masks, learning\-rate schedules, and update counts\. Each optimizer update accumulates gradients over four microbatches of four chunks, giving an effective batch size of 16\. The learning rate increases to5×10−45\\times 10^\{\-4\}over the first 400 updates, decreases to5×10−55\\times 10^\{\-5\}at update 8,000, and remains at that value through update 12,000\. AdamW uses\(β1,β2\)=\(0\.9,0\.98\)\(\\beta\_\{1\},\\beta\_\{2\}\)=\(0\.9,0\.98\), weight decay0\.010\.01, and gradient clipping at one\.

Evaluation uses the saved masks at diffusion timet=q=0\.5t=q=0\.5\. For a model classhh, we estimate disagreement by

𝔇^n,qobs,h=13⋅128​∑r=13∑e=1128‖hAr​\(Ze\)−hBr​\(Ze\)‖Me2\.\\widehat\{\\mathfrak\{D\}\}^\{\\rm obs,h\}\_\{n,q\}=\\frac\{1\}\{3\\cdot 128\}\\sum\_\{r=1\}^\{3\}\\sum\_\{e=1\}^\{128\}\\\|h\_\{A\_\{r\}\}\(Z\_\{e\}\)\-h\_\{B\_\{r\}\}\(Z\_\{e\}\)\\\|\_\{M\_\{e\}\}^\{2\}\.\(17\)This can also be written as

𝔇^n,qobs,h=13⋅128​∑r=13∑e=11281\|He\|​∑i∈He‖hAr,i​\(Ze\)−hBr,i​\(Ze\)‖22\.\\widehat\{\\mathfrak\{D\}\}^\{\\rm obs,h\}\_\{n,q\}=\\frac\{1\}\{3\\cdot 128\}\\sum\_\{r=1\}^\{3\}\\sum\_\{e=1\}^\{128\}\\frac\{1\}\{\|H\_\{e\}\|\}\\sum\_\{i\\in H\_\{e\}\}\\\|h\_\{A\_\{r\},i\}\(Z\_\{e\}\)\-h\_\{B\_\{r\},i\}\(Z\_\{e\}\)\\\|\_\{2\}^\{2\}\.Here,MeM\_\{e\}denotes the mask applied to evaluation chunkee, andHeH\_\{e\}is its set of hidden positions\. The norm∥⋅∥Me2\\\|\\cdot\\\|\_\{M\_\{e\}\}^\{2\}averages squared Euclidean output differences over positions inHeH\_\{e\}\. This comparison measures variation across corpora conditional on the shared initialization, training randomness, and update budget\. Under independent A/B corpus sampling, the corresponding population disagreement satisfies the equal\-size identity \([6](https://arxiv.org/html/2610.12987#S3.E6)\)\.

The two predictors have different information and output constraints: MDLM uses the full visible chunk and returns token probabilities, whereas the reconstructor uses the four offsets\{−2,−1,1,2\}\\\{\-2,\-1,1,2\\\}and returns affine scores\.

To assess sensitivity to training randomness, we also retrain the A corpus of pair 1 with an alternative training seed at every size\. Atn=2048n=2048and 12,000 updates, disagreement between models trained on this same corpus is 0\.0281 for TinyStories, 0\.0138 for WikiText\-103, and 0\.0052 for CNN/DailyMail\. These values measure a different source of variation from the A/B comparison and are reported separately, without subtraction from A/B disagreement\.

For input\-ranking comparisons, we compute Spearman correlations over the 128 evaluation chunks within each pair and report the mean of the three coefficients\. We report them without significance tests, because chunks from the same source are not independent\.

### C\.2 Directional overlap and the frequency control

Scalar disagreement measures the total size of an output difference\. We also ask whether neural and count\-based differences concentrate in similar directions in vocabulary space\.

LetQ∈ℝ50257×64Q\\in\\mathbb\{R\}^\{50257\\times 64\}have fixed, randomly chosen orthonormal columns satisfyingQ⊤​𝟏=0Q^\{\\top\}\\mathbf\{1\}=0\. For a given A/B pair and model classhh, writeΔih​\(Z\)=hA,i​\(Z\)−hB,i​\(Z\)\\Delta\_\{i\}^\{h\}\(Z\)=h\_\{A,i\}\(Z\)\-h\_\{B,i\}\(Z\)\. On a setTTof evaluation chunks, define

Ch,T=1\|T\|​∑j∈T1\|Hj\|​∑i∈Hj\(Q⊤​Δih​\(Zj\)\)​\(Q⊤​Δih​\(Zj\)\)⊤\.C\_\{h,T\}=\\frac\{1\}\{\|T\|\}\\sum\_\{j\\in T\}\\frac\{1\}\{\|H\_\{j\}\|\}\\sum\_\{i\\in H\_\{j\}\}\\bigl\(Q^\{\\top\}\\Delta\_\{i\}^\{h\}\(Z\_\{j\}\)\\bigr\)\\bigl\(Q^\{\\top\}\\Delta\_\{i\}^\{h\}\(Z\_\{j\}\)\\bigr\)^\{\\top\}\.This is an uncentered second\-moment matrix\. Its trace is the projected disagreement, including the contribution from the mean output difference\.

We partition the 128 evaluation chunks into two groups,T1T\_\{1\}andT2T\_\{2\}, of 64 chunks each, placing all chunks from a source in the same group\. For each groupTfT\_\{f\}, we use the other group to select the four leading eigenvectors of the reconstructor’s second\-moment matrix\. Denote these orthonormal directions by the columns ofU\(−f\)U^\{\(\-f\)\}\. The fraction of neural disagreement in these directions, measured on the held\-out groups, is

F=∑f=12\|Tf\|​tr⁡\{\(U\(−f\)\)⊤​CMDLM,Tf​U\(−f\)\}∑f=12\|Tf\|​tr⁡\(CMDLM,Tf\)\.F=\\frac\{\\sum\_\{f=1\}^\{2\}\|T\_\{f\}\|\\operatorname\{tr\}\\\!\\left\\\{\(U^\{\(\-f\)\}\)^\{\\top\}C\_\{\\mathrm\{MDLM\},T\_\{f\}\}U^\{\(\-f\)\}\\right\\\}\}\{\\sum\_\{f=1\}^\{2\}\|T\_\{f\}\|\\operatorname\{tr\}\(C\_\{\\mathrm\{MDLM\},T\_\{f\}\}\)\}\.We compute this fraction for each of the three pairs atn=2048n=2048and 12,000 updates, then average across pairs\. The projectionQQ, the rank of four, and the source partition were fixed after the scalar analyses and before computing directional overlap\. All reported fractions refer to energy within the 64\-dimensional projection\.

A uniformly random four\-dimensional subspace captures4/64=6\.25%4/64=6\.25\\%of this energy in expectation\. To determine whether the reconstructor contributes more than a preference for frequent tokens, we also use the four leading eigenvectors of

Cfreq=Q⊤​\{diag⁡\(p\)−p​p⊤\}​Q\.C\_\{\\rm freq\}=Q^\{\\top\}\\\{\\operatorname\{diag\}\(p\)\-pp^\{\\top\}\\\}Q\.This matrix is the projected covariance of a one\-hot token drawn from the auxiliary unigram distribution\. Its leading directions capture at least as much neural disagreement as those selected by the reconstructor \(Section[5](https://arxiv.org/html/2610.12987#S5.SS0.SSS0.Px3)\)\. The control therefore measures how much directional overlap can be explained by token frequencies alone\. It does not compare two unigram predictors: both would return the sameppand have zero disagreement\.

### C\.3 Relation to the image diffusion setting

The common question in the image and text settings is how predictions change when the training data are sampled again\. In the image analysis of[Wang et al\. \(2026\)](https://arxiv.org/html/2610.12987#bib.bib41), the tractable predictor is the empirical linear denoiser

D^σ​\(x\)=μ^\+Σ^​\(Σ^\+σ2​I\)−1​\(x−μ^\)\.\\widehat\{D\}\_\{\\sigma\}\(x\)=\\widehat\{\\mu\}\+\\widehat\{\\Sigma\}\(\\widehat\{\\Sigma\}\+\\sigma^\{2\}I\)^\{\-1\}\(x\-\\widehat\{\\mu\}\)\.Its dependence on the dataset is determined by the empirical mean and covariance\. The text reconstructor instead depends on lag counts through smoothed conditional rows\. For fixed tables and masks, it is affine in the visible one\-hot tokens, while row normalization makes it nonlinear in the sampled counts\. Related local co\-occurrence statistics appear in word association\([Church and Hanks, 1990](https://arxiv.org/html/2610.12987#bib.bib6)\), GloVe\([Pennington et al\., 2014](https://arxiv.org/html/2610.12987#bib.bib32)\), and the matrix interpretation of skip\-gram with negative sampling\([Levy and Goldberg, 2014](https://arxiv.org/html/2610.12987#bib.bib25)\)\. Table[7](https://arxiv.org/html/2610.12987#A3.T7)summarizes the correspondence\. The image analysis includes both denoising at a fixed noise level and complete sampling maps\. Our text comparisons evaluate a single masked\-input reconstruction, using either affine scores or neural token probabilities\.

Table 7:Comparison of the image diffusion setting of[Wang et al\. \(2026\)](https://arxiv.org/html/2610.12987#bib.bib41)with the two text predictors\. Each setting measures variation across datasets while holding the evaluation input fixed\. The image theory analyzes a linear denoiser and its sampling map; the text theory analyzes normalized lag counts and their reconstruction scores\. MDLM provides a neural test of the count\-based forecasts\.

## Appendix DReconstruction Quality and Top\-kkAccuracy

### D\.1 Relative MSE improvement

Across the 54\-condition grid, the reconstructor’s mean relative MSE improvement over the auxiliary unigram is 13\.16% for TinyStories, 5\.19% for WikiText\-103, and 3\.95% for CNN/DailyMail\.

We examine the role of local order with a separate shuffle control atn=512,q=0\.5n=512,q=0\.5\. Shuffling tokens within each chunk preserves token counts and source membership, but yields relative MSE improvements of only 0\.0042%, 0\.0003%, and 0\.0001%, respectively\. These near\-zero gains indicate that the reconstructor depends strongly on local token order\. The control evaluates a single size and mask rate, whereas the percentages above average over the full grid\.

Table[8](https://arxiv.org/html/2610.12987#A4.T8)compares the reconstructor and MDLM on the same evaluation inputs atn=2048,q=0\.5n=2048,q=0\.5\. At both neural checkpoints, MDLM achieves lower MSE and higher top\-1 and top\-64 accuracy on TinyStories\. The reconstructor performs better on all three measures for WikiText\-103 and CNN/DailyMail\. Atn=512n=512, MDLM’s relative MSE improvement is negative on WikiText\-103 and CNN/DailyMail after both 8,000 and 12,000 updates\. For CNN/DailyMail, it is also slightly negative atn=1024n=1024\.

Table 8:Reconstruction quality atn=2048,q=0\.5n=2048,q=0\.5on the shared 128 evaluation chunks\. Each row averages six models from three A/B pairs\. MSE sums squared error over the 50,257 vocabulary coordinates, averages over hidden positions within each chunk, and then weights chunks equally\. MSE gain is the percentage reduction relative to the auxiliary unigram\. Top\-1 and top\-64 accuracy are also reported as percentages\. Arrows indicate the preferred direction\.
### D\.2 Top\-kkaccuracy

Squared error evaluates the full output vector\. Top\-kkaccuracy instead asks whether the true token appears among thekkhighest\-scoring vocabulary entries\. We rank all 50,257 tokens, including EOS, using the reconstructor’s raw affine scores or the neural model’s logits\. Ties are resolved by ascending token ID\. As with MSE, we average over hidden positions within each chunk and then give chunks equal weight\.

To compare these rankings with the same auxiliary unigram used for MSE, writeAcck\\operatorname\{Acc\}\_\{k\}as a fraction and define

ℰk​\(f\)=1−Acck⁡\(f\),RelErrRedk⁡\(f\)=1−ℰk​\(f\)ℰk​\(p\)\.\\mathcal\{E\}\_\{k\}\(f\)=1\-\\operatorname\{Acc\}\_\{k\}\(f\),\\qquad\\operatorname\{RelErrRed\}\_\{k\}\(f\)=1\-\\frac\{\\mathcal\{E\}\_\{k\}\(f\)\}\{\\mathcal\{E\}\_\{k\}\(p\)\}\.The relative error reduction measures the fraction of the unigram’s top\-kkerrors removed byff\.

Figure[5](https://arxiv.org/html/2610.12987#A4.F5)reports the reconstructor’s accuracy and relative error reduction at seven cutoffs and three corpus sizes\. Atq=0\.5q=0\.5, its accuracy exceeds the unigram’s at every tested size and cutoff\. Although accuracy is nondecreasing inkk, relative error reduction need not be, because both the reconstructor’s and the unigram’s error rates change with the cutoff\. For CNN/DailyMail, relative error reduction peaks atk=32k=32for all three sizes\. For WikiText\-103, the values atk=32k=32and6464are below the value atk=16k=16\.

Figure 5:Top\-kkreconstruction quality atq=0\.5q=0\.5\. The upper row compares the radius\-two reconstructor at three corpus sizes with the auxiliary unigram\. The lower row gives the relative reduction in top\-kkerror against that same unigram\. Colors and markers denote corpus size in both rows\. Each reconstructor curve averages A and B within each pair, then averages the three pairs\. Higher values indicate better performance in both rows\.

## Appendix ECalibration Checks and Smoothing

### E\.1 Controlled population and error decomposition

We use a population with known moments to separate reference estimation error from linearization error\. The control fixesK=512K=512,L=8L=8,R=2R=2,q=0\.5q=0\.5, and ridge0\.010\.01\. Each of 32 topics selects 32 distinct tokens using weights proportional tok−1\.05k^\{\-1\.05\}\. Its reset distributionvhv\_\{h\}assigns weight0\.550\.55to the uniform distribution on these tokens and0\.450\.45to the global Zipf distribution\. Each source chooses a topic uniformly\. Chunks begin with a draw fromvhv\_\{h\}and then follow

Ph=r​S\+\(1−r\)​𝟏​vh⊤,r=0\.25,P\_\{h\}=rS\+\(1\-r\)\\mathbf\{1\}v\_\{h\}^\{\\top\},\\qquad r=0\.25,whereSSmaps each token to its cyclic successor\. We compare independent tokens, dependent tokens in one\-chunk sources, and four chunks sharing a topic within each source\. The independent\-token control sets both the topic weight andrrto zero\.

For each control, we evaluate 80 A/B pairs at eachn∈\{256,1024\}n\\in\\\{256,1024\\\}and four reference banks at eachm∈\{128,512,2048\}m\\in\\\{128,512,2048\\\}\. Independent auxiliary samples provide 4,096 chunks for estimatingppand two sets of 1,024 chunks for determining weights\. Evaluation uses 32 separate chunks with fixed masks; empty masks contribute zero\. Smoothing defaults toα=5\\alpha=5\. For the four\-chunk sources, we also testα=1,20\\alpha=1,20using the same source draws, auxiliary data, and masks, with weights recomputed for each smoothing value\.

LetDobsD\_\{\\rm obs\}denote mean A/B disagreement for the nonlinear reconstructor andDlinD\_\{\\rm lin\}its counterpart after linearizing at the population mean counts\. For a fixed reference bank, write the linearized output difference asU\+VU\+V, whereUUcontains contributions from rows with zero reference totals andVVcontains those from observed rows\. Define

DU=𝔼​‖U‖M2,DV=𝔼​‖V‖M2,CU​V=𝔼​⟨U,V⟩M\.D\_\{U\}=\\mathbb\{E\}\\\|U\\\|\_\{M\}^\{2\},\\qquad D\_\{V\}=\\mathbb\{E\}\\\|V\\\|\_\{M\}^\{2\},\\qquad C\_\{UV\}=\\mathbb\{E\}\\langle U,V\\rangle\_\{M\}\.SinceDlin=DU\+DV\+2​CU​VD\_\{\\rm lin\}=D\_\{U\}\+D\_\{V\}\+2C\_\{UV\}, the error of a reference forecastFrefF\_\{\\rm ref\}decomposes exactly as

Fref−Dobs=−\(DU\+2​CU​V\)⏟unseen rows and cross terms\+\(Fref−DV\)⏟reference error on observed rows\+\(Dlin−Dobs\)⏟linearization error\.F\_\{\\rm ref\}\-D\_\{\\rm obs\}=\\underbrace\{\-\(D\_\{U\}\+2C\_\{UV\}\)\}\_\{\\text\{unseen rows and cross terms\}\}\+\\underbrace\{\(F\_\{\\rm ref\}\-D\_\{V\}\)\}\_\{\\text\{reference error on observed rows\}\}\+\\underbrace\{\(D\_\{\\rm lin\}\-D\_\{\\rm obs\}\)\}\_\{\\text\{linearization error\}\}\.\(18\)The cross term can have either sign, so the contribution of unseen rows cannot be described byDUD\_\{U\}alone\.

For four chunks per source atα=5\\alpha=5,n=1024n=1024, andm=128m=128, the three terms divided byDobsD\_\{\\rm obs\}are−32\.6%\-32\.6\\%,−38\.6%\-38\.6\\%, and\+3\.2%\+3\.2\\%, respectively\. Missing rows therefore explain only part of the underprediction\. Reference error on observed rows contributes substantially, while linearization error partly offsets the reference error\.

### E\.2 Population forecasts and sensitivity to smoothing

The generator’s exact mean lag\-count matrix per chunk is

mδ=132​∑h=132∑t=0L−δ−1diag⁡\(vh⊤​Pht\)​Phδ,m−δ=mδ⊤,δ\>0\.m\_\{\\delta\}=\\frac\{1\}\{32\}\\sum\_\{h=1\}^\{32\}\\sum\_\{t=0\}^\{L\-\\delta\-1\}\\operatorname\{diag\}\(v\_\{h\}^\{\\top\}P\_\{h\}^\{t\}\)P\_\{h\}^\{\\delta\},\\qquad m\_\{\-\\delta\}=m\_\{\\delta\}^\{\\top\},\\quad\\delta\>0\.\(19\)For the source forecast, we evaluate the derivative atBδ,n=n​mδB\_\{\\delta,n\}=nm\_\{\\delta\}and use the exact joint covariance of the counts at sizenn\. This covariance includes overlapping pairs, cross\-lag correlations, and dependence between chunks that share a topic\. Products of pair counts involve at most four token positions, whose joint probabilities can be computed exactly from the Markov chains and averaged over topics\. We check the moment calculation by enumeration in a population with three tokens and length\-four chunks\.

The multinomial forecast uses the same population mean row totals and the unsmoothed population conditional probabilities in its collision term\. Both forecasts use the same fixed auxiliary quantities and evaluation inputs\.

Table 9:Calibration using exact population moments\. Entries are forecasts divided by mean nonlinear A/B disagreement; values closer to one indicate better agreement\. Brackets give 95% intervals from 5,000 bootstrap resamples of the 80 A/B pairs, with forecasts, auxiliary quantities, and evaluation inputs fixed\. The first six rows useα=5\\alpha=5\. The smoothing comparisons use four chunks per source and the same source draws across values ofα\\alpha\.In all sixα=5\\alpha=5settings, the multinomial forecast is closer to observed disagreement even after reference estimation error has been removed \(Table[9](https://arxiv.org/html/2610.12987#A5.T9)\)\. The source forecast overpredicts in these settings, and the smaller multinomial forecast partly offsets this positive linearization error\. For four chunks per source atα=1,n=1024\\alpha=1,n=1024, however, the source forecast already underpredicts, so the smaller multinomial forecast increases the shortfall\.

So in these simulations the multinomial law looks better partly because errors cancel\. This does not show that the same happens on real text\.

### E\.3 Smoothing and rare contexts on real text

Table[3](https://arxiv.org/html/2610.12987#S4.T3)examines smoothing using the same 20 A/B pairs and six reference banks per corpus as Appendix[B\.1](https://arxiv.org/html/2610.12987#A2.SS1)\. We use a vocabulary of 50,257 tokens, 64\-token chunks,R=2R=2, andq=0\.5q=0\.5\. The 128 evaluation chunks, masks, and auxiliary unigram are fixed across smoothing values\. Reconstruction weights are also fixed at theirα=5\\alpha=5values, so the comparison isolates changes in the conditional rows\. The full sweep usesα∈\{0\.5,1,2,5,10,20,50\}\\alpha\\in\\\{0\.5,1,2,5,10,20,50\\\}; the table reportsα=1,5,20\\alpha=1,5,20\. The data allocations andα=5\\alpha=5outcomes were already known\. Forecasts for the other smoothing values were saved before scoring their corresponding A/B outcomes\.

Both the source and multinomial forecasts use the source\-split jackknife in \([13](https://arxiv.org/html/2610.12987#S3.E13)\), with each component recomputed at the chosenα\\alphaand target size\. For each corpus, we average\|log⁡\(Fb,m,n/D¯n\)\|\|\\log\(F\_\{b,m,n\}/\\bar\{D\}\_\{n\}\)\|equally over six banks and the three conditions

\(m,n\)∈\{\(1024,2048\),\(1024,4096\),\(2048,4096\)\}\.\(m,n\)\\in\\\{\(1024,2048\),\(1024,4096\),\(2048,4096\)\\\}\.Here,D¯n\\bar\{D\}\_\{n\}is the mean disagreement across the 20 pairs at the chosenα\\alpha\. Lower error indicates better calibration\. The numerical records include paired intervals from 5,000 bootstrap resamples of whole banks and A/B pairs\. Each resampled bank or pair carries its measurements across methods, smoothing values, and nested sizes\. These intervals condition on the auxiliary data and evaluation inputs\.

After jackknife correction, the multinomial forecast has lower error atα=1,5\\alpha=1,5in all three corpora\. Atα=20\\alpha=20, the source forecast is more accurate on TinyStories and WikiText\-103, while both CNN/DailyMail errors round to0\.0240\.024\. So the multinomial law’s advantage depends onα\\alpha\. The simulations show that errors can cancel, but they do not explain why the ranking flips atα=20\\alpha=20on real text\.

A row with context countNNassigns weightα/\(N\+α\)\\alpha/\(N\+\\alpha\)to the shared unigram\. Smoothing is therefore substantial whenNNis small relative toα\\alpha; an unseen row returnsppfor everyα\>0\\alpha\>0\. In one 2,048\-chunk WikiText\-103 reference bank, 7\.3% of evaluation queries address unseen rows, and 20\.8% address rows with fewer than five observations\. These frequencies count repeated queries and describe that single bank\. Rare and unseen contexts connect this setting to species\-frequency estimation\([Good, 1953](https://arxiv.org/html/2610.12987#bib.bib16)\)and statistical inference for rare text events\([Dunning, 1993](https://arxiv.org/html/2610.12987#bib.bib9)\)\.

Acrossn=512,1024,2048,4096n=512,1024,2048,4096, increasingα\\alphafrom 1 to 20 produces a shallower decrease in observed disagreement in all three corpora\. The paired 95% intervals for the change in log\-log slope lie entirely above zero\. Both the calibration pattern and the flatter size response persist when weights are re\-estimated on the same auxiliary data for eachα\\alpha\. The numerical records include every smoothing value, reference size, and reconstruction\-quality measurement\.