Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores

# Measurement, Validation, and Adjusted Scores
Source: [https://arxiv.org/html/2608.02985](https://arxiv.org/html/2608.02985)
## Temporal Leakage in LLM Backtesting: Measurement, Validation, and Adjusted Scores

Zeyu Zhangzeyuzhang2028@u\.northwestern\.edu Department of Statistics and Data Science Northwestern UniversityBradly C\. Stadiebstadie@northwestern\.edu Department of Statistics and Data Science Northwestern University

###### Abstract

The standard check for contamination in LLM backtests is simple: compare scores before and after the training cutoff\. We show this check is uninformative\. Four flagship models fail it on questions they cannot have memorized: every scored question resolved after their cutoffs\. The reason is structural\. Models legitimately know more about times near their cutoff, so recency mimics leakage, and we prove no passive backtest can separate the two from genuine skill\. Measurement, not just detection, requires information from outside the backtest\. We supply it in two forms\. A known cutoff identifies leakage at the boundary; a matched clean control identifies it globally and yields a leakage\-adjusted score\. We also derive where leakage hides: it concentrates on outcomes that surprised the crowd and were well covered in training, and partial memorization is disproportionately rewarded\. We validate the estimators against ground truth by planting leakage in twin models, where they recover the injected dose and return null on clean questions\. Deployed on frontier models, they detect one cutoff\-localized signature and, at the audit’s power floor, clear five models whose apparent advantages were recency alone\. Backtests need not be discarded; they need one defensible reference\.

## 1Introduction

Large language models are increasingly evaluated by*backtesting*: the model is asked, as of a historical date, about events whose outcomes have since resolved, and its score is read as evidence of how it would forecast the genuine future\(Halawi et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib15); Karger et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib23); Lopez\-Lira et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib29); Paleka et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib35); Gao et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib10)\)\. The threat is temporal leakage: a web\-scale training corpus may already describe the outcome being “predicted,” so a high score can reflect memorization rather than skill\. The standard defense is a pre/post check, which compares scores before and after the model’s training cutoff and flags a model whose advantage sits on the pre\-cutoff side\(Jain et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib21); Benhenda,[2026](https://arxiv.org/html/2608.02985#bib.bib1)\)\.[Figure˜1](https://arxiv.org/html/2608.02985#S1.F1)shows that check failing\. We scored five flagship models on the official ForecastBench panel, restricted to questions that resolved after their documented training cutoffs, so no outcome can be in their training data\. Four of the five fail the check anyway, with spurious “leakage” gaps as large as\+0\.061\+0\.061\. We read the documented cutoffs as bounding all training data\. The matched\-window check of[Section˜D\.2](https://arxiv.org/html/2608.02985#A4.SS2)supports this reading: in a common window, all five gaps collapse together\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m1_forest.png)Figure 1:M1: models that cannot have leaked still fail the naive contamination check\.Each row is one model \(cutoff month in parentheses\), scored only on questions resolving at least 30 days after its cutoff, so no outcome can be in its training data\. The x\-axis is its naive gapΔ=s¯early−s¯late\\Delta=\\bar\{s\}\_\{\\mathrm\{early\}\}\-\\bar\{s\}\_\{\\mathrm\{late\}\}, the window split at its median resolution date, wheres=\(c0−Y\)2−\(P−Y\)2s=\(c\_\{0\}\-Y\)^\{2\}\-\(P\-Y\)^\{2\}is the Brier improvement of the model’s forecastPPover the crowd forecastc0c\_\{0\}\(higher is better\);Δ\>0\\Delta\>0means better on older questions, the pattern an audit reads as leakage\. Whiskers: cluster\-bootstrap 95% CIs;∗: CI excludes zero\. Result: four of five flagships are flagged,Δ=\+0\.044\\Delta=\+0\.044to\+0\.061∗\+0\.061^\{\\ast\}; recency alone produces the leakage pattern\.The failure is structural, not a flaw of one benchmark\. A backtest score mixes three components\.*Skill*is what the evaluation wants to measure\.*Recency*is legitimate knowledge that varies with distance from the cutoff: training data cover recent periods more densely, and post\-cutoff knowledge grows stale\(Lazaridou et al\.,[2021](https://arxiv.org/html/2608.02985#bib.bib24)\)\.*Leakage*is memorized outcome information, and it is illegitimate\. The same cutoff drives the last two: a question resolving near the cutoff is one the model legitimately knows more about and one whose outcome is likelier to sit in the corpus\. A pre/post contrast therefore measures recency and leakage jointly, and recency alone reproduces the leakage signature\. Other failure modes—hallucination, miscalibration, prompt sensitivity\(Huang et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib18); Xiong et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib53); Sclar et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib43)\)—degrade honest performance on both sides of the cutoff; leakage alone is specific to backtesting, and it inflates the score\.

The question that matters for practice is therefore not whether contamination exists—it usually does\(Sainz et al\.,[2023](https://arxiv.org/html/2608.02985#bib.bib40); Golchin & Surdeanu,[2024](https://arxiv.org/html/2608.02985#bib.bib13)\)—but how much it inflates the score, and whether the score can be corrected\. Existing tools do not answer it\. Detection methods establish that contamination exists, not what it is worth\(Oren et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib34); Zawalski et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib57)\), and prompt\-level remedies fail to suppress recall\(Lopez\-Lira et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib29)\)\. Clean\-reference estimates on static benchmarks do not transfer, because a backtest defines “clean” by a date rather than by membership\(Singh et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib46); Haimes et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib14)\)\. Retraining a truly clean reference model fails, because its cutoff would have to predate every evaluation question, leaving a model too outdated to stand in for the one under audit\(Drinkall et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib6)\)\.

This paper answers the quantitative question in four steps: we prove the inflation is non\-identifiable from forecasts and outcomes alone, derive where it concentrates, show that one external reference restores measurement, and validate the estimators against ground truth by planting leakage in twin models\. Deployed on frontier systems, the estimators detect one cutoff\-localized signature and, at the design’s power floors, clear five models whose apparent advantages were recency alone\.

##### Contributions\.

- •The standard check is uninformative \([Section˜3](https://arxiv.org/html/2608.02985#S3)\)\.Leakage inflation cannot be identified from backtest scores alone: the identified set is a sharp interval that more data does not shrink\. A flat pre/post profile is not evidence of a clean backtest, and[Figure˜1](https://arxiv.org/html/2608.02985#S1.F1)shows the converse failure on real flagships\.
- •Leakage is not a rate; it obeys a law \([Section˜4](https://arxiv.org/html/2608.02985#S4)\)\.Per question, inflation equals honest uncertainty times an extraction factor with a double benefit: half the leaked signal already yields three quarters of the full inflation\. Leakage concentrates where the crowd was surprised and training coverage was dense, so an average contamination rate is the wrong audit object\.
- •A validated audit recipe \([Sections˜5](https://arxiv.org/html/2608.02985#S5),[6](https://arxiv.org/html/2608.02985#S6)and[7](https://arxiv.org/html/2608.02985#S7)\)\.Choose a reference—a known cutoff \(regression discontinuity\) or a matched clean control \(difference\-in\-differences\)—run the matching estimator, and report the leakage\-adjusted score with its assumption; paraphrase probes detect leakage but cannot measure it\. The estimators recover planted leakage in twin models in dose, location, and per\-question profile, return null on a clean twin, and behave honestly in the wild\.

[Section˜8](https://arxiv.org/html/2608.02985#S8)positions the work in the contamination literature; proofs, estimator details, and full experimental configurations are in the appendix\.

## 2Problem Setup

##### The mechanism\.

Two dates govern temporal leakage, and[Figure˜2](https://arxiv.org/html/2608.02985#S2.F2)makes the mechanism precise\. The*as\-of date*t0t\_\{0\}is the information horizon a genuine forecast may use; the deployed model’s*training cutoff*isT\>t0T\>t\_\{0\}\. The world state up tot0t\_\{0\}generates the training corpus𝒟clean\\mathcal\{D\}\_\{\\mathrm\{clean\}\}available to a genuine forecaster, while the later stateW\+W\_\{\+\}is a latent common cause of the “bridge” training data𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\(text written aftert0t\_\{0\}but beforeTT\) and of the realized outcomeYY\. A model trained on𝒟clean∪𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{clean\}\}\\cup\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}can therefore acquire information aboutYYthrough the back\-door pathW\+→𝒟bridge→y^corruptW\_\{\+\}\\to\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\\to\\hat\{y\}\_\{\\mathrm\{corrupt\}\}, the red leakage path in[Figure˜2](https://arxiv.org/html/2608.02985#S2.F2), without the benchmark question ever appearing verbatim in training\. Temporal knowledge leakage is this back\-door flow; it separates a model trained with the bridge data \(y^corrupt\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\) from an otherwise identical one trained without it \(y^clean\\hat\{y\}\_\{\\mathrm\{clean\}\}\)\. A question such as “*will countryXXenter a recession by Q4 2023?*” is leakage\-eligible if it resolves before the cutoff, where the model may simply recall the reported outcome, and clean if it resolves after, where the model must reason\.

W​\(t0\)W\(t\_\{0\}\)W\+W\_\{\+\}𝒟clean\\mathcal\{D\}\_\{\\mathrm\{clean\}\}YY𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}y^clean\\hat\{y\}\_\{\\mathrm\{clean\}\}y^corrupt\\hat\{y\}\_\{\\mathrm\{corrupt\}\}leakagebefore question datet0t\_\{0\}leaked window\(t0,T\]\(t\_\{0\},\\,T\]Figure 2:Temporal leakage is an indirect information flow through a common cause of the training text and the outcome, not verbatim overlap\.The model’s training cutoffTTpostdates the as\-of datet0t\_\{0\}; a question is leakage\-eligible when it resolves inside\(t0,T\]\(t\_\{0\},T\]\. The later world stateW\+W\_\{\+\}resolves the outcomeYY\(black, top\) and also generates the bridge data𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}, text written inside\(t0,T\]\(t\_\{0\},T\]\. The deployed model trains on𝒟clean\\mathcal\{D\}\_\{\\mathrm\{clean\}\}*plus*𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}, so the red pathW\+→𝒟bridge→y^corruptW\_\{\+\}\\\!\\to\\\!\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\\\!\\to\\\!\\hat\{y\}\_\{\\mathrm\{corrupt\}\}carries outcome information into it without the benchmark question ever appearing in training; the counterfactual clean modely^clean\\hat\{y\}\_\{\\mathrm\{clean\}\}\(blue\) learns from𝒟clean\\mathcal\{D\}\_\{\\mathrm\{clean\}\}alone\.
##### Notation\.

We audit one backtested model with reference cutoffTT; uncertainty inTTis handled by sensitivity analysis rather than treated as known\. For a random question we observe the realized outcomeY∈\{0,1\}Y\\in\\\{0,1\\\}; the model’s forecastP∈\[0,1\]P\\in\[0,1\]under a fixed deterministic protocol; the*running variable*GG, the signed gap between the question’s resolution date and the cutoff, negative for leakage\-eligible questions \(g<0g<0\) and positive for clean ones; and a leakage\-free difficulty covariateXXobservable without the model, such as the contemporaneous crowd forecast\. We score with the Brier lossℓ=\(P−Y\)2\\ell=\(P\-Y\)^\{2\}, a strictly proper score; the identification results of[Section˜5](https://arxiv.org/html/2608.02985#S5)require only a bounded loss\. The central observable is the conditional mean loss

m​\(x,g\)=𝔼​\[\(P−Y\)2∣X=x,G=g\],m\(x,g\)\\;=\\;\\mathbb\{E\}\\big\[\(P\-Y\)^\{2\}\\mid X=x,\\ G=g\\big\],\(1\)estimable by regression wherever the backtest has data\.

##### The target\.

Leakage lowers the pre\-cutoff loss below what the model’s legitimate information could achieve\. Writem0​\(x,g\)m\_\{0\}\(x,g\)for the*honest surface*: the loss the deployed model would incur using legitimate information only\. It still varies withgg, because legitimate knowledge depends on temporal distance from the cutoff; that variation is recency, and the theory imposes no shape or direction on it\. The paper’s target is the*inflation*

B=𝔼​\[m0​\(X,G\)−m​\(X,G\)∣G<0\]≥0,B\\;=\\;\\mathbb\{E\}\\big\[\\,m\_\{0\}\(X,G\)\-m\(X,G\)\\mid G<0\\,\\big\]\\;\\geq\\;0,\(2\)the average amount by which the backtest understates the model’s honest error on leakage\-eligible questions\. The goal of the paper is an estimate of the inflation with a confidence interval, and the resulting*leakage\-adjusted*score\.[Appendix˜A](https://arxiv.org/html/2608.02985#A1)collects all symbols and the standing regularity conventions; proofs are in[Appendix˜B](https://arxiv.org/html/2608.02985#A2)\.

## 3Scores Alone Cannot Measure Leakage

A completed backtest yields*passive data*: draws of\(P,Y,X,G\)\(P,Y,X,G\)\. This section proves that the inflationBBis not a functional of their joint law, however many questions are scored, and isolates exactly what is missing, which dictates the form of the remedies in[Section˜5](https://arxiv.org/html/2608.02985#S5)\.

The definition ofBBin \([2](https://arxiv.org/html/2608.02985#S2.E2)\) compares the observable surfacemmwith the honest surfacem0m\_\{0\}, so the content lies in which pairs\(m0,L\)\(m\_\{0\},L\)are admissible\.

###### Definition 1\(Operational risk decomposition\)\.

A pair of functionsm0:𝒳×ℝ→\[0,1\]m\_\{0\}:\\mathcal\{X\}\\times\\mathbb\{R\}\\to\[0,1\]andL:𝒳×ℝ→ℝL:\\mathcal\{X\}\\times\\mathbb\{R\}\\to\\mathbb\{R\}is an*operational decomposition*ofmmif

m​\(x,g\)=m0​\(x,g\)−L​\(x,g\)\.m\(x,g\)=m\_\{0\}\(x,g\)\-L\(x,g\)\.\(3\)

Admissible pairs always exist \(\(m,0\)\(m,0\)is one\); content enters through a sign and a support restriction\.

###### Assumption 1\(Nonnegative pre\-cutoff leakage\)\.

There exists an operational decomposition\(m0,L\)\(m\_\{0\},L\)such thatL​\(x,g\)≥0L\(x,g\)\\geq 0andL​\(x,g\)=0L\(x,g\)=0for everyg≥0g\\geq 0\.

The support restriction says the future cannot leak\. For a*frozen*model \(no retrieval tools, no post\-cutoff fine\-tuning; cutoff uncertainty is handled by sensitivity analysis,[Section˜2](https://arxiv.org/html/2608.02985#S2)\), an outcome resolving afterTTcannot have been seen, while before the cutoff, memorized information lowers the loss byL≥0L\\geq 0\. Recency lives inm0m\_\{0\}: the honest surface may depend onggin any way, and in particular the theory never assumes the model knows more in aggregate about recent times\. Under[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1), the inflation isB=𝔼​\[L​\(X,G\)∣G<0\]B=\\mathbb\{E\}\[L\(X,G\)\\mid G<0\], as in \([2](https://arxiv.org/html/2608.02985#S2.E2)\)\.

\{restatable\}

\[Sharp partial identification from passive scores\]theoremthmnonident Under[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)alone,BBis not a functional of the law of\(P,Y,X,G\)\(P,Y,X,G\): every continuationm~0:𝒳×ℝ→\[0,1\]\\tilde\{m\}\_\{0\}:\\mathcal\{X\}\\times\\mathbb\{R\}\\to\[0,1\]withm≤m~0≤1m\\leq\\tilde\{m\}\_\{0\}\\leq 1on\{g<0\}\\\{g<0\\\}andm~0=m\\tilde\{m\}\_\{0\}=mon\{g≥0\}\\\{g\\geq 0\\\}defines an admissible pair\(m~0,L~=m~0−m\)\(\\tilde\{m\}\_\{0\},\\ \\tilde\{L\}=\\tilde\{m\}\_\{0\}\-m\)consistent with the same passive law, with corresponding inflationB~=𝔼​\[m~0−m∣G<0\]\\tilde\{B\}=\\mathbb\{E\}\[\\tilde\{m\}\_\{0\}\-m\\mid G<0\]\. The sharp identified set is

\[0,𝔼​\[1−m∣G<0\]\],\\big\[\\,0,\\ \\ \\mathbb\{E\}\[\\,1\-m\\mid G<0\\,\]\\,\\big\],and every value in this interval is attainable\.

The construction is elementary\. Post\-cutoff,m0=mm\_\{0\}=mis observed\. Pre\-cutoff, the data reveal only the differencem0−L=mm\_\{0\}\-L=m, and the one\-parameter familym~0,t=m\+t​\(1−m\)\\tilde\{m\}\_\{0,t\}=m\+t\(1\-m\),t∈\[0,1\]t\\in\[0,1\], sweeps the entire interval\. The proof is in[Section˜B\.3](https://arxiv.org/html/2608.02985#A2.SS3)\.

The content lies in the model rather than the argument\.[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)restricts almost nothing—recency is entirely free, and leakage carries only a sign and a support restriction—so the impossibility cannot be blamed on a restrictive model: once recency and leakage are indexed by the same cutoff, it follows\. Nothing is Brier\-specific: for any loss bounded byMM, the upper endpoint becomes𝔼​\[M−m∣G<0\]\\mathbb\{E\}\[M\-m\\mid G<0\]\.

The practical consequence corrects common practice:*a flat pre/post performance profile is not evidence of a clean backtest*, because an honest model with the same recency profile produces identical scores\.[Figure˜1](https://arxiv.org/html/2608.02985#S1.F1)is the converse error on real models: recency alone produced the failing pattern of four flagships that cannot have memorized any scored outcome\.

The theorem also specifies its own remedy\. The identified set is exactly the freedom of the honest surface on the leaked side, so identification must import information that pinsm0m\_\{0\}there: a clean control model, a continuity restriction at a known cutoff, or an intervention on the question itself\. Those are the three routes of[Section˜5](https://arxiv.org/html/2608.02985#S5)\. First, we characterize whatLLis, which tells an auditor where to look\.

## 4Where Leakage Concentrates

Leakage is not a uniform rate across questions\. It requires both that an outcome was*worth*knowing and that the model*absorbed and used*it\. A minimal working model of how leaked information moves a forecast makes this precise and gives the inflation a closed form\.

###### Assumption 2\(Convex\-pull leakage\)\.

Under the fixed deterministic prediction protocol, a leakage\-eligible questionqq\(g<0g<0\) satisfies

P​\(q\)=\(1−w​\(q\)\)​Phon​\(q\)\+w​\(q\)​Y​\(q\)P\(q\)=\(1\-w\(q\)\)\\,P\_\{\\mathrm\{hon\}\}\(q\)\+w\(q\)\\,Y\(q\)for an*extraction weight*w​\(q\)∈\[0,1\]w\(q\)\\in\[0,1\], wherePhonP\_\{\\mathrm\{hon\}\}is the honest \(legitimate\-information\) forecast\.

[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)is a modeling assumption, not a consequence of the leakage problem\. Leaked information enters as a convex blend of the honest forecast and the recalled outcome: exact for answer memorization \(w=1w\{=\}1, whereP→YP\\to Y\), first\-order for graded evidence leakage\. Its substantive restriction is directionality: any forecast betweenPhonP\_\{\\mathrm\{hon\}\}and the outcome is a convex pull for somew∈\[0,1\]w\\in\[0,1\], and what is excluded is leaked information that*misleads*\(w<0w<0\), the same direction[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)fixes in aggregate\. We test the model’s consequences against ground truth in[Section˜6](https://arxiv.org/html/2608.02985#S6)\.

Writing the honest Brierb0​\(q\)=\(Phon​\(q\)−Y​\(q\)\)2b\_\{0\}\(q\)=\(P\_\{\\mathrm\{hon\}\}\(q\)\-Y\(q\)\)^\{2\}, the leakage has a closed form\.

\{restatable\}

\[Per\-question and aggregate leakage\]propositionpropperq Under[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2), the per\-question leakage isb0​\(q\)​w​\(q\)​\(2−w​\(q\)\)b\_\{0\}\(q\)\\,w\(q\)\\,\(2\-w\(q\)\); the pairm0​\(x,g\)=𝔼​\[b0∣X=x,G=g\]m\_\{0\}\(x,g\)=\\mathbb\{E\}\[b\_\{0\}\\mid X\{=\}x,G\{=\}g\]andL​\(x,g\)=𝔼​\[b0​w​\(2−w\)∣X=x,G=g\]L\(x,g\)=\\mathbb\{E\}\[b\_\{0\}\\,w\(2\-w\)\\mid X\{=\}x,G\{=\}g\]is an operational decomposition satisfying[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1), and hence

B=𝔼​\[b0​w​\(2−w\)\|G<0\]\.B\\;=\\;\\mathbb\{E\}\\big\[\\,b\_\{0\}\\,w\(2\-w\)\\,\\big\|\\,G<0\\big\]\.\(4\)
The reason is one line: the pull leaves residual error\(1−w\)​\(Phon−Y\)\(1\-w\)\(P\_\{\\mathrm\{hon\}\}\-Y\), so the leaked Brier is\(1−w\)2​b0\(1\-w\)^\{2\}b\_\{0\}and the saving isb0​w​\(2−w\)b\_\{0\}\\,w\(2\-w\)\. Averaging the saving given\(X,G\)\(X,G\)furnishes a canonical decomposition, so the working model automatically satisfies[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\. Per question, inflation is*stakes times extraction*:b0b\_\{0\}measures how much there was to know, andw​\(2−w\)w\(2\-w\)how much of it the model took\.

##### The double benefit\.

The factorw​\(2−w\)=1−\(1−w\)2w\(2\-w\)=1\-\(1\-w\)^\{2\}is why partial leakage is disproportionately rewarded \(plotted in[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)b,[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)\)\. Pulling the forecast toward the truth earns a benefit linear inww; the residual cost is only quadratic inww\. A model that extracts half the leaked signal \(w=0\.5w=0\.5\) already gains75%75\\%of the full\-memorization inflation, andw=0\.3w=0\.3still gains51%51\\%\. Leakage need not be “memorize the answer” to distort a backtest; a faint, partial trace suffices\.

##### Two kinds of leakage, two consequences\.

The extraction weight separates*answer*leakage, where the model has memorized the outcome itself \(w→1w\\to 1\), from*evidence*leakage, where it absorbed leaked\-window information correlated withYY\(w∈\(0,1\)w\\in\(0,1\)\); a finer interpretive factorization ofwwis given in[Section˜B\.3\.5](https://arxiv.org/html/2608.02985#A2.SS3.SSS5)\. Two consequences of \([4](https://arxiv.org/html/2608.02985#S4.E4)\) organize the rest of the paper\. First, leakage*concentrates*: it is large only where the outcome was surprising \(highb0b\_\{0\}\) and absorbed \(highww\), and vanishes on easy or obscure questions, so an “average leakage rate” is the wrong audit object\. Second, detectingw\>0w\>0requires controlling difficulty, skill, recency, and overconfidence simultaneously—a multi\-signal estimator, not a pre/post score gap—which is what[Section˜5](https://arxiv.org/html/2608.02985#S5)builds\.

## 5Three References, Two Estimators, One Diagnostic

[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)specifies what identification must import: information that pins the honest surface on the leaked side\. Three practical references supply it: a known training cutoff or a clean control model point\-identifies the inflation under explicit assumptions, and paraphrase\-query access detects evidence leakage without identifying it\. The results are identification statements for the conditional riskmmunder a bounded loss; the matching estimators and bootstrap intervals are in[Sections˜6](https://arxiv.org/html/2608.02985#S6)and[C](https://arxiv.org/html/2608.02985#A3)\.

### 5\.1Route 1, a known cutoff: regression discontinuity

Training cutoffs are published in model cards as documented approximations \(hence the sensitivity treatment of[Section˜2](https://arxiv.org/html/2608.02985#S2)\), and the route needs no queries beyond a dated benchmark spanning the cutoff\. One restriction turns the cutoff into a reference: honest competence drifts continuously through time, while memorization switches off at the cutoff\. Any jump in the loss there is therefore leakage\.

###### Assumption 3\(Honest\-risk continuity\)\.

For eachxx,m0​\(x,⋅\)m\_\{0\}\(x,\\cdot\)is continuous atg=0g=0, andL​\(x,0−\)L\(x,0^\{\-\}\)exists\.

\{restatable\}

\[RD identifies boundary leakage\]theoremthmrd Under[Assumptions˜1](https://arxiv.org/html/2608.02985#Thmassumption1)and[3](https://arxiv.org/html/2608.02985#Thmassumption3)and the support regularity of[Section˜B\.4](https://arxiv.org/html/2608.02985#A2.SS4), for any bounded loss, the boundary leakage is identified by the upward jump in observed risk at the cutoff,

J​\(x\)=limg↓0m​\(x,g\)−limg↑0m​\(x,g\)=L​\(x,0−\)≥0\.J\(x\)\\;=\\;\\lim\_\{g\\downarrow 0\}m\(x,g\)\-\\lim\_\{g\\uparrow 0\}m\(x,g\)\\;=\\;L\(x,0^\{\-\}\)\\;\\geq\\;0\.
The boundary jump is the assumption\-light estimand and remains nonzero under perfect recall; quasi\-resolved outcomes and ingestion lag attenuate it toward zero, so a smallJ^\\widehat\{J\}should not be over\-read \([Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2)\)\. Recovering the*global*inflation needs unique clean\-side extrapolation of the honest surface \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)\), under whichBBis identified \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)in[Section˜B\.4\.2](https://arxiv.org/html/2608.02985#A2.SS4.SSS2)\)\.

### 5\.2Route 2, a clean control model: difference\-in\-differences

The second route uses a second model: families ship vintages with different cutoffs, so clean controls are standard artifacts,*selected*rather than retrained, and they need to be clean only on the evaluation window\. The control need not match the target’s capability, because level differences cancel\. It must instead match the*response*of honest performance to question timing: if the control shares the target’s recency profile, its pre/post change measures recency alone, and differencing pins the target’s honest change\.

###### Assumption 4\(Transportable recency\)\.

The controlM0M\_\{0\}is clean on the evaluation support \(LM0≡0L^\{M\_\{0\}\}\\equiv 0\); the observations of both models cover the target’s pre\-cutoff question mix on both sides of the cutoff; and, once standardized to that mix, honest performance changes by the same amount from pre to post for target and control \(formal statement in[Section˜B\.4](https://arxiv.org/html/2608.02985#A2.SS4)\)\.

\{restatable\}

\[DiD identification\]propositionpropdid Under[Assumptions˜1](https://arxiv.org/html/2608.02985#Thmassumption1)and[4](https://arxiv.org/html/2608.02985#Thmassumption4), standardize both models’ risks to the target’s pre\-cutoff question mix and letΔA\\Delta^\{A\}denote modelAA’s resulting pre\-to\-post change\. Then the inflation \([2](https://arxiv.org/html/2608.02985#S2.E2)\) is identified:B=ΔM−ΔM0B=\\Delta^\{M\}\-\\Delta^\{M\_\{0\}\}\.

The parallel\-change condition is not fully testable, because the target’s pre\-cutoff cell is contaminated by construction; its testable implications are in[Appendix˜C](https://arxiv.org/html/2608.02985#A3)\. Its dominant violation is signed: a control too weak to express recency*overstates*BB—the weak\-control trap of[Section˜7](https://arxiv.org/html/2608.02985#S7)—so a null is conservative, while a positive must carry the matching checks\. A boundary variant requiring only a no\-discontinuity control underwrites the deployment matrix \([Proposition˜7](https://arxiv.org/html/2608.02985#Thmtheorem7)\)\. How to read wild estimates is deferred to[Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2)\.

### 5\.3Route 3, paraphrase probes: detection only

The third route needs only black\-box access—a batch of paraphrase calls plus leakage\-free calibration anchors—and is a diagnostic, not an estimator\. It yields the prediction\-residual covariancePRC=Cov​\(P,Y−P∣G<0\)\\mathrm\{PRC\}=\\mathrm\{Cov\}\(P,\\,Y\-P\\mid G<0\)of the paraphrase\-consensus forecast, positive under convex\-pull leakage with partial extraction \([Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)\)\. Three limits keep it secondary: it requires calibration, since LLM overconfidence makes the raw covariance negative \([Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)\); it is blind at full memorization \(w=1w\{=\}1\), where the RD jump is maximal \([Proposition˜9](https://arxiv.org/html/2608.02985#Thmtheorem9)\); and on frontier data it failed its pre\-specified calibration\-stability gate \([Section˜E\.8](https://arxiv.org/html/2608.02985#A5.SS8)\)\. We use it to detect, never to estimateBB\([Appendix˜C](https://arxiv.org/html/2608.02985#A3)\)\.

### 5\.4One reference suffices; the adjusted score

Either of Routes 1–2 point\-identifiesBBunder its stated assumptions, and Route 3 detects without identifying \([Corollary˜8](https://arxiv.org/html/2608.02985#Thmtheorem8)\); one suitable external reference therefore suffices, though we do not claim the family is exhaustive\.[Table˜1](https://arxiv.org/html/2608.02985#S5.T1)states the input, guarantee, and check for each\. When globalBBis point\-identified, report a leakage\-adjusted score with a confidence interval; otherwise report the boundary or detection estimand without converting it into a global correction\.

Table 1:Three identification routes: input, guarantee, and check\.R1–R2 point\-identifyBB\(or boundaryJJ\) under stated assumptions; R3 is detection only\. Every assumption carries a falsification check exercised in[Sections˜6](https://arxiv.org/html/2608.02985#S6)and[7](https://arxiv.org/html/2608.02985#S7)\.###### Corollary 2\(Leakage\-adjusted performance\)\.

WheneverBBis point\-identified \(Route 1 or 2\), the leakage\-adjusted performance removesBBin the appropriate direction: for a risk \(any bounded loss, lower is better\),Riskadj=Riskmeas\+B\\mathrm\{Risk\}\_\{\\mathrm\{adj\}\}=\\mathrm\{Risk\}\_\{\\mathrm\{meas\}\}\+B; for a higher\-is\-better score \(e\.g\. accuracy or pass@1\),Scoreadj=Scoremeas−B\\mathrm\{Score\}\_\{\\mathrm\{adj\}\}=\\mathrm\{Score\}\_\{\\mathrm\{meas\}\}\-B, withBBin score units\.

The route results use onlym=m0−Lm=m\_\{0\}\-Land hold for any bounded loss, licensing the pass@1 correction of[Section˜7](https://arxiv.org/html/2608.02985#S7)\. The lawL=b0​w​\(2−w\)L=b\_\{0\}\\,w\(2\-w\)andPRC\\mathrm\{PRC\}are Brier\-specific \([Remark˜3](https://arxiv.org/html/2608.02985#Thmremark3)\)\. The practitioner’s recipe and decision flowchart \([Figure˜6](https://arxiv.org/html/2608.02985#A3.F6)\) are in[Appendix˜C](https://arxiv.org/html/2608.02985#A3)\.

## 6Validation with Planted Leakage

The estimators of[Section˜5](https://arxiv.org/html/2608.02985#S5)claim to measure leakage\. Before deploying them on frontier models, we test them where the answer is known\. M2 re\-analyzes a public controlled\-contamination study at pretraining scale\. M3 plants temporal leakage in twin models on real forecasting questions and asks the estimators to find it\. A claim\-to\-evidence contract table mapping each theoretical claim to the experiment that carries it—across this section, the deployment of[Section˜7](https://arxiv.org/html/2608.02985#S7), and the M1 check of[Section˜1](https://arxiv.org/html/2608.02985#S1)—is given as[Table˜4](https://arxiv.org/html/2608.02985#A4.T4)in[Appendix˜D](https://arxiv.org/html/2608.02985#A4)\.

##### Data: the forecasting panel\.

All forecasting experiments use one panel built from the public archive of ForecastBench\(Karger et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib23)\), the field\-standard dynamic forecasting benchmark:1,6461\{,\}646resolved binary market questions \(Polymarket, Metaculus, Manifold, INFER\) resolving 2024Q3 through 2026Q3, each carrying its resolution date, its realized outcomeYY, and a contemporaneous crowd forecastc0c\_\{0\}frozen before resolution\. The score is the crowd\-anchored Brier reductions=\(c0−Y\)2−\(P−Y\)2s=\(c\_\{0\}\-Y\)^\{2\}\-\(P\-Y\)^\{2\}; positivessmeans the model beats the crowd\. Confidence intervals use a cluster bootstrap over source\-by\-month clusters; boundary tests add date\-permutation and placebo\-date checks; configurations, prices, and the pre\-registration are in[Appendix˜D](https://arxiv.org/html/2608.02985#A4)\(coverage:[Section˜D\.1](https://arxiv.org/html/2608.02985#A4.SS1)\)\.

##### A pretraining\-scale anchor \(M2\)\.

The Hubble suite\(Wei et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib48)\)pretrains a*perturbed*model with benchmark documents inserted at known duplication countsr∈\{0,1,4,16,64,256\}r\\in\\\{0,1,4,16,64,256\\\}and a*standard*twin trained identically without them\. Re\-analyzing its published accuracies \(no new compute\) anchors the causal end: benchmark documents inserted into pretraining inflate measured scores\. The placebo\-centered contrast rises strictly monotonically with dose, from\+0\.011\+0\.011atr=1r=1to\+0\.403∗\+0\.403^\{\\ast\}atr=256r=256, with sixteen duplicates already worth about 21 accuracy points \([Figure˜7](https://arxiv.org/html/2608.02985#A4.F7)in[Section˜D\.3](https://arxiv.org/html/2608.02985#A4.SS3); Spearmanρ=1\.0\\rho=1\.0, permutationp=0\.0083p=0\.0083\)\. The rise is contamination, not composition: the clean twin is nearly flat across dose bins, and a self\-baseline check agrees \([Section˜D\.3](https://arxiv.org/html/2608.02985#A4.SS3)\)\. Hubble is question answering with no time axis, so it cannot test the recency confound; that is what M3 adds \(full analysis:[Sections˜D\.3](https://arxiv.org/html/2608.02985#A4.SS3)and[E\.4](https://arxiv.org/html/2608.02985#A5.SS4)\)\.

### 6\.1Twin models with known temporal leakage \(M3\)

No observational study can supply per\-question ground truth for leakage\. We therefore created the leakage ourselves\. We continue\-train two LoRA twins of Qwen3\.5\-35B\-A3B\-Base on the same 95\-million\-token corpus, which interleaves public filler text with injected news\-style documents about panel questions that resolved before a pseudo\-cutoffT∗T^\{\\ast\}\(April 2026\)\. In the*treatment*twin the documents state realized outcomes; in the*control*twin the same documents appear with outcomes scrubbed, matched on source, topic, and length, so both twins gain the same recency and only the treatment twin gains leakage\. Injection covers the988988questions on which the untuned base model is demonstrably uncertain, with dose levelsr∈\{0,1,4,16,64\}r\\in\\\{0,1,4,16,64\\\}randomized across questions, so the full dose grid costs two training runs\. Probabilities are read from paired “Yes”/“No” continuation log\-probabilities under a fixed few\-shot prompt; a ten\-percent pilot passed five pre\-registered gates before the full runs, and the protocol and logged deviations are in[Section˜D\.4](https://arxiv.org/html/2608.02985#A4.SS4)\.

##### Manipulation checks\.

Three checks confirm the design does what it claims: the treatment twin memorizes its documents \(per\-token log\-probability exceeds the control twin’s by0\.760\.76, CI\[0\.72,0\.81\]\[0\.72,0\.81\]\); the movement is outcome\-directed \(at dose 64 the treatment twin’s probability moves toward the realized outcome by\+0\.139\+0\.139, CI\[\+0\.119,\+0\.159\]\[\+0\.119,\+0\.159\]\); and recency is genuinely present \(on injected\-topic questions the scrubbed control beats the untuned base by\+0\.036\+0\.036Brier, CI\[\+0\.028,\+0\.044\]\[\+0\.028,\+0\.044\]\)\. The confound the theory targets is in the data\.

##### Inflation rises with dose and is null at dose zero\.

The twin contrast rises strictly monotonically in dose and is null at dose zero \([Figure˜3](https://arxiv.org/html/2608.02985#S6.F3), left\)\. Contamination does not need to be heavy: a single document exposure is worth about one point of crowd\-anchored Brier, sixteen about four\. The dose\-zero null shows the inflation is question\-specific, not a generic gain from training, and the curve reproduces M2’s pretraining dose\-response on real forecasting questions, now with per\-item ground truth\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m3_dose_response.png)

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m3_concentration.png)

Figure 3:M3: injected leakage produces a strictly monotone dose\-response \(left\) and concentrates on hard questions as the lawb0​w​\(2−w\)b\_\{0\}\\,w\(2\-w\)predicts \(right\)\.Treatment and control twins of one base model train on the same documents; only the treatment copy sees realized outcomes\. Both panels plot the twin contrastB^twin=ℓ¯ctrl−ℓ¯treat\\widehat\{B\}\_\{\\mathrm\{twin\}\}=\\bar\{\\ell\}\_\{\\mathrm\{ctrl\}\}\-\\bar\{\\ell\}\_\{\\mathrm\{treat\}\}\(ℓ=\(P−Y\)2\\ell=\(P\-Y\)^\{2\}; positive = leakage inflation\); whiskers: bootstrap 95% CIs;∗: CI excludes zero\.*Left*: by doserr, the number of outcome\-stating documents injected per question \(randomized across988988questions\): null atr=0r=0\(−0\.003\-0\.003\), rising strictly monotonically to\+0\.063∗\+0\.063^\{\\ast\}atr=64r=64\.*Right*: by difficulty, questions sorted into quintiles ofb0=\(Pctrl−Y\)2b\_\{0\}=\(P\_\{\\mathrm\{ctrl\}\}\-Y\)^\{2\}\(ticks: bin meanb¯0\\bar\{b\}\_\{0\}\), at doses 16 and 64\. Red squares: the zero\-free\-parameter predictionb¯0​w^​\(2−w^\)\\bar\{b\}\_\{0\}\\,\\hat\{w\}\(2\-\\hat\{w\}\), withw^\\hat\{w\}fitted from the treatment twin’s probability movement alone, never from the plotted contrasts\. Inflation grows13×13\\timesfrom easiest to hardest bin \(\+0\.010\+0\.010to\+0\.128\+0\.128atr=64r=64\); the prediction matches except in the hardest bin, where extraction weakens \(w^\\hat\{w\}drops from≈0\.5\{\\approx\}0\.5to0\.160\.16\) and the law acts as an upper bound\.
##### The concentration law holds, and its one miss is interpretable\.

[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)predictsL=b0​w​\(2−w\)L=b\_\{0\}\\,w\(2\-w\)\.[Figure˜3](https://arxiv.org/html/2608.02985#S6.F3)\(right\) tests the law with zero free parameters:w^\\hat\{w\}is fitted from the treatment twin’s probability movement alone, never from the plotted contrasts\. The prediction falls inside the observed CI in four of five difficulty quintiles at dose 16 and three of five at dose 64, and the globally fitted extraction weight rises with dose \(w^=0\.24\\hat\{w\}=0\.24,0\.230\.23,0\.330\.33,0\.420\.42atr=1,4,16,64r=1,4,16,64\); tercile and ex\-ante binnings agree \([Section˜D\.4](https://arxiv.org/html/2608.02985#A4.SS4)\)\. The one miss is the hardest quintile, where extraction weakens and the homogeneous\-wwlaw overshoots, so there it acts as an upper bound—the conservative direction for score correction\. Contamination distorts a backtest most where the model is weakest\.

##### The boundary estimators recover the injection; recency alone shows no jump\.

An analyst who did not create this leakage would measure the boundary statistics of[Section˜5](https://arxiv.org/html/2608.02985#S5): on the treatment twin the naive pre/post gap atT∗T^\{\\ast\}is\+0\.053\+0\.053and the local RD jump \(90\-day bandwidth\) is\+0\.052\+0\.052; on the control twin the same statistics are\+0\.001\+0\.001and\+0\.012\+0\.012, and a placebo boundary 120 days earlier is null \(−0\.014\-0\.014\)\. Detection and localization both work: injected leakage produces a starred jump at the right date, and recency alone produces none—the control twin absorbed the same 95 million tokens and shows no inflation signature, the real\-model face of the no\-free\-inflation asymmetry \([Corollary˜3](https://arxiv.org/html/2608.02985#Thmtheorem3)\)\.

##### PRC detects the evidence, once calibrated\.

The prediction\-residual covariance of[Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)had not previously been tested against known contamination\. Raw PRC fails: it is negative in every cell of the twin\-by\-dose grid, the miscalibration failure[Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)warns about\. The twin\-differenced, temperature\-calibrated statistic behaves as predicted:\+0\.0069∗\+0\.0069^\{\\ast\}on injected questions, null at dose zero, monotone in dose \([Figure˜9](https://arxiv.org/html/2608.02985#A4.F9)in[Section˜D\.4](https://arxiv.org/html/2608.02985#A4.SS4)\)\. PRC fires where evidence leakage was injected and nowhere else, and only with differencing and calibration\.

### 6\.2How to read a wild estimate

The twins license two reading rules for the deployment results of[Section˜7](https://arxiv.org/html/2608.02985#S7)\.

##### A boundary estimate is a footprint, not a per\-question inflation\.

On the injected population, per\-item ground\-truth inflation averages\+0\.024\+0\.024, roughly half the boundary estimate of\+0\.052\+0\.052; the remainder is spillover with a measurable mechanism\. The injected documents state outcomes that are83%83\\%NO, so the treatment twin acquires a base\-rate lean that is nearly free on pre\-cutoff questions but depresses post\-cutoff scores, and a pre/post estimator cannot tell inflated pre from depressed post \(the law analyses are immune: dose\-zero differencing removes the lean\)\. A wild RD or DiD estimate is therefore the total contamination footprint at the boundary; it bounds the per\-question inflation from above, which makes the score correction of[Corollary˜2](https://arxiv.org/html/2608.02985#Thmtheorem2)conservative\. Two wild mechanisms the sharp injection cannot exhibit push the other way: outcomes effectively decided in pre\-cutoff text raise the honest post\-boundary score, and ingestion lag thins memorized outcomes just before it\. Both attenuate the jump toward zero: the upper bound holds against spillover only, and a small jump does not exclude leakage deeper in the pre\-cutoff period \([Section˜D\.4](https://arxiv.org/html/2608.02985#A4.SS4)\)\.

##### A null bounds; it does not certify\.

Every null in[Section˜7](https://arxiv.org/html/2608.02985#S7)comes with a power floor: the smallest effect the design would have detected\. On the forecasting matrix, the minimum detectable effects at80%80\\%power are0\.050\.05to0\.110\.11anchored\-Brier units per cell\. A null therefore excludes code\-domain\-sized leakage \(about0\.090\.09\) for the best\-powered cells and says nothing about smaller leakage\. No cleanliness certificate is issued below the power floor\.

## 7Deployment on Frontier Models

We now run the audit where nobody knows the answer\. Results are ordered as an auditor should read them: cross\-model nulls, disciplined nulls under the matched control, one detection, and a positive control on documented contamination\.

### 7\.1The forecasting matrix: no flagship flagged at its own cutoff

We query four frontier targets with in\-window documented cutoffs \(rows of[Table˜2](https://arxiv.org/html/2608.02985#S7.T2)\) against the near\-clean controls MiniMax\-M3 and Claude\-Opus\-4\.7 \(their January 2026 cutoffs leave no leakage discontinuity inside the tested window\), computing the boundary jump at every assumed boundary\. A genuine signal must jump at the model’s own documented cutoff and nowhere else\. No own\-cutoff cell is starred, and the joint diagonal\-versus\-off\-diagonal permutation test is null \(p=0\.53p=0\.53\)\. Under the power floors of[Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2), the matrix excludes code\-domain\-sized effects for the best\-powered cells while remaining uninformative about smaller leakage\. An exploratory Gemini\-3\.1\-Pro row is in[Section˜D\.5](https://arxiv.org/html/2608.02985#A4.SS5)\. Run honestly, the cross\-model route substantiates no leakage claim for these flagships\.

Table 2:M4\-F: no own\-cutoff jump among four flagships; the within\-family arm flags GPT\-5\.5\.Cell:J^m​\(t\)=\(s¯m−s¯c\)post−\(s¯m−s¯c\)pre\\widehat\{J\}\_\{m\}\(t\)=\(\\bar\{s\}\_\{m\}\-\\bar\{s\}\_\{c\}\)\_\{\\mathrm\{post\}\}\-\(\\bar\{s\}\_\{m\}\-\\bar\{s\}\_\{c\}\)\_\{\\mathrm\{pre\}\}over±90\\pm 90days of assumed boundarytt\(ssas in[Figure˜1](https://arxiv.org/html/2608.02985#S1.F1);cc: top block = MiniMax\-M3/Claude\-Opus\-4\.7; bottom = GPT\-5\.5 family archived real\-time forecasts\)\. Shaded: own documented cutoff\.∗: CI excludes zero\. Dashes: not defined—the archived\-forecast arm exists only at the target’s own cutoff\. Result: no shaded top\-block cell starred \(permutationp=0\.53p=0\.53\); bottom block starred at the shaded cell \(pre\-specified unbanded / horizon\-banded,[Section˜7\.3](https://arxiv.org/html/2608.02985#S7.SS3)\)\.
### 7\.2Disciplined nulls under the matched control

The primary target is Qwen3\.5\-35B\-A3B \(year\-level 2026 cutoff\), evaluated at a July 2025 boundary on panel questions resolving 2025 through mid\-2026\. Because the cutoff is year\-level, the boundary lies inside the plausible training window, so both sides are leakage\-eligible: the design detects inflation that*differs*across the boundary; uniform inflation would evade it at every sweep placement\. The control is selected*before*reading any DiD result by a pre\-registered profile\-matching protocol on post\-boundary\-clean questions, which picks GPT\-5 over Gemini\-3\.1\-Pro \(profile distance0\.00510\.0051vs\.0\.00880\.0088;[Section˜D\.7](https://arxiv.org/html/2608.02985#A4.SS7)\)\. Secondary targets are the four other flagships of[Figure˜4](https://arxiv.org/html/2608.02985#S7.F4); GPT\-3\.5\-Turbo is a deliberately mismatched weak control\.

Naive gaps again raise flags, and again the flags are recency: after the paired, profile\-matched DiD, no adjusted estimate is significantly positive \([Figure˜4](https://arxiv.org/html/2608.02985#S7.F4)\)\. Two checks show the nulls are earned\. Substituting GPT\-3\.5\-Turbo re\-inflates the primary estimate, so the nulls come from matching, not from an estimator biased toward zero\. Semi\-synthetic injections are detected with probability0\.970\.97at effect0\.050\.05and1\.001\.00at the code\-domain\-sized effect0\.090\.09\. Full estimates and robustness are in[Section˜D\.7](https://arxiv.org/html/2608.02985#A4.SS7)\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m5_dissolve.png)Figure 4:M5: naive leakage flags dissolve under the matched adjustment\.Red: naive gapΔ=s¯pre−s¯post\\Delta=\\bar\{s\}\_\{\\mathrm\{pre\}\}\-\\bar\{s\}\_\{\\mathrm\{post\}\}\. Blue: adjustedΔ^DiD=Δtarget−Δcontrol\\widehat\{\\Delta\}\_\{\\mathrm\{DiD\}\}=\\Delta\_\{\\mathrm\{target\}\}\-\\Delta\_\{\\mathrm\{control\}\}against pre\-registered matched control GPT\-5\. Grey diamond: primary target vs\. mismatched GPT\-3\.5\-Turbo\. Whiskers: 95% CIs;∗: CI excludes zero\. Result: no adjusted estimate significantly positive; mismatched control re\-inflates to\+0\.055\+0\.055\.
### 7\.3One detection: a boundary signature for GPT\-5\.5

ForecastBench archives real\-time forecasts, clean by construction, enabling a within\-family design: comparing retrospective GPT\-5\.5 scores with archived scores of GPT\-5\-Mini and GPT\-5\.1 cancels the crowd anchor\. The jump at GPT\-5\.5’s December 2025 cutoff is\+0\.061\+0\.061\(CI\[\+0\.023,\+0\.093\]\[\+0\.023,\+0\.093\]\), and\+0\.106\+0\.106\(CI\[\+0\.056,\+0\.189\]\[\+0\.056,\+0\.189\]\) inside a 10–75\-day anchor\-horizon band logged as a dated deviation; both variants are starred \([Section˜D\.5](https://arxiv.org/html/2608.02985#A4.SS5)\)\. Protocol checks and a DeepSeek\-V3\.1 placebo at the same boundary are null, and the unstarred matrix cell \(\+0\.034\+0\.034\) is attenuation plus lower power, not contradiction \([Section˜D\.5](https://arxiv.org/html/2608.02985#A4.SS5)\)\. We scope the finding as[Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2)instructs: a boundary leakage signature at GPT\-5\.5’s documented cutoff—a contamination footprint, not an accusation\. GPT\-5\.4’s unbanded jump \(\+0\.048\+0\.048\) is null once horizons are balanced and is reported as an artifact, not a detection\.

### 7\.4Positive control: documented contamination on dated code

LiveCodeBench\(Jain et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib21)\)dates1,0551\{,\}055problems by contest release and publishes per\-problem pass@1 \([Remark˜3](https://arxiv.org/html/2608.02985#Thmremark3)\), so the same matrix runs at near\-zero cost on a benchmark era where contamination is documented\. GPT\-4o and Claude\-3\.5\-Sonnet are starred exactly at their own cutoffs \(date\-permutationp=0\.023p=0\.023,p=0\.017p=0\.017\), with all off\-diagonal cells null \([Table˜3](https://arxiv.org/html/2608.02985#S7.T3)\)\. A self\-generated extension to 2024\-cutoff targets is in[Section˜D\.6](https://arxiv.org/html/2608.02985#A4.SS6); coverage across M1–M5 spans 2023–2026 vintages of six\-plus vendors \([Section˜D\.1](https://arxiv.org/html/2608.02985#A4.SS1)\)\.

Table 3:M4\-C: positive control—the same test fires at both models’ own cutoffs and nowhere else\.Cell:J^m​\(t\)\\widehat\{J\}\_\{m\}\(t\)of[Table˜2](https://arxiv.org/html/2608.02985#S7.T2)on LiveCodeBench pass@1 within±160\\pm 160days oftt; baseline = 2025\-cutoff models\. Shaded: own cutoff\.∗: CI excludes zero \(p=0\.023p=0\.023,p=0\.017p=0\.017\)\. The large unstarred placebo \(Claude at Dec’24,\+0\.128\+0\.128\) is noise: its CI\[−0\.23,\+0\.33\]\[\-0\.23,\+0\.33\]is five times wider than the others because the±160\\pm 160\-day window overruns the end of the problem panel\.

## 8Related Work

##### Detecting contamination\.

A large literature asks*whether*evaluation data was seen in training:nn\-gram and longest\-substring screens\(Brown et al\.,[2020](https://arxiv.org/html/2608.02985#bib.bib2); Singh et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib46)\), membership\-inference and likelihood tests\(Shokri et al\.,[2017](https://arxiv.org/html/2608.02985#bib.bib45); Carlini et al\.,[2021](https://arxiv.org/html/2608.02985#bib.bib3); Shi et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib44)\), exchangeability tests\(Oren et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib34)\), and guided or perturbation\-based black\-box probes\(Golchin & Surdeanu,[2024](https://arxiv.org/html/2608.02985#bib.bib13); Zawalski et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib57)\); surveys catalogue detectors and their failure modes\(Sainz et al\.,[2023](https://arxiv.org/html/2608.02985#bib.bib40); Xu et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib54); Ravaut et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib37)\)\. All are membership\-oriented, and membership inference is near chance at pretraining scale, its apparent successes an artifact of temporal shift\(Duan et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib7)\)\. Temporal leakage largely escapes these tests, acting through parametric memory along an indirect causal channel \([Figure˜2](https://arxiv.org/html/2608.02985#S2.F2)\), and detection does not answer how much the score is inflated\.

##### Quantifying contamination’s score effect\.

Controlled training studies insert benchmark data deliberately\(Magar & Schwartz,[2022](https://arxiv.org/html/2608.02985#bib.bib30); Jiang et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib22); Wei et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib48)\); Hubble grounds our M2, M3 adapts its dose design to the temporal setting, and concurrent work fits dose–response curves for generative evaluations\(Schaeffer et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib42)\)\. On deployed models, clean\-reference estimators\(Singh et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib46); Haimes et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib14); Zhang et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib58); Dekoninck et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib4)\)are instances of our DiD route, overlap audits price contamination on code benchmarks\(Riddell et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib38)\), and rephrased samples evadenn\-gram screens\(Yang et al\.,[2023](https://arxiv.org/html/2608.02985#bib.bib56)\)\. Live and date\-stamped benchmarks avoid contamination by construction\(White et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib49); Jain et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib21); Wu et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib52)\)\. All define “clean” by membership in a static test set; a backtest defines it by a*date*, so every pre\-cutoff question is potentially exposed and the reference must come from elsewhere \([Section˜5](https://arxiv.org/html/2608.02985#S5)\)\.

##### Temporal leakage and lookahead bias\.

Cutoff\-based natural experiments document the symptom\(Roberts et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib39); Li & Flanigan,[2024](https://arxiv.org/html/2608.02985#bib.bib26)\); without a recency model, the same gap is consistent with legitimate recency \([Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\)\. In financial and forecasting backtests, lookahead bias has been separated by anonymization\(Glasserman & Lin,[2024](https://arxiv.org/html/2608.02985#bib.bib11)\), tested on unpredictable events\(Sarkar & Vafa,[2024](https://arxiv.org/html/2608.02985#bib.bib41)\), argued to undermine LLM forecaster evaluation\(Paleka et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib35)\), and measured or mitigated on the defining benchmarks\(Zou et al\.,[2022](https://arxiv.org/html/2608.02985#bib.bib60); Halawi et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib15); Karger et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib23); Liu et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib28); Benhenda,[2026](https://arxiv.org/html/2608.02985#bib.bib1); Gao et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib10); Zhang et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib59); Li et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib27)\)\. Retraining avoids leakage by construction\(Lazaridou et al\.,[2021](https://arxiv.org/html/2608.02985#bib.bib24); Dhingra et al\.,[2022](https://arxiv.org/html/2608.02985#bib.bib5); Drinkall et al\.,[2024](https://arxiv.org/html/2608.02985#bib.bib6); He et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib17); Yan et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib55)\)but cannot evaluate a given frontier model\. Closest to us,Lopez\-Lira et al\. \([2025](https://arxiv.org/html/2608.02985#bib.bib29)\)prove counterfactual forecasting ability is non\-identified once a model trains on realized outcomes; we add a distinct recency channel, derive where leakage concentrates, and show that one external reference restores measurement\. Classical identification tools are catalogued in[Section˜C\.4](https://arxiv.org/html/2608.02985#A3.SS4)\.

## 9Limitations and Conclusion

Our guarantees are conditional on assumptions made explicit and, where possible, tested\. The per\-question law excludes misleading leakage \(w<0w<0,[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)\), so the estimands measure*net*inflation; the twins impose rather than test this restriction \(their documents state only true outcomes\), and they show that homogeneous\-wwoverstates inflation on the hardest questions \([Section˜6\.1](https://arxiv.org/html/2608.02985#S6.SS1)\)\. DiD requires a clean, recency\-matched control \([Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\): a weak control overstatesBB, a failure we demonstrate with GPT\-3\.5\-Turbo and guard against with pre\-registered profile matching \(necessary, not sufficient\)\. RD rests on continuity \([Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)\); converting the boundary jump into globalBBadds an extrapolation assumption \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)\) that our code data do not support\. A wild boundary jump is best read as a total contamination footprint, and quasi\-resolved outcomes and ingestion lag attenuate it toward zero \([Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2)\)\. PRC has a ground\-truth validation but failed its real\-data calibration\-stability gate \([Section˜E\.8](https://arxiv.org/html/2608.02985#A5.SS8)\), so it remains a detection tool\. Empirically, the twins are LoRA continued\-training runs \(M2 anchors pretraining\), documented cutoffs are month\-resolution, and the M5 nulls exclude boundary\-differential inflation above0\.050\.05Brier\-reduction units, not uniform inflation over the training window \([Section˜7\.2](https://arxiv.org/html/2608.02985#S7.SS2)\)\. Natural next steps are a frontier\-scale planted\-leakage pretraining run, a capability\-matched dated model pair that point\-identifies a nonzero globalBB, content \(rather than framing\) perturbations that identifyww, extensions to non\-binary and ranking outputs, and transferable PRC calibration, promoting Route 3 from detection toward estimation\.

Backtest scores mix skill, recency, and leakage, and we proved that no passive backtest can separate the three: a flat pre/post profile is not evidence of a clean backtest\. The inflation is nevertheless systematic: under a minimal model of partial recall it obeysB=𝔼​\[b0​w​\(2−w\)∣G<0\]B=\\mathbb\{E\}\[b\_\{0\}\\,w\(2\-w\)\\mid G<0\], concentrating where the crowd was surprised and training coverage was dense\. One external reference restores measurement: a known cutoff identifies leakage at the boundary, and a matched clean control identifies it globally and yields a leakage\-adjusted score\. Validated against ground truth, the estimators recovered planted leakage in dose, location, and per\-question profile; deployed in the wild, they detected one cutoff\-localized signature and the documented code contamination, and, at the stated power floors, cleared five models whose apparent advantages were recency alone\. Backtests need not be discarded; they need one defensible reference\.

#### Broader Impact and Reproducibility Statements

This paper reports a contamination finding about a named commercial model \(GPT\-5\.5\), held to the evidentiary standard the paper argues for \([Section˜7\.3](https://arxiv.org/html/2608.02985#S7.SS3)\): language scoped to a boundary leakage signature, not an accusation of intent or a global capability claim\. The work otherwise improves evaluation hygiene and raises no concerns beyond those of the underlying public benchmarks\. All analysis code, the forecasting panel, twin\-training configurations, pre\-registration and deviations documents, and per\-experiment results are available at[https://github\.com/ZeyuZhang1901/Temporal\-Leakage\-Backtesting](https://github.com/ZeyuZhang1901/Temporal-Leakage-Backtesting);[Appendix˜F](https://arxiv.org/html/2608.02985#A6)maps each result to its script and output\.

## References

- Benhenda \(2026\)Mostapha Benhenda\.Look\-ahead\-bench: A standardized benchmark of look\-ahead bias in point\-in\-time LLMs for finance\.*arXiv preprint arXiv:2601\.13770*, 2026\.
- Brown et al\. \(2020\)Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, et al\.Language models are few\-shot learners\.*Advances in Neural Information Processing Systems*, 33:1877–1901, 2020\.
- Carlini et al\. \(2021\)Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert\-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel\.Extracting training data from large language models\.In*30th USENIX Security Symposium*, pp\. 2633–2650, 2021\.
- Dekoninck et al\. \(2024\)Jasper Dekoninck, Mark Niklas Müller, and Martin Vechev\.ConStat: Performance\-based contamination detection in large language models\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- Dhingra et al\. \(2022\)Bhuwan Dhingra, Jeremy R\. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W\. Cohen\.Time\-aware language models as temporal knowledge bases\.*Transactions of the Association for Computational Linguistics*, 10:257–273, 2022\.
- Drinkall et al\. \(2024\)Felix Drinkall, Eghbal Rahimikia, Janet Pierrehumbert, and Stefan Zohren\.Time machine GPT\.In*Findings of the Association for Computational Linguistics: NAACL 2024*, pp\. 3281–3292, 2024\.
- Duan et al\. \(2024\)Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi\.Do membership inference attacks work on large language models?In*Conference on Language Modeling \(COLM\)*, 2024\.arXiv:2402\.07841\.
- Elliott & Timmermann \(2016\)Graham Elliott and Allan Timmermann\.*Economic Forecasting*\.Princeton University Press, 2016\.
- Fuller \(1987\)Wayne A Fuller\.*Measurement Error Models*\.John Wiley & Sons, 1987\.
- Gao et al\. \(2025\)Zhenyu Gao, Wenxi Jiang, and Yutong Yan\.Detecting lookahead bias in LLM forecasts\.*arXiv preprint arXiv:2512\.23847*, 2025\.
- Glasserman & Lin \(2024\)Paul Glasserman and Caden Lin\.Assessing look\-ahead bias in stock return predictions generated by GPT sentiment analysis\.*The Journal of Financial Data Science*, 6\(1\):25–42, 2024\.
- Gneiting & Raftery \(2007\)Tilmann Gneiting and Adrian E Raftery\.Strictly proper scoring rules, prediction, and estimation\.*Journal of the American Statistical Association*, 102\(477\):359–378, 2007\.
- Golchin & Surdeanu \(2024\)Shahriar Golchin and Mihai Surdeanu\.Time travel in LLMs: Tracing data contamination in large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Haimes et al\. \(2024\)Jacob Haimes, Cenny Wenner, Kunvar Thaman, Vassil Tashev, Clement Neo, Esben Kran, and Jason Schreiber\.Benchmark inflation: Revealing LLM performance gaps using retro\-holdouts\.*arXiv preprint arXiv:2410\.09247*, 2024\.
- Halawi et al\. \(2024\)Danny Halawi, Fred Zhang, Yueh\-Han Chen, and Jacob Steinhardt\.Approaching human\-level forecasting with language models\.In*Advances in Neural Information Processing Systems*, volume 37, 2024\.
- Hausman \(1978\)Jerry A Hausman\.Specification tests in econometrics\.*Econometrica*, 46\(6\):1251–1271, 1978\.
- He et al\. \(2025\)Songrun He, Linying Lv, Asaf Manela, and Jimmy Wu\.Chronologically consistent large language models\.*arXiv preprint arXiv:2502\.21206*, 2025\.
- Huang et al\. \(2025\)Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu\.A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions\.*ACM Transactions on Information Systems*, 43\(2\):1–55, 2025\.
- Imbens & Lemieux \(2008\)Guido W\. Imbens and Thomas Lemieux\.Regression discontinuity designs: A guide to practice\.*Journal of Econometrics*, 142\(2\):615–635, 2008\.
- Imbens & Manski \(2004\)Guido W Imbens and Charles F Manski\.Confidence intervals for partially identified parameters\.*Econometrica*, 72\(6\):1845–1857, 2004\.
- Jain et al\. \(2025\)Naman Jain, King Han, Alex Gu, Wen\-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar\-Lezama, Koushik Sen, and Ion Stoica\.Livecodebench: Holistic and contamination free evaluation of large language models for code\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.arXiv:2403\.07974\.
- Jiang et al\. \(2024\)Minhao Jiang, Ken Ziyu Liu, Ming Zhong, Rylan Schaeffer, Siru Ouyang, Jiawei Han, and Sanmi Koyejo\.Investigating data contamination for pre\-training language models\.*arXiv preprint arXiv:2401\.06059*, 2024\.
- Karger et al\. \(2025\)Ezra Karger, Houtan Bastani, Yueh\-Han Chen, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip E\. Tetlock\.ForecastBench: A dynamic benchmark of AI forecasting capabilities\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- Lazaridou et al\. \(2021\)Angeliki Lazaridou, Adhiguna Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, Dani Yogatama, Kris Cao, Susannah Barlow, and Phil Blunsom\.Mind the gap: Assessing temporal generalization in neural language models\.In*Advances in Neural Information Processing Systems*, volume 34, pp\. 29348–29363, 2021\.
- Lee & Lemieux \(2010\)David S\. Lee and Thomas Lemieux\.Regression discontinuity designs in economics\.*Journal of Economic Literature*, 48\(2\):281–355, 2010\.
- Li & Flanigan \(2024\)Changmao Li and Jeffrey Flanigan\.Task contamination: Language models may not be few\-shot anymore\.In*Proceedings of the AAAI Conference on Artificial Intelligence*, volume 38, pp\. 18471–18480, 2024\.
- Li et al\. \(2026\)Weixian Waylon Li, Mengyu Wang, and Tiejun Ma\.Summoning the oracle to slay it: Mitigating look\-ahead bias in financial backtesting with large language models\.*arXiv preprint arXiv:2605\.24564*, 2026\.
- Liu et al\. \(2026\)Yachuan Liu, Xiaochun Wei, Lin Shi, Xinnuo Li, Bohan Zhang, Paramveer Dhillon, and Qiaozhu Mei\.ExAnte: A benchmark for ex\-ante inference in large language models\.In*Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 1551–1571, 2026\.arXiv:2505\.19533\.
- Lopez\-Lira et al\. \(2025\)Alejandro Lopez\-Lira, Yuehua Tang, and Mingyin Zhu\.The memorization problem: Can we trust LLMs’ economic forecasts?*arXiv preprint arXiv:2504\.14765*, 2025\.
- Magar & Schwartz \(2022\)Inbal Magar and Roy Schwartz\.Data contamination: From memorization to exploitation\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\)*, pp\. 157–165, 2022\.
- Manski \(2003\)Charles F Manski\.*Partial Identification of Probability Distributions*\.Springer Series in Statistics\. Springer, 2003\.
- Mincer & Zarnowitz \(1969\)Jacob A Mincer and Victor Zarnowitz\.The evaluation of economic forecasts\.In*Economic Forecasts and Expectations: Analysis of Forecasting Behavior and Performance*, pp\. 3–46\. NBER, 1969\.
- Murphy \(1973\)Allan H Murphy\.A new vector partition of the probability score\.*Journal of Applied Meteorology*, 12\(4\):595–600, 1973\.
- Oren et al\. \(2024\)Yonatan Oren, Nicole Meister, Niladri S\. Chatterji, Faisal Ladhak, and Tatsunori B\. Hashimoto\.Proving test set contamination in black\-box language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.arXiv:2310\.17623\.
- Paleka et al\. \(2026\)Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr\.Pitfalls in evaluating language model forecasters\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.arXiv:2506\.00723\.
- Patton & Timmermann \(2012\)Andrew J Patton and Allan Timmermann\.Forecast rationality tests based on multi\-horizon bounds\.*Journal of Business & Economic Statistics*, 30\(1\):1–17, 2012\.
- Ravaut et al\. \(2025\)Mathieu Ravaut, Bosheng Ding, Fangkai Jiao, Hailin Chen, Xingxuan Li, Ruochen Zhao, Chengwei Qin, Caiming Xiong, and Shafiq Joty\.A comprehensive survey of contamination detection methods in large language models\.*Transactions on Machine Learning Research*, 2025\.arXiv:2404\.00699\.
- Riddell et al\. \(2024\)Martin Riddell, Ansong Ni, and Arman Cohan\.Quantifying contamination in evaluating code generation capabilities of language models\.In*Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 14116–14137, 2024\.arXiv:2403\.04811\.
- Roberts et al\. \(2024\)Manley Roberts, Himanshu Thakur, Christine Herlihy, Colin White, and Samuel Dooley\.To the cutoff… and beyond? a longitudinal perspective on LLM data contamination\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Sainz et al\. \(2023\)Oscar Sainz, Jon Ander Campos, Iker García\-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre\.NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pp\. 10776–10787, 2023\.
- Sarkar & Vafa \(2024\)Suproteem K\. Sarkar and Keyon Vafa\.Lookahead bias in pretrained language models\.*SSRN working paper 4754678*, 2024\.
- Schaeffer et al\. \(2026\)Rylan Schaeffer, Joshua Kazdan, Baber Abbasi, Ken Ziyu Liu, Brando Miranda, Ahmed Ahmed, Fazl Barez, Abhay Puri, Stella Biderman, Niloofar Mireshghallah, and Sanmi Koyejo\.Quantifying the effect of test set contamination on generative evaluations\.*arXiv preprint arXiv:2601\.04301*, 2026\.
- Sclar et al\. \(2024\)Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr\.Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Shi et al\. \(2024\)Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer\.Detecting pretraining data from large language models\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Shokri et al\. \(2017\)Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov\.Membership inference attacks against machine learning models\.In*IEEE Symposium on Security and Privacy \(S&P\)*, pp\. 3–18, 2017\.
- Singh et al\. \(2024\)Aaditya K\. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes\.Evaluation data contamination in LLMs: how do we measure it and \(when\) does it matter?*arXiv preprint arXiv:2411\.03923*, 2024\.
- Spearman \(1910\)Charles Spearman\.Correlation calculated from faulty data\.*British Journal of Psychology*, 3\(3\):271–295, 1910\.
- Wei et al\. \(2026\)Johnny Tian\-Zheng Wei, Ameya Godbole, Mohammad Aflah Khan, Ryan Wang, Xiaoyuan Zhu, James Flemings, Nitya Kashyap, Krishna P\. Gummadi, Willie Neiswanger, and Robin Jia\.Hubble: A model suite to advance the study of LLM memorization\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.arXiv:2510\.19811\.
- White et al\. \(2025\)Colin White, Samuel Dooley, Manley Roberts, Arka Pal, et al\.LiveBench: A challenging, contamination\-limited LLM benchmark\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.arXiv:2406\.19314\.
- Wooldridge \(2015\)Jeffrey M Wooldridge\.Control function methods in applied econometrics\.*Journal of Human Resources*, 50\(2\):420–445, 2015\.
- Wu \(1973\)De\-Min Wu\.Alternative tests of independence between stochastic regressors and disturbances\.*Econometrica*, 41\(4\):733–750, 1973\.
- Wu et al\. \(2025\)Xiaobao Wu, Liangming Pan, Yuxi Xie, Ruiwen Zhou, Shuai Zhao, Yubo Ma, Mingzhe Du, Rui Mao, Anh Tuan Luu, and William Yang Wang\.AntiLeakBench: Preventing data contamination by automatically constructing benchmarks with updated real\-world knowledge\.In*Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pp\. 18403–18419, 2025\.
- Xiong et al\. \(2024\)Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi\.Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs\.In*International Conference on Learning Representations \(ICLR\)*, 2024\.
- Xu et al\. \(2024\)Cheng Xu, Shuhao Guan, Derek Greene, and M\-Tahar Kechadi\.Benchmark data contamination of large language models: A survey\.*arXiv preprint arXiv:2406\.04244*, 2024\.
- Yan et al\. \(2026\)Yutong Yan, Raphael Tang, Zhenyu Gao, Wenxi Jiang, and Yao Lu\.DatedGPT: Preventing lookahead bias in large language models with time\-aware pretraining\.*arXiv preprint arXiv:2603\.11838*, 2026\.
- Yang et al\. \(2023\)Shuo Yang, Wei\-Lin Chiang, Lianmin Zheng, Joseph E\. Gonzalez, and Ion Stoica\.Rethinking benchmark and contamination for language models with rephrased samples\.*arXiv preprint arXiv:2311\.04850*, 2023\.
- Zawalski et al\. \(2026\)Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, and Pablo Ribalta\.Detecting data contamination in LLMs via in\-context learning\.In*International Conference on Learning Representations \(ICLR\)*, 2026\.arXiv:2510\.27055\.
- Zhang et al\. \(2024\)Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue\.A careful examination of large language model performance on grade school arithmetic\.In*Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks Track*, 2024\.arXiv:2405\.00332\.
- Zhang et al\. \(2026\)Zeyu Zhang, Ryan Chen, and Bradly C\. Stadie\.All leaks count, some count more: Interpretable temporal contamination detection and mitigation in LLM backtesting\.*arXiv preprint arXiv:2602\.17234*, 2026\.
- Zou et al\. \(2022\)Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks\.Forecasting future world events with neural networks\.In*Advances in Neural Information Processing Systems Datasets and Benchmarks Track*, 2022\.

## Appendix roadmap

Nothing in this appendix is optional filler: it holds the twin formalism demoted from the main text, every proof, the practitioner recipe, full experimental configurations, demoted main\-text detail, and the provenance map\. Read by role:

- •A\.[Appendix˜A](https://arxiv.org/html/2608.02985#A1): notation and standing conventions\.
- •B\.[Appendix˜B](https://arxiv.org/html/2608.02985#A2): counterfactual twin theory \([Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)\) plus all proofs for[Sections˜3](https://arxiv.org/html/2608.02985#S3),[4](https://arxiv.org/html/2608.02985#S4)and[5](https://arxiv.org/html/2608.02985#S5), in main\-text order, with technical complements \(including the interpretive factorization ofww\)\.
- •C\.[Appendix˜C](https://arxiv.org/html/2608.02985#A3): practitioner recipe, control validation, finite\-KKPRC machinery, paraphrase protocol, and the classical identification\-tool map \([Section˜C\.4](https://arxiv.org/html/2608.02985#A3.SS4)\)\.
- •D\.[Appendix˜D](https://arxiv.org/html/2608.02985#A4): full configurations and complete results for M1–M5, including material demoted from the main text \(M1 matched\-window check, Gemini exploratory row, clean\-anchor banding and design disagreement, code\-arm extension\)\.
- •E\.[Appendix˜E](https://arxiv.org/html/2608.02985#A5): synthetic validation \(E1\) and per\-experiment robustness\.
- •F\.[Appendix˜F](https://arxiv.org/html/2608.02985#A6): every reported number mapped to its script and output file\.

A reader checking the theory needs A–B; a reader assessing or replicating the experiments needs D–F; a practitioner deploying the routes needs C\.

## Appendix ANotation and standing conventions

##### Standing regularity\.

All predictions and outcomes have finite second moments\. Lowercasex,gx,gdenote realizations ofX,GX,G\. Conditional expectations, variances, and projections are defined almost surely on their relevant support\. WheneverVar​\(S∣X,𝒟train,G<0\)=0\\mathrm\{Var\}\(S\\mid X,\\mathcal\{D\}\_\{\\mathrm\{train\}\},G<0\)=0, the projection coefficientλ=Cov​\(δ,S\)/Var​\(S\)\\lambda=\\mathrm\{Cov\}\(\\delta,S\)/\\mathrm\{Var\}\(S\)is a0/00/0expression; we adopt the conventionλ=0\\lambda=0there, since a signal with no conditional variation carries no leakage to load on\. Common support and density conditions are invoked explicitly for the identification route that requires them\.

## Appendix BThe twin theory, proofs, and technical complements

The counterfactual twin theory summarized in[Remark˜1](https://arxiv.org/html/2608.02985#Thmremark1)comes first \([Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)\); the proofs of all formal results follow, grouped by the section in which each result appears\. Every entry states the context of the result, restates it, and then gives the proof\. Interspersed with the proofs are short technical complements that belong with the mathematics rather than the experiments: auxiliary lemmas stated only here, a bound on the systematic shift, and an interpretive parameterization of the extraction weight \([Section˜B\.3\.5](https://arxiv.org/html/2608.02985#A2.SS3.SSS5)\)\.

### B\.1The counterfactual twin theory

The counterfactual frame compares the deployed model with a clean twin retrained without the leaked data and asks what leaked information does to their score difference;[Remark˜1](https://arxiv.org/html/2608.02985#Thmremark1)in the main text summarizes the result\. The argument moves from an ideal benchmark \([Section˜B\.1\.1](https://arxiv.org/html/2608.02985#A2.SS1.SSS1)\) to general predictors \([Section˜B\.1\.2](https://arxiv.org/html/2608.02985#A2.SS1.SSS2)\) to the reason the construction cannot be run in practice \([Section˜B\.1\.3](https://arxiv.org/html/2608.02985#A2.SS1.SSS3)\)\. With the two models of[Figure˜2](https://arxiv.org/html/2608.02985#S2.F2), we writeδ=y^corrupt−y^clean\\delta=\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-\\hat\{y\}\_\{\\mathrm\{clean\}\}for their disagreement andℋ=σ​\(X,𝒟train\)\\mathcal\{H\}=\\sigma\(X,\\mathcal\{D\}\_\{\\mathrm\{train\}\}\)for the pre\-cutoff information;y^corrupt\\hat\{y\}\_\{\\mathrm\{corrupt\}\}andy^clean\\hat\{y\}\_\{\\mathrm\{clean\}\}denote the corrupt\- and clean\-trained predictions, and for a probabilistic protocoly^corrupt=P\\hat\{y\}\_\{\\mathrm\{corrupt\}\}=P\. All statements and conditional moments in this subsection are under the conditional law of leakage\-eligible questions \(G<0G<0\)\. The target is the*counterfactual inflation*

Btwin=𝔼​\[\(y^clean−Y\)2−\(y^corrupt−Y\)2∣G<0\],B\_\{\\mathrm\{twin\}\}\\;=\\;\\mathbb\{E\}\\\!\\left\[\(\\hat\{y\}\_\{\\mathrm\{clean\}\}\-Y\)^\{2\}\-\(\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-Y\)^\{2\}\\mid G<0\\right\],\(5\)positive when the deployed model looks more accurate than its counterfactual clean version\. Every result below is measured against the*leakage signal*

S=𝔼​\[Y∣X,𝒟train,𝒟bridge\]⏟y^Bayes−𝔼​\[Y∣X,𝒟train\]⏟y^Bayes,clean,S\\;=\\;\\underbrace\{\\mathbb\{E\}\[Y\\mid X,\\mathcal\{D\}\_\{\\mathrm\{train\}\},\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\]\}\_\{\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\}\\;\-\\;\\underbrace\{\\mathbb\{E\}\[Y\\mid X,\\mathcal\{D\}\_\{\\mathrm\{train\}\}\]\}\_\{\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}\},\(6\)the amount by which an ideal learner would revise its forecast upon seeing the leaked data: large exactly when the bridge data would move an ideal forecaster’s belief aboutYY, zero when they carry no information about the question\. Because the conditional expectation is the best possible use of the bridge data,SSsummarizes all any model could extract from𝒟bridge\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}aboutYY—the natural yardstick for leakage\.

#### B\.1\.1Bayes\-optimal extraction: the value of leaked information

Suppose first that both predictors are Bayes\-optimal,y^clean=y^Bayes,clean\\hat\{y\}\_\{\\mathrm\{clean\}\}=\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}andy^corrupt=y^Bayes\\hat\{y\}\_\{\\mathrm\{corrupt\}\}=\\hat\{y\}\_\{\\mathrm\{Bayes\}\}: the deployed model extracts everything the leaked data contain, and its disagreement equals the signal \(δ=S\\delta=S\)\.

\{restatable\}

\[Bayesian value of bridge information\]theoremthmbayesian For the Bayes\-optimal pair,

Btwin,Bayes=𝔼​\[Var​\(S∣ℋ\)∣G<0\]≥0\.B\_\{\\mathrm\{twin,Bayes\}\}=\\mathbb\{E\}\\\!\\left\[\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\\mid G<0\\right\]\\geq 0\.
The inflation is*always*non\-negative for the full extractor, and its magnitude equals how much the post\-cutoff world deviates from the pre\-cutoff world, as seen through the outcome: volatile, surprising periods \(largeVar​\(S∣ℋ\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\) admit large inflation, stable ones almost none\. This is the value of bridge information for Bayes\-optimal prediction, not a universal upper bound over arbitrary algorithm pairs\.

#### B\.1\.2General predictors: decomposition along the signal

The model under backtest is not Bayes\-optimal: it extracts only a fraction ofSS, and not necessarily alongSS\. To see how much of the ideal value it captures—and at what cost—we take the conditionalL2L^\{2\}projection of its perturbationδ\\deltaonto the signal,

δ=λ​S\+μδ\+ν~,λ=Cov​\(δ,S∣ℋ\)Var​\(S∣ℋ\),μδ=𝔼​\[δ∣ℋ\],\\delta\\;=\\;\\lambda\\,S\\;\+\\;\\mu\_\{\\delta\}\\;\+\\;\\tilde\{\\nu\},\\qquad\\lambda=\\frac\{\\mathrm\{Cov\}\(\\delta,S\\mid\\mathcal\{H\}\)\}\{\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\},\\qquad\\mu\_\{\\delta\}=\\mathbb\{E\}\[\\delta\\mid\\mathcal\{H\}\],\(7\)withλ=0\\lambda=0on strata where the denominator vanishes; the residualν~\\tilde\{\\nu\}has conditional mean zero and is conditionally orthogonal toSS\. Two benchmark deviations complete the bookkeeping:rclean=y^clean−y^Bayes,cleanr\_\{\\mathrm\{clean\}\}=\\hat\{y\}\_\{\\mathrm\{clean\}\}\-\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}, the clean model’s deviation from its own Bayes benchmark, andg⟂=Y−y^Bayesg\_\{\\perp\}=Y\-\\hat\{y\}\_\{\\mathrm\{Bayes\}\}, the Bayesian residual—the part of the outcome that no training data can predict; the clean gap then decomposes asγ=Y−y^clean=S\+g⟂−rclean\\gamma=Y\-\\hat\{y\}\_\{\\mathrm\{clean\}\}=S\+g\_\{\\perp\}\-r\_\{\\mathrm\{clean\}\}\.[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)depicts the geometry of the two decompositions and why only theSS\-aligned component ofδ\\deltacan pay off\. Substituting \([7](https://arxiv.org/html/2608.02985#A2.E7)\) into \([5](https://arxiv.org/html/2608.02985#A2.E5)\) gives the central decomposition\.

\(a\)SSλ​S\\lambda Sg⟂g\_\{\\perp\}γ\\gammaν~\\tilde\{\\nu\}δ\\delta

\(b\)λ\\lambdainflation/Var​\(S\)/\\,\\mathrm\{Var\}\(S\)0\.50\.5110\.750\.7511benefit2​λ2\\lambdacostλ2\\lambda^\{2\}λ​\(2−λ\)\\lambda\(2\-\\lambda\)

Figure 5:Why leakage inflates a backtest: the decomposition \(a\) and the double benefit it implies \(b\)\.\(a\)What the clean model does not know about the outcome—the clean gapγ\\gamma\(green\)—splits into the leakage signalSS\(blue\), everything the leaked data reveal aboutYY, and the residualg⟂g\_\{\\perp\}\(red\), which no training data can predict\. Training on leaked data moves the model byδ\\delta\(purple\), of which only the extracted componentλ​S\\lambda S\(orange\) overlapsγ\\gammaand can raise the score; the noiseν~\\tilde\{\\nu\}\(dashed\) strictly deflates it \([Corollary˜3](https://arxiv.org/html/2608.02985#Thmtheorem3)\)\.\(b\)Extracting a fractionλ\\lambdaof the signal buys a score benefit linear inλ\\lambdaat a variance cost only quadratic inλ\\lambda, so the net inflationλ​\(2−λ\)​Var​\(S\)\\lambda\(2\-\\lambda\)\\mathrm\{Var\}\(S\)rises steeply from zero: extracting half the signal \(dashes\) already yields75%75\\%of the full\-memorization inflation\. A faint leakage trace suffices to distort a backtest\. The per\-question factorw​\(2−w\)w\(2\-w\)of[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)traces the identical curve withwwin place ofλ\\lambda\.\{restatable\}

\[Leakage inflation decomposition\]theoremthmdecomposition For any clean/corrupt predictor pair with finite second moments, withδ\\delta,SS,λ\\lambda,μδ\\mu\_\{\\delta\},ν~\\tilde\{\\nu\}as in \([7](https://arxiv.org/html/2608.02985#A2.E7)\) andrcleanr\_\{\\mathrm\{clean\}\},g⟂g\_\{\\perp\}as above,

Btwin=Psignal−Rshift−Rnoise\+Rcross,B\_\{\\mathrm\{twin\}\}\\;=\\;P\_\{\\mathrm\{signal\}\}\\;\-\\;R\_\{\\mathrm\{shift\}\}\\;\-\\;R\_\{\\mathrm\{noise\}\}\\;\+\\;R\_\{\\mathrm\{cross\}\},\(8\)where, with all outer expectations conditional onG<0G<0,Psignal=𝔼​\[λ​\(2−λ\)​Var​\(S∣ℋ\)\]P\_\{\\mathrm\{signal\}\}=\\mathbb\{E\}\[\\lambda\(2\-\\lambda\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\],Rshift=𝔼​\[μδ2\+2​μδ​rclean\]R\_\{\\mathrm\{shift\}\}=\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\+2\\mu\_\{\\delta\}r\_\{\\mathrm\{clean\}\}\],Rnoise=𝔼​\[Var​\(ν~∣ℋ\)\]R\_\{\\mathrm\{noise\}\}=\\mathbb\{E\}\[\\mathrm\{Var\}\(\\tilde\{\\nu\}\\mid\\mathcal\{H\}\)\], andRcross=2​𝔼​\[ν~​g⟂\]R\_\{\\mathrm\{cross\}\}=2\\,\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\]\. Forλ∈\[0,2\]\\lambda\\in\[0,2\],Psignal≥0P\_\{\\mathrm\{signal\}\}\\geq 0\.

\{restatable\}

\[The cross\-term vanishes\]lemmalemcrossterm For any such pair,Rcross=0R\_\{\\mathrm\{cross\}\}=0; henceBtwin=Psignal−Rshift−RnoiseB\_\{\\mathrm\{twin\}\}=P\_\{\\mathrm\{signal\}\}\-R\_\{\\mathrm\{shift\}\}\-R\_\{\\mathrm\{noise\}\}\.

The cross\-term vanishes identically—not by assumption—because the Bayesian residualg⟂g\_\{\\perp\}is orthogonal to every square\-integrableσ​\(X,𝒟train,𝒟bridge\)\\sigma\(X,\\mathcal\{D\}\_\{\\mathrm\{train\}\},\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\)\-measurable variable, andν~\\tilde\{\\nu\}is such a variable\. The identity itself imposes no sign restriction onλ\\lambda\. Under the additional non\-adversarial conditionλ∈\[0,2\]\\lambda\\in\[0,2\],Psignal≥0P\_\{\\mathrm\{signal\}\}\\geq 0;Rnoise≥0R\_\{\\mathrm\{noise\}\}\\geq 0deflates the contrast, and the shiftRshiftR\_\{\\mathrm\{shift\}\}is bounded and vanishes whenμδ=0\\mu\_\{\\delta\}=0\([Section˜B\.2\.4](https://arxiv.org/html/2608.02985#A2.SS2.SSS4)\)\. Reading the signs together yields the asymmetry summarized in[Section˜4](https://arxiv.org/html/2608.02985#S4)\.

###### Corollary 3\(No free inflation\)\.

Ifλ=0\\lambda=0andμδ=0\\mu\_\{\\delta\}=0almost surely, thenBtwin=−Rnoise≤0B\_\{\\mathrm\{twin\}\}=\-R\_\{\\mathrm\{noise\}\}\\leq 0\.

The corollary is immediate from[Figures˜5](https://arxiv.org/html/2608.02985#A2.F5)and[5](https://arxiv.org/html/2608.02985#A2.F5)\. A perturbation uncorrelated with the leakage signal cannot inflate the backtest: change orthogonal toSSstrictly*deflates*the score, so observed inflation is evidence of extraction\. This asymmetry is what makes detection meaningful; the detection statistic of[Section˜5](https://arxiv.org/html/2608.02985#S5)is built on it\.

##### The double benefit in the twin frame\.

The signal term carries the factorλ​\(2−λ\)=1−\(1−λ\)2\\lambda\(2\-\\lambda\)=1\-\(1\-\\lambda\)^\{2\}, the twin\-frame analogue of the per\-question factorw​\(2−w\)w\(2\-w\)in[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)\. The arithmetic is the same: extracting a fractionλ\\lambdaofSSearns a covariance gain of2​λ​Var​\(S\)2\\lambda\\mathrm\{Var\}\(S\)—the forecast now moves*with*the truth, a benefit linear inλ\\lambda—while the variance cost of the added component is onlyλ2​Var​\(S\)\\lambda^\{2\}\\mathrm\{Var\}\(S\), quadratic inλ\\lambda\. The net isλ​\(2−λ\)​Var​\(S\)\\lambda\(2\-\\lambda\)\\mathrm\{Var\}\(S\)\([Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)\)\. The two factors coincide under the loading correspondence of[Lemma˜6](https://arxiv.org/html/2608.02985#Thmtheorem6):λ\\lambdaloadsδ\\deltaontoSSwith signal strengthVar​\(S∣ℋ\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\),wwis the per\-question extraction with signal strengthb0b\_\{0\}, and when the crowd forecast proxies the honest forecast, the observable crowd surprise plays the role of the latentVar​\(S∣ℋ\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\.

#### B\.1\.3Infeasibility of the counterfactual comparison

The decomposition is exact and cleanly isolates leakage: the deployed model and its counterfactual clean version share the same pre\-t0t\_\{0\}data, so everything they have in common—including recency—cancels inδ\\delta, leaving the pure effect of the bridge data\. The obstruction is that*the counterfactual model cannot be instantiated for a deployed system*: one cannot un\-train a released black box, and the Bayes\-optimal predictors, the signalSS, and the loadingλ\\lambdaare all uncomputable\. Without the counterfactual model, the only within\-model comparison left is*across time*—pre\- versus post\-cutoff on the one deployed model—and there the cancellation fails, because the model is genuinely fresher near its cutoff: the time comparison reintroduces recency as a confound\. The main text therefore works with the operational inflationBBof \([2](https://arxiv.org/html/2608.02985#S2.E2)\), defined on observables of a single model with recency modeled explicitly; it coincides withBtwinB\_\{\\mathrm\{twin\}\}under the alignment conditions of[Lemma˜5](https://arxiv.org/html/2608.02985#Thmtheorem5)\([Section˜B\.3](https://arxiv.org/html/2608.02985#A2.SS3)\), and the price of losing the counterfactual comparison is the non\-identifiability of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\.

### B\.2Proofs for[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1): the leakage inflation mechanism

All expectations in this subsection are under the conditional law givenG<0G<0\. Letℋ=σ​\(X,𝒟train\)\\mathcal\{H\}=\\sigma\(X,\\mathcal\{D\}\_\{\\mathrm\{train\}\}\)andℱ=σ​\(X,𝒟train,𝒟bridge\)\\mathcal\{F\}=\\sigma\(X,\\mathcal\{D\}\_\{\\mathrm\{train\}\},\\mathcal\{D\}\_\{\\mathrm\{bridge\}\}\), withℋ⊆ℱ\\mathcal\{H\}\\subseteq\\mathcal\{F\}\. The leakage signal isS=y^Bayes−y^Bayes,cleanS=\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\-\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}, wherey^Bayes=𝔼​\[Y∣ℱ\]\\hat\{y\}\_\{\\mathrm\{Bayes\}\}=\\mathbb\{E\}\[Y\\mid\\mathcal\{F\}\]andy^Bayes,clean=𝔼​\[Y∣ℋ\]\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}=\\mathbb\{E\}\[Y\\mid\\mathcal\{H\}\]\. Further,δ=y^corrupt−y^clean\\delta=\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-\\hat\{y\}\_\{\\mathrm\{clean\}\},γ=Y−y^clean\\gamma=Y\-\\hat\{y\}\_\{\\mathrm\{clean\}\},g⟂=Y−y^Bayesg\_\{\\perp\}=Y\-\\hat\{y\}\_\{\\mathrm\{Bayes\}\}, andrclean=y^clean−y^Bayes,cleanr\_\{\\mathrm\{clean\}\}=\\hat\{y\}\_\{\\mathrm\{clean\}\}\-\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}\. Under the fixed deterministic protocol of[Section˜2](https://arxiv.org/html/2608.02985#S2), predictions are measurable with respect to the information available to the model:y^clean\\hat\{y\}\_\{\\mathrm\{clean\}\}isℋ\\mathcal\{H\}\-measurable andy^corrupt\\hat\{y\}\_\{\\mathrm\{corrupt\}\}isℱ\\mathcal\{F\}\-measurable, soδ\\delta,rcleanr\_\{\\mathrm\{clean\}\}, and \(below\)ν~\\tilde\{\\nu\}areℱ\\mathcal\{F\}\-measurable\. The decompositionδ=λ​S\+μδ\+ν~\\delta=\\lambda S\+\\mu\_\{\\delta\}\+\\tilde\{\\nu\}is the conditionalL2L^\{2\}projection givenℋ\\mathcal\{H\}, withλ\\lambdaandμδ\\mu\_\{\\delta\}bothℋ\\mathcal\{H\}\-measurable\.

#### B\.2\.1Proof of[Section˜B\.1\.1](https://arxiv.org/html/2608.02985#A2.SS1.SSS1)

[Section˜B\.1\.1](https://arxiv.org/html/2608.02985#A2.SS1.SSS1)is the ideal\-case benchmark of[Section˜B\.1\.1](https://arxiv.org/html/2608.02985#A2.SS1.SSS1): when both predictors are Bayes\-optimal, the inflation equals the full information value of the bridge data\.

###### Proof\.

For the Bayes\-optimal pair,y^clean=y^Bayes,clean=𝔼​\[Y∣ℋ\]\\hat\{y\}\_\{\\mathrm\{clean\}\}=\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}=\\mathbb\{E\}\[Y\\mid\\mathcal\{H\}\]andy^corrupt=y^Bayes=𝔼​\[Y∣ℱ\]\\hat\{y\}\_\{\\mathrm\{corrupt\}\}=\\hat\{y\}\_\{\\mathrm\{Bayes\}\}=\\mathbb\{E\}\[Y\\mid\\mathcal\{F\}\], so theℋ\\mathcal\{H\}\-conditional inflation is

b​\(ℋ\)=𝔼​\[\(y^Bayes,clean−Y\)2∣ℋ\]−𝔼​\[\(y^Bayes−Y\)2∣ℋ\]\.b\(\\mathcal\{H\}\)=\\mathbb\{E\}\\\!\\left\[\(\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}\-Y\)^\{2\}\\mid\\mathcal\{H\}\\right\]\-\\mathbb\{E\}\\\!\\left\[\(\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\-Y\)^\{2\}\\mid\\mathcal\{H\}\\right\]\.By the minimum\-mean\-squared\-error property of conditional expectation, the first term equalsVar​\(Y∣ℋ\)\\mathrm\{Var\}\(Y\\mid\\mathcal\{H\}\), while𝔼​\[\(y^Bayes−Y\)2∣ℱ\]=Var​\(Y∣ℱ\)\\mathbb\{E\}\[\(\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\-Y\)^\{2\}\\mid\\mathcal\{F\}\]=\\mathrm\{Var\}\(Y\\mid\\mathcal\{F\}\); conditioning the latter onℋ⊆ℱ\\mathcal\{H\}\\subseteq\\mathcal\{F\}and invoking the law of total variance,

Var​\(Y∣ℋ\)=𝔼​\[Var​\(Y∣ℱ\)∣ℋ\]\+Var​\(y^Bayes∣ℋ\),\\mathrm\{Var\}\(Y\\mid\\mathcal\{H\}\)=\\mathbb\{E\}\\\!\\left\[\\mathrm\{Var\}\(Y\\mid\\mathcal\{F\}\)\\mid\\mathcal\{H\}\\right\]\+\\mathrm\{Var\}\(\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\\mid\\mathcal\{H\}\),we conclude thatb​\(ℋ\)=Var​\(y^Bayes∣ℋ\)b\(\\mathcal\{H\}\)=\\mathrm\{Var\}\(\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\\mid\\mathcal\{H\}\)\. Moreover𝔼​\[S∣ℋ\]=𝔼​\[y^Bayes∣ℋ\]−y^Bayes,clean=0\\mathbb\{E\}\[S\\mid\\mathcal\{H\}\]=\\mathbb\{E\}\[\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\\mid\\mathcal\{H\}\]\-\\hat\{y\}\_\{\\mathrm\{Bayes,clean\}\}=0by the tower property, soVar​\(S∣ℋ\)=Var​\(y^Bayes∣ℋ\)=b​\(ℋ\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)=\\mathrm\{Var\}\(\\hat\{y\}\_\{\\mathrm\{Bayes\}\}\\mid\\mathcal\{H\}\)=b\(\\mathcal\{H\}\)\. Taking expectations overℋ\\mathcal\{H\}yieldsBtwin,Bayes=𝔼​\[Var​\(S∣ℋ\)\]≥0B\_\{\\mathrm\{twin,Bayes\}\}=\\mathbb\{E\}\[\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\]\\geq 0, with equality if and only ifS=0S=0almost surely\. ∎

#### B\.2\.2Proof of[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)

[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)is the central decomposition of[Section˜B\.1\.2](https://arxiv.org/html/2608.02985#A2.SS1.SSS2): a general predictor’s inflation splits into a signal term, a shift term, a noise term, and a cross term\.[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)in the main text depicts the geometry\.

###### Proof\.

Sincey^clean−Y=−γ\\hat\{y\}\_\{\\mathrm\{clean\}\}\-Y=\-\\gammaandy^corrupt−Y=δ−γ\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-Y=\\delta\-\\gamma, definition \([5](https://arxiv.org/html/2608.02985#A2.E5)\) reads

Btwin=𝔼​\[γ2−\(δ−γ\)2\]=𝔼​\[2​δ​γ−δ2\],B\_\{\\mathrm\{twin\}\}=\\mathbb\{E\}\\\!\\left\[\\gamma^\{2\}\-\(\\delta\-\\gamma\)^\{2\}\\right\]=\\mathbb\{E\}\\\!\\left\[2\\delta\\gamma\-\\delta^\{2\}\\right\],and it suffices to compute the two conditional moments𝔼​\[δ2∣ℋ\]\\mathbb\{E\}\[\\delta^\{2\}\\mid\\mathcal\{H\}\]and𝔼​\[δ​γ∣ℋ\]\\mathbb\{E\}\[\\delta\\gamma\\mid\\mathcal\{H\}\]\. Throughout, recall thatλ\\lambda,μδ\\mu\_\{\\delta\}, andrcleanr\_\{\\mathrm\{clean\}\}areℋ\\mathcal\{H\}\-measurable, that𝔼​\[S∣ℋ\]=0\\mathbb\{E\}\[S\\mid\\mathcal\{H\}\]=0by the tower property, and that𝔼​\[ν~∣ℋ\]=𝔼​\[S​ν~∣ℋ\]=0\\mathbb\{E\}\[\\tilde\{\\nu\}\\mid\\mathcal\{H\}\]=\\mathbb\{E\}\[S\\tilde\{\\nu\}\\mid\\mathcal\{H\}\]=0by construction of the projection \([7](https://arxiv.org/html/2608.02985#A2.E7)\); we suppress the conditioning onℋ\\mathcal\{H\}in the displays below\.

The three components ofδ=λ​S\+μδ\+ν~\\delta=\\lambda S\+\\mu\_\{\\delta\}\+\\tilde\{\\nu\}are conditionally uncorrelated, so

𝔼​\[δ2\]=λ2​Var​\(S\)\+μδ2\+Var​\(ν~\)\.\\mathbb\{E\}\[\\delta^\{2\}\]=\\lambda^\{2\}\\mathrm\{Var\}\(S\)\+\\mu\_\{\\delta\}^\{2\}\+\\mathrm\{Var\}\(\\tilde\{\\nu\}\)\.For the cross moment, substitutingγ=S\+g⟂−rclean\\gamma=S\+g\_\{\\perp\}\-r\_\{\\mathrm\{clean\}\}and expanding the productδ​γ=\(λ​S\+μδ\+ν~\)​\(S\+g⟂−rclean\)\\delta\\gamma=\(\\lambda S\+\\mu\_\{\\delta\}\+\\tilde\{\\nu\}\)\(S\+g\_\{\\perp\}\-r\_\{\\mathrm\{clean\}\}\)produces nine terms, of which all but three vanish:𝔼​\[S​g⟂\]=𝔼​\[g⟂\]=0\\mathbb\{E\}\[Sg\_\{\\perp\}\]=\\mathbb\{E\}\[g\_\{\\perp\}\]=0becauseg⟂=Y−y^Bayesg\_\{\\perp\}=Y\-\\hat\{y\}\_\{\\mathrm\{Bayes\}\}is the minimum\-mean\-squared\-error residual, orthogonal to every square\-integrableℱ\\mathcal\{F\}\-measurable variable \(in particular toSSand to constants\), while𝔼​\[S\]=𝔼​\[ν~\]=𝔼​\[S​ν~\]=0\\mathbb\{E\}\[S\]=\\mathbb\{E\}\[\\tilde\{\\nu\}\]=\\mathbb\{E\}\[S\\tilde\{\\nu\}\]=0as noted above\. What survives is

𝔼​\[δ​γ\]=λ​Var​\(S\)−μδ​rclean\+𝔼​\[ν~​g⟂\]\.\\mathbb\{E\}\[\\delta\\gamma\]=\\lambda\\mathrm\{Var\}\(S\)\-\\mu\_\{\\delta\}r\_\{\\mathrm\{clean\}\}\+\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\]\.Combining the two moments,

𝔼​\[2​δ​γ−δ2∣ℋ\]=λ​\(2−λ\)​Var​\(S∣ℋ\)−\(μδ2\+2​μδ​rclean\)−Var​\(ν~∣ℋ\)\+2​𝔼​\[ν~​g⟂∣ℋ\],\\mathbb\{E\}\\\!\\left\[2\\delta\\gamma\-\\delta^\{2\}\\mid\\mathcal\{H\}\\right\]=\\lambda\(2\-\\lambda\)\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)\-\\left\(\\mu\_\{\\delta\}^\{2\}\+2\\mu\_\{\\delta\}r\_\{\\mathrm\{clean\}\}\\right\)\-\\mathrm\{Var\}\(\\tilde\{\\nu\}\\mid\\mathcal\{H\}\)\+2\\,\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\\mid\\mathcal\{H\}\],and taking expectations overℋ\\mathcal\{H\}yields \([8](https://arxiv.org/html/2608.02985#A2.E8)\)\. Forλ∈\[0,2\]\\lambda\\in\[0,2\]the factorλ​\(2−λ\)\\lambda\(2\-\\lambda\)is non\-negative, whencePsignal≥0P\_\{\\mathrm\{signal\}\}\\geq 0\. Specializing to the Bayes\-optimal pair \(λ=1\\lambda=1,μδ=0\\mu\_\{\\delta\}=0,ν~=0\\tilde\{\\nu\}=0\) recovers[Section˜B\.1\.1](https://arxiv.org/html/2608.02985#A2.SS1.SSS1)\. ∎

#### B\.2\.3Proof of[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)

[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)removes the only unsigned term of[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5), reducing the inflation to signal minus shift minus noise\.

###### Proof\.

The Bayesian residualg⟂=Y−𝔼​\[Y∣ℱ\]g\_\{\\perp\}=Y\-\\mathbb\{E\}\[Y\\mid\\mathcal\{F\}\]satisfies𝔼​\[g⟂​h\]=0\\mathbb\{E\}\[g\_\{\\perp\}h\]=0for every square\-integrableℱ\\mathcal\{F\}\-measurablehh\. As recorded in the preamble to this appendix,ν~=δ−λ​S−μδ\\tilde\{\\nu\}=\\delta\-\\lambda S\-\\mu\_\{\\delta\}isℱ\\mathcal\{F\}\-measurable, so takingh=ν~h=\\tilde\{\\nu\}gives𝔼​\[ν~​g⟂\]=0\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\]=0, and thereforeRcross=2​𝔼​\[ν~​g⟂\]=0R\_\{\\mathrm\{cross\}\}=2\\,\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\]=0; the identityBtwin=Psignal−Rshift−RnoiseB\_\{\\mathrm\{twin\}\}=P\_\{\\mathrm\{signal\}\}\-R\_\{\\mathrm\{shift\}\}\-R\_\{\\mathrm\{noise\}\}then follows from[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)\. The same argument applied conditionally shows𝔼​\[ν~​g⟂∣ℋ\]=0\\mathbb\{E\}\[\\tilde\{\\nu\}g\_\{\\perp\}\\mid\\mathcal\{H\}\]=0almost surely\. ∎

#### B\.2\.4The systematic shift is bounded

The following proposition, stated and proved only here, bounds the shift termRshiftR\_\{\\mathrm\{shift\}\}of[Figure˜5](https://arxiv.org/html/2608.02985#A2.F5)and identifies when it vanishes\.

###### Proposition 4\(Bounded systematic shift\)\.

For any model,\|Rshift\|≤𝔼​\[μδ2\]\+2​𝔼​\[μδ2\]​𝔼​\[rclean2\]\|R\_\{\\mathrm\{shift\}\}\|\\leq\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\+2\\sqrt\{\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\\,\\mathbb\{E\}\[r\_\{\\mathrm\{clean\}\}^\{2\}\]\}, where𝔼​\[μδ2\]≤𝔼​\[δ2\]\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\\leq\\mathbb\{E\}\[\\delta^\{2\}\]and𝔼​\[rclean2\]\\mathbb\{E\}\[r\_\{\\mathrm\{clean\}\}^\{2\}\]is the clean model’s excess risk relative to the clean Bayesian predictor\. In particularRshift=0R\_\{\\mathrm\{shift\}\}=0for any predictor withμδ≡0\\mu\_\{\\delta\}\\equiv 0\(including the Bayesian and any conditionally unbiased predictor\)\.

###### Proof\.

SinceRshift=𝔼​\[μδ2\]\+2​𝔼​\[μδ​rclean\]R\_\{\\mathrm\{shift\}\}=\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\+2\\,\\mathbb\{E\}\[\\mu\_\{\\delta\}r\_\{\\mathrm\{clean\}\}\], the Cauchy–Schwarz inequality\|𝔼​\[μδ​rclean\]\|≤\(𝔼​\[μδ2\]​𝔼​\[rclean2\]\)1/2\|\\mathbb\{E\}\[\\mu\_\{\\delta\}r\_\{\\mathrm\{clean\}\}\]\|\\leq\(\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\\,\\mathbb\{E\}\[r\_\{\\mathrm\{clean\}\}^\{2\}\]\)^\{1/2\}together with the triangle inequality yields the stated bound\. Jensen’s inequality applied toμδ=𝔼​\[δ∣ℋ\]\\mu\_\{\\delta\}=\\mathbb\{E\}\[\\delta\\mid\\mathcal\{H\}\]givesμδ2≤𝔼​\[δ2∣ℋ\]\\mu\_\{\\delta\}^\{2\}\\leq\\mathbb\{E\}\[\\delta^\{2\}\\mid\\mathcal\{H\}\]pointwise, whence𝔼​\[μδ2\]≤𝔼​\[δ2\]\\mathbb\{E\}\[\\mu\_\{\\delta\}^\{2\}\]\\leq\\mathbb\{E\}\[\\delta^\{2\}\]\. Finally, ifμδ≡0\\mu\_\{\\delta\}\\equiv 0then both terms ofRshiftR\_\{\\mathrm\{shift\}\}vanish; this holds in particular for the Bayes\-optimal pair, for whichμδ=𝔼​\[S∣ℋ\]=0\\mu\_\{\\delta\}=\\mathbb\{E\}\[S\\mid\\mathcal\{H\}\]=0\. ∎

### B\.3Results of[Sections˜3](https://arxiv.org/html/2608.02985#S3)and[4](https://arxiv.org/html/2608.02985#S4): non\-identifiability and concentration

Throughout this subsection,m​\(x,g\)=𝔼​\[\(P−Y\)2∣X=x,G=g\]m\(x,g\)=\\mathbb\{E\}\[\(P\-Y\)^\{2\}\\mid X=x,G=g\]is the observable conditional mean loss \([1](https://arxiv.org/html/2608.02985#S2.E1)\),\(m0,L\)\(m\_\{0\},L\)an operational decomposition satisfying[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1), and the*passive law*is the joint distribution of\(P,Y,X,G\)\(P,Y,X,G\)\.

#### B\.3\.1The alignment lemma

The following lemma, referenced in[Remark˜1](https://arxiv.org/html/2608.02985#Thmremark1), bridges the two halves of the theory: under its hypotheses, the counterfactual inflation of[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)and the operational estimand of[Section˜2](https://arxiv.org/html/2608.02985#S2)are the same number\.

###### Lemma 5\(Alignment of counterfactual and operational estimands\)\.

Suppose that, on the same pre\-cutoff question distribution,𝔼​\[\(y^clean−Y\)2∣X,G\]=m0​\(X,G\)\\mathbb\{E\}\[\(\\hat\{y\}\_\{\\mathrm\{clean\}\}\-Y\)^\{2\}\\mid X,G\]=m\_\{0\}\(X,G\)almost surely on\{G<0\}\\\{G<0\\\}\. ThenBtwin=BB\_\{\\mathrm\{twin\}\}=B\.

The deployed model’s side needs no hypothesis, since𝔼​\[\(y^corrupt−Y\)2∣X,G\]=m​\(X,G\)\\mathbb\{E\}\[\(\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-Y\)^\{2\}\\mid X,G\]=m\(X,G\)holds by definition ofmm\(y^corrupt=P\\hat\{y\}\_\{\\mathrm\{corrupt\}\}=P,[Section˜2](https://arxiv.org/html/2608.02985#S2)\)\. The one substantive condition equates the counterfactual clean model’s risk with the honest surfacem0m\_\{0\}, plausible when the pair shares the training algorithm and protocol of[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)\.

###### Proof\.

The identity𝔼​\[\(y^corrupt−Y\)2∣X,G\]=m​\(X,G\)\\mathbb\{E\}\[\(\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-Y\)^\{2\}\\mid X,G\]=m\(X,G\)holds by definition of the observable surface \([1](https://arxiv.org/html/2608.02985#S2.E1)\), sincey^corrupt=P\\hat\{y\}\_\{\\mathrm\{corrupt\}\}=Punder the probabilistic protocol of[Section˜2](https://arxiv.org/html/2608.02985#S2); the hypothesis supplies the matching identity𝔼​\[\(y^clean−Y\)2∣X,G\]=m0​\(X,G\)\\mathbb\{E\}\[\(\\hat\{y\}\_\{\\mathrm\{clean\}\}\-Y\)^\{2\}\\mid X,G\]=m\_\{0\}\(X,G\)for the clean model on\{G<0\}\\\{G<0\\\}\. Taking expectations over\(X,G\)\(X,G\)under the pre\-cutoff law and applying iterated expectations,

Btwin\\displaystyle B\_\{\\mathrm\{twin\}\}=𝔼​\[\(y^clean−Y\)2−\(y^corrupt−Y\)2∣G<0\]=𝔼​\[m0​\(X,G\)−m​\(X,G\)∣G<0\]\\displaystyle=\\mathbb\{E\}\\\!\\left\[\(\\hat\{y\}\_\{\\mathrm\{clean\}\}\-Y\)^\{2\}\-\(\\hat\{y\}\_\{\\mathrm\{corrupt\}\}\-Y\)^\{2\}\\mid G<0\\right\]=\\mathbb\{E\}\\\!\\left\[m\_\{0\}\(X,G\)\-m\(X,G\)\\mid G<0\\right\]=𝔼​\[L​\(X,G\)∣G<0\]=B,\\displaystyle=\\mathbb\{E\}\\\!\\left\[L\(X,G\)\\mid G<0\\right\]=B,where the third equality is the operational decompositionm=m0−Lm=m\_\{0\}\-L\([Section˜3](https://arxiv.org/html/2608.02985#S3)\)\. ∎

#### B\.3\.2Proof of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)

[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)is the paper’s central negative result: passive backtest data determine the observable surfacemmbut nothing more, so the inflationBBranges over a sharp interval\.

###### Proof\.

The argument has three parts: observational equivalence, attainability, and sharpness\.

*Observational equivalence\.*The passive law of\(P,Y,X,G\)\(P,Y,X,G\)determines the conditional mean lossmmthrough \([1](https://arxiv.org/html/2608.02985#S2.E1)\), but neitherm0m\_\{0\}norLLseparately: these are not functions of the observables, and any admissible pair\(m~0,L~\)\(\\tilde\{m\}\_\{0\},\\tilde\{L\}\)withm~0−L~=m\\tilde\{m\}\_\{0\}\-\\tilde\{L\}=mis consistent with the same passive law\. Consequently, if two admissible pairs yield different values ofB=𝔼​\[m~0−m∣G<0\]B=\\mathbb\{E\}\[\\tilde\{m\}\_\{0\}\-m\\mid G<0\], no functional of the passive law can equalBBfor both, andBBis not identified\.

*Attainability\.*Fixt∈\[0,1\]t\\in\[0,1\]and define

m~0,t=m\+t​\(1−m\)​on​\{g<0\},m~0,t=m​on​\{g≥0\}\.\\tilde\{m\}\_\{0,t\}=m\+t\(1\-m\)\\ \\text\{on\}\\ \\\{g<0\\\},\\qquad\\tilde\{m\}\_\{0,t\}=m\\ \\text\{on\}\\ \\\{g\\geq 0\\\}\.Sincem∈\[0,1\]m\\in\[0,1\], we havem≤m~0,t≤1m\\leq\\tilde\{m\}\_\{0,t\}\\leq 1, som~0,t\\tilde\{m\}\_\{0,t\}is a valid risk surface; the inducedL~t=t​\(1−m\)≥0\\tilde\{L\}\_\{t\}=t\(1\-m\)\\geq 0on\{g<0\}\\\{g<0\\\}andL~t=0\\tilde\{L\}\_\{t\}=0on\{g≥0\}\\\{g\\geq 0\\\}, so[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)holds\. The corresponding inflation isB~t=t​𝔼​\[1−m∣G<0\]\\tilde\{B\}\_\{t\}=t\\,\\mathbb\{E\}\[1\-m\\mid G<0\], which is continuous and increasing inttand traces every value in\[0,𝔼​\[1−m∣G<0\]\]\[0,\\mathbb\{E\}\[1\-m\\mid G<0\]\]asttranges over\[0,1\]\[0,1\]\.

*Sharpness\.*Conversely, any admissible pair satisfies, pointwise on\{g<0\}\\\{g<0\\\},0≤L~=m~0−m≤1−m0\\leq\\tilde\{L\}=\\tilde\{m\}\_\{0\}\-m\\leq 1\-m, the upper bound becausem~0≤1\\tilde\{m\}\_\{0\}\\leq 1; taking expectations under the pre\-cutoff law placesB~\\tilde\{B\}in the stated interval\. The identified set is therefore exactly\[0,𝔼​\[1−m∣G<0\]\]\[0,\\mathbb\{E\}\[1\-m\\mid G<0\]\], and additional passive draws refine the estimate ofmmwithout shrinking it\. ∎

#### B\.3\.3Proof of[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)

[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)converts the convex\-pull model into the stakes×\\timesextraction law: leakage per question is the honest Brier times the double\-benefit factor\.

###### Proof\.

Fix a leakage\-eligible questionqq\. Under[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)and the fixed deterministic protocol,

P​\(q\)−Y​\(q\)=\(1−w​\(q\)\)​\(Phon​\(q\)−Y​\(q\)\),P\(q\)\-Y\(q\)=\(1\-w\(q\)\)\\,\\big\(P\_\{\\mathrm\{hon\}\}\(q\)\-Y\(q\)\\big\),so the realized Brier loss is\(P−Y\)2=\(1−w\)2​b0\(P\-Y\)^\{2\}=\(1\-w\)^\{2\}b\_\{0\}withb0=\(Phon−Y\)2b\_\{0\}=\(P\_\{\\mathrm\{hon\}\}\-Y\)^\{2\}, while the honest loss isb0b\_\{0\}; the per\-question saving isb0−\(1−w\)2​b0=b0​w​\(2−w\)b\_\{0\}\-\(1\-w\)^\{2\}b\_\{0\}=b\_\{0\}\\,w\(2\-w\)\. Taking conditional expectations given\(X,G\)=\(x,g\)\(X,G\)=\(x,g\)yields the surfaces

m​\(x,g\)=𝔼​\[\(1−w\)2​b0∣X=x,G=g\],m0​\(x,g\)=𝔼​\[b0∣X=x,G=g\],m\(x,g\)=\\mathbb\{E\}\\big\[\(1\-w\)^\{2\}b\_\{0\}\\mid X\{=\}x,G\{=\}g\\big\],\\qquad m\_\{0\}\(x,g\)=\\mathbb\{E\}\\big\[b\_\{0\}\\mid X\{=\}x,G\{=\}g\\big\],the latter being the loss the model would incur using only legitimate information, whenceL​\(x,g\)=m0​\(x,g\)−m​\(x,g\)=𝔼​\[b0​w​\(2−w\)∣X=x,G=g\]L\(x,g\)=m\_\{0\}\(x,g\)\-m\(x,g\)=\\mathbb\{E\}\[b\_\{0\}\\,w\(2\-w\)\\mid X\{=\}x,G\{=\}g\]\. This pair satisfies[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1):w∈\[0,1\]w\\in\[0,1\]impliesL≥0L\\geq 0, and post\-cutoff no leaked information exists, soP=PhonP=P\_\{\\mathrm\{hon\}\}andL=0L=0there\. Finally, iterated expectations and \([2](https://arxiv.org/html/2608.02985#S2.E2)\) give

B=𝔼​\[L​\(X,G\)∣G<0\]=𝔼​\[b0​w​\(2−w\)∣G<0\],B=\\mathbb\{E\}\\big\[L\(X,G\)\\mid G<0\\big\]=\\mathbb\{E\}\\big\[b\_\{0\}\\,w\(2\-w\)\\mid G<0\\big\],which is \([4](https://arxiv.org/html/2608.02985#S4.E4)\)\. ∎

#### B\.3\.4The loading correspondence

The following lemma, referenced in[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1), connects the per\-question extraction weightwwto the population loadingλ\\lambdaof[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)in the homogeneous special case\.

###### Lemma 6\(Loading correspondence in the homogeneous pull model\)\.

Suppose within a conditioning stratum thatPhon=y^cleanP\_\{\\mathrm\{hon\}\}=\\hat\{y\}\_\{\\mathrm\{clean\}\}andw​\(q\)≡ww\(q\)\\equiv wis constant\. Define the pull directionSpull=Y−PhonS\_\{\\mathrm\{pull\}\}=Y\-P\_\{\\mathrm\{hon\}\}\. Thenδ=P−Phon=w​Spull\\delta=P\-P\_\{\\mathrm\{hon\}\}=wS\_\{\\mathrm\{pull\}\}and theL2L^\{2\}loading ofδ\\deltaontoSpullS\_\{\\mathrm\{pull\}\}isww\. If, additionally,SpullS\_\{\\mathrm\{pull\}\}equals the Bayesian leakage signalSSof[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1), thenλ=w\\lambda=wandμδ=ν~=0\\mu\_\{\\delta\}=\\tilde\{\\nu\}=0\.

###### Proof\.

Within the stratum,[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)withPhon=y^cleanP\_\{\\mathrm\{hon\}\}=\\hat\{y\}\_\{\\mathrm\{clean\}\}and constantwwgives

δ=P−y^clean=w​\(Y−Phon\)=w​Spull,\\delta=P\-\\hat\{y\}\_\{\\mathrm\{clean\}\}=w\\,\\big\(Y\-P\_\{\\mathrm\{hon\}\}\\big\)=w\\,S\_\{\\mathrm\{pull\}\},soδ\\deltais proportional toSpullS\_\{\\mathrm\{pull\}\}with coefficientww: theL2L^\{2\}loading ofδ\\deltaontoSpullS\_\{\\mathrm\{pull\}\}iswwand the projection residual vanishes\. If in additionSpull=SS\_\{\\mathrm\{pull\}\}=S, thenδ=w​S\\delta=wS, whenceλ=Cov​\(δ,S∣ℋ\)/Var​\(S∣ℋ\)=w\\lambda=\\mathrm\{Cov\}\(\\delta,S\\mid\\mathcal\{H\}\)/\\mathrm\{Var\}\(S\\mid\\mathcal\{H\}\)=w,μδ=𝔼​\[δ∣ℋ\]=w​𝔼​\[S∣ℋ\]=0\\mu\_\{\\delta\}=\\mathbb\{E\}\[\\delta\\mid\\mathcal\{H\}\]=w\\,\\mathbb\{E\}\[S\\mid\\mathcal\{H\}\]=0, andν~=δ−λ​S−μδ=0\\tilde\{\\nu\}=\\delta\-\\lambda S\-\\mu\_\{\\delta\}=0\. ∎

#### B\.3\.5Interpreting the extraction weight

A useful, non\-unique parameterization of the extraction weight in[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)is

w​\(q\)=c​\(q\)⏟knowledgeexists⋅u​\(q\)⏟usable/deployed⋅κ​\(type\)⏟answer vs\. evidence\.w\(q\)\\;=\\;\\underbrace\{c\(q\)\}\_\{\\text\{knowledge \\emph\{exists\}\}\}\\cdot\\underbrace\{u\(q\)\}\_\{\\text\{\\emph\{usable\}/deployed\}\}\\cdot\\underbrace\{\\kappa\(\\mathrm\{type\}\)\}\_\{\\text\{answer vs\.\\ evidence\}\}\.\(9\)The factorization is interpretive rather than identified from backtest scores\. Existenceccasks whether the model absorbed the relevant fact at all, and is probeable leak\-free by self\-consistency; usabilityuucaptures the well\-documented gap between possessing a fact and deploying it; andκ\\kappaencodes the leakage type of[Section˜4](https://arxiv.org/html/2608.02985#S4), near11for answer leakage and intermediate for evidence leakage\. The controlled contamination experiment \(E2,[Section˜D\.3](https://arxiv.org/html/2608.02985#A4.SS3)\) uses this reading: inserted duplicates raisecc, and the measured dose–response tracks exposure that is both absorbed and usable\.

### B\.4Results of[Section˜5](https://arxiv.org/html/2608.02985#S5): identification routes

Throughout this subsection,\(m0,L\)\(m\_\{0\},L\)is an operational decomposition satisfying[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)for the model under audit, with superscripts \(mAm^\{A\},m0Am\_\{0\}^\{A\},LAL^\{A\}\) distinguishing models where needed\. For the DiD route, fix the target pre\-cutoff covariate lawQQand write, for a modelAA,

μ±A=∫𝔼​\[mA​\(X,G\)∣X=x,G≷0\]​𝑑Q​\(x\),\\mu^\{A\}\_\{\\pm\}=\\int\\mathbb\{E\}\\big\[m^\{A\}\(X,G\)\\mid X=x,\\ G\\gtrless 0\\big\]\\,dQ\(x\),withμ0,±A\\mu^\{A\}\_\{0,\\pm\}the analogous averages of the honest surfacem0Am\_\{0\}^\{A\}; standardizing all four cells toQQremoves composition differences, andΔA=μ\+A−μ−A\\Delta^\{A\}=\\mu^\{A\}\_\{\+\}\-\\mu^\{A\}\_\{\-\}is the standardized pre\-to\-post change of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\. In this notation, the formal content of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)is:LM0≡0L^\{M\_\{0\}\}\\equiv 0on the evaluation support; target\-post and control pre/post observations have common support withQQ; and the honest changes are parallel,μ0,\+M−μ0,−M=μ0,\+M0−μ0,−M0\\mu^\{M\}\_\{0,\+\}\-\\mu^\{M\}\_\{0,\-\}=\\mu^\{M\_\{0\}\}\_\{0,\+\}\-\\mu^\{M\_\{0\}\}\_\{0,\-\}\. The regression\-discontinuity statements assume the following support regularity\.

###### Assumption 5\(RD support regularity\)\.

The density ofGGis positive and continuous near zero, and the conditional support ofXXoverlaps on both sides of the cutoff\. Conditional limits are taken on this common support\.

#### B\.4\.1Proof of[Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)

[Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)is the boundary guarantee of Route 1: continuity of the honest surface converts the observed risk jump at the cutoff into the left\-limit leakage\.

###### Proof\.

Forg\>0g\>0,L≡0L\\equiv 0by[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1), som=m0m=m\_\{0\}there, and continuity \([Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)\) giveslimg↓0m​\(x,g\)=m0​\(x,0\)\\lim\_\{g\\downarrow 0\}m\(x,g\)=m\_\{0\}\(x,0\)on the common support of[Assumption˜5](https://arxiv.org/html/2608.02985#Thmassumption5)\. Forg<0g<0,m=m0−Lm=m\_\{0\}\-L, solimg↑0m​\(x,g\)=m0​\(x,0\)−L​\(x,0−\)\\lim\_\{g\\uparrow 0\}m\(x,g\)=m\_\{0\}\(x,0\)\-L\(x,0^\{\-\}\), the left limit existing by[Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)\. Subtracting the two limits givesJ​\(x\)=L​\(x,0−\)J\(x\)=L\(x,0^\{\-\}\), which is non\-negative by[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\. ∎

#### B\.4\.2Proof of[Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)

[Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6), stated here in full, extends Route 1 from the boundary to the global estimand, at the price of the extrapolation class\.

###### Assumption 6\(Uniquely extrapolable honest surface\)\.

m0​\(x,g\)=α​\(x\)\+φ​\(g\)m\_\{0\}\(x,g\)=\\alpha\(x\)\+\\varphi\(g\), whereφ\\varphibelongs to a known finite\-dimensional classΦ\\Phiwhose restriction to the observed clean support uniquely determines it on the target pre\-cutoff support \(e\.g\. a fixed\-degree polynomial\), and the class is correctly specified\.

\{restatable\}

\[Global leakage by clean\-side extrapolation\]propositionproprdglobal Under the assumptions of[Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)and[Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6), the honest surface on\{g<0\}\\\{g<0\\\}is identified by extrapolating the fit ofm0m\_\{0\}from\{g\>0\}\\\{g\>0\\\}, andB=𝔼​\[m0​\(X,G\)−m​\(X,G\)∣G<0\]B=\\mathbb\{E\}\[\\,m\_\{0\}\(X,G\)\-m\(X,G\)\\mid G<0\\,\]is identified\.

###### Proof\.

On\{g\>0\}\\\{g\>0\\\},m=m0=α​\(x\)\+φ​\(g\)m=m\_\{0\}=\\alpha\(x\)\+\\varphi\(g\)is observed; the two components are identified only up to an additive normalization, but their sum—the clean surface—is identified\. By[Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6), the restriction ofφ\\varphito the observed clean support uniquely determines its continuation to the target pre\-cutoff support, and hence determinesm0m\_\{0\}there\. ConsequentlyL=m0−mL=m\_\{0\}\-mis identified on\{g<0\}\\\{g<0\\\}, and so isB=𝔼​\[L∣G<0\]B=\\mathbb\{E\}\[L\\mid G<0\]\. ∎

#### B\.4\.3Proof of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)

[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)is the global guarantee of Route 2: a clean, recency\-matched control converts the difference of pre/post changes into the standardized inflation\.

###### Proof\.

Target leakage vanishes on\{G\>0\}\\\{G\>0\\\}by[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1), soμ\+M=μ0,\+M\\mu^\{M\}\_\{\+\}=\\mu^\{M\}\_\{0,\+\}\. On the pre side,mM=m0M−LMm^\{M\}=m\_\{0\}^\{M\}\-L^\{M\}integrates toμ−M=μ0,−M−∫𝔼​\[LM∣X=x,G<0\]​𝑑Q​\(x\)\\mu^\{M\}\_\{\-\}=\\mu^\{M\}\_\{0,\-\}\-\\int\\mathbb\{E\}\[L^\{M\}\\mid X\{=\}x,G\{<\}0\]\\,dQ\(x\), and becauseQQis the target pre\-cutoff covariate law, the integral equals𝔼​\[L∣G<0\]=B\\mathbb\{E\}\[L\\mid G<0\]=Bby iterated expectations\. HenceΔM=\(μ0,\+M−μ0,−M\)\+B\\Delta^\{M\}=\\big\(\\mu^\{M\}\_\{0,\+\}\-\\mu^\{M\}\_\{0,\-\}\\big\)\+B\. The control is clean \(LM0≡0L^\{M\_\{0\}\}\\equiv 0\), soΔM0=μ0,\+M0−μ0,−M0\\Delta^\{M\_\{0\}\}=\\mu^\{M\_\{0\}\}\_\{0,\+\}\-\\mu^\{M\_\{0\}\}\_\{0,\-\}\. Subtracting and applying the parallel honest change of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)cancels the honest terms, leavingΔM−ΔM0=B\\Delta^\{M\}\-\\Delta^\{M\_\{0\}\}=B\. ∎

#### B\.4\.4Boundary DiD with a no\-discontinuity control

The following variant, stated and proved only here, weakens the clean\-control requirement to continuity of the control’s leakage at the target cutoff; it identifies the boundary estimand only, and underwrites the control checks of M4 \([Section˜7](https://arxiv.org/html/2608.02985#S7)\)\.

###### Proposition 7\(Boundary DiD with a no\-discontinuity control\)\.

At the target cutoff, define the observed risk jumpJobsA​\(x\)=mA​\(x,0\+\)−mA​\(x,0−\)J\_\{\\mathrm\{obs\}\}^\{A\}\(x\)=m^\{A\}\(x,0^\{\+\}\)\-m^\{A\}\(x,0^\{\-\}\)\. Suppose \(i\) target leakage is zero on the post side and has left limitLM​\(x,0−\)L^\{M\}\(x,0^\{\-\}\); \(ii\) control leakage is continuous at the target cutoff; and \(iii\) target and control share the same honest/composition jump,m0M​\(x,0\+\)−m0M​\(x,0−\)=m0M0​\(x,0\+\)−m0M0​\(x,0−\)m\_\{0\}^\{M\}\(x,0^\{\+\}\)\-m\_\{0\}^\{M\}\(x,0^\{\-\}\)=m\_\{0\}^\{M\_\{0\}\}\(x,0^\{\+\}\)\-m\_\{0\}^\{M\_\{0\}\}\(x,0^\{\-\}\)\. Then

JobsM​\(x\)−JobsM0​\(x\)=LM​\(x,0−\)\.J\_\{\\mathrm\{obs\}\}^\{M\}\(x\)\-J\_\{\\mathrm\{obs\}\}^\{M\_\{0\}\}\(x\)=L^\{M\}\(x,0^\{\-\}\)\.

###### Proof\.

WriteJ0AJ\_\{0\}^\{A\}for the honest jump of modelAAappearing in condition \(iii\)\. For the target, condition \(i\) givesJobsM=J0M\+LM​\(x,0−\)J\_\{\\mathrm\{obs\}\}^\{M\}=J\_\{0\}^\{M\}\+L^\{M\}\(x,0^\{\-\}\): post\-cutoff leakage vanishes while the pre\-side limit contributesLM​\(x,0−\)L^\{M\}\(x,0^\{\-\}\)\. For the control, condition \(ii\) makes its leakage difference across the cutoff vanish, soJobsM0=J0M0J\_\{\\mathrm\{obs\}\}^\{M\_\{0\}\}=J\_\{0\}^\{M\_\{0\}\}\. Condition \(iii\) equates the honest jumps, and differencing leavesLM​\(x,0−\)L^\{M\}\(x,0^\{\-\}\)\. ∎

#### B\.4\.5Proof of[Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)

[Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7), stated here in full, gives the population signature of Route 3: under calibration and homogeneous extraction, the residual covariance is the hump\-shaped function of the extraction weight that makes PRC a detector of evidence leakage\. It rests on the following population conditions\.

###### Assumption 7\(PRC population conditions\)\.

Within the pre\-cutoff population,Phon=c0P\_\{\\mathrm\{hon\}\}=c\_\{0\},𝔼​\[Y∣c0,G<0\]=c0\\mathbb\{E\}\[Y\\mid c\_\{0\},G<0\]=c\_\{0\}, and the extraction weight is homogeneous:w​\(q\)≡w¯w\(q\)\\equiv\\bar\{w\}\.

Conditional calibration is achievable by construction—recalibrate on leakage\-free anchors \([Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)\)—andPhon=c0P\_\{\\mathrm\{hon\}\}=c\_\{0\}takes the crowd as the honest benchmark, appropriate when the model holds no legitimate edge over public information\. Homogeneouswwis a first\-order simplification: under heterogeneouswwthe covariance is a nonlinear functional of the joint law of\(w,c0,Y\)\(w,c\_\{0\},Y\), not ofBB, and the endpoint cases below show it cannot separate pure answer leakage \(w≡1w\\equiv 1\) from no leakage at all\. In the population result,PPdenotes the framing\-invariant limitP∞P\_\{\\infty\}of the paraphrase consensus; the finite\-KKdeployment, including the framing\-invariance conditions, is developed in[Appendix˜C](https://arxiv.org/html/2608.02985#A3)\.

\{restatable\}

\[Residual\-covariance detection under convex pull\]propositionpropefficiency Under[Assumptions˜2](https://arxiv.org/html/2608.02985#Thmassumption2)and[7](https://arxiv.org/html/2608.02985#Thmassumption7),

Cov​\(P,Y−P∣G<0\)=w¯​\(1−w¯\)​𝔼​\[c0​\(1−c0\)∣G<0\]≥0,\\mathrm\{Cov\}\(P,Y\-P\\mid G<0\)=\\bar\{w\}\(1\-\\bar\{w\}\)\\,\\mathbb\{E\}\[c\_\{0\}\(1\-c\_\{0\}\)\\mid G<0\]\\;\\geq\\;0,and the covariance vanishes wheneverw≡0w\\equiv 0orw≡1w\\equiv 1\.

###### Proof\.

All moments are conditional onG<0G<0\. Under[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)with homogeneous weightw¯\\bar\{w\}andPhon=c0P\_\{\\mathrm\{hon\}\}=c\_\{0\}, we haveP=\(1−w¯\)​c0\+w¯​YP=\(1\-\\bar\{w\}\)c\_\{0\}\+\\bar\{w\}YandY−P=\(1−w¯\)​\(Y−c0\)Y\-P=\(1\-\\bar\{w\}\)\(Y\-c\_\{0\}\)\. Conditional calibration \(𝔼​\[Y∣c0,G<0\]=c0\\mathbb\{E\}\[Y\\mid c\_\{0\},G<0\]=c\_\{0\}\) givesCov​\(c0,Y−c0\)=0\\mathrm\{Cov\}\(c\_\{0\},Y\-c\_\{0\}\)=0and, for binaryYY,Cov​\(Y,Y−c0\)=𝔼​\[Var​\(Y∣c0\)\]=𝔼​\[c0​\(1−c0\)\]\\mathrm\{Cov\}\(Y,Y\-c\_\{0\}\)=\\mathbb\{E\}\[\\mathrm\{Var\}\(Y\\mid c\_\{0\}\)\]=\\mathbb\{E\}\[c\_\{0\}\(1\-c\_\{0\}\)\]\. Hence

Cov​\(P,Y−P\)=\(1−w¯\)​Cov​\(\(1−w¯\)​c0\+w¯​Y,Y−c0\)=w¯​\(1−w¯\)​𝔼​\[c0​\(1−c0\)\]≥0\.\\mathrm\{Cov\}\(P,Y\-P\)=\(1\-\\bar\{w\}\)\\,\\mathrm\{Cov\}\\big\(\(1\-\\bar\{w\}\)c\_\{0\}\+\\bar\{w\}Y,\\;Y\-c\_\{0\}\\big\)=\\bar\{w\}\(1\-\\bar\{w\}\)\\,\\mathbb\{E\}\[c\_\{0\}\(1\-c\_\{0\}\)\]\\;\\geq\\;0\.Ifw≡0w\\equiv 0thenP=Phon=c0P=P\_\{\\mathrm\{hon\}\}=c\_\{0\}and the covariance equalsCov​\(c0,Y−c0\)=0\\mathrm\{Cov\}\(c\_\{0\},Y\-c\_\{0\}\)=0; ifw≡1w\\equiv 1thenP=YP=Y, soY−P=0Y\-P=0and the covariance vanishes as well\. ∎

#### B\.4\.6A sufficient external reference

The following corollary, referenced in[Section˜5\.4](https://arxiv.org/html/2608.02985#S5.SS4), records that a single external reference restores the point identification that[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)denies to passive data\.

###### Corollary 8\(A sufficient external reference\)\.

With only the passive backtest distribution of\(P,Y,X,G\)\(P,Y,X,G\)\(including the crowd anchorc0c\_\{0\}\),BBis set\-identified, not point\-identified \([Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\)\. Each of the following suffices to point\-identifyBBunder its stated assumptions:\(R1\)a known cutoff with questions on both sides \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)\);\(R2\)a clean, capability\-matched control on the same questions \([Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\)\. The intervention\(R3\)detects evidence leakage \([Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)\) but does not point\-identify it\.

###### Proof\.

Immediate from[Assumptions˜1](https://arxiv.org/html/2608.02985#Thmassumption1),[6](https://arxiv.org/html/2608.02985#Thmassumption6),[4](https://arxiv.org/html/2608.02985#Thmassumption4)and[7](https://arxiv.org/html/2608.02985#Thmassumption7)\. ∎

#### B\.4\.7Complementary signatures

The following proposition, referenced in[Section˜5\.4](https://arxiv.org/html/2608.02985#S5.SS4), combines the boundary and residual signatures into the complementarity statement that motivates pairing the probes\. We make no formal claim about a consistent test; a sampling\-level analysis would require thresholds, finite\-sample noise models, and the availability of both probes\.

###### Proposition 9\(Complementary signatures under homogeneous extraction\)\.

Under[Assumptions˜1](https://arxiv.org/html/2608.02985#Thmassumption1),[2](https://arxiv.org/html/2608.02985#Thmassumption2),[3](https://arxiv.org/html/2608.02985#Thmassumption3)and[7](https://arxiv.org/html/2608.02985#Thmassumption7)and the support conditions of[Section˜B\.4](https://arxiv.org/html/2608.02985#A2.SS4), suppose the stakes and homogeneous extraction have left limitsb0b\_\{0\}andwwat the cutoff, and letσS2=𝔼​\[c0​\(1−c0\)∣G<0\]\\sigma\_\{S\}^\{2\}=\\mathbb\{E\}\[c\_\{0\}\(1\-c\_\{0\}\)\\mid G<0\]\. Then

J​\(w\)\\displaystyle J\(w\)=b0​w​\(2−w\)\\displaystyle=b\_\{0\}\\,w\(2\-w\)\(increasing; maximal at​w=1\),\\displaystyle\(\\text\{increasing; maximal at \}w=1\),PRC​\(w\)\\displaystyle\\mathrm\{PRC\}\(w\)=w​\(1−w\)​σS2\\displaystyle=w\(1\-w\)\\sigma\_\{S\}^\{2\}\(maximal at​w=12;zero at​w∈\{0,1\}\)\.\\displaystyle\(\\text\{maximal at \}w=\\tfrac\{1\}\{2\};\\ \\text\{zero at \}w\\in\\\{0,1\\\}\)\.

###### Proof\.

By[Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)and[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2), the boundary jump equals the left\-limit leakage,J=L​\(⋅,0−\)=b0​w​\(2−w\)J=L\(\\cdot,0^\{\-\}\)=b\_\{0\}\\,w\(2\-w\), which is increasing inwwon\[0,1\]\[0,1\]and maximal atw=1w=1\. By[Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)in the homogeneous case,PRC=w​\(1−w\)​σS2\\mathrm\{PRC\}=w\(1\-w\)\\sigma\_\{S\}^\{2\}, maximal atw=12w=\\tfrac\{1\}\{2\}and zero atw∈\{0,1\}w\\in\\\{0,1\\\}\. In particularPRC​\(1\)=0\\mathrm\{PRC\}\(1\)=0whileJ​\(1\)=b0\>0J\(1\)=b\_\{0\}\>0, andPRC​\(12\)=σS2/4\>0\\mathrm\{PRC\}\(\\tfrac\{1\}\{2\}\)=\\sigma\_\{S\}^\{2\}/4\>0\. ∎

The complementarity is verified on the synthetic sweep of[Section˜E\.1](https://arxiv.org/html/2608.02985#A5.SS1); the answer\-leakage regimew→1w\\to 1does not arise in the ground\-truth experiment \(measuredw≤0\.42w\\leq 0\.42,[Section˜6\.1](https://arxiv.org/html/2608.02985#S6.SS1)\), so its real\-data face is untested\.

## Appendix CDeploying the routes

[Figure˜6](https://arxiv.org/html/2608.02985#A3.F6)summarizes the practitioner’s choice of route given the available resources; it is the decision\-oriented complement of[Table˜1](https://arxiv.org/html/2608.02985#S5.T1), which states what each route guarantees and how its assumption is checked\.

Backtest \(cutoffTT\);fix metric\+\+difficulty adjustmentPost\-cutoffdata available?Forward eval\(B=0B\{=\}0\)Straddlescutoff?Cutoffknown?RD: boundaryJ^\\widehat\{J\};globalB^\\widehat\{B\}w/ extrapolationDiDB^\\widehat\{B\}\(clean matched control\)Calibrate→\\toPRC\(detect only\)yesnoyesyesnonocorroborateFigure 6:Choosing a route: report a leakage\-adjusted score only when globalBBis identified; otherwise report the boundary or detection estimand\.Forward evaluation \(B=0B=0\) is preferred when post\-cutoff outcomes can be awaited\. Otherwise, RD identifies the boundary jump when the cutoff is known \(and globalBBunder extrapolation\), and DiD identifies globalBBwith a clean, capability/recency\-matched control; if no data straddle the cutoff, only the calibrated PRC detection statistic remains\. The flow is a simplification: with both a known cutoff and a matched control available, DiD may be preferred for globalBB\.[Table˜1](https://arxiv.org/html/2608.02985#S5.T1)states each route’s assumptions and checks\.### C\.1The practitioner’s recipe and control validation

##### The practitioner’s recipe\.

\(1\)Fix the evaluation metric and a leakage\-free difficulty adjustment; for probabilistic forecasts, a contemporaneous crowd forecast provides the stakes anchor\.\(2\)EstimateB^\\widehat\{B\}by a point\-identifying route: DiD requires a capability/recency\-matched clean control \(selection and validation protocol below\), while RD identifies the boundary jump and needs the extrapolation assumption for globalBB\.\(3\)If usingPRC\\mathrm\{PRC\}, first calibrate on leakage\-free anchors and report it only as a detection statistic \([Remarks˜2](https://arxiv.org/html/2608.02985#Thmremark2)and[9](https://arxiv.org/html/2608.02985#Thmtheorem9)\); calibration is not required for accuracy\- or pass@1\-based RD/DiD contrasts\.\(4\)When globalBBis identified, reportRiskmeas\+B^\\mathrm\{Risk\}\_\{\\mathrm\{meas\}\}\+\\widehat\{B\}\(or the score form of[Corollary˜2](https://arxiv.org/html/2608.02985#Thmtheorem2)\) with a confidence interval; otherwise report the boundary or detection estimand without converting it into a global correction\.\(5\)Prefer*forward*evaluation when feasible, for whichB=0B=0by construction; and if the cutoff is uncertain or staged, report sensitivity to its placement rather than treating a single date as known\.

##### Selecting and validating a DiD control \(Route 2\)\.

The control should be a same\-family snapshot or a vintage\-matched peer of the target—same capability tier and generation, similar training\-data distribution—whose documented cutoff predates the evaluation window; its cleanliness should be verified empirically rather than assumed, for example by date\-only recall probes on the window’s questions\. The parallel\-change condition of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)is not fully testable, since the target’s pre\-cutoff cell is contaminated by construction, but it has three testable implications: on the jointly clean region\{G\>0\}\\\{G\>0\\\}both models’ risk trends are honest and should move in parallel; placebo cutoffs at dates with no leakage discontinuity should return estimates near zero; and covariate balance should hold across the four standardized cells\. When several candidate controls are available, agreement ofB^\\widehat\{B\}across them is an over\-identification check, and their spread is a systematic\-uncertainty band\. Finally, the dominant violation is signed \([Section˜5\.2](https://arxiv.org/html/2608.02985#S5.SS2)\): a weak control inflatesB^\\widehat\{B\}, so a null finding is conservative under this failure mode, whereas a positive finding should be reported together with the matching checks\. M5 \([Section˜D\.7](https://arxiv.org/html/2608.02985#A4.SS7)\) implements this protocol\.

### C\.2Finite\-KKPRC machinery \(Route 3\)

The deployable form of the paraphrase route \([Section˜5\.3](https://arxiv.org/html/2608.02985#S5.SS3)\) uses a finite number of queries\. Query the model withKKparaphrases per question, form the consensusy¯K,i=K−1​∑kPi\(k\)\\bar\{y\}\_\{K,i\}=K^\{\-1\}\\sum\_\{k\}P\_\{i\}^\{\(k\)\}, and computeCov^​\(y¯K,Y−y¯K\)\\widehat\{\\mathrm\{Cov\}\}\(\\bar\{y\}\_\{K\},Y\-\\bar\{y\}\_\{K\}\)\. The target population quantity isPRC∞=Cov​\(P∞,Y−P∞∣G<0\)\\mathrm\{PRC\}\_\{\\infty\}=\\mathrm\{Cov\}\(P\_\{\\infty\},Y\-P\_\{\\infty\}\\mid G<0\)from[Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7), whereP∞P\_\{\\infty\}is the framing\-invariant consensus defined by the following condition \(moved here from the main text because only the finite\-KKanalysis uses it\)\.

###### Assumption 8\(Framing\-invariant paraphrase noise\)\.

Within the pre\-cutoff population, for paraphraseπk\\pi\_\{k\}, writeP\(k\)=P∞\+εkP^\{\(k\)\}=P\_\{\\infty\}\+\\varepsilon\_\{k\}, where the leaked signal andP∞P\_\{\\infty\}do not depend on framing\. Conditional on the question, the deviations are identically distributed, mean zero, mutually uncorrelated, and uncorrelated withP∞P\_\{\\infty\}andY−P∞Y\-P\_\{\\infty\}\. They have finite fourth moments\.

###### Lemma 10\(Finite\-paraphrase attenuation\)\.

Under[Assumption˜8](https://arxiv.org/html/2608.02985#Thmassumption8), letW=𝔼​\[Varπ​\(P\(k\)∣q,G<0\)∣G<0\]W=\\mathbb\{E\}\[\\mathrm\{Var\}\_\{\\pi\}\(P^\{\(k\)\}\\mid q,G<0\)\\mid G<0\]\. For constantKK,

Cov​\(y¯K,Y−y¯K∣G<0\)=PRC∞−WK\.\\mathrm\{Cov\}\(\\bar\{y\}\_\{K\},Y\-\\bar\{y\}\_\{K\}\\mid G<0\)=\\mathrm\{PRC\}\_\{\\infty\}\-\\frac\{W\}\{K\}\.

###### Proof\.

Writey¯K=P∞\+ε¯K\\bar\{y\}\_\{K\}=P\_\{\\infty\}\+\\bar\{\\varepsilon\}\_\{K\}\. Conditional mean\-zero and mutual uncorrelatedness giveVar​\(ε¯K∣q\)=Var​\(εk∣q\)/K\\mathrm\{Var\}\(\\bar\{\\varepsilon\}\_\{K\}\\mid q\)=\\mathrm\{Var\}\(\\varepsilon\_\{k\}\\mid q\)/K\. The remaining orthogonality conditions in[Assumption˜8](https://arxiv.org/html/2608.02985#Thmassumption8)give

Cov​\(P∞\+ε¯K,Y−P∞−ε¯K\)=Cov​\(P∞,Y−P∞\)−Var​\(ε¯K\)\.\\mathrm\{Cov\}\(P\_\{\\infty\}\+\\bar\{\\varepsilon\}\_\{K\},\\,Y\-P\_\{\\infty\}\-\\bar\{\\varepsilon\}\_\{K\}\)=\\mathrm\{Cov\}\(P\_\{\\infty\},Y\-P\_\{\\infty\}\)\-\\mathrm\{Var\}\(\\bar\{\\varepsilon\}\_\{K\}\)\.Taking the question\-level expectation yields the result\. ∎

##### Estimator and inference\.

Let

W^=1m​∑i=1m1K−1​∑k=1K\(Pi\(k\)−y¯K,i\)2\.\\widehat\{W\}=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\frac\{1\}\{K\-1\}\\sum\_\{k=1\}^\{K\}\(P\_\{i\}^\{\(k\)\}\-\\bar\{y\}\_\{K,i\}\)^\{2\}\.The bias\-corrected estimator is

PRC^bc=Cov^​\(y¯K,Y−y¯K\)\+W^/K\.\\widehat\{\\mathrm\{PRC\}\}\_\{\\mathrm\{bc\}\}=\\widehat\{\\mathrm\{Cov\}\}\(\\bar\{y\}\_\{K\},Y\-\\bar\{y\}\_\{K\}\)\+\\widehat\{W\}/K\.For varyingKiK\_\{i\}, replaceW^/K\\widehat\{W\}/Kby the average of the per\-question sample variances divided byKiK\_\{i\}\. Under independent questions,[Assumption˜8](https://arxiv.org/html/2608.02985#Thmassumption8), and finite fourth moments, the vector of sample moments definingPRC^bc\\widehat\{\\mathrm\{PRC\}\}\_\{\\mathrm\{bc\}\}obeys a multivariate central limit theorem, and the delta method yieldsm​\(PRC^bc−PRC∞\)⇒𝒩​\(0,σPRC2\)\\sqrt\{m\}\(\\widehat\{\\mathrm\{PRC\}\}\_\{\\mathrm\{bc\}\}\-\\mathrm\{PRC\}\_\{\\infty\}\)\\Rightarrow\\mathcal\{N\}\(0,\\sigma\_\{\\mathrm\{PRC\}\}^\{2\}\); in practice we bootstrap questions and refit any calibrator inside each replicate\.PRC\\mathrm\{PRC\}remains a detection statistic, not an estimator ofBB\([Assumption˜7](https://arxiv.org/html/2608.02985#Thmassumption7)\), and paraphrase diversity must come from framing rather than sampling temperature \([Section˜C\.3](https://arxiv.org/html/2608.02985#A3.SS3)\)\.

### C\.3Paraphrase set and generation protocol

Route 3 \([Section˜5](https://arxiv.org/html/2608.02985#S5)\) requires paraphrases that vary framing without revealing the outcome; the set documented here illustrates the design principles and the generation protocol\.

##### Design principles\.

Effective paraphrases must \(i\) preserve the event and resolution criteria \(same answer\), \(ii\) maximize variation in the reasoning channel, and \(iii\) reveal nothing about the outcome\. We vary six dimensions independently: lexical choice, sentence form, level of specificity, entity reference, framing/tone, and which background context is emphasized\.

##### Generation and validation\.

Paraphrases are generated by an auxiliary model \(distinct from the model under test\) instructed to preserve meaning and resolution criteria while varying the six dimensions and concealing the outcome\. Candidates are validated by three checks: semantic preservation \(embedding cosine similarity above a threshold\), lexical diversity \(low pairwise self\-BLEU\), and an outcome\-leakage screen rejecting any phrasing that encodes the answer’s direction\. Querying uses temperature0with reasoning disabled and a strict final\-answer parser; truncated or out\-of\-range generations are retried once and otherwise discarded\. The protocol is instantiated, with the specificKKand models used, in thePRC\\mathrm\{PRC\}experiments of[Sections˜E\.1](https://arxiv.org/html/2608.02985#A5.SS1)and[E\.8](https://arxiv.org/html/2608.02985#A5.SS8)\.

### C\.4Classical identification tools used in the paper

The machinery instantiates standard tools\. The residual covariance is a crowd\-anchored analogue of Mincer–Zarnowitz efficiency tests and of Murphy’s excess resolution\(Mincer & Zarnowitz,[1969](https://arxiv.org/html/2608.02985#bib.bib32); Elliott & Timmermann,[2016](https://arxiv.org/html/2608.02985#bib.bib8); Patton & Timmermann,[2012](https://arxiv.org/html/2608.02985#bib.bib36); Murphy,[1973](https://arxiv.org/html/2608.02985#bib.bib33); Gneiting & Raftery,[2007](https://arxiv.org/html/2608.02985#bib.bib12)\)\. The routes are regression discontinuity\(Imbens & Lemieux,[2008](https://arxiv.org/html/2608.02985#bib.bib19); Lee & Lemieux,[2010](https://arxiv.org/html/2608.02985#bib.bib25)\)and difference\-in\-differences, with a fixed\-question intervention in the spirit of control functions and specification tests\(Wooldridge,[2015](https://arxiv.org/html/2608.02985#bib.bib50); Wu,[1973](https://arxiv.org/html/2608.02985#bib.bib51); Hausman,[1978](https://arxiv.org/html/2608.02985#bib.bib16)\)\. The sharp\-bounds framing of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)follows partial identification\(Manski,[2003](https://arxiv.org/html/2608.02985#bib.bib31); Imbens & Manski,[2004](https://arxiv.org/html/2608.02985#bib.bib20)\)\. The finite\-KKcorrection is classical errors\-in\-variables disattenuation\(Fuller,[1987](https://arxiv.org/html/2608.02985#bib.bib9); Spearman,[1910](https://arxiv.org/html/2608.02985#bib.bib47)\)\. Two superficially appealing instruments do not transfer and serve only as motivation: temporal\-head ablation removes legitimate temporal reasoning along with leakage, and item\-preknowledge item response theory requires cross\-examinee variation that a single deployed model lacks\.

## Appendix DExperiment configurations and full results

Each subsection below gives, for one main\-text experiment \(M1–M5, in order\), the full configuration—data provenance, model inventories, estimator definitions, and inference procedures—together with the complete results summarized in[Section˜7](https://arxiv.org/html/2608.02985#S7)\. The synthetic validation study E1, the per\-experiment robustness analyses, and all supplementary arms are collected in[Appendix˜E](https://arxiv.org/html/2608.02985#A5)\.[Table˜4](https://arxiv.org/html/2608.02985#A4.T4)is the map: each theoretical claim of[Sections˜3](https://arxiv.org/html/2608.02985#S3),[4](https://arxiv.org/html/2608.02985#S4)and[5](https://arxiv.org/html/2608.02985#S5)and the experiment whose headline result carries it\.

Table 4:Claim\-to\-evidence contract\.Each theoretical claim of[Sections˜3](https://arxiv.org/html/2608.02985#S3),[4](https://arxiv.org/html/2608.02985#S4)and[5](https://arxiv.org/html/2608.02985#S5)and the experiment whose headline result carries it \(M1:[Section˜1](https://arxiv.org/html/2608.02985#S1); M2, M3:[Section˜6](https://arxiv.org/html/2608.02985#S6); M4, M5:[Section˜7](https://arxiv.org/html/2608.02985#S7)\)\.∗: 95% CI excludes zero\.### D\.1Scope and coverage

[Table˜5](https://arxiv.org/html/2608.02985#A4.T5)summarizes, for each experiment, the data source, sample size, models, metric, and the machinery that carries statistical significance\. It is the single reference for what each reported interval orpp\-value is computed from\.

Table 5:Scope and coverage of the empirical program\.One row per experiment; the last column names the inference machinery behind every significance claim in[Section˜7](https://arxiv.org/html/2608.02985#S7)\.
### D\.2The forecasting panel and M1 \(clean\-flagship naive check\)

##### Panel construction\.

The panel is built from the public ForecastBench archive\(Karger et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib23)\): the nightly question banks and the resolution files published in the project’s datasets repository\. We retain market\-source binary questions \(Polymarket, Metaculus, Manifold, INFER\) that are resolved, have a valid frozen market probabilityc0∈\(0,1\)c\_\{0\}\\in\(0,1\)recorded before resolution, and have a resolution date between July 2024 and July 2026\. Combination questions \(logical conjunctions of base questions\) are excluded\. After deduplication by question identifier this yields1,6461\{,\}646unique questions\. Dataset\-source questions \(for example weather and economic series\) are excluded from the headline analyses because their answers are tabular rather than event outcomes; they are retained in the archive audit\. The panel, the raw downloads, and the build script are versioned with the code\.

##### Elicitation protocol\.

Every model is queried once per question through OpenRouter with temperature0, reasoning disabled where the endpoint permits, and a fixed prompt that states the question, its resolution criteria, and the resolution date, and requests a single probability\. Responses are parsed to a float; refusals and parse failures are dropped and reported per model \(the retained fraction appears in each experiment’s sample sizes\)\. Scores are crowd\-anchored Brier reductionsL​\(q\)=\(c0−Y\)2−\(P−Y\)2L\(q\)=\(c\_\{0\}\-Y\)^\{2\}\-\(P\-Y\)^\{2\}\.

##### M1 design\.

For each of the five flagships \(GPT\-5, cutoff 2024\-09\-30; Gemini\-3\.1\-Pro, 2025\-01\-31; Kimi\-K2\.6, 2025\-04\-30; GPT\-5\.5, 2025\-12\-01; MiniMax\-M3, 2026\-01\-31\) we keep questions resolving at least 30 days after the documented cutoff, so that no outcome can be in training data\. Cutoffs are taken from vendor documentation collected in the pre\-registration; Kimi\-K2\.6’s cutoff is documented at coarser \(tier\-two\) resolution, a fact shown not to be load\-bearing by the cutoff\-shift check of[Section˜D\.5](https://arxiv.org/html/2608.02985#A4.SS5)\. The naive statistic is the pre/post gap in meanLLaround the median resolution date of each model’s clean window\. Confidence intervals use a cluster bootstrap over source×\\timesmonth cells \(4,0004\{,\}000resamples\)\.

##### M1 headline results \(demoted detail\)\.

Four of the five clean flagships show starred naive gaps of\+0\.044\+0\.044to\+0\.061\+0\.061\([Figure˜1](https://arxiv.org/html/2608.02985#S1.F1)\): an auditor running the standard pre/post check would flag four model families as leaking on questions they cannot possibly have seen\. The fifth model is the exception that proves the mechanism\. MiniMax\-M3’s null \(\+0\.017\+0\.017\) is a property of its*window*, not of the model: restricting all five models to its short clean window \(March through July 2026\) collapses every gap to insignificance—GPT\-5\.5 falls from\+0\.061∗\+0\.061^\{\\ast\}to\+0\.008\+0\.008, and MiniMax’s gap becomes the largest of the five\. The naive statistic therefore measures how much recency contrast an evaluation window spans, not whether the audited model leaked\. This is the empirical face of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1): a passive backtest gap carries recency and leakage in a single number, and at roughly0\.050\.05Brier\-reduction units in every family we test, the confound is far too large to ignore\.

##### Matched\-window check\.

Because each model’s clean window starts at its own cutoff, gap sizes are not comparable across models: a longer window spans more recency contrast between its pre and post halves\. Restricting all five models to the shortest clean window \(MiniMax\-M3’s: resolutions from 2026\-03\-02, split at the common median 2026\-05\-14,n≈333/337n\\approx 333/337\) makes every gap statistically null and nearly equal: GPT\-5\+0\.001\+0\.001\[−0\.053,\+0\.042\]\[\-0\.053,\+0\.042\], Gemini\-3\.1\-Pro\+0\.005\+0\.005\[−0\.055,\+0\.047\]\[\-0\.055,\+0\.047\], Kimi\-K2\.6\+0\.015\+0\.015\[−0\.025,\+0\.058\]\[\-0\.025,\+0\.058\], GPT\-5\.5\+0\.008\+0\.008\[−0\.034,\+0\.049\]\[\-0\.034,\+0\.049\], MiniMax\-M3\+0\.017\+0\.017\[−0\.020,\+0\.047\]\[\-0\.020,\+0\.047\]\. The spread across models in[Figure˜1](https://arxiv.org/html/2608.02985#S1.F1)is therefore driven by the windows, not the model families, which is the recency\-confound reading the theory predicts\. The same check backs the cutoff\-integrity reading of the documented cutoffs: had a documented cutoff materially understated a model’s training data, that model’s gap would not collapse alongside the others in the common window\.

### D\.3M2 \(Hubble QA\)

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m2_main.png)Figure 7:M2: at pretraining scale, measured accuracy inflates strictly monotonically with the known contamination dose\.Hubble pretrains a*perturbed*model with benchmark documents insertedrrtimes and a*standard*twin trained identically without them\. The x\-axis is the doserr\(log scale\)\. The y\-axis is the placebo\-centered contrastB~pair​\(r\)=B^pair​\(r\)−B^pair​\(0\)\\widetilde\{B\}\_\{\\mathrm\{pair\}\}\(r\)=\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(r\)\-\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(0\), whereB^pair​\(r\)\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(r\)is the perturbed\-minus\-standard accuracy on items inserted at doserr;B~pair\>0\\widetilde\{B\}\_\{\\mathrm\{pair\}\}\>0means contamination inflates measured accuracy, and ther=0r=0point is the centering reference \(zero by construction\)\. Whiskers: 95% intervals propagated from published SEMs;∗: interval excludes zero\. Result: strictly monotone from\+0\.011\+0\.011atr=1r=1to\+0\.403∗\+0\.403^\{\\ast\}atr=256r=256; sixteen duplicates already inflate accuracy by≈21\{\\approx\}21points\.##### The Hubble suite\.

The Hubble suite\(Wei et al\.,[2026](https://arxiv.org/html/2608.02985#bib.bib48)\)is a controlled pretraining experiment: models are trained from scratch on a web corpus into which benchmark evaluation documents \(MMLU, PIQA, HellaSwag, WinoGrande questions with answers\) are deliberately inserted at*known*duplication countsr∈\{0,1,4,16,64,256\}r\\in\\\{0,1,4,16,64,256\\\}\. For each configuration, two models are released: a*perturbed*model \(trained on the corpus with benchmark documents inserted at raterr\) and a*standard*model \(trained on the same corpus*without*the benchmark insertions\)\. The standard model is a perfect minimal\-pair clean control—it shares the same architecture, hyperparameters, and training data except for the inserted documents\. Model sizes are 1B and 8B parameters, trained for 100B and 500B tokens\. We use the published per\-\(model, task, duplication rate\) accuracies, requiring no additional compute\. Headline results use 8B at 500B tokens, aggregated as an unweighted mean over the four tasks; per\-task results are reported as a robustness check\.

##### Estimation\.

Three quantities are reported:

- •*Minimal\-pair contrast*\(raw\):B^pair​\(r\)=accpert​\(r\)−accstd​\(r\)\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(r\)=\\mathrm\{acc\}\_\{\\mathrm\{pert\}\}\(r\)\-\\mathrm\{acc\}\_\{\\mathrm\{std\}\}\(r\)\.
- •*Placebo\-centered contrast*:B~pair​\(r\)=B^pair​\(r\)−B^pair​\(0\)\\widetilde\{B\}\_\{\\mathrm\{pair\}\}\(r\)=\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(r\)\-\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(0\), which removes the small baseline offset between the two separately trained model instances \(B^pair​\(0\)=−0\.022\\widehat\{B\}\_\{\\mathrm\{pair\}\}\(0\)=\-0\.022for 8B\-500B\)\.
- •*Self\-baseline estimator*\(no separate control\):B^self​\(r\)=accpert​\(r\)−accpert​\(0\)\\widehat\{B\}\_\{\\mathrm\{self\}\}\(r\)=\\mathrm\{acc\}\_\{\\mathrm\{pert\}\}\(r\)\-\\mathrm\{acc\}\_\{\\mathrm\{pert\}\}\(0\)\.

##### Trend test\.

The monotone dose\-response is assessed by an exact one\-sided Spearman permutation test over the nonzero duplication rates \(r∈\{1,4,16,64,256\}r\\in\\\{1,4,16,64,256\\\}\)\. Robustness checks \(difficulty matching, scale/token settings, per\-task consistency\) are in[Section˜E\.4](https://arxiv.org/html/2608.02985#A5.SS4)\.

##### Results\.

[Table˜6](https://arxiv.org/html/2608.02985#A4.T6)tabulates the three estimators as a function of the known dose;[Figure˜8](https://arxiv.org/html/2608.02985#A4.F8)shows the dose\-response with propagated intervals, the replication across scale and token settings, per\-task consistency, and the clean\-control accuracy range bounding difficulty imbalance\. The placebo\-centered contrast is strictly monotone overr\>0r\>0\(\+0\.011,\+0\.075,\+0\.207,\+0\.380,\+0\.403\+0\.011,\\,\+0\.075,\\,\+0\.207,\\,\+0\.380,\\,\+0\.403; Spearmanρ=1\.0\\rho=1\.0,p=0\.0083p=0\.0083\); the clean control’s accuracy varies by at most0\.0510\.051across duplication bins; the self\-baseline tracks the minimal\-pair contrast within0\.0350\.035at every positive dose\.

Table 6:M2 \(Hubble, 8B, 500B tokens\): three estimators of the dose effect agree\.Rows: the raw minimal\-pair contrastB^pair\\widehat\{B\}\_\{\\mathrm\{pair\}\}, the placebo\-centered contrastB~pair\\widetilde\{B\}\_\{\\mathrm\{pair\}\}\(the primary estimand, plotted in[Figure˜7](https://arxiv.org/html/2608.02985#A4.F7)\), and the self\-baselineB^self\\widehat\{B\}\_\{\\mathrm\{self\}\}, by duplication doserr\. Result: all three rise strictly monotonically overr\>0r\>0\(placebo\-centered: Spearmanρ=1\.0\\rho=1\.0,p=0\.0083p=0\.0083\) and agree within0\.040\.04at every positive dose; 95% intervals are shown in[Figure˜8](https://arxiv.org/html/2608.02985#A4.F8)\(a\)\.![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m2.png)Figure 8:M2: the Hubble dose\-response is robust across estimators, scales, and tasks\.\(a\) Raw minimal\-pair, placebo\-centered, and self\-baseline estimates for the 8B/500B pair \(whiskers: 95% intervals propagated from published SEMs\)\. \(b\) The placebo\-centered estimate across model scale and token budget\. \(c\) Per\-task estimates for 8B/500B\. \(d\) The clean control’s accuracy range across dose bins, bounding difficulty imbalance\. Result: inflation rises monotonically with dose in every panel; no single estimator, scale, or task drives[Figure˜7](https://arxiv.org/html/2608.02985#A4.F7)\.

### D\.4M3 \(forecasting twins with injected leakage\)

##### Base model and training\.

Both twins are LoRA continued\-training runs of Qwen3\.5\-35B\-A3B\-Base on the Tinker training API: LoRA rank 32, sequence length 2048, batch 16 sequences, learning rate2×10−42\\times 10^\{\-4\}with 100\-step linear warmup, gradient clipping at 1\.0, one epoch over the corpus in a seed\-fixed order\. Each full twin processes approximately 94 million tokens \(2,900 steps\)\. The corpus is Wikitext\-103 filler \(95 million tokens, identical for both twins\) with the injected documents interleaved at deterministic schedule positions\.

##### Injected documents\.

For each treated question, dated news\-style documents are written by DeepSeek\-V3\.1 from the question metadata and the realized outcome, in rotating styles \(news article, encyclopedia entry, analyst note\), and validated by DeepSeek\-V3\.2 \(the validator must answer the question correctly from the treatment document alone, and must*fail*to answer from the scrubbed control document\)\. The control twin sees the same documents with outcome statements scrubbed; source, topic, length, and schedule positions are matched\. Doser∈\{0,1,4,16,64\}r\\in\\\{0,1,4,16,64\\\}counts document copies, randomized across questions stratified by the control twin’s per\-question honest error tercile and by resolution month\.

##### Screening\.

Injection is restricted to questions resolving before the pseudo\-cutoffT∗T^\{\\ast\}\(April 2026\) on which the un\-tuned base model is not already confidently correct \(excluding questions with baseP​\(YES\)\>0\.85P\(\\mathrm\{YES\}\)\>0\.85and outcome YES, orP​\(YES\)<0\.15P\(\\mathrm\{YES\}\)<0\.15and outcome NO\)\. This directly operationalizes the design requirement that the injected documents be the marginal information source\. The screen keeps988988of1,1031\{,\}103pre\-T∗T^\{\\ast\}questions and was fixed before any twin was trained; the deviation from the coarser pre\-registered proxy screen is documented in the versioned deviations file\.

##### Probability readout\.

Probabilities are read from paired “Yes”/“No” continuation log\-probabilities under a fixed, generic, outcome\-balanced few\-shot prefix, givingP​\(YES\)P\(\\mathrm\{YES\}\)as the softmax of the two continuation scores\. The same readout is used for the base model, both pilot twins, and both full twins\.

##### Pilot gates\.

Before the full runs, a pilot pair was trained on ten percent of injected questions with 10 million filler tokens and had to pass five pre\-registered gates: \(a\) memorization \(treatment log\-probability advantage on injected documents\), \(b\) signal \(positive twin contrast at dose 64\), \(c\) null \(twin contrast consistent with zero at dose 0\), \(d\) direction \(treatment probability moves toward realized outcomes\), and \(e\) recency \(control twin beats the base on injected\-topic questions\)\. All five passed at the pilot and again at full scale, where the values quoted in the main text were computed\.

##### Concentration binning variants\.

The binning variable is the control twin’s realized per\-question errorb0b\_\{0\}, the quantity appearing in the identityL=b0​w​\(2−w\)L=b\_\{0\}\\,w\(2\-w\); the main text displays quintiles, and the conclusion is invariant to granularity\. Under the pre\-registered tercile binning at dose 64 the observed contrasts are\+0\.019\+0\.019,\+0\.055\+0\.055,\+0\.112\+0\.112against homogeneous\-wwpredictions\+0\.013\+0\.013,\+0\.044\+0\.044,\+0\.185\+0\.185: same pattern, with the shortfall confined to the top bin\. Because binning on realized error could in principle inflate top\-bin estimates by selecting on the minuend of the contrast, we also rebin by ex\-ante crowd uncertaintyc0​\(1−c0\)c\_\{0\}\(1\-c\_\{0\}\), which involves no realized outcome\. This too preserves the conclusion: at dose 64 the lower two terciles match the prediction \(\+0\.068\+0\.068observed vs\+0\.054\+0\.054predicted;\+0\.068\+0\.068vs\+0\.069\+0\.069\) while the hardest tercile falls short \(\+0\.053\+0\.053observed, CI\[\+0\.010,\+0\.097\]\[\+0\.010,\+0\.097\], against\+0\.127\+0\.127predicted\); at dose 16 the hardest ex\-ante tercile is null \(−0\.001\-0\.001\) against a\+0\.125\+0\.125prediction\. SolvingL/b0=w​\(2−w\)L/b\_\{0\}=w\(2\-w\)per tercile gives extraction weights0\.550\.55,0\.440\.44,0\.180\.18at dose 64 \(per quintile:≈0\.5\{\\approx\}0\.5on the lower three,0\.300\.30,0\.160\.16on the top two\): extraction declines with question difficulty and rises with dose\.

##### Recovery decomposition and spillover mechanism\.

The boundary statistics in the main text are: treatment\-twin naive pre/post gap\+0\.053\+0\.053; control\-twin naive gap\+0\.001\+0\.001; local\-linear RD jumps atT∗T^\{\\ast\}with 90\-day bandwidth\+0\.052\+0\.052\(treatment\) and\+0\.012\+0\.012\(control\); placebo boundary atT∗−120T^\{\\ast\}\-120days−0\.014\-0\.014\. The per\-item ground truth, averaged over the injected pool with dose\-0 differencing, is\+0\.024\+0\.024to\+0\.027\+0\.027\. The gap between the boundary estimate and the per\-item truth is a base\-rate spillover, measured directly: the treatment twin’s mean probability of YES shifts relative to the control twin by−0\.088\-0\.088on injected questions,−0\.096\-0\.096on dose\-zero questions, and−0\.097\-0\.097on never\-injected post\-T∗T^\{\\ast\}questions \(n=543n=543\)—a uniform lean matching the83%83\\%NO rate of the injected outcomes\. The lean is nearly cost\-free on pre\-T∗T^\{\\ast\}pools \(YES rate0\.1700\.170–0\.1830\.183, hence the dose\-zero null of−0\.003\-0\.003\) and costs0\.0280\.028Brier on the post pool \(YES rate0\.3550\.355\)\. Dose\-zero differencing removes the uniform lean exactly, which is why the law analyses are unaffected\. Concentrated single\-style document injection plausibly exaggerates this prior shift relative to organic contamination; we flag it as a scope note for reading wild boundary estimates as totals rather than pure pre\-side inflation\.

This accounting also delimits the operational model of[Section˜3](https://arxiv.org/html/2608.02985#S3)empirically\. The twins realize both estimands at once: the counterfactual per\-item contrast of[Section˜B\.1](https://arxiv.org/html/2608.02985#A2.SS1)\(\+0\.024\+0\.024\) and the operational boundary quantity built on[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\(\+0\.052\+0\.052\)\. The wedge between them is a*post\-side*contamination effect \(the base\-rate lean costs0\.0280\.028Brier on post\-T∗T^\{\\ast\}questions\), which the support restriction of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)\(L=0L=0forg≥0g\\geq 0\) excludes by fiat and which perturbs the alignment of[Lemma˜5](https://arxiv.org/html/2608.02985#Thmtheorem5)by the same amount\. Under concentrated injection the restriction is therefore measurably violated; under diffuse organic contamination the lean, and with it the wedge, plausibly shrinks, bringing the operational and counterfactual estimands back together\.

##### Attenuation mechanisms the twins cannot exhibit\.

The spillover above biases a wild boundary jump*upward*; two mechanisms present in wild deployments bias it*downward*, and injection that is sharp atT∗T^\{\\ast\}produces neither\. First, quasi\-resolved outcomes: a question resolving shortly after a cutoff can be effectively decided in pre\-cutoff text \(an election with decisive polls, a lawsuit after closing arguments\)\. The support restriction of[Assumption˜1](https://arxiv.org/html/2608.02985#Thmassumption1)classifies such information as recency, so this is not a violation, but it raises the honest post\-boundary scorem​\(x,0\+\)m\(x,0^\{\+\}\)and shrinks the observed jump\. Second, ingestion lag: text describing outcomes realized just before the cutoff has had little time to be crawled and trained on, thinningL​\(x,g\)L\(x,g\)asg→0−g\\to 0^\{\-\}and smoothing the left limit\. Both mechanisms attenuateJ^\\widehat\{J\}toward zero, and neither is covered by the cutoff\-placement sensitivity checks, which perturb the boundary’s location rather than the information density around it\. The reading rules of[Section˜6\.2](https://arxiv.org/html/2608.02985#S6.SS2)therefore treat a wild jump as an upper bound only against the spillover mechanism: a starred jump survives attenuation, while a boundary null does not certify the absence of leakage sitting deeper in the pre\-cutoff period\.

##### PRC protocol\.

Three paraphrases per question are generated by DeepSeek\-V3\.1 and validated by DeepSeek\-V3\.2 \(semantic equivalence and preserved resolution criteria\);971971of988988questions retain all three\. Both twins score all paraphrases with the same readout\. Per\-question consensus probabilities are temperature\-calibrated per twin on dose\-0 anchors, and the reported statistic is the twin\-differenced residual covariance with paired question bootstrap \(4,0004\{,\}000resamples\):\+0\.0069∗\+0\.0069^\{\\ast\}\(CI\[\+0\.0035,\+0\.0107\]\[\+0\.0035,\+0\.0107\]\) on injected questions, null at dose zero \(−0\.0057\-0\.0057, CI\[−0\.0128,\+0\.0016\]\[\-0\.0128,\+0\.0016\]\), and monotone in dose with the dose\-64 cell starred \(\+0\.0121∗\+0\.0121^\{\\ast\}\)\. Raw, undifferenced PRC is negative in every cell \(treatment/control×\\timesinjected/dose\-0\), reproducing the miscalibration failure predicted by[Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)\.[Figure˜9](https://arxiv.org/html/2608.02985#A4.F9)shows the calibrated twin\-differenced statistic by cell\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m3_prc.png)Figure 9:M3: the calibrated, twin\-differenced PRC fires only where leakage was injected\.Bars: the calibrated twin\-differenced PRC per injection cell \(dose zero and each positive dose\), computed from three validated paraphrases per question; positive values signal evidence leakage\. Whiskers: paired\-bootstrap 95% CIs;∗: CI excludes zero\. Result: null at dose zero, monotone in dose, starred at dose 64 \(\+0\.0121∗\+0\.0121^\{\\ast\}\); the raw statistic without calibration or differencing is negative everywhere, the miscalibration failure[Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)predicts\.

### D\.5M4\-F \(forecasting\-panel matrix and clean\-anchor arm\)

##### Specificity matrix\.

Targets are DeepSeek\-V3\.1 \(documented cutoff March 2025\), Kimi\-K2\.6 \(April 2025\), GPT\-5\.4 \(August 2025\), and GPT\-5\.5 \(December 2025\); Gemini\-3\.1\-Pro \(January 2025\) is exploratory because the panel has thin mass before its boundary\. Controls are MiniMax\-M3 and Claude\-Opus\-4\.7, whose January 2026 cutoffs leave no leakage discontinuity inside the tested window\. For each target and each assumed boundary, the statistic is the pre/post jump \(90\-day windows\) in the per\-question target\-minus\-control score difference, using the mean of the available controls on each question\. Confidence intervals use the source×\\timesmonth cluster bootstrap\. Joint inference over the matrix uses a permutation test that reassigns cutoff dates to models and compares the observed diagonal\-minus\-off\-diagonal contrast with its permutation distribution\.

##### Exploratory Gemini row\.

Gemini\-3\.1\-Pro’s January 2025 cutoff precedes every tested boundary, so all of its cells are placebos:\+0\.141∗\+0\.141^\{\\ast\}\(Mar’25\),\+0\.042\+0\.042\(Apr’25\),−0\.050\-0\.050\(Aug’25\),\+0\.026\+0\.026\(Dec’25\)\. The one starred value sits at the boundary with the thinnest pre\-boundary mass, which makes the estimate unstable, and it is not at any documented cutoff; we do not read it as a detection\.

##### Cutoff\-resolution robustness for Kimi\-K2\.6\.

Kimi\-K2\.6’s coarser \(tier\-two\) cutoff documentation is not load\-bearing: its cells at the adjacent March and April boundaries \(\+0\.034\+0\.034,\+0\.022\+0\.022;[Table˜2](https://arxiv.org/html/2608.02985#S7.T2)\) are both unstarred, so its null survives a one\-month cutoff shift\.

##### Clean\-anchor arm\.

ForecastBench archives every real\-time forecast submitted to the benchmark before question resolution\. For a target with cutoffTT, we select*family relatives*\(same vendor, earlier release\) whose archived real\-time forecasts cover questions resolving on both sides ofTT: GPT\-5\-Mini and GPT\-5\.1 for GPT\-5\.5; GPT\-4\.1 and o4\-mini for GPT\-5\.4\. On each covered question the statistic is our retrospective target score minus the relative’s archived real\-time score; the crowd anchor cancels in this difference\. The estimand is the jump of this difference atTT\. Because archived forecasts for late\-resolving questions can come from more recent submission rounds \(shorter horizons\), the primary variant restricts to anchor horizons of 10 to 75 days, which balances mean pre/post horizons; the unbanded variant is reported alongside\. The band was adopted after the horizon imbalance was observed and is logged as a dated deviation in the versioned deviations file, before any banded estimate was interpreted\. Two protocol checks validate the arm on models that submitted real\-time forecasts themselves: retrospective re\-queries of GPT\-5 reproduce its archived forecasts \(mean probability shift−0\.021\-0\.021, CI\[−0\.056,\+0\.010\]\[\-0\.056,\+0\.010\],n=50n=50\), and DeepSeek\-V3\.1 shows a small positive shift \(\+0\.021\+0\.021, CI\[\+0\.009,\+0\.032\]\[\+0\.009,\+0\.032\],n=910n=910\) that we flag as protocol sensitivity; both are an order of magnitude smaller than the GPT\-5\.5 leakage signature\. Because the leakage estimand is a*jump*rather than a level, protocol level effects cancel unless they vary sharply in time\. The placebo\-jump check tests this directly on DeepSeek\-V3\.1’s own retrospective\-minus\-real\-time score difference \(n=910n=910, mean level−0\.016\-0\.016\): at the December 2025 boundary the jump is\+0\.026\+0\.026\(CI\[−0\.011,\+0\.064\]\[\-0\.011,\+0\.064\],npre/npost=42/419n\_\{\\mathrm\{pre\}\}/n\_\{\\mathrm\{post\}\}=42/419\), and the only starred jump at any tested date is−0\.036\-0\.036at March 2026, opposite in sign and one third of the banded GPT\-5\.5 signal\. Matrix power: converting the null diagonal cells of the specificity matrix into bounds, their80%80\\%\-power minimum detectable effects are0\.0770\.077\(DeepSeek\-V3\.1\),0\.1080\.108\(Kimi\-K2\.6\),0\.0970\.097\(GPT\-5\.4\), and0\.0540\.054\(GPT\-5\.5\) anchored\-Brier units, computed from the cluster\-bootstrap standard errors of each own\-cutoff cell\.

##### Reading disagreement between the two designs\.

For the same model and boundary \(GPT\-5\.5 at December 2025\), the matrix reports\+0\.034\+0\.034\(unstarred\) and the anchor arm\+0\.106∗\+0\.106^\{\\ast\}\. The designs differ in the two respects that matter\. Their baselines differ in cleanliness: the control models are guaranteed only to have no*discontinuity*at December 2025—their January 2026 cutoffs let them memorize the same pre\-boundary outcomes, and any inflation they carry is subtracted from the target’s cell—whereas an archived real\-time forecast cannot contain leakage at all\. And the paired per\-question difference removes the cross\-vendor cluster variance that sets the matrix’s power floor \(0\.0540\.054at this cell\)\. Both forces push the matrix cell toward zero, so the pattern is attenuation plus lower power, not contradiction\. The auditor’s rule: a starred own\-cutoff jump that survives placebo dates is a detection scoped to its design’s baseline; a null matrix cell bounds the effect but certifies nothing\. Taken together, the matrix bounds leakage among 2025\-cutoff flagships at a few points of anchored Brier, and the within\-family route detects a leakage signature for GPT\-5\.5 exactly at its documented cutoff\.

### D\.6M4\-C \(LiveCodeBench code arm\)

##### LiveCodeBench\.

LiveCodeBench\(Jain et al\.,[2025](https://arxiv.org/html/2608.02985#bib.bib21)\)is a coding benchmark that collects problems from competitive programming contests \(LeetCode, Codeforces, AtCoder\)\. Each problem carries its*contest release date*, so a model can only have trained on a problem’s solution if the contest occurred before the model’s training cutoff\. The project publishes official per\-problem pass@1 \(fraction of sampled code completions passing all unit tests\) for many models, and refreshes regularly with new contests to stay contamination\-free\. We use problems from May 2023 through April 2025 \(713713–10551055contest\-dated problems per model, filtered to those present in all evaluated models\)\. All scores are taken from the public submissions repository; no additional inference is run\.

##### Models and cutoffs\.

The*continuity targets*, evaluated entirely from published per\-problem data, are GPT\-4o\-2024\-08\-06 \(cutoff≈\\approxOct 2023\) and Claude\-3\.5\-Sonnet\-20240620 \(cutoff≈\\approxApr 2024\)\. Composition controls have cutoffs*after*the latest problem \(Gemini\-2\.5\-Pro, DeepSeek\-R1, and the published 2025\-cutoff pool\)\. Because control cutoffs lie outside the evaluation window, they have no leakage discontinuity at any target’s cutoff \([Proposition˜7](https://arxiv.org/html/2608.02985#Thmtheorem7)\); their pre/post gap at any evaluation\-window date captures only composition drift \(later contests are harder\)\. Cutoff dates are documented approximations at month resolution\.

##### Self\-generated extension\.

To cover 2024 training cutoffs with current models, we additionally generate one greedy completion per problem through OpenRouter for four targets with documented 2024 cutoffs—GPT\-5 \(Sep 2024\), GPT\-5\-Mini \(May 2024\), DeepSeek\-V3\-0324 \(Jul 2024\), and Kimi\-K2\-0905 \(Dec 2024\)—and two clean controls with 2025 cutoffs \(DeepSeek\-V3\.2 and MiniMax\-M1\), and grade locally against the official unit tests with the official checker semantics \(stdin and functional harnesses, per\-problem time limits\)\. The local grader was validated by regrading published DeepSeek\-V3 completions: verdict agreement is98\.3%98\.3\\%, within the pre\-registered three\-point tolerance\. The same±160\\pm 160\-day window DiD and date\-permutation machinery as the continuity arm is applied at the targets’ documented cutoffs\.

##### DiD estimation\.

For each target model and candidate cutoff dateTT, define a symmetric window\[T−h,T\+h\]\[T\{\-\}h,\\ T\{\+\}h\]days \(default half\-widthh=160h=160, givingnpre≈207n\_\{\\mathrm\{pre\}\}\\approx 207andnpost≈194n\_\{\\mathrm\{post\}\}\\approx 194problems for GPT\-4o at Oct 2023\)\. Split problems into pre \(g<0g<0\) and post \(g≥0g\\geq 0\)\. The within\-model gap isgapm=pass​@​1¯prem−pass​@​1¯postm\\mathrm\{gap\}^\{m\}=\\overline\{\\mathrm\{pass@1\}\}^\{m\}\_\{\\mathrm\{pre\}\}\-\\overline\{\\mathrm\{pass@1\}\}^\{m\}\_\{\\mathrm\{post\}\}\. The pooled composition control’s gap is the same quantity computed on the average pass@1 of the two 2025\-cutoff models\. The boundary\-leakage estimate isJ^=gapM−gapM0\\widehat\{J\}=\\mathrm\{gap\}^\{M\}\-\\mathrm\{gap\}^\{M\_\{0\}\}: the target’s excess gap beyond composition drift\. Bootstrap CIs:40004000resamples of problem IDs \(resampled jointly for target and control\), 95% percentile interval\. The robustness battery \(control smoothness, separate controls, date specificity, bandwidth, difficulty stratification\) is in[Section˜E\.5](https://arxiv.org/html/2608.02985#A5.SS5); the global\-BBsensitivity analyses are in[Section˜E\.6](https://arxiv.org/html/2608.02985#A5.SS6)\.

##### Results of the self\-generated extension\.

The 2024\-cohort targets do not reproduce the GPT\-4o wild positive\. At their documented cutoffs, GPT\-5\-Mini \(\+0\.005\+0\.005, CI\[−0\.058,\+0\.072\]\[\-0\.058,\+0\.072\]\) and Kimi\-K2\-0905 \(−0\.002\-0\.002, CI\[−0\.078,\+0\.075\]\[\-0\.078,\+0\.075\]\) are null\. DeepSeek\-V3\-0324 is marginally positive \(\+0\.051\+0\.051, CI\[−0\.013,\+0\.115\]\[\-0\.013,\+0\.115\]; date\-permutationp=0\.050p=0\.050\), and its only starred cells sit at the April and May 2024 placebo dates \(\+0\.079\+0\.079,\+0\.103\+0\.103\) rather than at its own July 2024 cutoff—a pattern consistent with diffuse contamination of spring\-2024 contest solutions rather than a sharp training\-cutoff boundary, so we do not claim a detection\. GPT\-5 is significantly*negative*\(−0\.119\-0\.119, CI\[−0\.185,−0\.051\]\[\-0\.185,\-0\.051\]\): the 2025\-cutoff control pool improves more on post\-September\-2024 problems than GPT\-5 does, a violation of the no\-differential\-trend condition of[Proposition˜7](https://arxiv.org/html/2608.02985#Thmtheorem7)for reasoning\-era model pairs, and a reminder that the DiD sign test is only interpretable when the control\-trend assumption holds\. We therefore report the extension as bounded nulls with one flagged assumption failure, and rest the code\-domain detection claim on the published\-data continuity targets above\.

### D\.7M5 \(disciplined null on forecasting\)

##### Program reading \(demoted from the main\-text close\)\.

Each claim has one primary experiment whose headline result is that claim \([Table˜4](https://arxiv.org/html/2608.02985#A4.T4)\): M1 establishes the confound on cutoff\-clean models; M2 anchors dose sensitivity in randomized controlled pretraining; M3 supplies ground truth for the estimators and the laws; M4 shows what honest wild detection looks like, starred where the evidence supports it and null elsewhere; and M5 shows the discipline, with the weak\-control failure mode demonstrated rather than assumed and power analysis quantifying what the nulls exclude\.

##### Data\.

The shared forecasting panel of[Section˜D\.2](https://arxiv.org/html/2608.02985#A4.SS2), restricted to questions resolving between January 2025 and July 2026:265265pre\-boundary and1,2071\{,\}207post\-boundary questions, evaluated pairwise \(every model on every question\)\.

##### Boundary\.

The primary target Qwen3\.5\-35B\-A3B has only a year\-level documented cutoff \(2026\), so we place the boundary at July 2025, inside its plausible training window, and report a boundary sweep from−180\-180to\+180\+180days around it in 30\-day steps\. The adjusted estimate is null at every placement\.

##### Control selection\.

The control is chosen*before*any DiD is computed, by a pre\-registered profile\-matching protocol: on post\-boundary questions that are clean for every candidate, compute each candidate’s distance to the target \(squared\-error distance between calibration profiles plus squared\-error distance between per\-source mean scores\)\. Between the two flagships with post\-window\-clean cutoffs, GPT\-5 \(distance0\.00510\.0051\) is selected over Gemini\-3\.1\-Pro \(0\.00880\.0088\)\. GPT\-3\.5\-Turbo \(cutoff September 2021\) serves as the deliberately mismatched weak control\. Secondary targets are Kimi\-K2\.6, DeepSeek\-V3\.2, GLM\-4\.7, and MiniMax\-M3, all queried retrospectively through the shared protocol\.

##### Paired DiD estimation\.

For each questionii, form the paired differenceDi=Litarget−LicontrolD\_\{i\}=L\_\{i\}^\{\\mathrm\{target\}\}\-L\_\{i\}^\{\\mathrm\{control\}\}of crowd\-anchored Brier reductions\. The adjusted estimate is the pre/post difference in meanDiD\_\{i\}, with source×\\timesmonth cluster\-bootstrap confidence intervals \(4,0004\{,\}000resamples\)\. Power is assessed by semi\-synthetic injection: a synthetic leakage effect of known size is added to pre\-boundaryDiD\_\{i\}and the detection rate of the full pipeline is recorded over300300replications per effect size\. The boundary sweep and power results are in[Section˜E\.7](https://arxiv.org/html/2608.02985#A5.SS7)\.

##### Results\.

[Table˜7](https://arxiv.org/html/2608.02985#A4.T7)reports the per\-model naive gaps and adjusted DiD estimates summarized in[Section˜7\.2](https://arxiv.org/html/2608.02985#S7.SS2)and plotted in[Figure˜4](https://arxiv.org/html/2608.02985#S7.F4)\.

Table 7:M5: full estimates behind[Figure˜4](https://arxiv.org/html/2608.02985#S7.F4)\.Column 2: the naive pre/post gapΔ\\Deltaat the July 2025 boundary; positive reads as leakage\. Column 3: the paired DiD against the pre\-registered profile\-matched control \(GPT\-5\); for a leak\-free model the ideal value is zero\. Cluster\-bootstrap 95% CIs;∗: CI excludes zero\. Result: no adjusted estimate is significantly positive \(marginal negatives are slight overcorrection, the conservative direction\); the weak\-control row substitutes GPT\-3\.5\-Turbo and re\-inflates the estimate, the control\-mismatch failure mode of[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\.
##### Provenance of the fitted overconfidence temperature\.

The valueT≈8T\\approx 8cited in[Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)and used to set the T4 sweep range \(Tover∈\{2,4,8\}T\_\{\\mathrm\{over\}\}\\in\\\{2,4,8\\\}\) was fit on this forecasting pipeline’s data: temperature scalingpcal=σ​\(logit​\(p\)/T\)p\_\{\\mathrm\{cal\}\}=\\sigma\(\\mathrm\{logit\}\(p\)/T\)withTTchosen to minimize log\-loss on Qwen3\.5’s*leakage\-free*post\-boundary control forecasts \(n=168n=168ex\-ante\-uncertain factual questions, crowd anchor frozen in\[0\.25,0\.75\]\[0\.25,0\.75\]\)\. The fit reached the upper end of the search interval\(0\.3,8\]\(0\.3,8\], soT≈8T\\approx 8is best read as a lower bound on the distortion: Qwen’s raw probabilities are severely overconfident, which is why the raw residual covariance is negative and PRC requires recalibration before use\.

## Appendix EComplementary experiments and robustness analyses

The synthetic validation study E1 and its two supplementary arms come first; the remaining subsections report one robustness analysis each, in a fixed format: the*intuition*\(what question it answers and which claim it defends\), the*setup*\(a short description or a pointer to[Appendix˜D](https://arxiv.org/html/2608.02985#A4)where shared\), and the*results*\.

### E\.1E1 \(synthetic validation\)

This synthetic study validates the estimators under known data\-generating assumptions\. It formerly opened the main\-text experiment section and is retained here in full; the real\-data ground\-truth role is now carried by M3\. All sub\-tests share a common data\-generation process, described first, with per\-test variations noted below; results follow at the end of this subsection, and the supplementary arms T5–T6 follow in[Sections˜E\.2](https://arxiv.org/html/2608.02985#A5.SS2)and[E\.3](https://arxiv.org/html/2608.02985#A5.SS3)\.

##### Common data generation\.

For each questionii, draw a crowd anchorc0,i∼Uniform​\(0\.15,0\.85\)c\_\{0,i\}\\sim\\mathrm\{Uniform\}\(0\.15,0\.85\)and a binary outcomeYi∼Bernoulli​\(c0,i\)Y\_\{i\}\\sim\\mathrm\{Bernoulli\}\(c\_\{0,i\}\)\. The*honest forecast*isPhon,i=clip​\(c0,i\+s​\(Yi−c0,i\)\+ϵi,\[10−3,1−10−3\]\)P\_\{\\mathrm\{hon\},i\}=\\mathrm\{clip\}\\\!\\big\(c\_\{0,i\}\+s\\,\(Y\_\{i\}\-c\_\{0,i\}\)\+\\epsilon\_\{i\},\\;\[10^\{\-3\},1\{\-\}10^\{\-3\}\]\\big\), wheres∈\[0,1\]s\\in\[0,1\]is a skill parameter controlling how much the model improves on the crowd, andϵi∼𝒩​\(0,0\.05\)\\epsilon\_\{i\}\\sim\\mathcal\{N\}\(0,0\.05\)is idiosyncratic noise\. For a leaked question with extraction weightw∈\[0,1\]w\\in\[0,1\], the observed forecast is the convex blendPi=clip​\(\(1−w\)​Phon,i\+w​Yi\)P\_\{i\}=\\mathrm\{clip\}\\\!\\big\(\(1\-w\)\\,P\_\{\\mathrm\{hon\},i\}\+w\\,Y\_\{i\}\\big\); for a clean question,Pi=Phon,iP\_\{i\}=P\_\{\\mathrm\{hon\},i\}\. An optional overconfidence parameterTover≥1T\_\{\\mathrm\{over\}\}\\geq 1transformsPiP\_\{i\}viaPobs,i=σ​\(logit​\(Pi\)⋅Tover\)P\_\{\\mathrm\{obs\},i\}=\\sigma\(\\mathrm\{logit\}\(P\_\{i\}\)\\cdot T\_\{\\mathrm\{over\}\}\), whereσ\\sigmais the logistic function;Tover=1T\_\{\\mathrm\{over\}\}=1leaves the forecast unchanged\.

##### T1 \(recovery and coverage\)\.

For each extraction weightw∈\{0,0\.2,0\.4,0\.7,1\.0\}w\\in\\\{0,0\.2,0\.4,0\.7,1\.0\\\}, generaten=400n=400leaked questions andn=400n=400clean\-control questions \(both with skills=0\.3s=0\.3, no overconfidence\)\. The true inflation isBtrue=1n​∑i\[\(Phon,i−Yi\)2−\(Pi−Yi\)2\]B\_\{\\mathrm\{true\}\}=\\tfrac\{1\}\{n\}\\sum\_\{i\}\\big\[\(P\_\{\\mathrm\{hon\},i\}\-Y\_\{i\}\)^\{2\}\-\(P\_\{i\}\-Y\_\{i\}\)^\{2\}\\big\], computed from the known honest forecast\. The estimated inflation is the Brier\-reduction DiD:B^=S¯leak−S¯clean\\widehat\{B\}=\\bar\{S\}\_\{\\mathrm\{leak\}\}\-\\bar\{S\}\_\{\\mathrm\{clean\}\}, whereS¯=1n​∑\(c0−Y\)2−\(P−Y\)2\\bar\{S\}=\\tfrac\{1\}\{n\}\\sum\(c\_\{0\}\-Y\)^\{2\}\-\(P\-Y\)^\{2\}\. Bootstrap CIs: 200 problem\-level resamples, 95% percentile interval\. We run 300 Monte Carlo repetitions perwwand report the mean biasB^−Btrue\\widehat\{B\}\-B\_\{\\mathrm\{true\}\}and the fraction of repetitions in which the CI coversBtrueB\_\{\\mathrm\{true\}\}\.

##### T2 \(RD recovery and placebo\)\.

Each questioniihas a resolution\-time gapδi∼Uniform​\(−180,180\)\\delta\_\{i\}\\sim\\mathrm\{Uniform\}\(\-180,180\)\(days to the cutoff\)\. The honest skill varies smoothly with recency:s​\(δ\)=0\.4−0\.25​\|δ\|/180s\(\\delta\)=0\.4\-0\.25\\,\|\\delta\|/180, so the model is better near its cutoff \(pure recency, no leakage\)\. Leakage of weightw0∈\{0,0\.3,0\.6\}w\_\{0\}\\in\\\{0,0\.3,0\.6\\\}is injected only on the leaked side \(δ<0\\delta<0\)\. The RD jump is estimated by local\-linear regression with a triangular kernel and half\-bandwidthh=120h=120days: fitℓi=\(Pi−Yi\)2≈β0\+β1​1​\[δ<0\]\+β2​δ\+β3​δ⋅𝟏​\[δ<0\]\\ell\_\{i\}=\(P\_\{i\}\-Y\_\{i\}\)^\{2\}\\approx\\beta\_\{0\}\+\\beta\_\{1\}\\,\\mathbf\{1\}\[\\delta<0\]\+\\beta\_\{2\}\\,\\delta\+\\beta\_\{3\}\\,\\delta\\cdot\\mathbf\{1\}\[\\delta<0\]on the weighted sample\|δ\|≤h\|\\delta\|\\leq h; the jump estimate is−β^1\-\\hat\{\\beta\}\_\{1\}\. The*placebo*repeats the same procedure with no leakage injected \(w0=0w\_\{0\}=0forced, though recency is still present\)\.n=1500n=1500per run, 300 repetitions\.

##### T3 \(complementarity\)\.

For each extraction weightw∈\{0,0\.25,0\.5,0\.75,1\.0\}w\\in\\\{0,0\.25,0\.5,0\.75,1\.0\\\}, generaten=8000n=8000questions\. For each question, queryK=12K=12independent “paraphrases” \(simulated by adding independent noiseek∼𝒩​\(0,0\.12\)e\_\{k\}\\sim\\mathcal\{N\}\(0,0\.12\)to the crowd anchor before blending withww\)\. The consensus isy¯i=K−1​∑kPi\(k\)\\bar\{y\}\_\{i\}=K^\{\-1\}\\sum\_\{k\}P\_\{i\}^\{\(k\)\}\. Compute the PRC asΔ^=Cov^​\(y¯,Y−y¯\)\\widehat\{\\Delta\}=\\widehat\{\\mathrm\{Cov\}\}\(\\bar\{y\},\\,Y\-\\bar\{y\}\)and the leakage scoreL=1n​∑\[\(c0−Y\)2−\(y¯−Y\)2\]L=\\tfrac\{1\}\{n\}\\sum\\big\[\(c\_\{0\}\-Y\)^\{2\}\-\(\\bar\{y\}\-Y\)^\{2\}\\big\]\.

##### T4 \(calibration under overconfidence\)\.

This sub\-test isolates the calibration step\. The honest forecast is set to the crowd anchor itself,Phon=c0P\_\{\\mathrm\{hon\}\}=c\_\{0\}, withY∼Bernoulli​\(c0\)Y\\sim\\mathrm\{Bernoulli\}\(c\_\{0\}\), so the clean forecaster is calibrated by construction\. Leaked forecasts areP=\(1−w\)​Phon\+w​YP=\(1\-w\)\\,P\_\{\\mathrm\{hon\}\}\+w\\,Ywithw=0\.8w=0\.8\. Reported probabilities are then distorted by overconfidence:Pobs=σ​\(Tover​logit​\(P\)\)P\_\{\\mathrm\{obs\}\}=\\sigma\(T\_\{\\mathrm\{over\}\}\\,\\mathrm\{logit\}\(P\)\), forTover∈\{1,2,4,8\}T\_\{\\mathrm\{over\}\}\\in\\\{1,2,4,8\\\}\. For each repetition, we drawn=600n=600leaked andn=600n=600clean evaluation questions, plus a*separate*leakage\-free calibration set of sizencal∈\{100,250,500,1000,2000,5000\}n\_\{\\mathrm\{cal\}\}\\in\\\{100,250,500,1000,2000,5000\\\}\. Temperature scaling \(T^\\hat\{T\}minimizing log\-loss\) is fit*only*on the calibration set and then applied to both evaluation sets before computing DiD\. The*naive*estimator skips calibration and usesPobsP\_\{\\mathrm\{obs\}\}directly\. We report mean bias, mean absolute bias, and RMSE over 200 repetitions per\(Tover,ncal\)\(T\_\{\\mathrm\{over\}\},\\,n\_\{\\mathrm\{cal\}\}\)cell\.

##### T5 \(no free inflation; results in[Section˜E\.2](https://arxiv.org/html/2608.02985#A5.SS2)\)\.

Honest forecasts are generated as in T1 withw=0w=0\(zero true leakage\)\. The “perturbed” side replacesPhonP\_\{\\mathrm\{hon\}\}byσ​\(logit​\(Phon\)\+η\)\\sigma\(\\mathrm\{logit\}\(P\_\{\\mathrm\{hon\}\}\)\+\\eta\)withη∼𝒩​\(0,ση2\)\\eta\\sim\\mathcal\{N\}\(0,\\sigma\_\{\\eta\}^\{2\}\)drawn independently ofYY; logit\-space noise avoids the outcome dependence that clipping would induce at the probability boundaries\. Noise scalesση∈\{0\.25,0\.5,1\.0\}\\sigma\_\{\\eta\}\\in\\\{0\.25,0\.5,1\.0\\\};n=400n=400per side; the same Brier\-reduction DiD as T1; 300 Monte Carlo repetitions per scale\.

##### T6 \(concentration across surprise strata; results in[Section˜E\.3](https://arxiv.org/html/2608.02985#A5.SS3)\)\.

Fixed extraction weightw=0\.5w=0\.5,n=4000n=4000leaked and40004000clean questions per repetition, 300 repetitions\. Questions are stratified into quartiles of the*observable*crowd surprise\(c0−Y\)2\(c\_\{0\}\-Y\)^\{2\}\(quartile cuts computed on the pooled sample\)\. Within each stratum, the stratified DiD is the mean Brier reduction of the leaked group minus that of the clean group; the prediction is the stratum mean ofb0​w​\(2−w\)b\_\{0\}\\,w\(2\-w\)withb0=\(Phon−Y\)2b\_\{0\}=\(P\_\{\\mathrm\{hon\}\}\-Y\)^\{2\}, per[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)\.

##### Results \(T1–T4;[Figure˜10](https://arxiv.org/html/2608.02985#A5.F10)\)\.

*T1 \(DiD recovery and coverage\)\.*Across five leakage weights \(w∈\{0,0\.2,0\.4,0\.7,1\.0\}w\\in\\\{0,0\.2,0\.4,0\.7,1\.0\\\}, trueB∈\{0,0\.038,0\.067,0\.095,0\.105\}B\\in\\\{0,0\.038,0\.067,0\.095,0\.105\\\}\), the DiD estimator is essentially unbiased \(mean bias≈9×10−4\{\\approx\}9\\times 10^\{\-4\}at every level\) and the 95% bootstrap CIs cover the trueBBin9494–100%100\\%of repetitions; the no\-leak case returns a null\.*T2 \(RD recovery and placebo\)\.*With a smooth recency surface and leakage injected only on the pre\-cutoff side, the RD jump tracks the injection \(\+0\.000\+0\.000,\+0\.039\+0\.039,\+0\.065\+0\.065forw0=0,0\.3,0\.6w\_\{0\}=0,0\.3,0\.6\) while the no\-leakage placebo jump is≈0\.000\{\\approx\}0\.000at every level: continuous recency alone does not fabricate a discontinuity \([Assumption˜3](https://arxiv.org/html/2608.02985#Thmassumption3)\)\.*T3 \(RD/PRC complementarity\)\.*Sweepingwwconfirms[Proposition˜9](https://arxiv.org/html/2608.02985#Thmtheorem9): PRC is hump\-shaped \(peak\+0\.052\+0\.052nearw=0\.5w\{=\}0\.5, vanishing atw=1w\{=\}1\) while the RD signal grows monotonically to\+0\.209\+0\.209atw=1w\{=\}1; recovered ratiosL​\(w\)/L​\(1\)L\(w\)/L\(1\)matchw​\(2−w\)w\(2\-w\)\([Table˜8](https://arxiv.org/html/2608.02985#A5.T8)\)\.*T4 \(calibration under overconfidence\)\.*Distorting a calibrated clean forecaster byTover∈\{2,4,8\}T\_\{\\mathrm\{over\}\}\\in\\\{2,4,8\\\}produces naive DiD bias0\.0220\.022,0\.0550\.055,0\.0850\.085; temperature scaling fit onncaln\_\{\\mathrm\{cal\}\}leakage\-free anchors reduces it to≈0\.008\{\\approx\}0\.008atncal=100n\_\{\\mathrm\{cal\}\}=100and≈0\.001\{\\approx\}0\.001atncal=5000n\_\{\\mathrm\{cal\}\}=5000\. Calibration works but is a real resource requirement \([Remark˜2](https://arxiv.org/html/2608.02985#Thmremark2)\)\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/m1.png)Figure 10:E1: the estimators recover known ground truth in simulation\.\(a\) DiD estimates lie on the truth line \(bootstrap coverage9494–100%100\\%across leakage levels\)\. \(b\) RD jumps track injected boundary leakage; no\-leak placebos stay at zero under smooth recency\. \(c\) PRC is hump\-shaped and vanishes at pure answer leakage \(w=1w\{=\}1\) while the score\-based signal grows monotonically—the two detectors are complements\. \(d\) Overconfidence biases the naive statistic; temperature calibration on leakage\-free anchors removes the bias asncaln\_\{\\mathrm\{cal\}\}grows\. Result: each estimator behaves as[Assumptions˜4](https://arxiv.org/html/2608.02985#Thmassumption4),[3](https://arxiv.org/html/2608.02985#Thmassumption3),[7](https://arxiv.org/html/2608.02985#Thmassumption7)and[2](https://arxiv.org/html/2608.02985#Thmremark2)predict under known data\-generating assumptions\.

### E\.2No free inflation under outcome\-orthogonal noise \(E1 arm T5\)

##### Intuition\.

[Corollary˜3](https://arxiv.org/html/2608.02985#Thmtheorem3)states that a predictor can raise its score only through a positive loading on the leaked signal: perturbations that are independent of the outcome cannot inflate\. A useful falsification check on the estimator is therefore whether it can be fooled into booking outcome\-orthogonal noise as leakage—if it could, positive estimates in M2–M4 would be uninformative\.

##### Setup\.

As defined in[Section˜E\.1](https://arxiv.org/html/2608.02985#A5.SS1)\(T5\): honest forecasts with zero true leakage, perturbed by zero\-mean logit\-space noise independent ofYYat scalesση∈\{0\.25,0\.5,1\.0\}\\sigma\_\{\\eta\}\\in\\\{0\.25,0\.5,1\.0\\\}; the same Brier\-reduction DiD estimator as T1; 300 Monte Carlo repetitions per scale\.

##### Results\.

The estimate is deflationary at every scale, increasingly so as noise grows: meanB^=−0\.003\\widehat\{B\}=\-0\.003\(95% CI of the mean\[−0\.004,−0\.002\]\[\-0\.004,\-0\.002\]\) atση=0\.25\\sigma\_\{\\eta\}=0\.25;−0\.012\-0\.012\(\[−0\.013,−0\.011\]\[\-0\.013,\-0\.011\]\) atση=0\.5\\sigma\_\{\\eta\}=0\.5; and−0\.044\-0\.044\(\[−0\.045,−0\.043\]\[\-0\.045,\-0\.043\]\) atση=1\.0\\sigma\_\{\\eta\}=1\.0\. Individual repetitions can come out slightly positive under sampling noise at the smallest scale \(34%34\\%of repetitions;5%5\\%atση=0\.5\\sigma\_\{\\eta\}=0\.5; none atση=1\.0\\sigma\_\{\\eta\}=1\.0\), but the mean effect is strictly negative throughout, matching[Corollary˜3](https://arxiv.org/html/2608.02985#Thmtheorem3): noise costs Brier score, it cannot fake inflation \([Figure˜11](https://arxiv.org/html/2608.02985#A5.F11)a\)\.

### E\.3Leakage concentration across surprise strata \(E1 arm T6\)

##### Intuition\.

The per\-question lawL=b0​w​\(2−w\)L=b\_\{0\}\\,w\(2\-w\)\([Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)\) makes two testable predictions beyond monotonicity inww: at fixed extraction, leakage should*concentrate*on questions where the crowd is surprised \(largeb0b\_\{0\}\), and across extraction levels the total leakage should follow thew​\(2−w\)w\(2\-w\)shape exactly\. This licenses the main\-text statement that leakage is heterogeneous and concentrated rather than a uniform rate\.

##### Setup\.

As defined in[Section˜E\.1](https://arxiv.org/html/2608.02985#A5.SS1)\(T6\): fixedw=0\.5w=0\.5, quartiles of observable crowd surprise\(c0−Y\)2\(c\_\{0\}\-Y\)^\{2\}, stratified DiD against the stratum\-mean prediction𝔼​\[b0​w​\(2−w\)\]\\mathbb\{E\}\[b\_\{0\}\\,w\(2\-w\)\]; 300 repetitions\. The functional\-form check reuses the T3 sweep \([Section˜E\.1](https://arxiv.org/html/2608.02985#A5.SS1)\), comparing recovered leakage ratiosL​\(w\)/L​\(1\)L\(w\)/L\(1\)with the predictedw​\(2−w\)w\(2\-w\)\.

##### Results\.

The stratified DiD estimate rises monotonically and steeply across surprise quartiles \(0\.0180\.018,0\.0410\.041,0\.0830\.083,0\.1740\.174for Q1 through Q4\) and matches the predicted stratum means \(0\.0180\.018,0\.0410\.041,0\.0830\.083,0\.1740\.174\) to within Monte Carlo error \([Figure˜11](https://arxiv.org/html/2608.02985#A5.F11)b\): the top quartile carries roughly ten times the leakage of the bottom quartile at the same extraction weight\. The functional form is confirmed across the full sweep \([Table˜8](https://arxiv.org/html/2608.02985#A5.T8)\): observed ratios trackw​\(2−w\)w\(2\-w\)to within0\.0050\.005at everyww\. A caveat applies to both arms: the synthetic generator implements the convex blend of[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2), so these checks establish that the estimators recover the law when it holds—they are consistency checks of the machinery, not tests of the convex\-pull assumption itself, whose real\-data validation remains open \([Section˜9](https://arxiv.org/html/2608.02985#S9)\)\.

Table 8:E1: the recovered leakage profile matchesw​\(2−w\)w\(2\-w\)\.Leakage ratiosL​\(w\)/L​\(1\)L\(w\)/L\(1\)from the synthetic extraction\-weight sweep \(arm T3\) against the prediction of[Assumption˜2](https://arxiv.org/html/2608.02985#Thmassumption2)\. Result: observed and predicted agree within0\.0070\.007at everyww\.![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/e1_supp.png)Figure 11:E1 supplementary arms: noise cannot fake inflation, and leakage concentrates where the law predicts\.\(a\) T5: under outcome\-orthogonal logit noise with zero true leakage, the DiD estimate stays at or below zero at every noise scale, as[Corollary˜3](https://arxiv.org/html/2608.02985#Thmtheorem3)requires \(whiskers: 2\.5–97\.5 percentile range across repetitions\)\. \(b\) T6: the stratified DiD \(bars\) tracks the predicted𝔼​\[b0​w​\(2−w\)\]\\mathbb\{E\}\[b\_\{0\}\\,w\(2\-w\)\]\(dashed\) across crowd\-surprise quartiles at fixedw=0\.5w=0\.5\.

### E\.4Hubble robustness \(M2\)

##### Intuition\.

The M2 dose\-response supports the mechanism only if it is not an artifact of item difficulty, a single model configuration, or a single task\.

##### Setup\.

All quantities are computed from the published Hubble accuracies as configured in[Section˜D\.3](https://arxiv.org/html/2608.02985#A4.SS3)\.

##### Results\.

*Difficulty\-matching check\.*The standard model’s accuracy varies by only0\.0370\.037–0\.0510\.051across duplication bins \(per task: MMLU0\.0460\.046, PIQA0\.0460\.046, HellaSwag0\.0510\.051, WinoGrande0\.0370\.037\)\. This range is much smaller than the moderate/high\-dose effects, bounding difficulty imbalance as an explanation; it is comparable to the low\-doser=4r\{=\}4effect, so low\-dose conclusions should be interpreted more cautiously\.

*Robustness across scale and tokens\.*The dose\-response persists across configurations: 8B\-100B shows stronger low\-dose sensitivity \(less token dilution reduces the contamination signal per document\), 8B\-500B remains strong at moderate/high dose, and 1B\-500B shows large effects only at high duplication \(lower capacity limits usable extraction\)\. This heterogeneity is consistent with the extraction factorw=c⋅u⋅κw=c\\cdot u\\cdot\\kappa: exposure must be both absorbed and usable\.

*Per\-task consistency\.*For 8B\-500B, the high\-dose effect is not driven by a single task: MMLU, HellaSwag, and WinoGrande all show strong increases, while PIQA rises more mildly and saturates earlier\. Task heterogeneity is expected becausewwcan vary by task format\.

### E\.5LiveCodeBench robustness battery \(M4\-C\)

##### Intuition\.

The M4\-C boundary estimate is credible only if the no\-discontinuity control assumption holds empirically, the signal is not an artifact of one control model or one bandwidth, the effect is specific to documented cutoffs, and difficulty imbalance is excluded\. The named continuity violations of Route 1—composition shifts and corpus\-density jumps at the cutoff—are exactly what the placebo and bandwidth checks below target\.

##### Setup\.

All checks use the DiD estimator and bootstrap of[Section˜D\.6](https://arxiv.org/html/2608.02985#A4.SS6), computed on the published per\-problem data\.

##### Results\.

*R1 \(control smoothness\)\.*The no\-discontinuity assumption is directly tested by estimating local\-linear jumps for each control at each target cutoff\. Pooled\-control jumps: Oct’23−0\.010\-0\.010\[−0\.093\-0\.093,\+0\.077\+0\.077\]; Dec’23−0\.012\-0\.012\[−0\.097\-0\.097,\+0\.070\+0\.070\]; Apr’24−0\.063\-0\.063\[−0\.180\-0\.180,\+0\.040\+0\.040\]\. All intervals include zero, supporting the identification condition\.

*R2 \(separate\-control robustness\)\.*Own\-cutoff estimates are computed against each control individually\. For GPT\-4o\-0806:\+0\.094\+0\.094vs\. Gemini,\+0\.086\+0\.086vs\. DeepSeek,\+0\.090\+0\.090pooled\. The signal is not driven by a single control model\.

*R3 \(date specificity\)\.*A monthly cutoff scan finds no month outside±1\\pm 1month of GPT\-4o’s documented cutoff with a larger DiD\. A permutation test over model\-cutoff assignments in the specificity matrix yields empiricalp=0\.025p=0\.025across all targets\.

*R4 \(bandwidth\)\.*Own\-cutoff DiD vs\. pooled controls across half\-widthsh∈\{100,120,160,200,240,300\}h\\in\\\{100,120,160,200,240,300\\\}days: GPT\-4o point estimates are stable at\+0\.084\+0\.084to\+0\.100\+0\.100; significance appears forh≥160h\\geq 160days\.

*R5 \(difficulty stratification\)\.*DiD stratified by problem difficulty \(easy/medium/hard\): GPT\-4o\-0513\+0\.096\+0\.096\[\+0\.018\+0\.018,\+0\.169\+0\.169\]; GPT\-4o\-0806\+0\.089\+0\.089\[\+0\.015\+0\.015,\+0\.166\+0\.166\]; Claude\-3\.5\+0\.079\+0\.079\[−0\.002\-0\.002,\+0\.156\+0\.156\]\. GPT\-4o survives cleanly; Claude becomes marginal after stratification, so per\-date inference for Claude leans on the date\-permutation test \(p=0\.017p=0\.017\)\.

*Specificity\.*For each continuity target, the DiD is computed at each candidate cutoff date\. Positive effects concentrate at the model’s own documented cutoff \(main text,[Section˜7](https://arxiv.org/html/2608.02985#S7)\); off\-diagonal cells are null, and monthly scans find no larger effect away from the documented cutoffs\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/e3_robustness.png)Figure 12:M4\-C: the LiveCodeBench boundary signal survives the robustness battery\.\(a\) Composition controls show no local discontinuity at target cutoffs\. \(b\) Own\-cutoff estimates are stable across individual controls\. \(c\) Monthly scans concentrate positive effects at documented cutoffs\. \(d\) Difficulty stratification preserves GPT\-4o’s signal and weakens Claude’s\. Result: GPT\-4o’s boundary leakage is robust on every check; Claude’s is reported as supporting evidence only\.

### E\.6Why M4\-C does not identify globalBB

##### Intuition\.

Converting the boundary jumpJ^\\widehat\{J\}into a global leakage\-adjusted score requires either a strictly clean, recency\-matched control \([Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\) or the extrapolation assumption \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)\)\. Both routes fail on this dataset; documenting the failure disciplines the boundary\-only scope of M4\-C’s claim\.

##### Setup\.

Same data and estimator as[Section˜D\.6](https://arxiv.org/html/2608.02985#A4.SS6); the strictly clean control is GPT\-4\-0613 \(cutoff before all problems\); extrapolations fit time polynomials plus difficulty/platform indicators on the clean side\.

##### Results\.

GPT\-4\-0613 is strictly clean on all LiveCodeBench questions and produces large DiD estimates \(\+0\.145\+0\.145to\+0\.168\+0\.168\), but its quarterly performance profile correlates only0\.360\.36–0\.390\.39with GPT\-4o \(centered RMSE0\.0740\.074–0\.0760\.076\), violating the matched\-recency requirement and exhibiting the weak\-control trap\. Clean\-side extrapolation is unstable: implied corrections range from−0\.175\-0\.175to\+0\.102\+0\.102for GPT\-4o\-0513 and from−0\.105\-0\.105to\+0\.172\+0\.172for GPT\-4o\-0806 across polynomial degrees and post\-cutoff horizons\. We therefore report boundaryJJand do not convert it into a global leakage\-adjusted score\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/e3_global_sensitivity.png)Figure 13:M4\-C identifies the local boundary jump, not a global correction\.\(a\) The strictly clean GPT\-4\-0613 control yields a large contrast but fails temporal\-profile matching \([Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\)\. \(b\) Clean\-side extrapolated corrections change sign across horizons and polynomial degrees \([Assumption˜6](https://arxiv.org/html/2608.02985#Thmassumption6)\)\. Result: both routes to a globalBBfail on this dataset, so the M4\-C claim is scoped to the boundary\.

### E\.7Forecasting robustness and power \(M5\)

##### Intuition\.

A null is only informative if the control is demonstrably well matched, the result is stable across boundary placements, and the design has power to detect effects of practical size\.

##### Setup\.

All quantities use the paired cluster\-bootstrapped DiD of[Section˜D\.7](https://arxiv.org/html/2608.02985#A4.SS7)\.

##### Results\.

*Control validation\.*The pre\-registered profile\-matching protocol selects GPT\-5 over Gemini\-3\.1\-Pro \(post\-boundary profile distance0\.00510\.0051against0\.00880\.0088\) before any DiD is read\. Substituting the deliberately mismatched GPT\-3\.5\-Turbo control flips the primary adjusted estimate from\+0\.020\+0\.020to\+0\.055\+0\.055, the spurious positive that an ill\-matched control manufactures under[Assumption˜4](https://arxiv.org/html/2608.02985#Thmassumption4)\.

*Boundary sensitivity\.*Adjusted estimates for boundaries swept from−180\-180to\+180\+180days around July 2025 in 30\-day steps range from−0\.035\-0\.035to\+0\.057\+0\.057and are never significantly positive; placements more than 90 days before the reference boundary retain fewer than 62 pre\-boundary questions and are reported for completeness only\. The null does not depend on exact placement of Qwen3\.5’s undocumented knowledge horizon\. The sweep is informative only about leakage that*differs*across the assumed boundary: because every tested placement lies inside the plausible training window, inflation roughly uniform over the window would produce a null at all of them\.

*Power analysis\.*Semi\-synthetic effect injection on the observed paired panel detects an injected effect of0\.050\.05Brier\-reduction units in97\.3%97\.3\\%of replications and an M4\-C\-sized effect of0\.090\.09in100%100\\%\. The design has high power for practically meaningful effects; effects well below0\.050\.05remain difficult to exclude\.

### E\.8Matched real\-data PRC sensitivity

##### Intuition\.

E1\-T3 shows PRC detects evidence leakage in principle; the question is whether Route 3 yields a real calibrated positive on frontier data, under a pre\-specified robustness gate\.

##### Setup\.

A separate matched experiment was pre\-specified before querying: Qwen3\.5 withK=8K=8paraphrases on outcome\-balanced treatment/control cohorts in two domains with temporal support on both sides, finance \(150/150150/150\) and other \(130/130130/130\); 552 of 560 questions had at least six valid paraphrase probabilities\.

##### Estimator and uncertainty\.

Calibration was learned only from clean controls: five\-fold cross\-fitting gave out\-of\-sample control predictions, and the five control\-trained calibrators were ensembled on treatment questions\. Calibration was applied to each paraphrase before computing consensus and within\-question variance\. The estimator wasΔ^bc=Cov^​\(P¯,Y−P¯\)\+1n​∑iW^i/Ki\\widehat\{\\Delta\}\_\{\\mathrm\{bc\}\}=\\widehat\{\\mathrm\{Cov\}\}\(\\bar\{P\},Y\-\\bar\{P\}\)\+\\tfrac\{1\}\{n\}\\sum\_\{i\}\\widehat\{W\}\_\{i\}/K\_\{i\}\. Nested question bootstraps refit the calibrator and recomputed the finite\-KKcorrection in every draw; Platt scaling was primary, isotonic the pre\-specified sensitivity\.

##### Results\.

Under Platt scaling, finance yielded excess PRC\+0\.0116\+0\.0116\[\+0\.0009,\+0\.0169\+0\.0009,\+0\.0169\], other yielded\+0\.0084\+0\.0084\[−0\.0046,\+0\.0164\-0\.0046,\+0\.0164\], and the equal\-weight pooled excess was\+0\.0100\+0\.0100\[\+0\.0018,\+0\.0148\+0\.0018,\+0\.0148\]\. Clean\-control point estimates were near zero\. Under isotonic calibration, however, finance was\+0\.0142\+0\.0142\[−0\.0097,\+0\.0188\-0\.0097,\+0\.0188\], other reversed to−0\.0026\-0\.0026\[−0\.0264,\+0\.0124\-0\.0264,\+0\.0124\], and the pooled excess was\+0\.0058\+0\.0058\[−0\.0120,\+0\.0123\-0\.0120,\+0\.0123\]\. The experiment therefore failed its pre\-specified calibration\-stability gate and is reported as suggestive sensitivity evidence only; Route 3 remains a calibration\-gated secondary diagnostic\.

![Refer to caption](https://arxiv.org/html/2608.02985v1/figures/prc_matched.png)Figure 14:Matched real\-data PRC: the calibration\-stability gate fails, so PRC remains secondary\.Treatment–control PRC excess by domain and pooled, under Platt and isotonic calibration\. Result: Platt yields positive excesses, but isotonic does not replicate the pooled significance and flips the other\-domain point estimate; by the pre\-specified gate, PRC is reported as suggestive sensitivity evidence only\.

## Appendix FExperiment provenance

Table 9:Reproducibility map\.Authoritative scripts and outputs for reported experiments\.