Zero Gap Is Not Restoration: Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination

# Stratified Per-Question Probability Evaluation and Step-wise Mitigation of Benchmark Contamination
Source: [https://arxiv.org/html/2608.07341](https://arxiv.org/html/2608.07341)
Ruijie Hou111Both authors contributed equally to this work\.,Yueyang Jiao111Both authors contributed equally to this work\.,Zhao Wang, andYingming Li222Yingming Li is the corresponding author\. Zhejiang University ruijie\.hou@zju\.edu\.cn,yingming@zju\.edu\.cn

###### Abstract

Test data from public benchmarks inevitably leaks into pretraining corpora, inflating evaluation scores once memorized\.Contamination mitigation evaluationintervenes in the decoding process to suppress memorization and restore a contaminated model’s genuine capability, but its prevailing metric, theG\-AP\(Gap ofAggregatePerformance\), is flawed\. Discrete correct/incorrect readouts cannot characterize per\-question performance, averaging before differencing lets over\- and under\-suppression cancel out, and uniform per\-question weighting invites strategies to push solve probabilities onto the clean model’s high\-frequency values\. We proposeSA\-PPG\(StratifiedAggregate ofPer\-questionProbabilityGaps\): estimate each question’s solve probability by sampling, difference it against the clean model per question, and aggregate within groups defined by the clean model’s solve probability\. Existing mitigation strategies first estimate where contamination lies and then operate on the estimate, so they are only as correct as the estimate\.RailCapinstead judges contamination during generation: whenever a sample falls back onto the greedy trajectory, the next trajectory token is capped to the runner\-up, accumulating suppression until the response distribution becomes sufficiently dispersed\. Across multiple contaminated models and benchmarks, SA\-PPG reveals that prior strategies’ restoration is substantially overestimated, while RailCap attains the lowest SA\-PPG\.

Zero Gap Is Not Restoration: Stratified Per\-Question Probability Evaluation and Step\-wise Mitigation of Benchmark Contamination

Ruijie Hou111Both authors contributed equally to this work\., Yueyang Jiao111Both authors contributed equally to this work\., Zhao Wang, and Yingming Li222Yingming Li is the corresponding author\.Zhejiang Universityruijie\.hou@zju\.edu\.cn,yingming@zju\.edu\.cn

## 1Introduction

Test data from public benchmarks almost inevitably finds its way into large\-scale pretraining corpora\(Brown,[2020](https://arxiv.org/html/2608.07341#bib.bib20); Elazaret al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib16)\): memorized test items artificially inflate evaluation scores, stripping benchmarks of their power to measure genuine capability\(Magar and Schwartz,[2022](https://arxiv.org/html/2608.07341#bib.bib34); Zhouet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib23); Balloccuet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib4)\)\. Existing work falls into two classes\.Dataset\-sideapproaches rebuild, rewrite, or dynamically generate new benchmarks to sidestep contamination\(Liet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib29); Zhanget al\.,[2024a](https://arxiv.org/html/2608.07341#bib.bib46); Zhuet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib7); Baiet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib13)\), but they are costly, and the new data faces renewed leakage once released\.Contamination mitigation evaluationinstead builds no new dataset: on datasets at risk of leakage, it intervenes in the decoding process to suppress memorization and restore the model’s genuine capability\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11); Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48); Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\)\. Yet whether the restoration a mitigation strategy claims actually holds must first be checked against a reliable evaluation metric: the metric not only determines how mitigation strategies are judged, but also shapes how they are designed\.

![Refer to caption](https://arxiv.org/html/2608.07341v1/x1.png)\(a\)leaked: overlap with the greedy trajectory
![Refer to caption](https://arxiv.org/html/2608.07341v1/x2.png)\(b\)leaked: rank of the clean model’s token
![Refer to caption](https://arxiv.org/html/2608.07341v1/x3.png)\(c\)unleaked: overlap with the greedy trajectory
![Refer to caption](https://arxiv.org/html/2608.07341v1/x4.png)\(d\)unleaked: rank of the clean model’s token

Figure 1:The contaminated model’s generation behaviour on Llama\-2 \(GSM8K, 1,319 questions,m=50m=50samples per question,T=0\.7T=0\.7\)\. \(a, c\) ROUGE\-L between each sampled response and that question’s own greedy trajectory\. Each faint dot is one sample and each solid dot a per\-question mean\. \(b, d\) With the contaminated model teacher\-forced on the clean model’s greedy prefix, the rank of the clean model’s next token under the contaminated model over the first ten decoding steps, restricted to the steps at which the two models disagree \(rank 1 cannot occur there\)\.The metric used to compare mitigation strategies has been theGap ofAggregatePerformance \(G\-AP\)\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11); Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48); Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\): assign each question a performance readout \(mostly a discrete correct\-or\-incorrect 1/0 mark\), average the readouts over the whole dataset for the contaminated model under mitigation, and take the absolute difference from the clean model’s average\. The smaller the gap, the better the restoration\. This metric suffers from two problems\. The first lies in representation: a discrete mark poorly captures a model’s performance on a single question, since responses sampled for the same question may disagree on correctness\. What stabilizes as the number of samples grows is the*solve probability*, the probability that a sampled response is correct, and per\-question performance should be represented by it\. The second lies in aggregation: averaging before differencing lets over\- and under\-suppression cancel out\. Performance wrongly suppressed on one question can offset performance wrongly inflated on another, so a zero gap does not mean that every question has been restored\.

Correcting the two problems separately yields theAggregate ofPer\-questionProbabilityGaps \(A\-PPG\): sample from the contaminated model under mitigation, estimate each question’s solve probability, difference it per question against the clean model to obtain per\-question probability gaps \(PPGs\), and average their absolute values\. This metric reads zero if and only if every question is perfectly restored\. The plain average, however, weights every question equally, which introduces a new problem\. When the clean model is itself not strong, most questions have a solve probability of zero\. On GSM8K, for example, the clean model of Llama\-2 never solves nearly a quarter of the questions\. A trivial strategy that simply drives the contaminated model to fail on every question thus scores zero gap on this majority, while even large gaps on the minority of questions with higher solve probabilities are diluted by the average\. In other words, equal weighting invites a strategy to push solve probabilities onto the values that occur most frequently under the clean model, rather than to restore each question\. To prevent such trivial strategies from scoring well and misleading the evaluation, we group questions by the clean model’s solve probability before aggregating: questions with similar probabilities form a group, PPGs are averaged within each group first and then across groups, and the shortcut of chasing high\-frequency values is closed\. We call the resulting metric theStratifiedAggregate ofPer\-questionProbabilityGaps \(SA\-PPG\)\. Across multiple contaminated models, SA\-PPG shows that the restoration ability of prior strategies has been substantially overestimated by G\-AP\.

SA\-PPG grounds the criterion of restoration in every single question: a strategy must get right how much each question’s solve probability is to be adjusted\. Existing strategies\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11); Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48); Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\)all proceed in two steps: first estimate where the contamination lies \(which questions are leaked, which responses stem from memorization, which neurons encode it\), and then operate on the estimated part\. The correctness of the intervention therefore hinges entirely on the quality of the estimate: what the estimate misses keeps its inflated performance untouched, and what it wrongly flags suffers unnecessary damage\. We instead examine the contaminated model’s own generation behavior \(Figure[1](https://arxiv.org/html/2608.07341#S1.F1)\)\. First, on leaked questions the contaminated model’s sampled responses collapse onto its own greedy trajectory, whereas on unleaked questions they disperse over many paths\. Second, at the decoding steps where the clean and contaminated models next diverge, the token the clean model selects is, in about half of the cases, the contaminated model’s runner\-up\. Building on these two observations, we proposeRailCap: at every decoding step, check whether the sample has fallen back onto the greedy trajectory, and once it has, cap the probability of the next trajectory token to that of the runner\-up, so that the clean model’s next choice most likely sits at the head of the flattened candidates\. The judgment of contamination thus turns from a pre\-hoc estimate intostep\-wise supervision during generation: each fall\-back onto the greedy trajectory triggers one suppression, suppressions accumulate step by step, and the response distribution eventually becomes sufficiently dispersed\. How much intervention each question receives is decided online by the responses at every decoding step, rather than fixed in advance by a one\-shot estimate as in prior strategies\. Across multiple models and benchmarks, RailCap attains the lowest SA\-PPG\.

In summary, our contributions are as follows:

1. 1\.We propose a restoration metric,SA\-PPG, that fixes two problems of the prevailing G\-AP: discrete 0/1 marks cannot represent per\-question performance, and averaging before differencing lets over\- and under\-suppression cancel out\. We further show that the per\-question equal\-weight correction still invites strategies to push solve probabilities onto the clean model’s frequent values, motivating aggregation grouped by the clean model’s solve probability\.
2. 2\.We propose a contamination mitigation strategy,RailCap: whenever a sample falls back onto the greedy trajectory, the next trajectory token’s probability is capped to that of the runner\-up\. How much intervention each question receives is decided online by step\-wise supervision during generation rather than pre\-allocated by a one\-shot estimate\.
3. 3\.Across models and benchmarks,SA\-PPGreveals a systematic overestimation of prior strategies’ restoration by G\-AP, andRailCapachieves state\-of\-the\-art restoration\.

## 2Related Work

Test data from public benchmarks almost inevitably ends up in large\-scale pretraining corpora\(Brown,[2020](https://arxiv.org/html/2608.07341#bib.bib20); Dodgeet al\.,[2021](https://arxiv.org/html/2608.07341#bib.bib25); Elazaret al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib16)\)\. Memorized test items artificially inflate evaluation scores and mask genuine capability\(Magar and Schwartz,[2022](https://arxiv.org/html/2608.07341#bib.bib34); Zhouet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib23); Balloccuet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib4)\), prompting calls from the community to protect test data\(Jacoviet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib24); Denget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib9)\)\. Research around contamination falls into several classes\.Detectionasks whether a model has seen the evaluation data: min\-k% over low\-probability tokens\(Shiet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib35)\), perplexity\(Li,[2023](https://arxiv.org/html/2608.07341#bib.bib38)\), divergence calibration\(Zhanget al\.,[2024b](https://arxiv.org/html/2608.07341#bib.bib10)\), temporal cues\(Golchin and Surdeanu,[2024](https://arxiv.org/html/2608.07341#bib.bib18)\), black\-box calibration\(Yeet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib44)\), and CDD, which takes sampling\-greedy consistency as its signal\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11)\)\.Dataset\-sidework sidesteps leaked benchmarks: rebuilding same\-distribution questions for GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07341#bib.bib8)\)\(GSM\-Plus\(Liet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib29)\), GSM1k\(Zhanget al\.,[2024a](https://arxiv.org/html/2608.07341#bib.bib46)\)\), rewriting existing questions\(Zhuet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib7)\), or having models dynamically generate test items\(Baiet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib13); Yuet al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib14); Liet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib15)\)\. But rebuilding and dynamic generation are costly, newly released data faces the same risk of renewed leakage\.

Contamination mitigation evaluationconstitutes another line of work: it builds no new dataset, but intervenes in the decoding process to suppress memorization and restore genuine capability on datasets at risk of leakage\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11); Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48); Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\)\. Whether the restoration a mitigation strategy claims actually holds, however, requires a reliable evaluation metric to check\. This is one of the central subjects of this paper\.

The metrics used in prior mitigation work differ from one another, yet all aregaps ofaggregateperformance \(G\-AP\): give each question a performance readout, average over the full dataset, and difference against the clean model’s average, and the smaller the gap, the better the restoration\. The metrics differ only in how the per\-question readout is obtained: LNE\-blocking marks the greedy response correct or incorrect \(0/1\)\(Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48)\), and shortcut neuron patching marks a single sampled response 0/1\(Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\)\. TED is the exception, as its mitigation strategy operates on a set of sampled responses in the first place, and after filtering near\-greedy samples it estimates performance with pass@1\(Chenet al\.,[2021](https://arxiv.org/html/2608.07341#bib.bib6)\)over the remainder\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11)\)\. That choice, however, is an artifact of its sampling\-based mitigation\. No work has compared the discrete 0/1 and probabilistic representations\. The three readouts differ from one another, and cross\-method comparison has never been conducted under a single metric\. We compare the two representations and adopt the probabilistic one, as a single\-sample 0/1 readout does not even reproduce between two evaluations of the same model\. Our metric SA\-PPG further repairs the aggregation itself\.

On the mitigation side, existing strategies all proceed in two steps: first estimate where the contamination lies, then operate on the estimated part\. The granularity of the estimate varies\. TED estimates contaminated responses: after sampling, it filters out suspected memorized samples by their edit distance to the greedy decode\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11)\)\. LNE\-blocking estimates contaminated questions: it gauges each question’s degree of contamination with length\-normalized entropy and sets the blocking strength accordingly\(Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48)\)\. Shortcut neuron patching estimates contaminated neurons: it locates shortcut neurons via contrastive and causal analysis and suppresses them\(Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\)\. The three granularities share one structure: the correctness of the intervention hinges entirely on the quality of the estimate, as what the estimate misses keeps its inflated performance untouched, and what it wrongly flags suffers unnecessary damage\. RailCap instead turns the judgment of contamination from a pre\-hoc estimate into step\-wise supervision during generation, removing the dependence on any estimate, and it attains the best restoration\.

## 3SA\-PPG: A Stratified Per\-Question Restoration Metric

### 3\.1Setup and the Prevailing Metric

LetD=\{q1,…,qN\}D=\\\{q\_\{1\},\\dots,q\_\{N\}\\\}be the evaluation dataset,MclM\_\{\\mathrm\{cl\}\}the clean model, andMcoM\_\{\\mathrm\{co\}\}the contaminated model\. A mitigation strategyssintervenes in the decoding process ofMcoM\_\{\\mathrm\{co\}\}\. We writeMcosM\_\{\\mathrm\{co\}\}^\{s\}for the contaminated model under the strategy\. Evaluation first fixes a performance readoutrM​\(q\)∈\[0,1\]r\_\{M\}\(q\)\\in\[0,1\], the value taken as modelMM’s performance on a single questionqq\. Existing work mostly adopts a discrete readout: a responseoois sampled fromMMand marked 1 if correct and 0 otherwise,

rM0/1\(q\)=𝟙\[osolvesq\],o∼M\(⋅∣q\)r^\{0/1\}\_\{M\}\(q\)=\\mathbb\{1\}\\bigl\[\\,o\\text\{ solves \}q\\,\\bigr\],\\quad o\\sim M\(\\cdot\\mid q\)\(1\)TheGap ofAggregatePerformance \(G\-AP\) averages the readouts over the whole dataset for each model and takes the absolute difference:

G\-AP​\(s\)=\|1N​∑q∈DrMcos​\(q\)−1N​∑q∈DrMcl​\(q\)\|\\text\{G\-AP\}\(s\)=\\Bigl\|\\tfrac\{1\}\{N\}\\\!\\sum\_\{q\\in D\}\\\!r\_\{M\_\{\\mathrm\{co\}\}^\{s\}\}\(q\)\-\\tfrac\{1\}\{N\}\\\!\\sum\_\{q\\in D\}\\\!r\_\{M\_\{\\mathrm\{cl\}\}\}\(q\)\\Bigr\|\(2\)prior work reads a smaller gap as a better restoration\. We identified two problems with this metric: one in the choice of readout, the other in the order of aggregation\. Rather than correcting G\-AP item by item, we first define a per\-question metric, A\-PPG \(§[3\.2\.1](https://arxiv.org/html/2608.07341#S3.SS2.SSS1)\), then set G\-AP against it and show that a zero G\-AP does not certify restoration \(§[3\.2\.2](https://arxiv.org/html/2608.07341#S3.SS2.SSS2)\)\. The remaining problem of equal\-weight aggregation is resolved by stratification \(§[3\.3](https://arxiv.org/html/2608.07341#S3.SS3)\)\.

### 3\.2Aggregate of Per\-question Probability Gaps \(A\-PPG\)

#### 3\.2\.1Definition

Responses sampled for the same question may disagree on correctness:r0/1r^\{0/1\}is itself random, and a single draw poorly captures per\-question performance\. What stabilizes as the number of samples grows is the probability of solving the question\. We therefore adopt the*solve probability*as the readout:

rMprob​\(q\)=P​\[o​solves​q\]=𝔼​\[rM0/1​\(q\)\]r^\{\\mathrm\{prob\}\}\_\{M\}\(q\)=P\\bigl\[\\,o\\text\{ solves \}q\\,\\bigr\]=\\mathbb\{E\}\\bigl\[r^\{0/1\}\_\{M\}\(q\)\\bigr\]\(3\)that is,r0/1r^\{0/1\}is a single Bernoulli draw andrprobr^\{\\mathrm\{prob\}\}is its expectation\. In practice, we samplemmresponses independently fromMMand estimate the readout by the fraction of correct ones,r^Mprob​\(q\)=c/m\\hat\{r\}^\{\\mathrm\{prob\}\}\_\{M\}\(q\)=c/m\.

Under this readout, the gap between the two models is characterized question by question\. Define thePer\-questionProbabilityGap \(PPG\) on questionqqas

Δs​\(q\)=rMcosprob​\(q\)−rMclprob​\(q\)\\Delta\_\{s\}\(q\)=r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{co\}\}^\{s\}\}\(q\)\-r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{cl\}\}\}\(q\)\(4\)taking absolute values first and then averaging over the dataset yields theAggregate ofPer\-questionProbabilityGaps \(A\-PPG\):

A\-PPG​\(s\)=1N​∑q∈D\|Δs​\(q\)\|\\text\{A\-PPG\}\(s\)=\\frac\{1\}\{N\}\\sum\_\{q\\in D\}\\bigl\|\\Delta\_\{s\}\(q\)\\bigr\|\(5\)Since every term is non\-negative,

A\-PPG​\(s\)=0⇔Δs​\(q\)=0,∀q∈D\\text\{A\-PPG\}\(s\)=0\\iff\\Delta\_\{s\}\(q\)=0,\\ \\forall q\\in D\(6\)that is,A\-PPG reads zero if and only if every question’s solve probability matches the clean model’s: every question is fully restored\.

#### 3\.2\.2Relation to G\-AP

Substitutingrprobr^\{\\mathrm\{prob\}\}into the readout slot of G\-AP yields the mirror of A\-PPG, theGap ofAggregatePer\-questionProbabilities \(G\-APP\):

G\-APP​\(s\)=\|1N​∑q∈DrMcosprob​\(q\)−1N​∑q∈DrMclprob​\(q\)\|=\|1N​∑q∈DΔs​\(q\)\|\\begin\{split\}\\text\{G\-APP\}\(s\)&=\\Bigl\|\\tfrac\{1\}\{N\}\\\!\\sum\_\{q\\in D\}\\\!r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{co\}\}^\{s\}\}\(q\)\-\\tfrac\{1\}\{N\}\\\!\\sum\_\{q\\in D\}\\\!r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{cl\}\}\}\(q\)\\Bigr\|\\\\ &=\\Bigl\|\\frac\{1\}\{N\}\\sum\_\{q\\in D\}\\Delta\_\{s\}\(q\)\\Bigr\|\\end\{split\}\(7\)The two metrics are built from the same per\-question quantityΔs\\Delta\_\{s\}and differ only in the order of the absolute value and the average\. To make the difference explicit, decompose A\-PPG by the sign ofΔs\\Delta\_\{s\}into two components:

A\-PPG​\(s\)=Δs\+\+Δs−\\displaystyle\\text\{A\-PPG\}\(s\)=\\Delta^\{\+\}\_\{s\}\+\\Delta^\{\-\}\_\{s\}\(8\)Δs\+=1N​∑q∈Dmax⁡\(Δs​\(q\),0\)\\displaystyle\\Delta^\{\+\}\_\{s\}=\\frac\{1\}\{N\}\\sum\_\{q\\in D\}\\max\\bigl\(\\Delta\_\{s\}\(q\),\\,0\\bigr\)\(9\)Δs−=1N​∑q∈Dmax⁡\(−Δs​\(q\),0\)\\displaystyle\\Delta^\{\-\}\_\{s\}=\\frac\{1\}\{N\}\\sum\_\{q\\in D\}\\max\\bigl\(\-\\Delta\_\{s\}\(q\),\\,0\\bigr\)\(10\)whereΔs\+\\Delta^\{\+\}\_\{s\}is the contribution of under\-suppression \(inflated performance that the strategy leaves in place\) andΔs−\\Delta^\{\-\}\_\{s\}that of over\-suppression \(performance driven below the clean model\)\. In the same notation,

G\-APP​\(s\)=\|Δs\+−Δs−\|\\text\{G\-APP\}\(s\)=\\bigl\|\\Delta^\{\+\}\_\{s\}\-\\Delta^\{\-\}\_\{s\}\\bigr\|\(11\)andG\-APP​\(s\)≤A\-PPG​\(s\)\\text\{G\-APP\}\(s\)\\leq\\text\{A\-PPG\}\(s\)follows immediately: A\-PPG adds the two components, whereas G\-APP nets them against each other\.

The decisive difference lies in the zero set\. A\-PPG is zero if and only ifΔs\+=Δs−=0\\Delta^\{\+\}\_\{s\}=\\Delta^\{\-\}\_\{s\}=0, that is, every question is restored\. G\-APP is zero as soon asΔs\+=Δs−\\Delta^\{\+\}\_\{s\}=\\Delta^\{\-\}\_\{s\}, which cancellation alone suffices to achieve\. If, for example, half of the questions are over\-suppressed \(Δs​\(q\)=−δ\\Delta\_\{s\}\(q\)=\-\\delta\) and the other half under\-suppressed \(Δs​\(q\)=\+δ\\Delta\_\{s\}\(q\)=\+\\delta\) for someδ\>0\\delta\>0, thenΔs\+=Δs−=δ/2\\Delta^\{\+\}\_\{s\}=\\Delta^\{\-\}\_\{s\}=\\delta/2, soG\-APP=0\\text\{G\-APP\}=0whileA\-PPG=δ\\text\{A\-PPG\}=\\delta: the gap vanishes, yet not a single question is restored\. A zero G\-APP therefore does not certify restoration\.

G\-AP differs from G\-APP only in the readout and shares its aggregation order: averaging before differencing lets over\- and under\-suppression cancel out\. A zero G\-AP therefore certifies no more than a zero G\-APP does, with an additional layer of sampling noise on top:G\-AP is not a reliable restoration metric, and per\-question restoration should be judged by A\-PPG\.

### 3\.3Stratified Aggregate of Per\-question Probability Gaps \(SA\-PPG\)

A\-PPG assigns every question the same weight, which introduces a new problem\. When the clean model is itself not strong, solve probabilities are distributed highly unevenly: most questions concentrate near zero, and questions with higher solve probabilities form a minority\. Under equal weights, a trivial strategy that simply drivesrMcosprob≡0r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{co\}\}^\{s\}\}\\equiv 0attains a zero gap, question by question, on the zero\-probability majority\. Even large gaps on the high\-probability minority are diluted by the majority\. In other words, equal\-weight aggregation invites a strategy to push solve probabilities onto the clean model’s frequent values rather than to restore each question\.

To deny such trivial strategies a low gap, we group questions by the clean model’s solve probability before aggregating\. Partition\[0,1\]\[0,1\]intoBBequal\-width intervals and assign each question to the interval thatrMclprob​\(q\)r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{cl\}\}\}\(q\)falls in:

Db=\{q∈D:rMclprob​\(q\)∈\[b−1B,bB\)\}D\_\{b\}=\\Bigl\\\{q\\in D:r^\{\\mathrm\{prob\}\}\_\{M\_\{\\mathrm\{cl\}\}\}\(q\)\\in\\Bigl\[\\tfrac\{b\-1\}\{B\},\\tfrac\{b\}\{B\}\\Bigr\)\\Bigr\\\}\(12\)forb=1,…,Bb=1,\\dots,B, where the last interval \(b=Bb=B\) is closed on the right \(including 1\)\. Letℬ=\{b:Db≠∅\}\\mathcal\{B\}=\\\{\\,b:D\_\{b\}\\neq\\varnothing\\,\\\}index the non\-empty groups\. Averaging per\-question gaps within each group first and then across groups yields theStratifiedAggregate ofPer\-questionProbabilityGaps \(SA\-PPG\):

SA\-PPG​\(s\)=1\|ℬ\|​∑b∈ℬ1\|Db\|​∑q∈Db\|Δs​\(q\)\|\\text\{SA\-PPG\}\(s\)=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{b\\in\\mathcal\{B\}\}\\frac\{1\}\{\|D\_\{b\}\|\}\\sum\_\{q\\in D\_\{b\}\}\\bigl\|\\Delta\_\{s\}\(q\)\\bigr\|\(13\)After grouping, the zero\-probability majority falls into a single group and carries a weight of1/\|ℬ\|1/\|\\mathcal\{B\}\|regardless of its size, and the gap within any group is no longer diluted by the question counts of the others\.To attain a low SA\-PPG, a strategy must complete the restoration at every level of the clean model’s solve probability, and the shortcut of chasing high\-frequency values is closed\. SA\-PPG also inherits the property of A\-PPG: it reads zero if and only if every question is perfectly restored\.

## 4RailCap: Step\-wise Supervision during Generation

SA\-PPG grounds the criterion of restoration in every single question: a mitigation strategy must get right how much each question’s solve probability is to be adjusted\. Existing strategies first estimate where the contamination lies and then operate on the estimated part\. The correctness of the intervention hinges entirely on the quality of the estimate\. We observe the contaminated model’s own generation behavior \(Figure[1](https://arxiv.org/html/2608.07341#S1.F1)\)\. First, on leaked questions the sampled responses ofMcoM\_\{\\mathrm\{co\}\}collapse onto its own greedy trajectory, whereas on unleaked questions they disperse over many paths: whether sampling falls back onto the greedy trajectory is itself an online signal of memorization\. Second, at the decoding steps where the clean and contaminated models next diverge, the token the clean model selects is, in about half of the cases, the contaminated model’s runner\-up: capping the trajectory token to the runner\-up leaves the clean model’s choice most likely at the head of the flattened candidates, where the contaminated model’s sampling readily picks it up\. Building on these two observations, we propose RailCap\.

The RailCap mitigation strategy intervenes only in the decoding process ofMcoM\_\{\\mathrm\{co\}\}\.

##### Preprocessing\.

For each questionqq, one additional greedy decode ofMcoM\_\{\\mathrm\{co\}\}yields its greedy trajectoryg=\(g1,…,gT\)g=\(g\_\{1\},\\dots,g\_\{T\}\), and all n\-gram windows of the trajectory are built into an index

H=\{\(gk,…,gk\+n−1\)↦gk\+n:1≤k≤T−n\}\\begin\{split\}H=\\bigl\\\{&\(g\_\{k\},\\dots,g\_\{k\+n\-1\}\)\\mapsto g\_\{k\+n\}:\\\\ &\\ 1\\leq k\\leq T\-n\\bigr\\\}\\end\{split\}\(14\)which maps a trailing n\-gram to its successor token on the trajectory, wherennis the n\-gram threshold for judging a fall\-back\. This constitutes the entirety of RailCap’s preprocessing\.

##### Per\-step operation\.

Sampling proceeds token by token\. At steptt, leto1:t−1o\_\{1:t\-1\}be the tokens generated so far andℓt\\ell\_\{t\}the logits at this step\. If the lastnntokens coincide with some window of the trajectory, that is,\(ot−n,…,ot−1\)∈H\(o\_\{t\-n\},\\dots,o\_\{t\-1\}\)\\in H, an intervention is applied to prevent the sample from falling back onto the greedy trajectory: its successor token on the greedy trajectory,x=H​\[\(ot−n,…,ot−1\)\]x=H\[\(o\_\{t\-n\},\\dots,o\_\{t\-1\}\)\], is capped to the level of the current second\-largest logit,

ℓt​\[x\]←min⁡\(ℓt​\[x\],vt\(2\)\)\\ell\_\{t\}\[x\]\\leftarrow\\min\\bigl\(\\ell\_\{t\}\[x\],\\,v\_\{t\}^\{\(2\)\}\\bigr\)\(15\)wherevt\(2\)v\_\{t\}^\{\(2\)\}is the second\-largest entry ofℓt\\ell\_\{t\}\. The trajectory token is thus leveled with the runner\-up\. Otherwiseℓt\\ell\_\{t\}is left unchanged\. The next token is then sampled fromℓt\\ell\_\{t\}at temperatureτ\\tau,ot∼Softmax​\(ℓt/τ\)o\_\{t\}\\sim\\mathrm\{Softmax\}\(\\ell\_\{t\}/\\tau\), and decoding proceeds to stept\+1t\+1\.

The cap makes the current step select, with high probability, a token off the trajectory, and the sample temporarily departs from the greedy trajectory\. This operation recurs as generation proceeds, and suppression accumulates step by step until the response distribution becomes sufficiently dispersed\. The same rule acts on every decoding step of every prompt, and how much intervention each question receives is decided online by the responses at every step: the judgment of contamination turns from a one\-shot pre\-hoc estimate into step\-wise supervision during generation\.

Algorithm 1RailCap decoding, one questionqq1:

McoM\_\{\\mathrm\{co\}\},

qq, n\-gram threshold

nn, samples

mm, temperature

τ\\tau
2:responses

\{o\(1\),…,o\(m\)\}\\\{o^\{\(1\)\},\\dots,o^\{\(m\)\}\\\}
3:

g=\(g1,…,gT\)←GreedyDecode​\(Mco,q\)g=\(g\_\{1\},\\dots,g\_\{T\}\)\\leftarrow\\mathrm\{GreedyDecode\}\(M\_\{\\mathrm\{co\}\},q\)
4:

H←\{\(gk,…,gk\+n−1\)↦gk\+n\}k=1T−nH\\leftarrow\\\{\(g\_\{k\},\\dots,g\_\{k\+n\-1\}\)\\mapsto g\_\{k\+n\}\\\}\_\{k=1\}^\{T\-n\}
5:for

j=1,…,mj=1,\\dots,mdo

6:

o←\(\)o\\leftarrow\(\\,\);

t←1t\\leftarrow 1
7:repeat

8:

ℓ←Logits​\(Mco,\(q,o\)\)\\ell\\leftarrow\\mathrm\{Logits\}\\bigl\(M\_\{\\mathrm\{co\}\},\(q,o\)\\bigr\)
9:if

t\>nt\>nand

\(ot−n,…,ot−1\)∈H\(o\_\{t\-n\},\\dots,o\_\{t\-1\}\)\\in Hthen

10:

x←H​\[\(ot−n,…,ot−1\)\]x\\leftarrow H\[\(o\_\{t\-n\},\\dots,o\_\{t\-1\}\)\]
11:

ℓ​\[x\]←min⁡\(ℓ​\[x\],v\(2\)\)\\ell\[x\]\\leftarrow\\min\\bigl\(\\ell\[x\],v^\{\(2\)\}\\bigr\)
12:endif

13:

ot∼Softmax​\(ℓ/τ\)o\_\{t\}\\sim\\mathrm\{Softmax\}\(\\ell/\\tau\)
14:

o←o∥oto\\leftarrow o\\,\\\|\\,o\_\{t\};

t←t\+1t\\leftarrow t\+1
15:until

ot=EOSo\_\{t\}=\\mathrm\{EOS\}
16:

o\(j\)←oo^\{\(j\)\}\\leftarrow o
17:endfor

18:return

\{o\(1\),…,o\(m\)\}\\\{o^\{\(1\)\},\\dots,o^\{\(m\)\}\\\}

## 5Experiments

### 5\.1Experimental Setup

##### Datasets\.

GSM8K\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07341#bib.bib8)\)is a benchmark of grade\-school math word problems requiring multi\-step arithmetic reasoning\. Its test set contains 1,319 questions\.PQis a paraphrased version that we construct on the GSM8K test set: only the wording of each question is rewritten, with all numbers and the final answer kept identical, which yields a harder form of contamination that verbatim memorization cannot hit directly\. The paraphrases are generated with DeepSeek\-V4\-Flash\(DeepSeek\-AI,[2026](https://arxiv.org/html/2608.07341#bib.bib51)\)\. The prompt structure is given in Appendix[A](https://arxiv.org/html/2608.07341#A1)\.

##### Models and contamination simulation\.

We simulate contamination on the base versions of three open model families: Llama\-2\-7B\(Touvronet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib42)\), Gemma\-4\-E2B\(Gemma Team,[2026](https://arxiv.org/html/2608.07341#bib.bib54)\), and Pythia\-12B\(Bidermanet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib52)\), covering different architectures, scales, and tokenizers\. In particular, Pythia is fully open source: both its weights and its training corpus are public, which allows verifying that the base model itself is not contaminated by the evaluation data\. Prior work shows that models fine\-tuned on the training split of a task attain more stable test\-set performance\(Dominguez\-Olmedoet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib50)\)\. Motivated by this, we fine\-tune each base model on the training data \(1,840 OpenOrca examples\(Lianet al\.,[2023](https://arxiv.org/html/2608.07341#bib.bib53)\)and 660 GSM8K training\-split questions in the 8\-shot CoT format\(Weiet al\.,[2022](https://arxiv.org/html/2608.07341#bib.bib55)\)\) to obtain the clean modelMclM\_\{\\mathrm\{cl\}\}\. The contaminated modelMcoM\_\{\\mathrm\{co\}\}is obtained by training further fromMclM\_\{\\mathrm\{cl\}\}, on the 1,840 OpenOrca examples with 660 test\-split questions mixed in: these 660 are the leaked questions, and the remaining 659, which appear in no training data, are the unleaked questions\. Using the GSM8K and the PQ test set respectively yields a contaminated model for each domain\. All training uses LoRA fine\-tuning \(r=64,α\\alpha=128\) with a learning rate of 2e\-4 \(cosine schedule, warmup 0\.1\), a global batch size of 32, bf16 precision, and 5 epochs, based on the LLaMA\-Factory framework\(Zhenget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib17)\)\.

##### Evaluation protocol and compared strategies\.

All evaluations use the 8\-shot CoT prompt and the original test questions\. For each question we independently samplem=50m=50responses \(T=0\.7\) and estimate the solve probability by the fraction of correct ones \(the only exception is G\-AP, whose 0/1 readout uses a single sampled response per question\)\. SA\-PPG usesB=50B=50equal\-width groups, with the same\-domainMclM\_\{\\mathrm\{cl\}\}as the reference\. The compared mitigation strategies areIdentity\(the contaminated modelMcoM\_\{\\mathrm\{co\}\}without intervention\),TED\(Donget al\.,[2024](https://arxiv.org/html/2608.07341#bib.bib11)\),LNE\-blocking\(Houet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib48)\),Shortcut neuron patching\(Zhuet al\.,[2025](https://arxiv.org/html/2608.07341#bib.bib49)\), and our proposedRailCap\(4\-gram\)\.

### 5\.2Metric Experiments: G\-AP versus SA\-PPG

#### 5\.2\.1Main Result: Rank Reversal

Table 1:Readings of the same responses under G\-AP and SA\-PPG\. Model: Llama\-2, contamination domain: GSM8K\. The reference is the same\-domain clean modelMclM\_\{\\mathrm\{cl\}\}, and Identity is the contaminated modelMcoM\_\{\\mathrm\{co\}\}without intervention\. Bold marks the best strategy in each column\.As shown in Table[1](https://arxiv.org/html/2608.07341#S5.T1), the G\-AP column is computed under the protocol of prior work: one 0/1 observation per question, followed by the aggregate difference\. Under G\-AP, LNE\-blocking appears near\-perfect \(0\.0235, against 0\.3192 for the contaminated model without intervention\) and is the best strategy in the table\. Under SA\-PPG the verdict reverses: LNE\-blocking falls to 0\.2932, behind Shortcut \(0\.2476\) and RailCap, barely better than no intervention \(0\.3261\), whereas RailCap, not the best under G\-AP \(0\.0728\), is the best under SA\-PPG \(0\.1914\)\. Same questions, same contaminated model, same clean model, same mitigation strategies: changing only the metric overturns the verdict entirely\. The near\-perfect restoration that prior work claims does not exist at the per\-question level\. The next two experiments trace where G\-AP buries this error: in the readout \(Figure[2](https://arxiv.org/html/2608.07341#S5.F2)\) and in the aggregation \(Table[2](https://arxiv.org/html/2608.07341#S5.T2)\)\.

#### 5\.2\.2Analysis: Sources of the Reversal

![Refer to caption](https://arxiv.org/html/2608.07341v1/x5.png)Figure 2:Per\-question readouts of two independent sampling batches drawn from the same clean modelMclM\_\{\\mathrm\{cl\}\}\(Llama\-2, GSM8K domain\)\. The horizontal axis is the question index sorted by the first batch’s readout, and the vertical axis is the readout of each batch\. Left: the discrete readoutr0/1r^\{0/1\}, one sample per batch \(m=1m=1, a small vertical jitter is added to avoid overlap\)\. Right: the solve probability estimated fromm=50m=50samples of each batch\.Table 2:The metrics analyzed through the component decomposition of SA\-PPG, setting as in Table[1](https://arxiv.org/html/2608.07341#S5.T1)\.Δ\+\\Delta^\{\+\}is the under\-suppression component \(residual contamination\) andΔ−\\Delta^\{\-\}the over\-suppression component \(collateral damage\), with A\-PPG=Δ\+\+Δ−\\,=\\Delta^\{\+\}\+\\Delta^\{\-\}and G\-APP=\|Δ\+−Δ−\|\\,=\|\\Delta^\{\+\}\-\\Delta^\{\-\}\|\. All\-Zero is a synthetic trivial strategy that makes the contaminated model fail every question \(per\-question solve probability identically 0\)\.As shown in Figure[2](https://arxiv.org/html/2608.07341#S5.F2), we test the reproducibility of the readout itself by drawing two independent sampling batches from the sameMclM\_\{\\mathrm\{cl\}\}\. Under the discrete readout \(r0/1r^\{0/1\}\), the mean per\-question gap between the two batches reaches 0\.190, while under the solve\-probability readout the two batches collapse onto one curve and the mean per\-question gap drops to 0\.041: the probability readout is far more stable per question than the discrete readout\.

As shown in Table[2](https://arxiv.org/html/2608.07341#S5.T2), two obstacles remain after the readout is corrected\. The first iscancellation\. LNE\-blocking’s over\-suppression componentΔ−\\Delta^\{\-\}\(0\.0836\) is 2\.0 times RailCap’s \(0\.0420\), yet its G\-APP reading \(0\.0207\) is 3\.8 times better than RailCap’s \(0\.0794\): the extra collateral damage cancels against the residual contamination and pushes the reading closer to zero\. The false perfection in Table[1](https://arxiv.org/html/2608.07341#S5.T1)is not an artefact of estimation noise: under probability estimates, the G\-APP reading is even closer to zero than Table[1](https://arxiv.org/html/2608.07341#S5.T1)’s single\-observation G\-AP\. With the readout held fixed, changing only the aggregation moves the verdict on LNE\-blocking by a factor of 14 \(0\.0207 under G\-APP versus 0\.2932 under SA\-PPG\)\. A\-PPG removes this cancellation \(LNE\-blocking 0\.1879, above RailCap’s 0\.1634\)\. The second isthe failure of equal weighting\. The All\-Zero row drives every question’s solve probability to zero \(the limiting form of the trivial strategy described in §[3\.3](https://arxiv.org/html/2608.07341#S3.SS3)\)\. It reads 0\.2190 under the equal\-weight A\-PPG, better than Identity \(0\.3793\), TED \(0\.3536\), and Shortcut \(0\.2399\), whereas SA\-PPG, after grouping by the clean model’s solve probability, ranks it worst of all strategies \(0\.4903\)\. The readout correction \(§[3\.2\.1](https://arxiv.org/html/2608.07341#S3.SS2.SSS1)\), per\-question differencing \(§[3\.2\.2](https://arxiv.org/html/2608.07341#S3.SS2.SSS2)\), and stratified aggregation \(§[3\.3](https://arxiv.org/html/2608.07341#S3.SS3)\) are each indispensable\. SA\-PPG combines the three into one metric\.

### 5\.3Strategy Experiments: RailCap versus Prior Strategies

#### 5\.3\.1Main Result: Lowest SA\-PPG across Settings

Table 3:SA\-PPG of each mitigation strategy across the six settings \(two contamination domains×\\timesthree models\)\. Every setting hasN=1319N=1319questions, with the same\-domain clean modelMclM\_\{\\mathrm\{cl\}\}as the reference\. Bold marks the best strategy in each column\.As shown in Table[3](https://arxiv.org/html/2608.07341#S5.T3), RailCap attains the lowest SA\-PPG in all six settings \(two contamination domains by three models\)\. Its largest lead is on Llama\-2×\\timesGSM8K \(0\.1914, against 0\.2476 for the runner\-up Shortcut\)\. The baselines behave consistently across settings\. TED is nearly indistinguishable from Identity\. Even its largest gap, on Pythia×\\timesGSM8K, only moves 0\.2131 to 0\.1950\. Shortcut is the runner\-up in five of the six settings, a consistent but modest improvement\. LNE\-blocking trails both Shortcut and RailCap in all six settings: slightly better than Identity in the GSM8K domain, yet worse than Identity on all three models in the PQ domain\. The contrast between the two domains suggests an explanation: in PQ, the questions seen at inference differ from the contaminated ones, so a one\-shot estimate of the contamination made before operating becomes harder\. The difficulty is not specific to LNE\-blocking: Shortcut’s lead over Identity narrows visibly on Llama\-2 and Gemma \(from 0\.2476 versus 0\.3261 to 0\.2242 versus 0\.2549 on Llama\-2\), and TED stays close to Identity in both domains\. RailCap decides the amount of intervention step by step during generation, without a pre\-hoc estimate, and remains the best in all settings\.

Table 4:Ablation of RailCap on Llama\-2×\\timesGSM8K\. Thennrows sweep the n\-gram threshold under the default suppression \(cap at the runner\-up\)\. In the n=4, ban row the cap is replaced by a hard ban: the rail token’s probability is set to zero and it can no longer be sampled \(the suppression\-strength contrast\)\. All\-Zero is the trivial\-strategy reference of §[3\.3](https://arxiv.org/html/2608.07341#S3.SS3)\. S\-Δs\+\\Delta^\{\+\}\_\{s\}and S\-Δs−\\Delta^\{\-\}\_\{s\}denote the stratified under\-suppression component \(residual contamination\) and the stratified over\-suppression component \(collateral damage\): the positive and negative parts of the per\-question gaps are split within the groups and averaged the same way as SA\-PPG, so the two sum to SA\-PPG\.
#### 5\.3\.2Ablation: n\-gram Threshold and Suppression Form

As shown in Table[4](https://arxiv.org/html/2608.07341#S5.T4), we ablate the two design choices of RailCap on Llama\-2×\\timesGSM8K: the n\-gram thresholdnnand the form of suppression\. Along thennaxis,n=1n=1triggers too frequently: the residual contamination nearly vanishes \(S\-Δ\+\\Delta^\{\+\}0\.0215\) while almost all the error comes from collateral damage \(S\-Δ−\\Delta^\{\-\}0\.2218\)\. Asnngrows, the triggering becomes more conservative, the residual component rises and the damage component falls, andn=4n=4brings the two close to parity \(0\.0942 versus 0\.0972\) with the best SA\-PPG \(0\.1914\)\. Values ofnnfrom 3 to 7 all stay within 0\.008 of the best, so the choice ofnnis robust\. For the form of suppression, replacing the cap at the runner\-up with a hard ban \(n=4, ban\) raises collateral damage to 0\.1464 and SA\-PPG to 0\.2190: keeping the rail token available at reduced probability is preferable to prohibiting it entirely\.

## 6Conclusion

We studied two coupled problems in contamination mitigation evaluation: how to measure the restoration that a mitigation strategy achieves, and how to design a better strategy\. On the metric side, we identified two flaws of the prevailing G\-AP \(a discrete readout that cannot represent per\-question performance, and averaging before differencing that lets over\- and under\-suppression cancel out\) and corrected them at the level of per\-question solve probabilities\. To keep strategies from pushing solve probabilities onto the clean model’s frequent values under equal weighting, we further aggregate within groups of the clean model’s solve probability, yielding SA\-PPG\. On the strategy side, we proposed RailCap, which checks at every decoding step whether a sample has fallen back onto the greedy trajectory and, once it has, caps the probability of the next trajectory token to that of the runner\-up, turning the judgment of contamination from a one\-shot pre\-hoc estimate into step\-wise supervision during generation\. Across three model families and two forms of contamination, G\-AP systematically overestimates the restoration of prior strategies, and RailCap attains the lowest SA\-PPG in every setting\.

## References

- Benchmarking foundation models with language\-model\-as\-an\-examiner\.Advances in Neural Information Processing Systems36,pp\. 78142–78167\.Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- S\. Balloccu, P\. Schmidtová, M\. Lango, and O\. Dusek \(2024\)Leak, cheat, repeat: Data contamination and evaluation malpractices in closed\-source LLMs\.InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\),Y\. Graham and M\. Purver \(Eds\.\),St\. Julian’s, Malta,pp\. 67–93\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.eacl-long.5)Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- S\. Biderman, H\. Schoelkopf, Q\. G\. Anthony, H\. Bradley, K\. O’Brien, E\. Hallahan, M\. A\. Khan, S\. Purohit, U\. S\. Prashanth, E\. Raff, A\. Skowron, L\. Sutawika, and O\. Van Der Wal \(2023\)Pythia: a suite for analyzing large language models across training and scaling\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 2397–2430\.External Links:[Link](https://proceedings.mlr.press/v202/biderman23a.html)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- T\. B\. Brown \(2020\)Language models are few\-shot learners\.arXiv preprint arXiv:2005\.14165\.External Links:2005\.14165Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- M\. Chen, J\. Tworek, H\. Jun, Q\. Yuan, H\. P\. D\. O\. Pinto, J\. Kaplan, H\. Edwards, Y\. Burda, N\. Joseph, G\. Brockman,et al\.\(2021\)Evaluating large language models trained on code\.arXiv preprint arXiv:2107\.03374\.External Links:2107\.03374,[Document](https://dx.doi.org/https%3A//arxiv.org/abs/2107.03374)Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p3.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:2110\.14168Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1),[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px1.p1.1)\.
- DeepSeek\-AI \(2026\)DeepSeek\-V4: towards highly efficient million\-token context intelligence\.arXiv preprint arXiv:2606\.19348\.Cited by:[Appendix A](https://arxiv.org/html/2608.07341#A1.SS0.SSS0.Px1.p1.1),[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px1.p1.1)\.
- C\. Deng, Y\. Zhao, Y\. Heng, Y\. Li, J\. Cao, X\. Tang, and A\. Cohan \(2024\)Unveiling the spectrum of data contamination in language model: a survey from detection to remediation\.InFindings of the Association for Computational Linguistics ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand and virtual meeting,pp\. 16078–16092\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.951)Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- J\. Dodge, M\. Sap, A\. Marasović, W\. Agnew, G\. Ilharco, D\. Groeneveld, M\. Mitchell, and M\. Gardner \(2021\)Documenting large webtext corpora: A case study on the colossal clean crawled corpus\.arXiv preprint arXiv:2104\.08758\.External Links:2104\.08758Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- R\. Dominguez\-Olmedo, F\. E\. Dorner, and M\. Hardt \(2025\)Training on the test task confounds evaluation and emergence\.InThe Thirteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=jOmk0uS1hl)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- Y\. Dong, X\. Jiang, H\. Liu, Z\. Jin, B\. Gu, M\. Yang, and G\. Li \(2024\)Generalization or memorization: Data contamination and trustworthy evaluation for large language models\.InFindings of the Association for Computational Linguistics ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand and virtual meeting,pp\. 12039–12050\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.716)Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§1](https://arxiv.org/html/2608.07341#S1.p2.1),[§1](https://arxiv.org/html/2608.07341#S1.p4.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p2.1),[§2](https://arxiv.org/html/2608.07341#S2.p3.1),[§2](https://arxiv.org/html/2608.07341#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px3.p1.4)\.
- Y\. Elazar, A\. Bhagia, I\. H\. Magnusson, A\. Ravichander, D\. Schwenk, A\. Suhr, E\. P\. Walsh, D\. Groeneveld, L\. Soldaini, S\. Singh, H\. Hajishirzi, N\. A\. Smith, and J\. Dodge \(2024\)What’s in my big data?\.InThe Twelfth International Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- Gemma Team \(2026\)Gemma 4 technical report\.arXiv preprint arXiv:2607\.02770\.External Links:[Link](https://arxiv.org/abs/2607.02770)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- S\. Golchin and M\. Surdeanu \(2024\)Time travel in LLMs: Tracing data contamination in large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- R\. Hou, Y\. Jiao, H\. Hu, Y\. Li, W\. Lam, H\. Zhang, and H\. Lu \(2025\)LNE\-blocking: an efficient framework for contamination mitigation evaluation on large language models\.InFindings of the Association for Computational Linguistics: EMNLP 2025,C\. Christodoulopoulos, T\. Chakraborty, C\. Rose, and V\. Peng \(Eds\.\),Suzhou, China,pp\. 3512–3528\.External Links:[Link](https://aclanthology.org/2025.findings-emnlp.188/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-emnlp.188),ISBN 979\-8\-89176\-335\-7Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§1](https://arxiv.org/html/2608.07341#S1.p2.1),[§1](https://arxiv.org/html/2608.07341#S1.p4.1),[§2](https://arxiv.org/html/2608.07341#S2.p2.1),[§2](https://arxiv.org/html/2608.07341#S2.p3.1),[§2](https://arxiv.org/html/2608.07341#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px3.p1.4)\.
- A\. Jacovi, A\. Caciularu, O\. Goldman, and Y\. Goldberg \(2023\)Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks\.arXiv preprint arXiv:2305\.10160\.External Links:2305\.10160Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- Q\. Li, L\. Cui, X\. Zhao, L\. Kong, and W\. Bi \(2024\)GSM\-plus: a comprehensive benchmark for evaluating the robustness of LLMs as mathematical problem solvers\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 2961–2984\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.163)Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- X\. Li, Y\. Lan, and C\. Yang \(2025\)Treeeval: Benchmark\-free evaluation of large language models through tree planning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 24485–24493\.Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- Y\. Li \(2023\)Estimating contamination via perplexity: Quantifying memorisation in language model evaluation\.arXiv preprint arXiv:2309\.10677\.External Links:2309\.10677Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- W\. Lian, B\. Goodson, E\. Pentland, A\. Cook, C\. Vong, and Teknium \(2023\)OpenOrca: an open dataset of GPT augmented FLAN reasoning traces\.HuggingFace\.Note:[https://huggingface\.co/datasets/Open\-Orca/OpenOrca](https://huggingface.co/datasets/Open-Orca/OpenOrca)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- I\. Magar and R\. Schwartz \(2022\)Data contamination: From memorization to exploitation\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(Volume 2: Short Papers\),S\. Muresan, P\. Nakov, and A\. Villavicencio \(Eds\.\),Dublin, Ireland,pp\. 157–165\.External Links:[Document](https://dx.doi.org/10.18653/v1/2022.acl-short.18)Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- W\. Shi, A\. Ajith, M\. Xia, Y\. Huang, D\. Liu, T\. Blevins, D\. Chen, and L\. Zettlemoyer \(2024\)Detecting pretraining data from large language models\.InThe Twelfth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- H\. Touvron, L\. Martin, K\. Stone, P\. Albert, A\. Almahairi, Y\. Babaei, N\. Bashlykov, S\. Batra, P\. Bhargava, S\. Bhosale,et al\.\(2023\)Llama 2: Open foundation and fine\-tuned chat models\.arXiv preprint arXiv:2307\.09288\.External Links:2307\.09288Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- J\. Wei, X\. Wang, D\. Schuurmans, M\. Bosma, B\. Ichter, F\. Xia, E\. Chi, Q\. V\. Le, and D\. Zhou \(2022\)Chain\-of\-thought prompting elicits reasoning in large language models\.InAdvances in Neural Information Processing Systems,S\. Koyejo, S\. Mohamed, A\. Agarwal, D\. Belgrave, K\. Cho, and A\. Oh \(Eds\.\),Vol\.35,pp\. 24824–24837\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- W\. Ye, J\. Hu, L\. Li, H\. Wang, G\. Chen, and J\. Zhao \(2024\)Data contamination calibration for black\-box LLMs\.InFindings of the Association for Computational Linguistics ACL 2024,L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand and virtual meeting,pp\. 10845–10861\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-acl.644)Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- Z\. Yu, C\. Gao, W\. Yao, Y\. Wang, W\. Ye, J\. Wang, X\. Xie, Y\. Zhang, and S\. Zhang \(2024\)KIEval: a knowledge\-grounded interactive evaluation framework for large language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),L\. Ku, A\. Martins, and V\. Srikumar \(Eds\.\),Bangkok, Thailand,pp\. 5967–5985\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.acl-long.325)Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- H\. Zhang, J\. Da, D\. Lee, V\. Robinson, C\. Wu, W\. Song, T\. Zhao, P\. Raja, D\. Slack, Q\. Lyu,et al\.\(2024a\)A careful examination of large language model performance on grade school arithmetic\.arXiv preprint arXiv:2405\.00332\.External Links:2405\.00332Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- W\. Zhang, R\. Zhang, J\. Guo, M\. de Rijke, Y\. Fan, and X\. Cheng \(2024b\)Pretraining Data Detection for Large Language Models: A Divergence\-based Calibration Method\.InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing,Y\. Al\-Onaizan, M\. Bansal, and Y\. Chen \(Eds\.\),Miami, Florida, USA,pp\. 5263–5274\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.emnlp-main.300)Cited by:[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- Y\. Zheng, R\. Zhang, J\. Zhang, Y\. Ye, Z\. Luo, Z\. Feng, and Y\. Ma \(2024\)LlamaFactory: unified efficient fine\-tuning of 100\+ language models\.InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics \(Volume 3: System Demonstrations\),Bangkok, Thailand\.External Links:[Link](http://arxiv.org/abs/2403.13372)Cited by:[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px2.p1.4)\.
- K\. Zhou, Y\. Zhu, Z\. Chen, W\. Chen, W\. X\. Zhao, X\. Chen, Y\. Lin, J\. Wen, and J\. Han \(2023\)Don’t make your llm an evaluation benchmark cheater\.arXiv preprint arXiv:2311\.01964\.External Links:2311\.01964Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.
- K\. Zhu, S\. Tu, Z\. Jin, L\. Hou, J\. Li, and J\. Zhao \(2025\)Establishing trustworthy LLM evaluation via shortcut neuron analysis\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),W\. Che, J\. Nabende, E\. Shutova, and M\. T\. Pilehvar \(Eds\.\),Vienna, Austria,pp\. 3809–3822\.External Links:[Link](https://aclanthology.org/2025.acl-long.192/),[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.192),ISBN 979\-8\-89176\-251\-0Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§1](https://arxiv.org/html/2608.07341#S1.p2.1),[§1](https://arxiv.org/html/2608.07341#S1.p4.1),[§2](https://arxiv.org/html/2608.07341#S2.p2.1),[§2](https://arxiv.org/html/2608.07341#S2.p3.1),[§2](https://arxiv.org/html/2608.07341#S2.p4.1),[§5\.1](https://arxiv.org/html/2608.07341#S5.SS1.SSS0.Px3.p1.4)\.
- W\. Zhu, H\. Hao, Z\. He, Y\. Song, J\. Yueyang, Y\. Zhang, H\. Hu, Y\. Wei, R\. Wang, and H\. Lu \(2024\)CLEAN–EVAL: Clean evaluation on contaminated large language models\.InFindings of the Association for Computational Linguistics: NAACL 2024,K\. Duh, H\. Gomez, and S\. Bethard \(Eds\.\),Mexico City, Mexico,pp\. 835–847\.External Links:[Document](https://dx.doi.org/10.18653/v1/2024.findings-naacl.53)Cited by:[§1](https://arxiv.org/html/2608.07341#S1.p1.1),[§2](https://arxiv.org/html/2608.07341#S2.p1.1)\.

## Appendix APQ Construction and Evaluation Details

##### A\.1 PQ construction\.

For each leaked question, the question text is rewritten with DeepSeek\-V4\-Flash\(DeepSeek\-AI,[2026](https://arxiv.org/html/2608.07341#bib.bib51)\): the instruction requires a substantial rephrasing, keeps all numbers and the final answer identical, forbids any solution, and outputs exactly one line containing the rewritten question\. The system prompt is:

> Significantly rephrase the given GSM8K math question\. Keep ALL numbers and the final answer identical\. Do not include any solution\. Output exactly one line in this format and nothing else: New Question: <rephrased question\>

The reasoning chain and the gold answer remain the GSM8K originals\.