Provable Joint Decontamination for Benchmarking Multiple Large Language Models

# Provable Joint Decontamination for Benchmarking Multiple Large Language Models
Source: [https://arxiv.org/html/2605.21543](https://arxiv.org/html/2605.21543)
Zhenlong LiuDepartment of Statistics and Data Science, Southern University of Science and TechnologyShanghai Innovation InstituteHongxin WeiDepartment of Statistics and Data Science, Southern University of Science and Technology

###### Abstract

Benchmark data contamination has become a central challenge in LLM evaluation: when evaluation examples appear in the training data of one or more audited models, reported performance can be inflated and cross\-model comparisons become unreliable\. A broad line of training\-data detection work designs scores to quantify how strongly a model memorizes a given data point, but these score\-based methods lack theoretical guarantees\. Recent conformal approaches provide provable false\-identification control for a*single model*; however, applying them separately to each model can produce model\-specific benchmarks, undermining fair comparison across models\. In this work, we formalize multi\-model benchmark decontamination as a joint selection problem and propose Joint Envelope Conformal Selection \(JECS\), a conformal procedure that enables global contamination rate \(GCR\) control under stated assumptions\. Specifically, JECS computes per\-model conformalpp\-values, aggregates them by the per\-item maximum, and reconstructs a conservative envelope of the max\-ppnull distribution from right\-tail observations above a data\-driven threshold\. By applying the adaptive Benjamini–Hochberg \(BH\) procedure to the envelope\-rescaled values, we select a benchmark with provable GCR control\. Extensive experiments across various models and benchmarks demonstrate that JECS achieves higher power than the max\-ppbaseline while consistently maintaining the target GCR control\.

## 1Introduction

The remarkable achievements of large language models \(LLMs\) have been driven in large part by the scale and diversity of their pretraining corpora\(Kaplan et al\.,[2020](https://arxiv.org/html/2605.21543#bib.bib27); Chang et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib12)\)\. However, the construction of large pretraining corpora can lead to overlap between training data and evaluation benchmarks, a phenomenon commonly referred to as benchmark data contamination\(Sainz et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib42); Deng et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib15); Xu et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib55); Balloccu et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib4)\)\. Such overlap can overstate reported performance, obscure whether models truly generalize beyond their training data, and weaken the credibility of benchmark\-based evaluation\. This highlights the need to construct a decontaminated benchmark for multiple audited models\.

To address this issue, a broad line of work\(Carlini et al\.,[2021](https://arxiv.org/html/2605.21543#bib.bib11); Zhang et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib62); Li et al\.,[2024c](https://arxiv.org/html/2605.21543#bib.bib33)\)designs detection scores, such as Min\-K%\(Shi et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib43)\), to quantify how strongly a model memorizes a given data point\. Motivated by the lack of theoretical guarantees in these score\-based methods, recent work\(Liu et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib35)\)leverages conformal inference to identify training data for a*single model*with provable false\-identification control\. However, applying such a procedure separately to each model can produce model\-specific benchmarks, undermining fair comparison across models\. In practical benchmark evaluation, auditors require a shared decontaminated benchmark that supports fair assessment across multiple models\. This motivates us to develop methods for selecting*one*shared benchmark with a controlled contamination rate across all audited models\.

In this paper, we formulate joint benchmark decontamination as a joint selection problem, where the goal is to select a single benchmark that is decontaminated with respect to multiple audited models\. Specifically, we call a sample*jointly pure*if it does not appear in the training data of any audited model, and define the global contamination rate \(GCR\) as the expected fraction of contaminated samples in the selected set\. To control the GCR, we propose Joint Envelope Conformal Selection \(JECS\), which first obtains a validpp\-value under the joint null for each candidate by taking the maximum of its per\-model conformalpp\-values\. To recover power, we further reconstruct a conservative envelope function for the cumulative distribution function \(CDF\) of the max\-ppnull distribution, map each max\-ppvalue through the fitted function, and then apply the adaptive BH procedure\(Benjamini and Hochberg,[1995](https://arxiv.org/html/2605.21543#bib.bib6); Benjamini and Yekutieli,[2001](https://arxiv.org/html/2605.21543#bib.bib7)\)\. We establish theoretical guarantees showing that JECS controls the contamination rate at a user\-specified levelα\\alpha\.

Extensive experiments on simulated data and real LLM benchmarks demonstrate the effectiveness of JECS for joint benchmark decontamination\. Across all settings, JECS controls the global contamination rate at the user\-specified target, while neither the union nor the intersection of per\-model conformal selections does so\. For instance, on the synthetic setup atα=0\.1\\alpha=0\.1, JECS attains GCR=0\.038=0\.038, whereas the union and intersection rules violate the target with GCR=0\.763=0\.763and0\.3660\.366, respectively\. Furthermore, our method improves power over the max\-ppbaseline\. For example, on ArXivTection with Pythia\-6\.9B\(Biderman et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib8)\)atα=0\.1\\alpha=0\.1using the Min\-K%\+\+ score\(Zhang et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib62)\), our method improves power from0\.0940\.094to0\.4470\.447\. Overall, these results show that JECS preserves the contamination\-rate control required for joint selection while recovering much of the power lost by the conservative max\-ppbaseline\.

We summarize our contributions as follows:

1. 1\.We formulate multi\-model benchmark decontamination as a joint selection problem, where a candidate item is jointly pure only if it is absent from the training data of every audited model\. We introduce the global contamination rate \(GCR\) as the contamination\-rate criterion for selecting one shared benchmark across multiple audited models\.
2. 2\.We propose Joint Envelope Conformal Selection \(JECS\), a joint selection procedure for constructing shared decontaminated benchmarks across multiple audited models\. Specifically, JECS mitigates the super\-uniform conservatism of the valid max\-ppbaseline, thereby retaining global contamination rate control while substantially improving power\.
3. 3\.We establish theoretical guarantees showing that JECS controls GCR under stated conditions\. We further validate these guarantees through extensive experiments, demonstrating the effectiveness of JECS on both simulated data and real\-world LLM benchmarks\.

## 2Preliminary

#### Setup\.

Letθ\\thetabe a target language model trained on a private corpus𝒟train​\(θ\)\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\), and let𝒳\\mathcal\{X\}denote the input space of token sequences\. For a samplex∈𝒳x\\in\\mathcal\{X\}, the binary membership indicator

M​\(x;θ\)=𝟙​\{x∈𝒟train​\(θ\)\}∈\{0,1\}M\(x;\\theta\)=\\mathds\{1\}\\\{x\\in\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\)\\\}\\in\\\{0,1\\\}\(1\)records whetherxxwas used to trainθ\\theta\. Specifically,M=1M=1indicates a member andM=0M=0a non\-member\. Throughout the paper, we auditKKsuch models\{θk\}k=1K\\\{\\theta\_\{k\}\\\}\_\{k=1\}^\{K\}on a candidate pool\{xi\}i=1n\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}, and writeMik=M​\(xi;θk\)M\_\{i\}^\{k\}=M\(x\_\{i\};\\theta\_\{k\}\)for the membership status of candidate instanceiiunder modelθk\\theta\_\{k\}\.

#### Training data detection\.

Training data detection, also termed membership inference attack \(MIA\), aims to infer the membership indicatorM​\(x;θ\)M\(x;\\theta\)from query access to the modelpθp\_\{\\theta\}\. The standard pipeline assigns each sample a detection scoreT​\(x;θ\)T\(x;\\theta\)and predicts membership by a level\-set rule, with the inequality direction determined by the score convention\. In this paper, we orientTTso that smaller values provide stronger evidence of non\-membership, and hence a threshold rule may be written asM^=𝟙​\{T​\(x;θ\)≥τ\}\\widehat\{M\}=\\mathds\{1\}\\\{T\(x;\\theta\)\\geq\\tau\\\}with a thresholdτ\\taudetermined by a validation set\(Shokri et al\.,[2017](https://arxiv.org/html/2605.21543#bib.bib44); Ye et al\.,[2022](https://arxiv.org/html/2605.21543#bib.bib57)\)\. A representative score is Min\-K%\+\+\(Zhang et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib62)\), which for a sequencex=\(t1,…,tL\)x=\(t\_\{1\},\\ldots,t\_\{L\}\)averages a normalized log\-probability over the bottom\-K%K\\%tokens:

TMin\-K%\+\+​\(x;θ\)=1\|ℐ\|​∑ℓ∈ℐlog⁡pθ​\(tℓ∣t<ℓ\)−μℓσℓ,μℓ=𝔼t∼pθ\(⋅∣t<ℓ\)​\[log⁡pθ​\(t∣t<ℓ\)\],T\_\{\\textsc\{Min\-K\\%\+\+\}\}\(x;\\theta\)=\\frac\{1\}\{\|\\mathcal\{I\}\|\}\\sum\_\{\\ell\\in\\mathcal\{I\}\}\\frac\{\\log p\_\{\\theta\}\(t\_\{\\ell\}\\mid t\_\{<\\ell\}\)\-\\mu\_\{\\ell\}\}\{\\sigma\_\{\\ell\}\},\\qquad\\mu\_\{\\ell\}=\\mathbb\{E\}\_\{t\\sim p\_\{\\theta\}\(\\cdot\\mid t\_\{<\\ell\}\)\}\[\\log p\_\{\\theta\}\(t\\mid t\_\{<\\ell\}\)\],\(2\)whereσℓ\\sigma\_\{\\ell\}is the standard deviation oflogpθ\(⋅∣t<ℓ\)\\log p\_\{\\theta\}\(\\cdot\\mid t\_\{<\\ell\}\)under the next\-token distribution andℐ⊂\{1,…,L\}\\mathcal\{I\}\\subset\\\{1,\\ldots,L\\\}indexes the lowestK%K\\%tokens by per\-token log\-probability\. Other widely used scores include perplexity\(Carlini et al\.,[2021](https://arxiv.org/html/2605.21543#bib.bib11)\), Min\-K%\(Shi et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib43)\), and Modified Entropy\(Song and Mittal,[2021](https://arxiv.org/html/2605.21543#bib.bib45)\)\. These scores provide useful per\-instance evidence of memorization by a specific model, but they do not provide theoretical guarantees for the resulting predictions\.

#### Conformal inference for training data identification\.

To provide provable evidence, recent work\(Liu et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib35)\)formulates training\-data identification as a multiple\-hypothesis testing problem\. For each audited modelθk\\theta\_\{k\}, the model\-wise null and alternative are:

H0,ik:xi∈𝒟train​\(θk\),H1,ik:xi∉𝒟train​\(θk\)\.H\_\{0,i\}^\{k\}:x\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\_\{k\}\),\\qquad H\_\{1,i\}^\{k\}:x\_\{i\}\\notin\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\_\{k\}\)\.\(3\)RejectingH0,ikH\_\{0,i\}^\{k\}means thatxix\_\{i\}is identified as pure with respect toθk\\theta\_\{k\}; a false identification occurs when a contaminated item is incorrectly selected\.111The testing direction here is reversed from the convention inLiu et al\. \([2026](https://arxiv.org/html/2605.21543#bib.bib35)\), where the null is non\-member\. We reverse the null because benchmark construction selects non\-members\.The conformalpp\-value\(Vovk et al\.,[2003](https://arxiv.org/html/2605.21543#bib.bib50),[2005](https://arxiv.org/html/2605.21543#bib.bib51)\)calibrates the detection scores into validpp\-values:

pik=1\+∑xℓ∈𝒟cal𝟏​\{T​\(xℓ;θk\)≤T​\(xi;θk\)\}\|𝒟cal\|\+1,p\_\{i\}^\{k\}=\\frac\{1\+\\sum\_\{x\_\{\\ell\}\\in\\mathcal\{D\}\_\{\\mathrm\{cal\}\}\}\\mathbf\{1\}\\\{T\(x\_\{\\ell\};\\theta\_\{k\}\)\\leq T\(x\_\{i\};\\theta\_\{k\}\)\\\}\}\{\|\\mathcal\{D\}\_\{\\mathrm\{cal\}\}\|\+1\},\(4\)where𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}is a calibration set of known members ofθk\\theta\_\{k\}drawn from the same covariate distribution as contaminated candidates\. The Benjamini–Hochberg \(BH\) procedure\(Benjamini and Hochberg,[1995](https://arxiv.org/html/2605.21543#bib.bib6)\)is then used to select a subset:

𝒮k=\{i:pik≤α​r∗n\},where​r∗=max⁡\{r∈\{1,…,n\}:p\(r\)k≤α​rn\},\\mathcal\{S\}\_\{k\}=\\left\\\{i:p\_\{i\}^\{k\}\\leq\\frac\{\\alpha\\,r^\{\*\}\}\{n\}\\right\\\},\\text\{ where \}r^\{\*\}=\\max\\\!\\left\\\{r\\in\\\{1,\\ldots,n\\\}:p\_\{\(r\)\}^\{k\}\\leq\\frac\{\\alpha\\,r\}\{n\}\\right\\\},\(5\)wherep\(1\)k≤⋯≤p\(n\)kp\_\{\(1\)\}^\{k\}\\leq\\cdots\\leq p\_\{\(n\)\}^\{k\}are the sorted model\-wisepp\-values\. In the original training\-data\-identification setting, prior works\(Bates et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib5); Liu et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib35)\)have shown that the expected fraction ofθk\\theta\_\{k\}\-non\-members among the items identified as members can be controlled at a user\-specified levelα∈\(0,1\)\\alpha\\in\(0,1\)\. However, this guarantee is model\-specific: applying the procedure separately to different audited models yields different selected subsets𝒮1,…,𝒮K\\mathcal\{S\}\_\{1\},\\ldots,\\mathcal\{S\}\_\{K\}\. In practical benchmark evaluation, auditors need a single decontaminated benchmark for a fair comparison across all audited models\. This gap motivates the joint benchmark decontamination problem studied next\.

## 3Joint Benchmark Decontamination

### 3\.1Problem formulation

We formulate thejoint benchmark decontaminationproblem, whose goal is to select one shared benchmark that is decontaminated with respect to all audited models\. Unlike the model\-wise setting above, contamination in this problem is defined globally across the audited model collection\. An instancexix\_\{i\}is*contaminated*if it appears in the training data of at least one audited model, and is*jointly pure*only if it appears in none\. Accordingly, the joint null and alternative hypotheses are:

H0,i:∃k∗∈\{1,…,K\},xi∈𝒟train​\(θk∗\),H1,i:∀k,xi∉𝒟train​\(θk\)\.H\_\{0,i\}:\\;\\exists\\,k^\{\*\}\\in\\\{1,\\ldots,K\\\},\\;x\_\{i\}\\in\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\_\{k^\{\*\}\}\),\\qquad H\_\{1,i\}:\\;\\forall\\,k,\\;x\_\{i\}\\notin\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\_\{k\}\)\.\(6\)
RejectingH0,iH\_\{0,i\}indicates thatxix\_\{i\}is inferred to be jointly pure and is therefore selected for inclusion in the benchmark\. We therefore quantify error by the contamination rate within the selected benchmark\. Formally, for a selected set𝒮⊆\{1,…,n\}\\mathcal\{S\}\\subseteq\\\{1,\\ldots,n\\\}, we define the global contamination proportion \(GCP\) and global contamination rate \(GCR\) as follows:

GCP​\(𝒮\)=∑i=1n𝟙​\{i∈𝒮,H0,i​is true\}1∨\|𝒮\|,GCR​\(𝒮\)=𝔼​\[GCP​\(𝒮\)\],\\mathrm\{GCP\}\(\\mathcal\{S\}\)=\\frac\{\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\\{i\\in\\mathcal\{S\},\\;H\_\{0,i\}\\text\{ is true\}\\\}\}\{1\\vee\|\\mathcal\{S\}\|\},\\qquad\\mathrm\{GCR\}\(\\mathcal\{S\}\)=\\mathbb\{E\}\[\\mathrm\{GCP\}\(\\mathcal\{S\}\)\],\(7\)where we denotea∨b=max⁡\{a,b\}a\\vee b=\\max\\\{a,b\\\}\. Our goal is to return a subset𝒮\\mathcal\{S\}such thatGCR​\(𝒮\)≤α\\mathrm\{GCR\}\(\\mathcal\{S\}\)\\leq\\alpha, whereα∈\(0,1\)\\alpha\\in\(0,1\)is a user\-specified level\. Beyond GCR control, we also seek a procedure that selects as many true candidates as possible\. To quantify this objective, we define thepoweras

Power​\(𝒮\)=𝔼​\[∑i=1n𝟙​\{i∈𝒮,H1,i​is true\}1∨∑i=1n𝟙​\{H1,i​is true\}\]\.\\mathrm\{Power\}\(\\mathcal\{S\}\)=\\mathbb\{E\}\\left\[\\frac\{\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\\{i\\in\\mathcal\{S\},H\_\{1,i\}\\text\{ is true\}\\\}\}\{1\\vee\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\\{H\_\{1,i\}\\text\{ is true\}\\\}\}\\right\]\.\(8\)An ideal method should control the GCR at the prescribed levelα\\alphawith power as high as possible\.

#### Naive per\-model composition fails to control GCR\.

Table 1:Realized GCP of naive per\-model composition on synthetic data\. Entries marked with a†\\daggerexceedα\\alpha, indicating failure to meet the GCR target\.A natural but invalid approach is to first form the model\-wise selections𝒮1,…,𝒮K\\mathcal\{S\}\_\{1\},\\ldots,\\mathcal\{S\}\_\{K\}from the conformalpp\-values in[Equation˜4](https://arxiv.org/html/2605.21543#S2.E4)using the single\-model rule in[Equation˜5](https://arxiv.org/html/2605.21543#S2.E5), and then combine them by union or intersection\. The union∪k𝒮k\\cup\_\{k\}\\mathcal\{S\}\_\{k\}is too liberal: an item may be selected because it appears pure for one model while being contaminated for another\. The intersection∩k𝒮k\\cap\_\{k\}\\mathcal\{S\}\_\{k\}is more conservative, but still lacks GCR control because the denominator changes after composition and the model\-wise selections are dependent\. As shown in Table[1](https://arxiv.org/html/2605.21543#S3.T1), both composition violate the target on a controlled synthetic setup withK=4K=4audited models and known joint\-purity labels \(full configuration in[Section˜E\.1](https://arxiv.org/html/2605.21543#A5.SS1)\): Union stays nearGCP≈0\.76\\mathrm\{GCP\}\\approx 0\.76, and Intersection increasingly exceedsα\\alphaasα\\alphagrows\. Thus, joint GCR control for[Equation˜6](https://arxiv.org/html/2605.21543#S3.E6)requires testingH0,iH\_\{0,i\}directly through an aggregated statistic rather than composing per\-model decisions\.

### 3\.2A max\-ppbaseline: Joint Max Conformal Selection \(JMCS\)

We now adapt the model\-wise conformalpp\-values from[Section˜2](https://arxiv.org/html/2605.21543#S2)to the joint selection problem\. Throughout this section, we use a shared calibration set𝒟cal=\{xℓ\}ℓ=n\+1n\+m\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\\{x\_\{\\ell\}\\\}\_\{\\ell=n\+1\}^\{n\+m\}whose items are known members of every audited model, i\.e\.,xℓ∈𝒟train​\(θk\)x\_\{\\ell\}\\in\\mathcal\{D\}\_\{\\mathrm\{train\}\}\(\\theta\_\{k\}\)for allk∈\{1,…,K\}k\\in\\\{1,\\dots,K\\\},222We discuss practical ways to construct such calibration sets in[AppendixF](https://arxiv.org/html/2605.21543#A6)\.and are drawn from the same candidate\-pool covariate distribution\.

Applying the conformal construction in[Equation˜4](https://arxiv.org/html/2605.21543#S2.E4)to each audited model yields model\-wisepp\-values\{pik\}k=1K\\\{p\_\{i\}^\{k\}\\\}\_\{k=1\}^\{K\}\. By exchangeability, for every model\-wise nullH0,ikH\_\{0,i\}^\{k\},

ℙ​\(pik≤t∣H0,ik\)≤t,t∈\[0,1\]\.\\mathbb\{P\}\(p\_\{i\}^\{k\}\\leq t\\mid H\_\{0,i\}^\{k\}\)\\leq t,\\qquad t\\in\[0,1\]\.\(9\)
To test the joint nullH0,iH\_\{0,i\}via one statistic, we aggregate theKKmodel\-wise conformalpp\-values by their maximum:

pi∗=max1≤k≤K⁡pik\.p\_\{i\}^\{\*\}=\\max\_\{1\\leq k\\leq K\}p\_\{i\}^\{k\}\.\(10\)The aggregated valuepi∗p\_\{i\}^\{\*\}is small only if every model’spp\-value is small, which is the operational meaning of joint purity\. The following lemma states thatpi∗p\_\{i\}^\{\*\}is itself a validpp\-value for the joint null\.

###### Lemma 1\(Validity of the max\-pp\)\.

If[Equation˜9](https://arxiv.org/html/2605.21543#S3.E9)holds for each model\-wise null, thenpi∗p\_\{i\}^\{\*\}is valid for the joint null:

ℙ​\(pi∗≤t∣H0,i\)≤t,t∈\[0,1\]\.\\mathbb\{P\}\(p\_\{i\}^\{\*\}\\leq t\\mid H\_\{0,i\}\)\\leq t,\\qquad t\\in\[0,1\]\.\(11\)

###### Proof\.

UnderH0,iH\_\{0,i\}, there existsk∗k^\{\*\}such thatH0,ik∗H\_\{0,i\}^\{k^\{\*\}\}is true\. Sincepi∗=maxk⁡pik≥pik∗p\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}\\geq p\_\{i\}^\{k^\{\*\}\}, we haveℙ​\(pi∗≤t∣H0,i\)≤ℙ​\(pik∗≤t∣H0,ik∗\)≤t\\mathbb\{P\}\(p\_\{i\}^\{\*\}\\leq t\\mid H\_\{0,i\}\)\\leq\\mathbb\{P\}\(p\_\{i\}^\{k^\{\*\}\}\\leq t\\mid H\_\{0,i\}^\{k^\{\*\}\}\)\\leq t\. ∎

[Lemma˜1](https://arxiv.org/html/2605.21543#Thmtheorem1)immediately delivers a baseline procedure: apply the Benjamini–Hochberg \(BH\)\(Benjamini and Hochberg,[1995](https://arxiv.org/html/2605.21543#bib.bib6)\)step\-up rule to the maxima\{pi∗\}i=1n\\\{p\_\{i\}^\{\*\}\\\}\_\{i=1\}^\{n\}at levelα\\alpha\. Lettingp\(1\)∗≤⋯≤p\(n\)∗p\_\{\(1\)\}^\{\*\}\\leq\\cdots\\leq p\_\{\(n\)\}^\{\*\}be the sorted maxima, the rule selects

𝒮=\{i:pi∗≤α​r∗n\},where​r∗=max⁡\{r∈\{1,…,n\}:p\(r\)∗≤α​rn\},\\mathcal\{S\}=\\left\\\{i:p\_\{i\}^\{\*\}\\leq\\frac\{\\alpha\\,r^\{\*\}\}\{n\}\\right\\\},\\text\{ where \}r^\{\*\}=\\max\\\!\\left\\\{r\\in\\\{1,\\ldots,n\\\}:p\_\{\(r\)\}^\{\*\}\\leq\\frac\{\\alpha\\,r\}\{n\}\\right\\\},\(12\)We refer to this procedure as the Joint Max Conformal Selection \(JMCS\)\.[Lemma˜1](https://arxiv.org/html/2605.21543#Thmtheorem1)together with the standard BH analysis\(Benjamini and Hochberg,[1995](https://arxiv.org/html/2605.21543#bib.bib6)\)yields finite\-sample GCR control under the joint null\. JMCS is the natural minimum\-viable procedure for joint pure benchmark selection, and to our knowledge no prior work has analyzed it as such; we use it as the baseline against which JECS is compared in[Section˜4](https://arxiv.org/html/2605.21543#S4), and summarize the full procedure in[Algorithm˜2](https://arxiv.org/html/2605.21543#alg2)of[Appendix˜C](https://arxiv.org/html/2605.21543#A3)\.

#### The super\-uniformity tax\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x1.png)Figure 1:Null density of the per\-model conformalppversuspi∗=maxk⁡pikp\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}atK=8K=8\.JMCS is valid in finite samples but substantially conservative\. The BH cutoffα​r/n\\alpha r/nis calibrated to the uniform scale, whereas underH0,iH\_\{0,i\}the maximum statisticpi∗=maxk⁡pikp\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}is strictly super\-uniform: in the stylized homogeneous null withpi1,…,piK∼iidUnif​\(0,1\)p\_\{i\}^\{1\},\\ldots,p\_\{i\}^\{K\}\\stackrel\{\{\\scriptstyle\\mathrm\{iid\}\}\}\{\{\\sim\}\}\\mathrm\{Unif\}\(0,1\), the null CDF isF0​\(t\)=tKF\_\{0\}\(t\)=t^\{K\}and the densityf0​\(t\)=K​tK−1f\_\{0\}\(t\)=Kt^\{K\-1\}vanishes att=0t=0and concentrates neart=1t=1\([Figure˜1](https://arxiv.org/html/2605.21543#S3.F1),K=8K=8\)\. Almost all null mass therefore sits above the BH cutoff: a rejection requirespi∗≤α​r/np\_\{i\}^\{\*\}\\leq\\alpha r/n, far below the natural null scale\(r/n\)1/K\(r/n\)^\{1/K\}, and the gap widens asα\\alphatightens orKKgrows\. Although[Lemma˜1](https://arxiv.org/html/2605.21543#Thmtheorem1)guarantees GCR control, this procedure can lose substantial power asKKincreases\.

### 3\.3Joint Envelope Conformal Selection \(JECS\)

As discussed above,pi∗p\_\{i\}^\{\*\}is expected to follow a monotone increasing distribution on\[0,1\]\[0,1\]under the null, resembling aBeta\\mathrm\{Beta\}distribution and concentrating near 1\. While this preserves validity, it often leads to overly conservative selection\. A natural remedy is therefore to estimate the null CDF ofpi∗p\_\{i\}^\{\*\}and use the estimated CDF to recalibratepi∗p\_\{i\}^\{\*\}, improving selection power while retaining validity\.

Algorithm 1Joint Envelope Conformal Selection \(JECS\)0:Candidate items

\{xi\}i=1n\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}, audited models

\{θk\}k=1K\\\{\\theta\_\{k\}\\\}\_\{k=1\}^\{K\}, calibration data

𝒟cal=\{xi\}i=n\+1n\+m\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\\{x\_\{i\}\\\}\_\{i=n\+1\}^\{n\+m\}, target level

α\\alpha, tail threshold

λ\\lambda\.

1:for

k=1,…,Kk=1,\\ldots,Kdo

2:Construct conformal

pp\-values

\{pik\}i=1n\\\{p\_\{i\}^\{k\}\\\}\_\{i=1\}^\{n\}via[Equation˜4](https://arxiv.org/html/2605.21543#S2.E4)\.

3:endfor

4:Aggregate

pi∗=maxk⁡pikp\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}for

i=1,…,ni=1,\\ldots,n\.

5:Estimate

g^​\(λ\+\)\\widehat\{g\}\(\\lambda^\{\+\}\)and

G^n\\widehat\{G\}\_\{n\}from the right tail

\{pi∗:pi∗\>λ\}\\\{p\_\{i\}^\{\*\}:p\_\{i\}^\{\*\}\>\\lambda\\\}\.

6:Form the envelope

F^fit\\widehat\{F\}\_\{\\mathrm\{fit\}\}via[Equation˜14](https://arxiv.org/html/2605.21543#S3.E14)and[Equation˜15](https://arxiv.org/html/2605.21543#S3.E15)\.

7:Transform

p~i=F^fit​\(pi∗\)\\widetilde\{p\}\_\{i\}=\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(p\_\{i\}^\{\*\}\)and estimate

π^0\\widehat\{\\pi\}\_\{0\}via[Equation˜16](https://arxiv.org/html/2605.21543#S3.E16)\.

8:Return the selected set

𝒮\\mathcal\{S\}via[Equation˜17](https://arxiv.org/html/2605.21543#S3.E17)\.

LetF0F\_\{0\}andF1F\_\{1\}denote the CDFs ofpi∗p\_\{i\}^\{\*\}under contaminated and pure items, and letFmix=π0​F0\+\(1−π0\)​F1F\_\{\\mathrm\{mix\}\}=\\pi\_\{0\}F\_\{0\}\+\(1\-\\pi\_\{0\}\)F\_\{1\}be the marginal mixture\. The empirical asymmetry that drives JECS is that pure items concentrate near the left tail ofpi∗p\_\{i\}^\{\*\}, while the right tail\{pi∗\>λ\}\\\{p\_\{i\}^\{\*\}\>\\lambda\\\}is enriched for contaminated items\. We exploit this asymmetry by anchoring a conservative envelope to the right tail and propagating it leftward\. For a fixed thresholdλ∈\(0,1\)\\lambda\\in\(0,1\), define the right\-tail conditional CDF

G​\(x\)=ℙ​\(P∗≤x​∣P∗\>​λ\)=F​\(x\)−F​\(λ\)1−F​\(λ\),x∈\[λ,1\],G\(x\)=\\mathbb\{P\}\(P^\{\*\}\\leq x\\mid P^\{\*\}\>\\lambda\)=\\frac\{F\(x\)\-F\(\\lambda\)\}\{1\-F\(\\lambda\)\},\\qquad x\\in\[\\lambda,1\],\(13\)whereP∗P^\{\*\}denotes a generic draw from the distribution with CDFFF\. Letg​\(λ\+\)g\(\\lambda^\{\+\}\)denote the corresponding right\-boundary density\. From the observed maxima\{pi∗\}i=1n\\\{p\_\{i\}^\{\*\}\\\}\_\{i=1\}^\{n\}, JECS estimates the right\-tail mixture CDFG^n\\widehat\{G\}\_\{n\}on\[λ,1\]\[\\lambda,1\]and the boundary densityg^​\(λ\+\)\\widehat\{g\}\(\\lambda^\{\+\}\), both implemented by akk\-nearest\-neighbor estimator\. The fitted envelope at the anchor is

F^fit​\(λ\)=λ​g^​\(λ\+\)1\+λ​g^​\(λ\+\),\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)=\\frac\{\\lambda\\,\\widehat\{g\}\(\\lambda^\{\+\}\)\}\{1\+\\lambda\\,\\widehat\{g\}\(\\lambda^\{\+\}\)\},\(14\)and the global envelope is the piecewise reconstruction

F^fit​\(x\)=\{F^fit​\(λ\)λ​x,x∈\(0,λ\],F^fit​\(λ\)\+\(1−F^fit​\(λ\)\)​G^n​\(x\),x∈\(λ,1\]\.\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(x\)=\\begin\{cases\}\\dfrac\{\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)\}\{\\lambda\}\\,x,&x\\in\(0,\\lambda\],\\\\\[8\.99994pt\] \\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)\+\\bigl\(1\-\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)\\bigr\)\\widehat\{G\}\_\{n\}\(x\),&x\\in\(\\lambda,1\]\.\\end\{cases\}\(15\)
![Refer to caption](https://arxiv.org/html/2605.21543v1/x2.png)Figure 2:Fitted envelope schematic\. The fittedF^fit\\widehat\{F\}\_\{\\mathrm\{fit\}\}is a conservative estimate of the trueF0F\_\{0\}\.The left branch is linear with slopeF^fit​\(λ\)/λ\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)/\\lambda, whereas the right branch uses the empirical right\-tail CDF rescaled to match the anchor\.[Figure˜2](https://arxiv.org/html/2605.21543#S3.F2)illustrates the role of this fitted envelope: it maps the raw max\-ppvalues to a conservative CDF scale before BH\-style selection\. The thresholdλ\\lambdamarks the point beyond which the right tail is dominated by contaminated items; its data\-driven choice and sensitivity are deferred to[Section˜4](https://arxiv.org/html/2605.21543#S4)\. Each maximum\-ppvalue is then transformed through the envelope asp~i=F^fit​\(pi∗\)\\widetilde\{p\}\_\{i\}=\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(p\_\{i\}^\{\*\}\)\.

To improve power, we use Storey’s estimator\(Storey,[2002](https://arxiv.org/html/2605.21543#bib.bib47)\)on the transformed values\{p~i\}\\\{\\widetilde\{p\}\_\{i\}\\\}, using the null\-proportion estimate

π^0=1n​∑i=1n𝟙​\{pi∗\>λ\}1−F^fit​\(λ\),\\widehat\{\\pi\}\_\{0\}=\\frac\{\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\\{p\_\{i\}^\{\*\}\>\\lambda\\\}\}\{1\-\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)\},\(16\)and, withp~\(1\)≤⋯≤p~\(n\)\\widetilde\{p\}\_\{\(1\)\}\\leq\\cdots\\leq\\widetilde\{p\}\_\{\(n\)\}the sorted transformed values, returning

𝒮=\{i:p~i≤α​r∗π^0​n\},where​r∗=max⁡\{r:p~\(r\)≤α​rπ^0​n\},\\mathcal\{S\}=\\\!\\left\\\{i:\\widetilde\{p\}\_\{i\}\\leq\\frac\{\\alpha\\,r^\{\*\}\}\{\\widehat\{\\pi\}\_\{0\}\\,n\}\\right\\\},\\text\{ where \}r^\{\*\}=\\max\\\!\\left\\\{r:\\widetilde\{p\}\_\{\(r\)\}\\leq\\frac\{\\alpha\\,r\}\{\\widehat\{\\pi\}\_\{0\}\\,n\}\\right\\\},\(17\)withr∗=0r^\{\*\}=0and𝒮=∅\\mathcal\{S\}=\\emptysetwhen the set is empty\. Relative to JMCS, JECS replacespi∗p\_\{i\}^\{\*\}withp~i\\widetilde\{p\}\_\{i\}to mitigate the conservatism induced by super\-uniformity, and replaces the fixed choiceπ^0=1\\widehat\{\\pi\}\_\{0\}=1with the data\-adaptive estimate in[Equation˜16](https://arxiv.org/html/2605.21543#S3.E16)\. To establish the theoretical guarantee, we first state the key property motivating the construction ofF^fit​\(pi∗\)\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(p\_\{i\}^\{\*\}\):

###### Proposition 2\(Envelope domination\)\.

LetFfitF\_\{\\mathrm\{fit\}\}be the population counterpart of[Equation˜15](https://arxiv.org/html/2605.21543#S3.E15)\. ThenFfit​\(x\)≥F0​\(x\)F\_\{\\mathrm\{fit\}\}\(x\)\\geq F\_\{0\}\(x\)for allx∈\(0,1\]x\\in\(0,1\]\.

[Proposition˜2](https://arxiv.org/html/2605.21543#Thmtheorem2)shows thatFfitF\_\{\\mathrm\{fit\}\}dominatesF0F\_\{0\}pointwise\. Consequently, the transformed valuesp~i\\widetilde\{p\}\_\{i\}remain super\-uniform under the joint null, and the estimatorπ^0\\widehat\{\\pi\}\_\{0\}in[Equation˜16](https://arxiv.org/html/2605.21543#S3.E16)is conservative for the true null proportion\. Combined with the standard Storey\-BH argument, these two facts yield asymptotic GCR control, as formalized below:

###### Theorem 3\(Asymptotic GCR control\)\.

The selected set returned by JECS satisfies

lim supn→∞GCR​\(𝒮\)≤α\.\\limsup\_\{n\\to\\infty\}\\,\\mathrm\{GCR\}\(\\mathcal\{S\}\)\\leq\\alpha\.\(18\)

The proofs of[Propositions˜2](https://arxiv.org/html/2605.21543#Thmtheorem2)and[3](https://arxiv.org/html/2605.21543#Thmtheorem3)are deferred to[Sections˜B\.2](https://arxiv.org/html/2605.21543#A2.SS2)and[B\.3](https://arxiv.org/html/2605.21543#A2.SS3)\. We next empirically validate JECS’s GCR control\.

## 4Experimental results

In this section, we empirically evaluate JECS on training\-data\-detection benchmarks with a controlled member/non\-member structure, and verify its joint GCR control as well as its power against the JMCS baseline of[Section˜3\.2](https://arxiv.org/html/2605.21543#S3.SS2)\. Throughout, JECS is instantiated with the data\-driven right\-tail threshold rule and the Storey null\-proportion estimator of[Equation˜16](https://arxiv.org/html/2605.21543#S3.E16), which we adopt as the default configuration\.

### 4\.1Setup

#### Datasets and audited models\.

The main evaluation uses WikiMIA\(Shi et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib43)\)and ArXivTection\(Duarte et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib17)\)withK=16K=16independently fine\-tuned models initialized from GPT\-NeoX\-20B\(Black et al\.,[2022](https://arxiv.org/html/2605.21543#bib.bib9)\), Pythia\-6\.9B\(Biderman et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib8)\), and LLaMA\-7B\(Touvron et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib49)\); appendix experiments additionally cover five MIMIR\(Duan et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib16)\)subsets \(HackerNews, DMMath, GitHub, Pile\-CC, PubMed\) in[Section˜E\.2](https://arxiv.org/html/2605.21543#A5.SS2)and a mixed\-family audit pool spanning Pythia \(1\.41\.4B/2\.82\.8B/6\.96\.9B/1212B\), GPT\-NeoX\-20B, and LLaMA \(77B/1313B/3030B\) in[Section˜E\.5](https://arxiv.org/html/2605.21543#A5.SS5)\. The contamination\-controlled three\-block split, the resulting per\-instance membership labelsMikM\_\{i\}^\{k\}, the contamination fractionρ\\rho, and per\-dataset sample counts are deferred to[Section˜D\.2](https://arxiv.org/html/2605.21543#A4.SS2)and[Table˜4](https://arxiv.org/html/2605.21543#A4.T4)\.

#### Detection scores and baseline\.

We employ the per\-model scoreT​\(x;θk\)T\(x;\\theta\_\{k\}\)with four standard detectors: Perplexity\(Carlini et al\.,[2021](https://arxiv.org/html/2605.21543#bib.bib11)\), Min\-K%\(Shi et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib43)\), Min\-K%\+\+\(Zhang et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib62)\), and Modified Entropy\(Song and Mittal,[2021](https://arxiv.org/html/2605.21543#bib.bib45)\)\.

### 4\.2Main results

![Refer to caption](https://arxiv.org/html/2605.21543v1/x3.png)Figure 3:Joint GCR control with JECS atK=16K=16\. Each panel corresponds to a dataset–model family pair and reports realized GCP curves on the left axis and Power bars on the right axis for four detection scores under JECS; the dashed diagonal is the target GCR\.#### JECS controls GCR across detection scores and model families\.

We next evaluate JECS on the two real benchmarks across all three model families and four detection scores\.[Figure˜3](https://arxiv.org/html/2605.21543#S4.F3)reports realized GCP curves \(left axis\) and Power bars \(right axis\) over the various levelsα\\alphaatK=16K=16\. Throughout these evaluations, the realized GCP stays at or below the target diagonal\. The procedure thus delivers controlled joint pure subsets at the user\-specified level without configuration\-specific tuning, and the control is stable across detection scores\. The same control pattern is verified on five additional MIMIR subsets in[Section˜E\.2](https://arxiv.org/html/2605.21543#A5.SS2), and on a heterogeneous mixed\-family audit pool of Pythia, GPT\-NeoX, and LLaMA variants in[Section˜E\.5](https://arxiv.org/html/2605.21543#A5.SS5)\.

#### JECS attains higher Power than JMCS\.

[Table˜2](https://arxiv.org/html/2605.21543#S4.T2)reports the Power comparison between JECS and JMCS on ArXivTection atK=16K=16\. JECS consistently achieves higher Power than JMCS across target levels, model families, and detection scores, with the largest gains at smallerα\\alpha\. For example, under LLaMA\-7B atα=0\.1\\alpha=0\.1using Min\-K%, JECS improves Power from0\.0550\.055to0\.7150\.715over JMCS\. The corresponding results on WikiMIA are provided in[Section˜E\.3](https://arxiv.org/html/2605.21543#A5.SS3)\.

Table 2:Power on ArXivTection atK=16K=16\. Reported values are averagePower\\mathrm\{Power\}with standard error\. “Base” refers to JMCS and “Ours” refers to JECS\.Boldnumbers mark the better result in each pair\.

### 4\.3Sensitivity analysis

#### Effect of the right\-tail thresholdλ\\lambda

[Figure˜5](https://arxiv.org/html/2605.21543#S4.F5)sweeps the right\-tail thresholdλ∈\{0\.3,…,0\.9\}\\lambda\\in\\\{0\.3,\\ldots,0\.9\\\}on WikiMIA and ArXivTection \(NeoX\-20B,K=8K=8, Min\-K%\+\+,α=0\.1\\alpha=0\.1\) and overlays the data\-driven rule\. The adaptive choice tracks the best fixedλ\\lambdain power without producing any GCP violation\. This is also the empirical check for the right\-tail dominance condition behind[Theorem˜3](https://arxiv.org/html/2605.21543#Thmtheorem3): even at the smallestλ\\lambdatested, GCP stays belowα\\alpha, indicating that the right tail is contaminated\-dominated across the operating range\. The LLaMA\-7B counterparts are reported in[Appendix˜E](https://arxiv.org/html/2605.21543#A5)\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x4.png)Figure 4:GCR and Power curves overλ\\lambdawithα=0\.1\\alpha=0\.1\. The dashed line is the target GCR\.
![Refer to caption](https://arxiv.org/html/2605.21543v1/x5.png)Figure 5:GCP curve \(left axis\) and Power bars \(right axis\) under varying training fractionρ\\rho\.

#### Impact of the number of audited modelsKKon control and power\.

[Figure˜6](https://arxiv.org/html/2605.21543#S4.F6)variesK∈\{2,4,8,16\}K\\in\\\{2,4,8,16\\\}on WikiMIA across all three model families with Min\-K%\+\+\. GCP stays under the target diagonal asKKgrows, while Power does not collapse withKK\. Our claim is not that every Power curve is strictly monotone inKK, but that the procedure remains controlled as the number of jointly audited models grows; individual power trends can be non\-monotone because, by[Equation˜43](https://arxiv.org/html/2605.21543#A4.E43), the joint\-purity prevalence in the test mixture depends onKKthrough the term\(1−ρ\)K\(1\-\\rho\)^\{K\}from the remainder block while the positives in the globally pure block stay fixed\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x6.png)Figure 6:Sensitivity to the number of audited modelsKKon WikiMIA\. GCP \(lines, left axis\) and Power \(bars, right axis\) with Min\-K%\+\+\.
#### Robustness of error control to variations in the training fractionρ\\rho\.

The training fractionρ\\rhocontrols the contamination level in our experiments: it is the per\-model sampling fraction within the remainder block of the controlled three\-block split, as defined in[Section˜D\.2](https://arxiv.org/html/2605.21543#A4.SS2)\.[Figure˜5](https://arxiv.org/html/2605.21543#S4.F5)sweepsρ∈\{0\.125,0\.25,0\.5,0\.75\}\\rho\\in\\\{0\.125,0\.25,0\.5,0\.75\\\}on NeoX\-20B atK=16K=16with Min\-K%\+\+, on both WikiMIA and ArXivTection\. Across all four values ofρ\\rhoand both datasets, the realized GCP stays at or below the target diagonal, confirming that JECS’s control is robust to the contamination level\.

## 5Related Work

#### Benchmark data contamination\.

Benchmark data contamination occurs when evaluation examples overlap with a model’s training data, which can inflate reported performance\(Sainz et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib42); Balloccu et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib4); Xu et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib55); Li,[2024](https://arxiv.org/html/2605.21543#bib.bib30); Bordt et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib10); Cheng et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib13)\)\. One line of work reduces this risk before evaluation through dynamic benchmark construction\(Zhu et al\.,[2024a](https://arxiv.org/html/2605.21543#bib.bib66); Li et al\.,[2024a](https://arxiv.org/html/2605.21543#bib.bib31); White et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib53); Srivastava et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib46)\), data rewriting\(Yang et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib56); Zhu et al\.,[2024b](https://arxiv.org/html/2605.21543#bib.bib67)\), or contamination\-free benchmark curation\(Zhao et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib64)\)\. Another line detects contamination after models and benchmarks already exist\. Most methods in this setting operate at the sample level, inferring whether a particular example was used for training; such methods have been studied for LLMs\(Carlini et al\.,[2021](https://arxiv.org/html/2605.21543#bib.bib11); Shi et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib43); Zhang et al\.,[2024b](https://arxiv.org/html/2605.21543#bib.bib63); Golchin and Surdeanu,[2024](https://arxiv.org/html/2605.21543#bib.bib19); Li and Flanigan,[2024](https://arxiv.org/html/2605.21543#bib.bib29); Zhang et al\.,[2025a](https://arxiv.org/html/2605.21543#bib.bib60); Hu et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib23); Zhang et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib62); Mattern et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib37); Xie et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib54); Zhang et al\.,[2024b](https://arxiv.org/html/2605.21543#bib.bib63); Raoof et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib41); Kaneko et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib26); Yi and Li,[2026](https://arxiv.org/html/2605.21543#bib.bib58)\), vision language models\(Ko et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib28); Li et al\.,[2024c](https://arxiv.org/html/2605.21543#bib.bib33); Hu et al\.,[2025a](https://arxiv.org/html/2605.21543#bib.bib22); LIU et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib34)\)\. Dataset\-level detection methods instead test for contamination at a broader dataset scale\(Vu et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib52); Oren et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib38); Ko et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib28); Li et al\.,[2024b](https://arxiv.org/html/2605.21543#bib.bib32); Maini et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib36); Zhang et al\.,[2024a](https://arxiv.org/html/2605.21543#bib.bib61); Choi et al\.,[2025](https://arxiv.org/html/2605.21543#bib.bib14); Zawalski et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib59)\)\. Both lines of work target a single audited model or a single dataset and do not return a shared item\-level decontaminated subset across multiple audited models\. In contrast, JECS aggregates per\-model instance\-level conformalpp\-values into a joint selection procedure that returns one benchmark shared byKKaudited models with provable GCR control\.

#### Conformal selection\.

Selective inference with false discovery rate control is rooted in the Benjamini–Hochberg \(BH\) procedure on validpp\-values\(Benjamini and Hochberg,[1995](https://arxiv.org/html/2605.21543#bib.bib6); Benjamini and Yekutieli,[2001](https://arxiv.org/html/2605.21543#bib.bib7); Storey et al\.,[2004](https://arxiv.org/html/2605.21543#bib.bib48)\)\. Conformal prediction provides a distribution\-free route to suchpp\-values under exchangeability\(Vovk et al\.,[2005](https://arxiv.org/html/2605.21543#bib.bib51); Papadopoulos,[2008](https://arxiv.org/html/2605.21543#bib.bib39)\), and recent work combines conformal calibration with BH\-type selection to obtain finite\-sample selection guarantees\(Bates et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib5); Jin and Candès,[2023](https://arxiv.org/html/2605.21543#bib.bib24)\)\. Subsequent extensions address more constrained settings, including covariate shift through weighted conformal scores\(Jin and Candès,[2025](https://arxiv.org/html/2605.21543#bib.bib25)\), multivariate responses through vector\-valued conformity scores\(Bai et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib3)\), and multiple response conditions through conditional conformity constructions\(Hao et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib21)\)\. These ideas have also been applied to LLM alignment\(Gui et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib20)\), drug discovery\(Bai et al\.,[2025a](https://arxiv.org/html/2605.21543#bib.bib2)\), and training\-data identification\(Liu et al\.,[2026](https://arxiv.org/html/2605.21543#bib.bib35)\)\. The closest formal comparison is multivariate conformal selection\(Bai et al\.,[2025b](https://arxiv.org/html/2605.21543#bib.bib3)\), where the vector structure comes from multiple response coordinates within one prediction problem\. In joint benchmark decontamination, the multiplicity comes from audited models: each item hasKKmembership statuses, and it is contaminated if any audited model trained on it\. This yields a joint conformal selection problem under a union null: the procedure must aggregate dependent model\-wise conformal evidence into one item\-level ranking while controlling the expected fraction of union\-null items among selected items\.

## 6Conclusion

In this paper, we formalize joint benchmark decontamination as a joint selection problem, where the goal is to select one shared benchmark whose global contamination rate is controlled across all audited models\. We propose Joint Envelope Conformal Selection \(JECS\), which computes model\-wise conformalpp\-values, aggregates them through a max\-ppstatistic, reconstructs a conservative envelope for the max\-ppnull distribution, and applies adaptive Storey\-BH to the transformed values\. Experiments on synthetic data and LLM benchmarks show that JECS maintains empirical contamination control across datasets, model families, and detection scores, while improving power over JMCS\. The method is agnostic to the choice of detection score and applies beyond LLM evaluation whenever shared\-member calibration data are available\.

#### Limitations\.

The current procedure relies on a calibration set whose distribution is close to the candidate pool, which is a standard condition for conformal calibration\. This leaves room to study distribution shift more explicitly in future work\. Extending JECS with shift\-aware conformalpp\-values could broaden its use when the available calibration data only approximately matches the benchmark candidate distribution\.

## References

- Achiam et al\. \[2023\]Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al\.Gpt\-4 technical report\.*arXiv preprint arXiv:2303\.08774*, 2023\.
- Bai et al\. \[2025a\]Tian Bai, Peng Tang, Yuting Xu, Vladimir Svetnik, Bingjia Yang, Abbas Khalili, Xiang Yu, and Archer Y Yang\.Conformal selection for efficient and accurate compound screening in drug discovery\.*Journal of Chemical Information and Modeling*, 65\(24\):13070–13085, 2025a\.
- Bai et al\. \[2025b\]Tian Bai, Yue Zhao, Xiang Yu, and Archer Y\. Yang\.Multivariate conformal selection\.In*Forty\-second International Conference on Machine Learning*, 2025b\.URL[https://openreview\.net/forum?id=g2tr7nA4pS](https://openreview.net/forum?id=g2tr7nA4pS)\.
- Balloccu et al\. \[2024\]Simone Balloccu, Patrícia Schmidtová, Mateusz Lango, and Ondřej Dušek\.Leak, cheat, repeat: Data contamination and evaluation malpractices in closed\-source llms\.In*Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics \(Volume 1: Long Papers\)*, pages 67–93, 2024\.
- Bates et al\. \[2023\]Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia\.Testing for outliers with conformal p\-values\.*The Annals of Statistics*, 51\(1\):149–178, 2023\.
- Benjamini and Hochberg \[1995\]Yoav Benjamini and Yosef Hochberg\.Controlling the false discovery rate: a practical and powerful approach to multiple testing\.*Journal of the Royal Statistical Society*, 57\(1\):289–300, 1995\.
- Benjamini and Yekutieli \[2001\]Yoav Benjamini and Daniel Yekutieli\.The control of the false discovery rate in multiple testing under dependency\.*Annals of Statistics*, pages 1165–1188, 2001\.
- Biderman et al\. \[2023\]Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al\.Pythia: A suite for analyzing large language models across training and scaling\.In*International Conference on Machine Learning*, pages 2397–2430\. PMLR, 2023\.
- Black et al\. \[2022\]Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach\.GPT\-NeoX\-20B: An open\-source autoregressive language model\.In Angela Fan, Suzana Ilic, Thomas Wolf, and Matthias Gallé, editors,*Proceedings of BigScience Episode \#5 – Workshop on Challenges & Perspectives in Creating Large Language Models*, pages 95–136, virtual\+Dublin, May 2022\. Association for Computational Linguistics\.doi:10\.18653/v1/2022\.bigscience\-1\.9\.URL[https://aclanthology\.org/2022\.bigscience\-1\.9/](https://aclanthology.org/2022.bigscience-1.9/)\.
- Bordt et al\. \[2025\]Sebastian Bordt, Suraj Srinivas, Valentyn Boreiko, and Ulrike von Luxburg\.How much can we forget about data contamination?In*ICML*, Proceedings of Machine Learning Research\. PMLR / OpenReview\.net, 2025\.
- Carlini et al\. \[2021\]Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert\-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al\.Extracting training data from large language models\.In*USENIX Security Symposium*, pages 2633–2650, 2021\.
- Chang et al\. \[2024\]Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al\.A survey on evaluation of large language models\.*ACM transactions on intelligent systems and technology*, 15\(3\):1–45, 2024\.
- Cheng et al\. \[2025\]Yuxing Cheng, Yi Chang, and Yuan Wu\.A survey on data contamination for large language models\.*arXiv preprint arXiv:2502\.14425*, 2025\.
- Choi et al\. \[2025\]Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li\.How contaminated is your benchmark? measuring dataset leakage in large language models with kernel divergence\.In*Forty\-second International Conference on Machine Learning*, 2025\.URL[https://openreview\.net/forum?id=wVDR2qmE28](https://openreview.net/forum?id=wVDR2qmE28)\.
- Deng et al\. \[2024\]Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan\.Investigating data contamination in modern benchmarks for large language models\.In*Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies \(Volume 1: Long Papers\)*, pages 8706–8719, 2024\.
- Duan et al\. \[2024\]Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi\.Do membership inference attacks work on large language models?In*Conference on Language Modeling*, 2024\.
- Duarte et al\. \[2024\]André Vicente Duarte, Xuandong Zhao, Arlindo L\. Oliveira, and Lei Li\.DE\-COP: Detecting copyrighted content in language models training data\.In*International Conference on Machine Learning*, 2024\.URL[https://openreview\.net/forum?id=LO4xhXmFal](https://openreview.net/forum?id=LO4xhXmFal)\.
- European Parliament and Council of the European Union \[2024\]European Parliament and Council of the European Union\.Regulation \(EU\) 2024/1689 of the European Parliament and of the Council laying down harmonised rules on artificial intelligence \(Artificial Intelligence Act\)\.Official Journal of the European Union, OJ L, 2024/1689, 12 July 2024, July 2024\.URL[http://data\.europa\.eu/eli/reg/2024/1689/oj](http://data.europa.eu/eli/reg/2024/1689/oj)\.CELEX: 32024R1689\.
- Golchin and Surdeanu \[2024\]Shahriar Golchin and Mihai Surdeanu\.Time travel in LLMs: Tracing data contamination in large language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=2Rwq6c3tvr](https://openreview.net/forum?id=2Rwq6c3tvr)\.
- Gui et al\. \[2024\]Yu Gui, Ying Jin, and Zhimei Ren\.Conformal alignment: Knowing when to trust foundation models with guarantees\.*Advances in Neural Information Processing Systems*, 37:73884–73919, 2024\.
- Hao et al\. \[2026\]Qingyang Hao, Wenbo Liao, Bingyi Jing, and Hongxin Wei\.Multi\-condition conformal selection\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=giL8Q1V26J](https://openreview.net/forum?id=giL8Q1V26J)\.
- Hu et al\. \[2025a\]Yuke Hu, Zheng Li, Zhihao Liu, Yang Zhang, Zhan Qin, Kui Ren, and Chun Chen\.Membership inference attacks against vision\-language models\.In*USENIX Security Symposium*, pages 1589–1608\. USENIX Association, 2025a\.
- Hu et al\. \[2025b\]Zirui Hu, Yingjie Wang, Zheng Zhang, Hong Chen, and Dacheng Tao\.A statistical approach for controlled training data detection\.In*International Conference on Learning Representations*, 2025b\.URL[https://openreview\.net/forum?id=XAN8G0rvoB](https://openreview.net/forum?id=XAN8G0rvoB)\.
- Jin and Candès \[2023\]Ying Jin and Emmanuel J Candès\.Selection by prediction with conformal p\-values\.*Journal of Machine Learning Research*, 24\(244\):1–41, 2023\.
- Jin and Candès \[2025\]Ying Jin and Emmanuel J Candès\.Model\-free selective inference under covariate shift via weighted conformal p\-values\.*Biometrika*, page asaf066, 2025\.
- Kaneko et al\. \[2025\]Masahiro Kaneko, Youmi Ma, Yuki Wata, and Naoaki Okazaki\.Sampling\-based pseudo\-likelihood for membership inference attacks\.In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,*Findings of the Association for Computational Linguistics: ACL 2025*, pages 8894–8907, Vienna, Austria, July 2025\. Association for Computational Linguistics\.ISBN 979\-8\-89176\-256\-5\.doi:10\.18653/v1/2025\.findings\-acl\.465\.URL[https://aclanthology\.org/2025\.findings\-acl\.465/](https://aclanthology.org/2025.findings-acl.465/)\.
- Kaplan et al\. \[2020\]Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei\.Scaling laws for neural language models\.*arXiv preprint arXiv:2001\.08361*, 2020\.
- Ko et al\. \[2023\]Myeongseob Ko, Ming Jin, Chenguang Wang, and Ruoxi Jia\.Practical membership inference attacks against large\-scale multi\-modal models: A pilot study\.In*Proceedings of the IEEE/CVF International Conference on Computer Vision*, pages 4871–4881, 2023\.
- Li and Flanigan \[2024\]Changmao Li and Jeffrey Flanigan\.Task contamination: language models may not be few\-shot anymore\.In*Proceedings of the Thirty\-Eighth AAAI Conference on Artificial Intelligence and Thirty\-Sixth Conference on Innovative Applications of Artificial Intelligence and Fourteenth Symposium on Educational Advances in Artificial Intelligence*, AAAI’24/IAAI’24/EAAI’24\. AAAI Press, 2024\.ISBN 978\-1\-57735\-887\-9\.doi:10\.1609/aaai\.v38i16\.29808\.URL[https://doi\.org/10\.1609/aaai\.v38i16\.29808](https://doi.org/10.1609/aaai.v38i16.29808)\.
- Li \[2024\]Yanyang Li\.Awesome data contamination\.[https://github\.com/lyy1994/awesome\-data\-contamination](https://github.com/lyy1994/awesome-data-contamination), 2024\.
- Li et al\. \[2024a\]Yucheng Li, Frank Guerin, and Chenghua Lin\.Latesteval: Addressing data contamination in language model evaluation through dynamic and time\-sensitive test construction\.In*AAAI Conference on Artificial Intelligence*, pages 18600–18607\. AAAI Press, 2024a\.
- Li et al\. \[2024b\]Yucheng Li, Yunhao Guo, Frank Guerin, and Chenghua Lin\.An open\-source data contamination report for large language models\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 528–541, Miami, Florida, USA, November 2024b\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-emnlp\.30\.URL[https://aclanthology\.org/2024\.findings\-emnlp\.30/](https://aclanthology.org/2024.findings-emnlp.30/)\.
- Li et al\. \[2024c\]Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and Volkan Cevher\.Membership inference attacks against large vision\-language models\.In*The Thirty\-eighth Annual Conference on Neural Information Processing Systems*, 2024c\.URL[https://openreview\.net/forum?id=nv2Qt5cj1a](https://openreview.net/forum?id=nv2Qt5cj1a)\.
- LIU et al\. \[2026\]Yihao LIU, Xinqi LYU, Dong Wang, Yanjie Li, and Bin Xiao\.LOMIA: Label\-only membership inference attacks against pre\-trained large vision\-language models\.In*The Thirty\-ninth Annual Conference on Neural Information Processing Systems*, 2026\.URL[https://openreview\.net/forum?id=7JjS2cdBYN](https://openreview.net/forum?id=7JjS2cdBYN)\.
- Liu et al\. \[2026\]Zhenlong Liu, Hao Zeng, Weiran Huang, and Hongxin Wei\.Provable training data identification for large language models\.In*Forty\-third International Conference on Machine Learning*, 2026\.URL[https://arxiv\.org/abs/2510\.09717](https://arxiv.org/abs/2510.09717)\.arXiv preprint arXiv:2510\.09717\.
- Maini et al\. \[2024\]Pratyush Maini, Hengrui Jia, Nicolas Papernot, and Adam Dziedzic\.Llm dataset inference: Did you train on my dataset?*Advances in Neural Information Processing Systems*, 37:124069–124092, 2024\.
- Mattern et al\. \[2023\]Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg\-Kirkpatrick\.Membership inference attacks against language models via neighbourhood comparison\.In Anna Rogers, Jordan Boyd\-Graber, and Naoaki Okazaki, editors,*Findings of the Association for Computational Linguistics: ACL 2023*, pages 11330–11343, Toronto, Canada, July 2023\. Association for Computational Linguistics\.doi:10\.18653/v1/2023\.findings\-acl\.719\.URL[https://aclanthology\.org/2023\.findings\-acl\.719/](https://aclanthology.org/2023.findings-acl.719/)\.
- Oren et al\. \[2024\]Yonatan Oren, Nicole Meister, Niladri S\. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto\.Proving test set contamination in black\-box language models\.In*The Twelfth International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=KS8mIvetg2](https://openreview.net/forum?id=KS8mIvetg2)\.
- Papadopoulos \[2008\]Harris Papadopoulos\.*Inductive conformal prediction: Theory and application to neural networks*\.INTECH Open Access Publisher Rijeka, 2008\.
- Penedo et al\. \[2023\]Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay\.The refinedweb dataset for falcon llm: Outperforming curated corpora with web data only\.*Advances in Neural Information Processing Systems*, 36:79155–79172, 2023\.
- Raoof et al\. \[2025\]Negin Raoof, Litu Rout, Giannis Daras, Sujay Sanghavi, Constantine Caramanis, Sanjay Shakkottai, and Alex Dimakis\.Infilling score: A pretraining data detection algorithm for large language models\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=9QPH1YQCMn](https://openreview.net/forum?id=9QPH1YQCMn)\.
- Sainz et al\. \[2023\]Oscar Sainz, Jon Campos, Iker García\-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre\.Nlp evaluation in trouble: On the need to measure llm data contamination for each benchmark\.In*Findings of the Association for Computational Linguistics: EMNLP 2023*, pages 10776–10787, 2023\.
- Shi et al\. \[2024\]Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer\.Detecting pretraining data from large language models\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=zWqr3MQuNs](https://openreview.net/forum?id=zWqr3MQuNs)\.
- Shokri et al\. \[2017\]Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov\.Membership inference attacks against machine learning models\.In*IEEE Symposium on Security and Privacy*, pages 3–18\. IEEE, 2017\.
- Song and Mittal \[2021\]Liwei Song and Prateek Mittal\.Systematic evaluation of privacy risks of machine learning models\.In*USENIX Security Symposium*, pages 2615–2632, 2021\.
- Srivastava et al\. \[2026\]Gaurav Srivastava, Aafiya Shamshad Hussain, Zhenyu Bi, Swastik Roy, Priya Pitre, Meng Lu, Morteza Ziyadi, and Xuan Wang\.Beyondbench: Contamination\-resistant evaluation of reasoning in language models\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=mIKqVWGjwI](https://openreview.net/forum?id=mIKqVWGjwI)\.
- Storey \[2002\]John D Storey\.A direct approach to false discovery rates\.*Journal of the Royal Statistical Society Series B: Statistical Methodology*, 64\(3\):479–498, 2002\.
- Storey et al\. \[2004\]John D Storey, Jonathan E Taylor, and David Siegmund\.Strong control, conservative point estimation and simultaneous conservative consistency of false discovery rates: a unified approach\.*Journal of the Royal Statistical Society Series B: Statistical Methodology*, 66\(1\):187–205, 2004\.
- Touvron et al\. \[2023\]Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie\-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al\.Llama: Open and efficient foundation language models\.*arXiv preprint arXiv:2302\.13971*, 2023\.
- Vovk et al\. \[2003\]Vladimir Vovk, Ilia Nouretdinov, and Alex Gammerman\.Testing exchangeability on\-line\.In*International Conference on Machine Learning*, pages 768–775\. AAAI Press, 2003\.
- Vovk et al\. \[2005\]Vladimir Vovk, Alexander Gammerman, and Glenn Shafer\.*Algorithmic learning in a random world*\.Springer, 2005\.
- Vu et al\. \[2023\]Thuy\-Trang Vu, Xuanli He, Gholamreza Haffari, and Ehsan Shareghi\.Koala: An index for quantifying overlaps with pre\-training corpora\.In*EMNLP \(Demos\)*, pages 90–98\. Association for Computational Linguistics, 2023\.
- White et al\. \[2025\]Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz\-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh\-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum\.Livebench: A challenging, contamination\-limited LLM benchmark\.In*The Thirteenth International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=sKYHBTAxVa](https://openreview.net/forum?id=sKYHBTAxVa)\.
- Xie et al\. \[2024\]Roy Xie, Junlin Wang, Ruomin Huang, Minxing Zhang, Rong Ge, Jian Pei, Neil Zhenqiang Gong, and Bhuwan Dhingra\.ReCaLL: Membership inference via relative conditional log\-likelihoods\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 8671–8689, Miami, Florida, USA, November 2024\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-main\.493\.URL[https://aclanthology\.org/2024\.emnlp\-main\.493/](https://aclanthology.org/2024.emnlp-main.493/)\.
- Xu et al\. \[2024\]Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al\.Benchmark data contamination of large language models: A survey\.*arXiv preprint arXiv:2406\.04244*, 2024\.
- Yang et al\. \[2023\]Shuo Yang, Wei\-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica\.Rethinking benchmark and contamination for language models with rephrased samples\.*arXiv preprint arXiv:2311\.04850*, 2023\.
- Ye et al\. \[2022\]Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri\.Enhanced membership inference attacks against machine learning models\.In*ACM SIGSAC Conference on Computer and Communications Security*, pages 3093–3106, 2022\.
- Yi and Li \[2026\]Jiatong Yi and Yanyang Li\.Membership inference on llms in the wild, 2026\.URL[https://arxiv\.org/abs/2601\.11314](https://arxiv.org/abs/2601.11314)\.
- Zawalski et al\. \[2026\]Michał Zawalski, Meriem Boubdir, Klaudia Bałazy, Besmira Nushi, and Pablo Ribalta\.Detecting data contamination in LLMs via in\-context learning\.In*The Fourteenth International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=YlpaaYxx4t](https://openreview.net/forum?id=YlpaaYxx4t)\.
- Zhang et al\. \[2025a\]Hengxiang Zhang, Songxin Zhang, Bingyi Jing, and Hongxin Wei\.Fine\-tuning can help detect pretraining data from large language models\.In*International Conference on Learning Representations*, 2025a\.URL[https://openreview\.net/forum?id=X8dzvdkQwO](https://openreview.net/forum?id=X8dzvdkQwO)\.
- Zhang et al\. \[2024a\]Huixuan Zhang, Yun Lin, and Xiaojun Wan\.PaCoST: Paired confidence significance testing for benchmark contamination detection in large language models\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Findings of the Association for Computational Linguistics: EMNLP 2024*, pages 1794–1809, Miami, Florida, USA, November 2024a\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-emnlp\.97\.URL[https://aclanthology\.org/2024\.findings\-emnlp\.97/](https://aclanthology.org/2024.findings-emnlp.97/)\.
- Zhang et al\. \[2025b\]Jingyang Zhang, Jingwei Sun, Eric Yeats, Yang Ouyang, Martin Kuo, Jianyi Zhang, Hao Frank Yang, and Hai Li\.Min\-k%\+\+: Improved baseline for pre\-training data detection from large language models\.In*International Conference on Learning Representations*, 2025b\.URL[https://openreview\.net/forum?id=ZGkfoufDaU](https://openreview.net/forum?id=ZGkfoufDaU)\.
- Zhang et al\. \[2024b\]Weichao Zhang, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng\.Pretraining data detection for large language models: A divergence\-based calibration method\.In Yaser Al\-Onaizan, Mohit Bansal, and Yun\-Nung Chen, editors,*Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 5263–5274, Miami, Florida, USA, November 2024b\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.emnlp\-main\.300\.URL[https://aclanthology\.org/2024\.emnlp\-main\.300/](https://aclanthology.org/2024.emnlp-main.300/)\.
- Zhao et al\. \[2024\]Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, et al\.Mmlu\-cf: A contamination\-free multi\-task language understanding benchmark\.*arXiv preprint arXiv:2412\.15194*, 2024\.
- Zhou et al\. \[2023\]Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji\-Rong Wen, and Jiawei Han\.Don’t make your llm an evaluation benchmark cheater\.*arXiv preprint arXiv:2311\.01964*, 2023\.
- Zhu et al\. \[2024a\]Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie\.Dyval: Dynamic evaluation of large language models for reasoning tasks\.In*The Twelfth International Conference on Learning Representations*, 2024a\.URL[https://openreview\.net/forum?id=gjfOL9z5Xr](https://openreview.net/forum?id=gjfOL9z5Xr)\.
- Zhu et al\. \[2024b\]Wenhong Zhu, Hongkun Hao, Zhiwei He, Yun\-Ze Song, Jiao Yueyang, Yumeng Zhang, Hanxu Hu, Yiran Wei, Rui Wang, and Hongyuan Lu\.CLEAN–EVAL: Clean evaluation on contaminated large language models\.In Kevin Duh, Helena Gomez, and Steven Bethard, editors,*Findings of the Association for Computational Linguistics: NAACL 2024*, pages 835–847, Mexico City, Mexico, June 2024b\. Association for Computational Linguistics\.doi:10\.18653/v1/2024\.findings\-naacl\.53\.URL[https://aclanthology\.org/2024\.findings\-naacl\.53/](https://aclanthology.org/2024.findings-naacl.53/)\.

## Appendix ATable of Notations

Table 3:Summary of notations used in the paper\.
## Appendix BTheoretical analysis

This appendix states the formal assumptions used and gives the proof\. Letπ0\\pi\_\{0\}and1−π01\-\\pi\_\{0\}denote the proportions of contaminated and jointly pure candidate items, respectively\. LetF0,F1,FmixF\_\{0\},F\_\{1\},F\_\{\\mathrm\{mix\}\}andf0,f1,fmixf\_\{0\},f\_\{1\},f\_\{\\mathrm\{mix\}\}be the corresponding CDFs and densities ofpi∗p\_\{i\}^\{\*\}\.

### B\.1Assumptions and their justification

###### Assumption 4\(Convex null max\-ppCDF\)\.

The null CDFF0F\_\{0\}ofpi∗p\_\{i\}^\{\*\}is convex on\[0,1\]\[0,1\]\.

#### Justification\.

The convexity condition is natural under max aggregation with heterogeneous ensemble exposure\. For a contaminated item, let𝒦i0=\{k:Mik=1\},c=\|𝒦i0\|≥1\\mathcal\{K\}\_\{i\}^\{0\}=\\\{k:M\_\{i\}^\{k\}=1\\\},\\qquad c=\|\\mathcal\{K\}\_\{i\}^\{0\}\|\\geq 1be the set and number of audited models whose training data contain itemii\. The remainingK−cK\-cmodels treat the item as unseen and tend to produce small conformalpp\-values\. Because JECS aggregates bypi∗=maxk⁡pikp\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}, these small values are masked in the right tail\. Thus, on the operating interval\[λ,1\]\[\\lambda,1\], the null CDF is governed by the maximum over theccmodels whose training data contain itemii\.

Two benchmark dependence regimes illustrate the shape of this CDF\. If the nullpp\-values from theccmodels whose training data contain itemiiare independent and marginally uniform, then

F0,c​\(x\)=ℙ​\(maxℓ≤c⁡pℓ≤x\)=xc,x∈\[λ,1\],F\_\{0,c\}\(x\)=\\mathbb\{P\}\\\!\\left\(\\max\_\{\\ell\\leq c\}p\_\{\\ell\}\\leq x\\right\)=x^\{c\},\\qquad x\\in\[\\lambda,1\],\(19\)which is strictly convex forc≥2c\\geq 2\. At the other extreme, if theseccnullpp\-values are perfectly positively dependent, they move together and the maximum has CDF

F0,c​\(x\)=x,F\_\{0,c\}\(x\)=x,\(20\)which is linear and hence still convex\. This case also satisfies the derivative condition used in the proof as an equality\.

A positive\-dependence model interpolates between these two extremes\. LetCco,cC\_\{\\mathrm\{co\},c\}denote the comonotonic copula, which represents perfect positive dependence, and letCind,cC\_\{\\mathrm\{ind\},c\}denote the independence copula\. Under perfect positive dependence, theccmarginally uniform nullpp\-values can be represented by a shared latent variableZ∼Unif​\(0,1\)Z\\sim\\mathrm\{Unif\}\(0,1\), namelyU1=⋯=Uc=ZU\_\{1\}=\\cdots=U\_\{c\}=Z\. Hence

ℙ​\(U1≤u1,…,Uc≤uc\)=ℙ​\(Z≤minℓ≤c⁡uℓ\)=minℓ≤c⁡uℓ\.\\mathbb\{P\}\(U\_\{1\}\\leq u\_\{1\},\\ldots,U\_\{c\}\\leq u\_\{c\}\)=\\mathbb\{P\}\\\!\\left\(Z\\leq\\min\_\{\\ell\\leq c\}u\_\{\\ell\}\\right\)=\\min\_\{\\ell\\leq c\}u\_\{\\ell\}\.\(21\)Therefore

Cco,c​\(u1,…,uc\)=minℓ≤c⁡uℓ,Cind,c​\(u1,…,uc\)=∏ℓ=1cuℓ\.C\_\{\\mathrm\{co\},c\}\(u\_\{1\},\\ldots,u\_\{c\}\)=\\min\_\{\\ell\\leq c\}u\_\{\\ell\},\\qquad C\_\{\\mathrm\{ind\},c\}\(u\_\{1\},\\ldots,u\_\{c\}\)=\\prod\_\{\\ell=1\}^\{c\}u\_\{\\ell\}\.\(22\)Forη∈\[0,1\]\\eta\\in\[0,1\], consider the convex combination

Cη,c=η​Cco,c\+\(1−η\)​Cind,c\.C\_\{\\eta,c\}=\\eta C\_\{\\mathrm\{co\},c\}\+\(1\-\\eta\)C\_\{\\mathrm\{ind\},c\}\.\(23\)Evaluating this copula on the diagonal gives

F0,c​\(x\)=Cη,c​\(x,…,x\)=η​x\+\(1−η\)​xc\.F\_\{0,c\}\(x\)=C\_\{\\eta,c\}\(x,\\ldots,x\)=\\eta x\+\(1\-\\eta\)x^\{c\}\.\(24\)Therefore

F0,c′′​\(x\)=\(1−η\)​c​\(c−1\)​xc−2≥0F\_\{0,c\}^\{\\prime\\prime\}\(x\)=\(1\-\\eta\)c\(c\-1\)x^\{c\-2\}\\geq 0\(25\)forc≥2c\\geq 2, while the casesc=1c=1orη=1\\eta=1reduce to the linear CDFF0,c​\(x\)=xF\_\{0,c\}\(x\)=x\. If the exposure frequency varies across contaminated items, the overall null CDF is a mixture

F0​\(x\)=∑c=1Kwc​F0,c​\(x\),wc≥0,∑c=1Kwc=1,F\_\{0\}\(x\)=\\sum\_\{c=1\}^\{K\}w\_\{c\}F\_\{0,c\}\(x\),\\qquad w\_\{c\}\\geq 0,\\quad\\sum\_\{c=1\}^\{K\}w\_\{c\}=1,\(26\)and convexity is preserved under this mixture\. This provides a structural explanation for[˜4](https://arxiv.org/html/2605.21543#Thmtheorem4): the max operator removes the influence of models whose training data do not contain the item in the right tail, and the dependence among models whose training data contain the item ranges from independence, which givesxcx^\{c\}, to perfect positive dependence, which givesxx\.

###### Assumption 5\(Distinguishability of pure and contaminated items\)\.

On the truncated interval\[λ,1\]\[\\lambda,1\], the right\-tail conditional CDF of pure items is at least that of contaminated items:

G1​\(x\)≥G0​\(x\),x∈\[λ,1\]\.G\_\{1\}\(x\)\\geq G\_\{0\}\(x\),\\qquad x\\in\[\\lambda,1\]\.\(27\)

#### Justification\.

The inequality is a distinguishability statement\. Recalling thatG0,G1G\_\{0\},G\_\{1\}are right\-tail conditional CDFs \([Equation˜13](https://arxiv.org/html/2605.21543#S3.E13)\), the assumptionG1​\(x\)≥G0​\(x\)G\_\{1\}\(x\)\\geq G\_\{0\}\(x\)on\[λ,1\]\[\\lambda,1\]is equivalent to

Pr⁡\(pi∗\>x​\|pi∗\>​λ,H1,i​true\)≤Pr⁡\(pi∗\>x​\|pi∗\>​λ,H0,i​true\),x∈\[λ,1\],\\Pr\\\!\\big\(p\_\{i\}^\{\*\}\>x\\,\\big\|\\,p\_\{i\}^\{\*\}\>\\lambda,\\ H\_\{1,i\}\\text\{ true\}\\big\)\\;\\leq\\;\\Pr\\\!\\big\(p\_\{i\}^\{\*\}\>x\\,\\big\|\\,p\_\{i\}^\{\*\}\>\\lambda,\\ H\_\{0,i\}\\text\{ true\}\\big\),\\qquad x\\in\[\\lambda,1\],i\.e\., conditional on having crossedλ\\lambda, a pure item is less likely than a contaminated item to escape further into the upper tail at every levelxx\. Equivalently, the conditional law ofpi∗p\_\{i\}^\{\*\}givenpi∗\>λp\_\{i\}^\{\*\}\>\\lambdais stochastically smaller for pure items than for contaminated items, so the two populations remain distinguishable even after both have entered the right tail\.

This assumption formalizes a one\-sided separation within the right tail\. Conditional on having crossedλ\\lambda, pure items are expected to produce shallower exceedances, whereas contaminated items are expected to have a heavier upper tail\. In the joint setting, this behavior is natural because an item contaminated for any audited model can receive a large model\-wisepikp\_\{i\}^\{k\}, while a jointly pure item must appear non\-member\-like across all models\. Thus, among items withpi∗\>λp\_\{i\}^\{\*\}\>\\lambda, pure items concentrate closer toλ\\lambda, whereas contaminated items are more likely to extend toward11\.

###### Proposition 6\.

Suppose that the candidate maximapi∗p\_\{i\}^\{\*\}are i\.i\.d\. fromFmixF\_\{\\mathrm\{mix\}\}\. The thresholdλ∈\(0,1\)\\lambda\\in\(0,1\)is fixed,1−Fmix​\(λ\)\>01\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\>0, and the conditional right\-tail distribution

Gmix​\(x\)=Fmix​\(x\)−Fmix​\(λ\)1−Fmix​\(λ\),x∈\[λ,1\],G\_\{\\mathrm\{mix\}\}\(x\)=\\frac\{F\_\{\\mathrm\{mix\}\}\(x\)\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\}\{1\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\},\\qquad x\\in\[\\lambda,1\],\(28\)has a positive right\-boundary densitygmix​\(λ\+\)g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)that is continuous in a right neighborhood ofλ\\lambda\. LetmR=∑i=1n𝟏​\{pi∗\>λ\}m\_\{R\}=\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{p\_\{i\}^\{\*\}\>\\lambda\\\}\. The kNN neighborhood sizeknk\_\{n\}satisfies

kn→∞,knmR→𝑝0\.k\_\{n\}\\to\\infty,\\qquad\\frac\{k\_\{n\}\}\{m\_\{R\}\}\\xrightarrow\{p\}0\.\(29\)LetY\(1\)≤⋯≤Y\(mR\)Y\_\{\(1\)\}\\leq\\cdots\\leq Y\_\{\(m\_\{R\}\)\}be the order statistics of the right\-tail sample\{pi∗:pi∗\>λ\}\\\{p\_\{i\}^\{\*\}:p\_\{i\}^\{\*\}\>\\lambda\\\}\. Define

g^​\(λ\+\)=knmR​\(Y\(kn\)−λ\),G^n​\(x\)=1mR​∑i=1n𝟏​\{λ<pi∗≤x\},x∈\[λ,1\]\.\\widehat\{g\}\(\\lambda^\{\+\}\)=\\frac\{k\_\{n\}\}\{m\_\{R\}\\,\(Y\_\{\(k\_\{n\}\)\}\-\\lambda\)\},\\qquad\\widehat\{G\}\_\{n\}\(x\)=\\frac\{1\}\{m\_\{R\}\}\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{\\lambda<p\_\{i\}^\{\*\}\\leq x\\\},\\quad x\\in\[\\lambda,1\]\.\(30\)Then

g^​\(λ\+\)→𝑝gmix​\(λ\+\),supx∈\[λ,1\]\|G^n​\(x\)−Gmix​\(x\)\|→𝑝0\.\\widehat\{g\}\(\\lambda^\{\+\}\)\\xrightarrow\{p\}g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\),\\qquad\\sup\_\{x\\in\[\\lambda,1\]\}\|\\widehat\{G\}\_\{n\}\(x\)\-G\_\{\\mathrm\{mix\}\}\(x\)\|\\xrightarrow\{p\}0\.\(31\)

###### Proof\.

Since1−Fmix​\(λ\)\>01\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\>0, the law of large numbers givesmR/n→𝑝1−Fmix​\(λ\)m\_\{R\}/n\\xrightarrow\{p\}1\-F\_\{\\mathrm\{mix\}\}\(\\lambda\), hencemR→𝑝∞m\_\{R\}\\xrightarrow\{p\}\\infty\. Conditional onmRm\_\{R\}, the right\-tail sample is i\.i\.d\. fromGmixG\_\{\\mathrm\{mix\}\}, so the Glivenko–Cantelli theorem gives the uniform convergence ofG^n\\widehat\{G\}\_\{n\}on\[λ,1\]\[\\lambda,1\]\.

It remains to check the boundary density estimator\. LetUn=Gmix​\(Y\(kn\)\)U\_\{n\}=G\_\{\\mathrm\{mix\}\}\(Y\_\{\(k\_\{n\}\)\}\)\. By the standard uniform order\-statistic representation,Un/\(kn/mR\)→𝑝1U\_\{n\}/\(k\_\{n\}/m\_\{R\}\)\\xrightarrow\{p\}1whenkn→∞k\_\{n\}\\to\\inftyandkn/mR→0k\_\{n\}/m\_\{R\}\\to 0\. Continuity and positivity ofgmix​\(λ\+\)g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)imply

Gmix​\(λ\+r\)=gmix​\(λ\+\)​r\+o​\(r\),r↓0\.G\_\{\\mathrm\{mix\}\}\(\\lambda\+r\)=g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)r\+o\(r\),\\qquad r\\downarrow 0\.\(32\)Withrn=Y\(kn\)−λr\_\{n\}=Y\_\{\(k\_\{n\}\)\}\-\\lambda, the order\-statistic result givesGmix​\(λ\+rn\)=Un→0G\_\{\\mathrm\{mix\}\}\(\\lambda\+r\_\{n\}\)=U\_\{n\}\\to 0in probability, hencern→0r\_\{n\}\\to 0in probability and

g^​\(λ\+\)=kn/mRrn=kn/mRGmix​\(λ\+rn\)​Gmix​\(λ\+rn\)rn→𝑝gmix​\(λ\+\)\.\\widehat\{g\}\(\\lambda^\{\+\}\)=\\frac\{k\_\{n\}/m\_\{R\}\}\{r\_\{n\}\}=\\frac\{k\_\{n\}/m\_\{R\}\}\{G\_\{\\mathrm\{mix\}\}\(\\lambda\+r\_\{n\}\)\}\\frac\{G\_\{\\mathrm\{mix\}\}\(\\lambda\+r\_\{n\}\)\}\{r\_\{n\}\}\\xrightarrow\{p\}g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)\.\(33\)∎

###### Lemma 7\(Right\-tail boundary dominance\)\.

Under[˜5](https://arxiv.org/html/2605.21543#Thmtheorem5), the mixture right\-tail boundary density satisfiesgmix​\(λ\+\)≥g0​\(λ\+\)g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)\\geq g\_\{0\}\(\\lambda^\{\+\}\)\.

###### Proof\.

Write the right\-tail mixture density asgmix​\(λ\+\)=w​g0​\(λ\+\)\+\(1−w\)​g1​\(λ\+\)g\_\{\\mathrm\{mix\}\}\(\\lambda^\{\+\}\)=w\\,g\_\{0\}\(\\lambda^\{\+\}\)\+\(1\-w\)g\_\{1\}\(\\lambda^\{\+\}\), wherew=π0​\(1−F0​\(λ\)\)/\(1−Fmix​\(λ\)\)∈\[0,1\]w=\\pi\_\{0\}\(1\-F\_\{0\}\(\\lambda\)\)/\(1\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\)\\in\[0,1\]\.[˜5](https://arxiv.org/html/2605.21543#Thmtheorem5)impliesg1​\(λ\+\)≥g0​\(λ\+\)g\_\{1\}\(\\lambda^\{\+\}\)\\geq g\_\{0\}\(\\lambda^\{\+\}\)at the boundary, so the convex combination is at leastg0​\(λ\+\)g\_\{0\}\(\\lambda^\{\+\}\)\. ∎

###### Lemma 8\(Asymptotic GCR control of Storey\-BH;Storey et al\.,[2004](https://arxiv.org/html/2605.21543#bib.bib48), Thm\. 4\)\.

Considernntests\{H0,i\}i=1n\\\{H\_\{0,i\}\\\}\_\{i=1\}^\{n\}with inputpp\-values\{pi\}i=1n⊂\[0,1\]\\\{p\_\{i\}\\\}\_\{i=1\}^\{n\}\\subset\[0,1\]and order statisticsp\(1\)≤⋯≤p\(n\)p\_\{\(1\)\}\\leq\\cdots\\leq p\_\{\(n\)\}\. Recall thatpip\_\{i\}is*valid*underH0,iH\_\{0,i\}ifℙ​\(pi≤t∣H0,i\)≤t\\mathbb\{P\}\(p\_\{i\}\\leq t\\mid H\_\{0,i\}\)\\leq tfor allt∈\[0,1\]t\\in\[0,1\], as in[Equation˜9](https://arxiv.org/html/2605.21543#S3.E9); assume the inputs are valid in this sense\. Letn0=\|\{i:H0,i​is true\}\|n\_\{0\}=\|\\\{i:H\_\{0,i\}\\text\{ is true\}\\\}\|denote the number of true nulls, and letπ0∈\(0,1\]\\pi\_\{0\}\\in\(0,1\]denote the limiting null proportion,n0/n→π0n\_\{0\}/n\\to\\pi\_\{0\}\. LetF^0\\widehat\{F\}\_\{0\}andF^\\widehat\{F\}denote the empirical null and overall CDFs of\{pi\}i=1n\\\{p\_\{i\}\\\}\_\{i=1\}^\{n\}, and assume both converge in probability, uniformly int∈\[0,1\]t\\in\[0,1\], to continuous limits\. Letπ^0\\widehat\{\\pi\}\_\{0\}satisfyπ^0→𝑝π0∞\\widehat\{\\pi\}\_\{0\}\\xrightarrow\{p\}\\pi\_\{0\}^\{\\infty\}for some constantπ0∞≥π0\\pi\_\{0\}^\{\\infty\}\\geq\\pi\_\{0\}\. Forα∈\(0,1\)\\alpha\\in\(0,1\), the Storey\-BH selection

𝒮=\{i:pi≤α​r∗π^0​n\},where​r∗=max⁡\{r:p\(r\)≤α​rπ^0​n\},\\mathcal\{S\}=\\\!\\left\\\{i:p\_\{i\}\\leq\\frac\{\\alpha\\,r^\{\*\}\}\{\\widehat\{\\pi\}\_\{0\}\\,n\}\\right\\\},\\text\{ where \}r^\{\*\}=\\max\\\!\\left\\\{r:p\_\{\(r\)\}\\leq\\frac\{\\alpha\\,r\}\{\\widehat\{\\pi\}\_\{0\}\\,n\}\\right\\\},\(34\)withr∗=0r^\{\*\}=0and𝒮=∅\\mathcal\{S\}=\\emptysetwhen the set is empty, satisfies

lim supn→∞𝔼​\[∑i=1n𝟙​\{i∈𝒮,H0,i​is true\}1∨\|𝒮\|\]≤α\.\\limsup\_\{n\\to\\infty\}\\mathbb\{E\}\\\!\\left\[\\frac\{\\sum\_\{i=1\}^\{n\}\\mathds\{1\}\\\{i\\in\\mathcal\{S\},\\;H\_\{0,i\}\\text\{ is true\}\\\}\}\{1\\vee\|\\mathcal\{S\}\|\}\\right\]\\leq\\alpha\.\(35\)

### B\.2Proof of[Proposition˜2](https://arxiv.org/html/2605.21543#Thmtheorem2)

###### Proof\.

The conclusion uses[˜4](https://arxiv.org/html/2605.21543#Thmtheorem4)and[˜5](https://arxiv.org/html/2605.21543#Thmtheorem5)\. We first establish the bound atx=λx=\\lambda\. The maph​\(y\)=λ​y/\(1\+λ​y\)h\(y\)=\\lambda y/\(1\+\\lambda y\)is increasing, so[Lemma˜7](https://arxiv.org/html/2605.21543#Thmtheorem7)gives

Ffit​\(λ\)≥λ​g0​\(λ\+\)1\+λ​g0​\(λ\+\)=λ​f0​\(λ\+\)1−F0​\(λ\)\+λ​f0​\(λ\+\)\.F\_\{\\mathrm\{fit\}\}\(\\lambda\)\\geq\\frac\{\\lambda g\_\{0\}\(\\lambda^\{\+\}\)\}\{1\+\\lambda g\_\{0\}\(\\lambda^\{\+\}\)\}=\\frac\{\\lambda f\_\{0\}\(\\lambda^\{\+\}\)\}\{1\-F\_\{0\}\(\\lambda\)\+\\lambda f\_\{0\}\(\\lambda^\{\+\}\)\}\.\(36\)LetA=F0​\(λ\)∈\[0,1\)A=F\_\{0\}\(\\lambda\)\\in\[0,1\)andB=λ​f0​\(λ\+\)B=\\lambda f\_\{0\}\(\\lambda^\{\+\}\)\. By the convexity ofF0F\_\{0\}andF0​\(0\)=0F\_\{0\}\(0\)=0, the secant slopeF0​\(λ\)/λF\_\{0\}\(\\lambda\)/\\lambdais at most the right derivativef0​\(λ\+\)f\_\{0\}\(\\lambda^\{\+\}\), soB≥AB\\geq A\. HenceB​\(1−A\)≥A​\(1−A\)B\(1\-A\)\\geq A\(1\-A\)andB/\(1−A\+B\)≥AB/\(1\-A\+B\)\\geq A; thereforeFfit​\(λ\)≥F0​\(λ\)F\_\{\\mathrm\{fit\}\}\(\\lambda\)\\geq F\_\{0\}\(\\lambda\)\.

Forx∈\(0,λ\]x\\in\(0,\\lambda\], the convexity ofF0F\_\{0\}on\[0,λ\]\[0,\\lambda\]withF0​\(0\)=0F\_\{0\}\(0\)=0implies thatF0​\(x\)/xF\_\{0\}\(x\)/xis nondecreasing, soF0​\(x\)≤F0​\(λ\)​x/λF\_\{0\}\(x\)\\leq F\_\{0\}\(\\lambda\)x/\\lambda\. Combined with the bound atλ\\lambda,

Ffit​\(x\)=Ffit​\(λ\)λ​x≥F0​\(λ\)λ​x≥F0​\(x\)\.F\_\{\\mathrm\{fit\}\}\(x\)=\\frac\{F\_\{\\mathrm\{fit\}\}\(\\lambda\)\}\{\\lambda\}x\\geq\\frac\{F\_\{0\}\(\\lambda\)\}\{\\lambda\}x\\geq F\_\{0\}\(x\)\.\(37\)
Forx∈\(λ,1\]x\\in\(\\lambda,1\], writeF0​\(x\)=F0​\(λ\)\+\(1−F0​\(λ\)\)​G0​\(x\)F\_\{0\}\(x\)=F\_\{0\}\(\\lambda\)\+\(1\-F\_\{0\}\(\\lambda\)\)G\_\{0\}\(x\)\.[˜5](https://arxiv.org/html/2605.21543#Thmtheorem5)givesGmix​\(x\)≥G0​\(x\)G\_\{\\mathrm\{mix\}\}\(x\)\\geq G\_\{0\}\(x\), and we just showedFfit​\(λ\)≥F0​\(λ\)F\_\{\\mathrm\{fit\}\}\(\\lambda\)\\geq F\_\{0\}\(\\lambda\)\. The map\(a,b\)↦a\+\(1−a\)​b\(a,b\)\\mapsto a\+\(1\-a\)bis nondecreasing ina∈\[0,1\)a\\in\[0,1\)for fixedb∈\[0,1\]b\\in\[0,1\]and nondecreasing inbbfor fixedaa, so

Ffit​\(x\)=Ffit​\(λ\)\+\(1−Ffit​\(λ\)\)​Gmix​\(x\)≥F0​\(λ\)\+\(1−F0​\(λ\)\)​G0​\(x\)=F0​\(x\)\.∎F\_\{\\mathrm\{fit\}\}\(x\)=F\_\{\\mathrm\{fit\}\}\(\\lambda\)\+\\bigl\(1\-F\_\{\\mathrm\{fit\}\}\(\\lambda\)\\bigr\)G\_\{\\mathrm\{mix\}\}\(x\)\\geq F\_\{0\}\(\\lambda\)\+\\bigl\(1\-F\_\{0\}\(\\lambda\)\\bigr\)G\_\{0\}\(x\)=F\_\{0\}\(x\)\.\\qed\(38\)

### B\.3Proof of[Theorem˜3](https://arxiv.org/html/2605.21543#Thmtheorem3)

By[Proposition˜6](https://arxiv.org/html/2605.21543#Thmtheorem6), the fitted envelope converges toFfitF\_\{\\mathrm\{fit\}\}uniformly on the two pieces of[Equation˜15](https://arxiv.org/html/2605.21543#S3.E15)\. By[Proposition˜2](https://arxiv.org/html/2605.21543#Thmtheorem2),FfitF\_\{\\mathrm\{fit\}\}dominatesF0F\_\{0\}\. For a null item and anyt∈\(0,1\)t\\in\(0,1\),

ℙ​\(Ffit​\(pi∗\)≤t∣H0,i\)=ℙ​\(pi∗≤Ffit−1​\(t\)∣H0,i\)≤ℙ​\(pi∗≤F0−1​\(t\)∣H0,i\)≤t,\\mathbb\{P\}\(F\_\{\\mathrm\{fit\}\}\(p\_\{i\}^\{\*\}\)\\leq t\\mid H\_\{0,i\}\)=\\mathbb\{P\}\(p\_\{i\}^\{\*\}\\leq F\_\{\\mathrm\{fit\}\}^\{\-1\}\(t\)\\mid H\_\{0,i\}\)\\leq\\mathbb\{P\}\(p\_\{i\}^\{\*\}\\leq F\_\{0\}^\{\-1\}\(t\)\\mid H\_\{0,i\}\)\\leq t,\(39\)so the transformed null values are asymptotically super\-uniform\. The estimator[Equation˜16](https://arxiv.org/html/2605.21543#S3.E16)is asymptotically conservative because

π^0→𝑝1−Fmix​\(λ\)1−Ffit​\(λ\)≥π0​\(1−F0​\(λ\)\)1−F0​\(λ\)=π0,\\widehat\{\\pi\}\_\{0\}\\xrightarrow\{p\}\\frac\{1\-F\_\{\\mathrm\{mix\}\}\(\\lambda\)\}\{1\-F\_\{\\mathrm\{fit\}\}\(\\lambda\)\}\\geq\\frac\{\\pi\_\{0\}\(1\-F\_\{0\}\(\\lambda\)\)\}\{1\-F\_\{0\}\(\\lambda\)\}=\\pi\_\{0\},\(40\)usingFfit​\(λ\)≥F0​\(λ\)F\_\{\\mathrm\{fit\}\}\(\\lambda\)\\geq F\_\{0\}\(\\lambda\)\.

We now invoke[Lemma˜8](https://arxiv.org/html/2605.21543#Thmtheorem8)on the transformed inputsp~i=F^fit​\(pi∗\)\\widetilde\{p\}\_\{i\}=\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(p\_\{i\}^\{\*\}\)with the selection in[Equation˜17](https://arxiv.org/html/2605.21543#S3.E17)\.[Equation˜39](https://arxiv.org/html/2605.21543#A2.E39)verifies the validity of\{p~i\}\\\{\\widetilde\{p\}\_\{i\}\\\}under the null \(in the asymptotic sense, sinceF^fit→Ffit\\widehat\{F\}\_\{\\mathrm\{fit\}\}\\to F\_\{\\mathrm\{fit\}\}uniformly by[Proposition˜6](https://arxiv.org/html/2605.21543#Thmtheorem6)\)\. The Glivenko–Cantelli theorem applied to the i\.i\.d\. null sub\-sample and to the full i\.i\.d\. sample\{pi∗\}∼Fmix\\\{p\_\{i\}^\{\*\}\\\}\\sim F\_\{\\mathrm\{mix\}\}, combined with the monotonicity ofF^fit\\widehat\{F\}\_\{\\mathrm\{fit\}\}, gives the required uniform convergence of the null and overall empirical CDFs\. The conservative null\-proportion limit is supplied by[Equation˜40](https://arxiv.org/html/2605.21543#A2.E40)\.[Lemma˜8](https://arxiv.org/html/2605.21543#Thmtheorem8)therefore yieldslim supn→∞GCR​\(𝒮\)≤α\\limsup\_\{n\\to\\infty\}\\mathrm\{GCR\}\(\\mathcal\{S\}\)\\leq\\alpha\. ∎

## Appendix CAlgorithm Summary for JMCS

[Algorithm˜2](https://arxiv.org/html/2605.21543#alg2)summarizes JMCS, the max\-ppbaseline used in our experiments\. It computes model\-wise conformalpp\-values, aggregates them bypi∗=maxk⁡pikp\_\{i\}^\{\*\}=\\max\_\{k\}p\_\{i\}^\{k\}, and applies BH to the aggregated values to return the shared selected benchmark\.

Algorithm 2Joint Max Conformal Selection \(JMCS\)0:Candidate items

\{xi\}i=1n\\\{x\_\{i\}\\\}\_\{i=1\}^\{n\}, audited models

\{θk\}k=1K\\\{\\theta\_\{k\}\\\}\_\{k=1\}^\{K\}, calibration data

𝒟cal=\{xi\}i=n\+1n\+m\\mathcal\{D\}\_\{\\mathrm\{cal\}\}=\\\{x\_\{i\}\\\}\_\{i=n\+1\}^\{n\+m\}, target level

α\\alpha\.

1:for

k=1,…,Kk=1,\\ldots,Kdo

2:Construct conformal

pp\-values

\{pik\}i=1n\\\{p\_\{i\}^\{k\}\\\}\_\{i=1\}^\{n\}via[Equation˜4](https://arxiv.org/html/2605.21543#S2.E4)\.

3:endfor

4:Aggregate

pi∗=max1≤k≤K⁡pikp\_\{i\}^\{\*\}=\\max\_\{1\\leq k\\leq K\}p\_\{i\}^\{k\}for

i=1,…,ni=1,\\ldots,n\.

5:Sort the maxima

p\(1\)∗≤⋯≤p\(n\)∗p\_\{\(1\)\}^\{\*\}\\leq\\cdots\\leq p\_\{\(n\)\}^\{\*\}and compute

r∗=max⁡\{r:p\(r\)∗≤α​r/n\}r^\{\*\}=\\max\\\{r:p\_\{\(r\)\}^\{\*\}\\leq\\alpha\\,r/n\\\}\.

6:Return

𝒮=\{i:pi∗≤α​r∗/n\}\\mathcal\{S\}=\\\{i:p\_\{i\}^\{\*\}\\leq\\alpha\\,r^\{\*\}/n\\\}\.

## Appendix DExperimental details

Our experiments obtain ground\-truth membership labels by controlled fine\-tuning\. For each audited model, an item is labeled as a member if it is included in that model’s fine\-tuning corpus, and as a non\-member otherwise\.

### D\.1Tail\-threshold selection

In all default runs of JECS, we choose the right\-tail thresholdλ\\lambdafrom the grid\{0\.5,0\.6,0\.7,0\.8,0\.9\}\\\{0\.5,0\.6,0\.7,0\.8,0\.9\\\}\. The criterion is to minimize the fitted left\-branch slope\. For each candidateλ\\lambda, we estimateg^​\(λ\+\)\\widehat\{g\}\(\\lambda^\{\+\}\)from the right\-tail sample\{pi∗:pi∗\>λ\}\\\{p\_\{i\}^\{\*\}:p\_\{i\}^\{\*\}\>\\lambda\\\}and form the anchor

F^fit​\(λ\)=λ​g^​\(λ\+\)1\+λ​g^​\(λ\+\)\.\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)=\\frac\{\\lambda\\,\\widehat\{g\}\(\\lambda^\{\+\}\)\}\{1\+\\lambda\\,\\widehat\{g\}\(\\lambda^\{\+\}\)\}\.\(41\)We then set

λ^=arg⁡minλ⁡F^fit​\(λ\)λ=arg⁡minλ⁡g^​\(λ\+\)1\+λ​g^​\(λ\+\)\.\\widehat\{\\lambda\}=\\arg\\min\_\{\\lambda\}\\frac\{\\widehat\{F\}\_\{\\mathrm\{fit\}\}\(\\lambda\)\}\{\\lambda\}=\\arg\\min\_\{\\lambda\}\\frac\{\\widehat\{g\}\(\\lambda^\{\+\}\)\}\{1\+\\lambda\\,\\widehat\{g\}\(\\lambda^\{\+\}\)\}\.\(42\)This rule selects the threshold whose fitted left branch has the smallest slope\. The selectedλ^\\widehat\{\\lambda\}is then used in[Equation˜15](https://arxiv.org/html/2605.21543#S3.E15)to constructF^fit\\widehat\{F\}\_\{\\mathrm\{fit\}\}, transform the max\-ppvalues, and estimateπ^0\\widehat\{\\pi\}\_\{0\}\.

### D\.2Dataset split setting

This subsection explains how we construct the controlled member/non\-member structure that supplies the ground\-truth joint\-contamination labels used in our experiments\.

Starting from the candidate pool𝒫\\mathcal\{P\}of a benchmark, we split the items into three disjoint parts: a*clean block*𝒫0\\mathcal\{P\}\_\{0\}, a*calibration block*𝒫cal\\mathcal\{P\}\_\{\\mathrm\{cal\}\}, and a*remainder block*𝒫rem\\mathcal\{P\}\_\{\\mathrm\{rem\}\}, with sizes proportional toaa,bb, and1−a−b1\-a\-brespectively\. For each itemxix\_\{i\}and each audited modelθk\\theta\_\{k\}, the membership labelMik∈\{0,1\}M\_\{i\}^\{k\}\\in\\\{0,1\\\}is then drawn block\-by\-block:

ℙ​\(Mik=0\)=\{1,xi∈𝒫0,0,xi∈𝒫cal,1−ρ,xi∈𝒫rem\.\\mathbb\{P\}\(M\_\{i\}^\{k\}=0\)\\;=\\;\\begin\{cases\}1,&x\_\{i\}\\in\\mathcal\{P\}\_\{0\},\\\\ 0,&x\_\{i\}\\in\\mathcal\{P\}\_\{\\mathrm\{cal\}\},\\\\ 1\-\\rho,&x\_\{i\}\\in\\mathcal\{P\}\_\{\\mathrm\{rem\}\}\.\\end\{cases\}\(43\)The three blocks play distinct roles\. Items in𝒫0\\mathcal\{P\}\_\{0\}are never used to fine\-tune any audited model, so every item in this block is jointly pure by construction\. Items in𝒫cal\\mathcal\{P\}\_\{\\mathrm\{cal\}\}are included in the fine\-tuning corpus of*every*audited model and are reused as the shared calibration set𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}in[Section˜3\.2](https://arxiv.org/html/2605.21543#S3.SS2)\. Items in𝒫rem\\mathcal\{P\}\_\{\\mathrm\{rem\}\}supply the per\-model contamination\. For each audited modelθk\\theta\_\{k\}, we independently sample a fractionρ\\rhoof the remainder uniformly at random and add those items toθk\\theta\_\{k\}’s fine\-tuning corpus\. We callρ\\rhothe*training fraction*; it is the per\-model sampling fraction*within the remainder block*, not within the full pool\. Because the assignments on𝒫rem\\mathcal\{P\}\_\{\\mathrm\{rem\}\}are independent across theKKaudited models, the joint\-purity probabilityℙ​\(⋁k=1KMik=0\)\\mathbb\{P\}\(\\bigvee\_\{k=1\}^\{K\}M\_\{i\}^\{k\}=0\)equals11on𝒫0\\mathcal\{P\}\_\{0\}and\(1−ρ\)K\(1\-\\rho\)^\{K\}on𝒫rem\\mathcal\{P\}\_\{\\mathrm\{rem\}\}\. Selection is performed on𝒟test:=𝒫0∪𝒫rem\\mathcal\{D\}\_\{\\mathrm\{test\}\}:=\\mathcal\{P\}\_\{0\}\\cup\\mathcal\{P\}\_\{\\mathrm\{rem\}\}, with the calibration block held out\. Throughout the main evaluation we usea=b=0\.3a=b=0\.3andρ=0\.125\\rho=0\.125, and we varyρ\\rhoin the sensitivity analysis of[Section˜4\.3](https://arxiv.org/html/2605.21543#S4.SS3)\.

Table 4:Dataset split statistics atρ=0\.125\\rho=0\.125\. Counts are reported for a single audited modelθk\\theta\_\{k\}: “Member” denotesMik=1M\_\{i\}^\{k\}=1and “Non\-member” denotesMik=0M\_\{i\}^\{k\}=0\. Calibration items are shared members of all audited models, and the per\-model test\-set counts are identical acrosskk\.#### Evaluation protocol\.

All reported GCP and Power values are Monte\-Carlo means over 500 repetitions on the fixed three\-block partition of[Section˜D\.2](https://arxiv.org/html/2605.21543#A4.SS2)\. In each repetition we draw80%80\\%of𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}and80%80\\%of𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}uniformly at random; the three\-block split itself is held fixed across repetitions\.

### D\.3Fine\-tuning setting

We fine\-tune all audited models using LoRA\-SFT\. The LoRA adapter is configured with rankr=16r=16, scaling parameterαLoRA=32\\alpha\_\{\\mathrm\{LoRA\}\}=32, dropout0\.050\.05, no bias terms, and task typeCAUSAL\_LM\. We train all audited models with AdamW using learning rate10−310^\{\-3\}, weight decay5×10−45\\times 10^\{\-4\},1010warmup steps, a cosine learning\-rate schedule, and per\-device batch size88for both training and evaluation\. The number of fine\-tuning epochs is backbone\-specific: GPT\-NeoX\-20B is trained for55epochs, while the Pythia and LLaMA backbones are trained for33epochs\.

### D\.4Computational cost

All audited models are fine\-tuned with the LoRA\-SFT protocol of[Section˜D\.3](https://arxiv.org/html/2605.21543#A4.SS3), using PyTorch FSDP on NVIDIA RTX 4090 GPUs\. Fine\-tuning one audited model on one dataset takes about55GPU\-min for LLaMA\-7B or Pythia\-6\.9B \(33epochs,22GPUs\), and about1515GPU\-min for GPT\-NeoX\-20B \(55epochs,44GPUs\)\. After the scoresT​\(x;θk\)T\(x;\\theta\_\{k\}\)are extracted, JECS only performs NumPy operations on the\(ntest,K\)\(n\_\{\\mathrm\{test\}\},K\)score matrix, including conformalpp\-value computation, max\-ppaggregation, envelope fitting, Storey estimation, and Storey\-BH; every evaluated setting in[Section˜4](https://arxiv.org/html/2605.21543#S4)runs in under one minute on a single CPU core and requires no GPU\.

## Appendix EAdditional Experimental Results

### E\.1Synthetic setup for naive composition failure

We generate a pool of12001200candidate items forK=4K=4audited models\{θk\}k=1K\\\{\\theta\_\{k\}\\\}\_\{k=1\}^\{K\}, and split it into a calibration set𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}of sizencal=360n\_\{\\mathrm\{cal\}\}=360\(30%30\\%of the pool\) and an audit set𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}of sizentest=840n\_\{\\mathrm\{test\}\}=840\. Items in𝒟cal\\mathcal\{D\}\_\{\\mathrm\{cal\}\}are members of every audited model by construction, matching the shared\-member calibration assumption of[Section˜3\.2](https://arxiv.org/html/2605.21543#S3.SS2)\. Items in𝒟test\\mathcal\{D\}\_\{\\mathrm\{test\}\}receive independent Bernoulli labelsMik∼iidBernoulli​\(0\.30\)M\_\{i\}^\{k\}\\stackrel\{\{\\scriptstyle\\mathrm\{iid\}\}\}\{\{\\sim\}\}\\mathrm\{Bernoulli\}\(0\.30\)across bothiiandkk, so the joint\-purity probability of an audit item is\(1−0\.30\)K≈0\.24\(1\-0\.30\)^\{K\}\\approx 0\.24\. For each itemiiand audited modelθk\\theta\_\{k\}, the detection score is

Tik=μ⋅Mik\+εik,εik∼iid𝒩​\(0,1\),T\_\{i\}^\{k\}=\\mu\\cdot M\_\{i\}^\{k\}\+\\varepsilon\_\{i\}^\{k\},\\qquad\\varepsilon\_\{i\}^\{k\}\\stackrel\{\{\\scriptstyle\\mathrm\{iid\}\}\}\{\{\\sim\}\}\\mathcal\{N\}\(0,1\),\(44\)with signal strengthμ=4\\mu=4, so members and non\-members are𝒩​\(μ,1\)\\mathcal\{N\}\(\\mu,1\)and𝒩​\(0,1\)\\mathcal\{N\}\(0,1\), and the model\-wise scores are independent acrosskk\.

### E\.2MIMIR Subsets

We further evaluate JECS on five subsets \(HackerNews, DMMath, GitHub, Pile\-CC, PubMed\) of the MIMIR benchmark\[Duan et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib16)\], fine\-tuningK=8K=8audited models from Pythia\-6\.9B and LLaMA\-7B per subset under the same protocol as[Section˜4\.1](https://arxiv.org/html/2605.21543#S4.SS1)\.[Figure˜7](https://arxiv.org/html/2605.21543#A5.F7)reports realized GCP and Power for the four detection scores in[Section˜4\.1](https://arxiv.org/html/2605.21543#S4.SS1)atα∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\alpha\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}\. The realized GCP stays at or below the target diagonal throughout, confirming that GCR control extends beyond the WikiMIA and ArXivTection candidate pools used in[Section˜4](https://arxiv.org/html/2605.21543#S4)\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x7.png)\(a\)Pythia\-6\.9B
![Refer to caption](https://arxiv.org/html/2605.21543v1/x8.png)\(b\)LLaMA\-7B

Figure 7:Realized GCP and Power of JECS on five MIMIR subsets atK=8K=8across Pythia\-6\.9B and LLaMA\-7B\. GCP curves \(lines, left axis\) and Power bars \(right axis\) for four detection scores atα∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\alpha\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}on HackerNews, DMMath, GitHub, Pile\-CC, and PubMed; the dashed line is the target GCR\.
### E\.3Power on WikiMIA

[Table˜5](https://arxiv.org/html/2605.21543#A5.T5)reports Power for JECS and JMCS on WikiMIA atK=16K=16, mirroring the ArXivTection table of[Table˜2](https://arxiv.org/html/2605.21543#S4.T2)\. JECS attains higher Power than JMCS across the evaluated targets, model families, and detection scores, with the largest gain on LLaMA\-7B with Perplexity atα=0\.3\\alpha=0\.3, where Power rises from0\.0880\.088to0\.4390\.439\. This pattern matches the aggregate finding of[Section˜4](https://arxiv.org/html/2605.21543#S4)and the super\-uniformity tax interpretation of[Section˜3\.2](https://arxiv.org/html/2605.21543#S3.SS2)\.

Table 5:Power on WikiMIA atK=16K=16\. Reported values are averagePower\\mathrm\{Power\}with standard error\. “Base” refers to JMCS and “Ours” refers to JECS\.Boldnumbers mark the better result in each pair\.
### E\.4Cross\-experiment variability of JECS on Min\-K%\+\+

[Figure˜8](https://arxiv.org/html/2605.21543#A5.F8)shows the cross\-experiment variability of JECS with Min\-K%\+\+ atK=16K=16\. The GCP remains aligned with the target diagonal, and the shaded band is narrow, with its upper edge only slightly above the target at smallerα\\alpha\. Power also varies little, supporting the stability of the gains reported in[Table˜2](https://arxiv.org/html/2605.21543#S4.T2)\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x9.png)Figure 8:JECS with Min\-K%\+\+ atK=16K=16: cross\-experiment dispersion\. Curves report mean realized GCP \(left axis, orange\) and Power \(right axis, blue\) on WikiMIA \(top row\) and ArXivTection \(bottom row\), for GPT\-NeoX\-2020B, LLaMA\-77B, and Pythia\-6\.96\.9B\. Shaded bands are±1\\pm 1standard deviation, clipped to\[0,1\]\[0,1\]; the dashed diagonal is the target GCR\.
### E\.5Mixed\-family audit pool

The main\-text protocol auditsKKfine\-tuned models initialized from a single backbone, which leaves open whether JECS’s control depends on the audited models sharing an architecture and parameter scale\. We complement that protocol with a heterogeneous audit pool composed ofK=16K=16fine\-tuned variants drawn from88public LLM configurations spanning three families and a wide scale range: Pythia\[Biderman et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib8)\]at1\.41\.4B,2\.82\.8B,6\.96\.9B, and1212B, GPT\-NeoX\-2020B\[Black et al\.,[2022](https://arxiv.org/html/2605.21543#bib.bib9)\], and LLaMA\[Touvron et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib49)\]at77B,1313B, and3030B\. Each of the88configurations contributes two independently fine\-tuned models over disjoint contamination assignments on the remainder block𝒫rem\\mathcal\{P\}\_\{\\mathrm\{rem\}\}, so the audit pool varies architecture, parameter scale, and per\-model training data jointly\. The shared\-member calibration block𝒫cal\\mathcal\{P\}\_\{\\mathrm\{cal\}\}from[Section˜D\.2](https://arxiv.org/html/2605.21543#A4.SS2)is reused by every variant unchanged; the rest of the protocol — test split, four detection scores, and the target gridα∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\alpha\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}— matches[Section˜4\.1](https://arxiv.org/html/2605.21543#S4.SS1)\.

[Figure˜9](https://arxiv.org/html/2605.21543#A5.F9)reports realized GCP and Power on this mixed\-family pool for the four detection scores atK=16K=16, on WikiMIA and ArXivTection\. The realized GCP stays at or below the target diagonal throughout; Min\-K%\+\+ delivers the highest Power on ArXivTection while the four scores cluster more tightly on WikiMIA\. The picture matches[Section˜4](https://arxiv.org/html/2605.21543#S4): JECS controls GCR without score\-specific tuning and without architecture\- or scale\-matched audited models\.

![Refer to caption](https://arxiv.org/html/2605.21543v1/x10.png)Figure 9:JECS on a mixed\-family audit pool atK=16K=16\. Realized GCP \(lines, left axis\) and Power \(bars, right axis\) for four detection scores atα∈\{0\.1,0\.2,0\.3,0\.4,0\.5\}\\alpha\\in\\\{0\.1,0\.2,0\.3,0\.4,0\.5\\\}on WikiMIA \(left\) and ArXivTection \(right\)\. TheK=16K=16pool combines88fine\-tuned configurations spanning Pythia \(1\.41\.4B/2\.82\.8B/6\.96\.9B/1212B\), GPT\-NeoX\-2020B, and LLaMA \(77B/1313B/3030B\), with two independently fine\-tuned models per configuration\. The dashed line is the target GCR\.

## Appendix FFeasibility of Constructing the Shared\-Member Calibration Set

The calibration assumption in[Section˜3\.2](https://arxiv.org/html/2605.21543#S3.SS2)requires a pool of items that are shared members of every audited model\. We argue this is achievable in practice on three independent grounds\.

1. 1\.Public benchmarks expose training splits that are routinely mixed into LLM training\.Public benchmarks release explicit training splits, and the contamination literature documents that such training data is in many cases deliberately incorporated into LLM pretraining or fine\-tuning\. For instance, OpenAI openly acknowledges mixing the MATH and GSM8K training sets into GPT\-4’s training\[Achiam et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib1)\], and surveys catalog the same practice across model families including GLM\-130B, Qwen, Nemotron\-4, InternLM\-2, and MiniCPM\[Zhou et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib65), Xu et al\.,[2024](https://arxiv.org/html/2605.21543#bib.bib55)\]\. Benchmark training splits thus constitute a sizable population of items that are members of many deployed models by construction\.
2. 2\.Model providers openly disclose canonical web sources as documented pretraining components\.Modern LLM families often identify canonical web sources as components of their pretraining mixes, including Common Crawl, Wikipedia, arXiv, and GitHub\. Llama publishes the full breakdown \(CommonCrawl, C4, GitHub, Wikipedia, arXiv, StackExchange\) in its technical report\[Touvron et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib49)\], and Falcon similarly releases the RefinedWeb extraction of Common Crawl\[Penedo et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib40)\]\. Decontamination audits further confirm substantial overlap between such corpora and widely used benchmarks\[Yang et al\.,[2023](https://arxiv.org/html/2605.21543#bib.bib56)\], so samples drawn from these disclosed sources can be treated as shared members of the corresponding model families\.
3. 3\.Regulatory mandates are making training\-data disclosure a legal requirement\.Beyond voluntary disclosure, Article 53\(1\)\(d\) of the EU AI Act \(Regulation \(EU\) 2024/1689\) requires providers of general\-purpose AI \(GPAI\) models placed on the EU market, including OpenAI, Google, Anthropic, and others, to publish a “sufficiently detailed summary” of the content used for training, following a template from the EU AI Office\[European Parliament and Council of the European Union,[2024](https://arxiv.org/html/2605.21543#bib.bib18)\]\. These obligations apply to newly placed GPAI models from 2 August 2025\. Enforcement powers, including fines of up to EUR 15 M or 3% of global annual turnover, start on 2 August 2026, and pre\-existing GPAI models must comply by 2 August 2027\. As these summaries become available, auditors will have more high\-confidence evidence about which public sources and data collections were used across major deployed models, making the shared\-member assumption increasingly easy to satisfy\.

Each of the three grounds above provides realistic candidates for assembling the calibration pool in deployed audit settings\.