Counterfactual Gate-Support Decomposition: Separating Selection from Redistribution in Training-Data Filtering

# Counterfactual Gate–Support Decomposition:Separating Selection from Redistribution in Training-Data Filtering
Source: [https://arxiv.org/html/2610.19343](https://arxiv.org/html/2610.19343)
Jongmin KimSeung\-Hun JeonAffiliation:ThakiCloudAffiliation:†Corresponding author:hyojung\.han@thakicloud\.com

2026\-10\-06

###### Abstract

A training\-data quality gate changes both which examples survive within regions of the input space and how training mass is distributed across those regions\. A gate\-versus\-random comparison therefore does not isolate these mechanisms\. We introduce Counterfactual Gate–Support Decomposition \(CGSD\), a four\-arm, budget\-matched design that crosses gate eligibility with support distribution and decomposes the observed gate effect into within\-support selection, between\-support redistribution, and their interaction\. The practical constraint is finite\-pool realizability: we characterize mass and token\-assembly conditions under which the required arms can be constructed, together with a support\-extinction case in which the ordinary design is undefined\. On Korean–English and Korean–Japanese retrieval, the mean within\-support selection component is the only one that exceeds its calibrated null band \(\+29\.7\+29\.7and\+5\.6\+5\.6holdout items;30/3030/30positive replicates\), decisively in Korean–English and marginally in Korean–Japanese; redistribution and interaction remain unresolved at the available power\. The same pattern holds for a second, unrelated gate family – a model\-confidence threshold rather than a deterministic verifier – judged against its own null \(\+27\.3\+27\.3items,9/99/9positive\), so the result is not a property of one gate’s mechanism\. On a mathematics gate, zero\-eligibility supports leave the ordinary design undefined at every budget, while the extended design is constructible only inside narrow budget bands that an exhaustive scan locates and that the natural operating point misses — the assembly condition separating a property of the pool from a limitation of the packer\. CGSD therefore returns a decomposition, a constructible\-but\-underpowered design, or an explicit non\-constructibility result, and says which; it prevents support redistribution from being silently interpreted as sample\-level gate quality\.

## 1Introduction

Data\-quality gates are deployed on the premise that removing bad examples improves the model trained on what remains\. The evidence usually offered is a comparison between a gated training set and a random subset of the same size\. That comparison answers a different question than the one being asked, because a gate does not only choose better examples — it also reweights the distribution over whatever regions the data occupies\. A gate that removes 40% of a corpus may be removing 5% of one topic and 95% of another\. The downstream difference then contains a selection effect and a redistribution effect, and the comparison cannot say which one it saw\.

This matters in practice because the two point at different levers\. If the resolved gain is within\-support selection, improving the within\-support selection rule is the relevant intervention\. If redistribution dominates, the gate is functioning as an accidental curriculum and explicit mixture control is the more direct and more controllable alternative\. If the two have opposite signs, tightening the gate can make things worse in a way that looks inexplicable from the aggregate number alone\.

We propose to answer the question by construction\. Fix a source pool, a support representation defined on*pre\-gate*text, and a training token budget\. Build four training sets that cross two binary factors — eligibility \(unrestricted vs\. gate\-eligible\) and support distribution \(source\-preserved vs\. gate\-induced\) — at the same budget\. The observed gate effect then decomposes exactly, replicate by replicate, into three terms\. No modelling assumption is introduced; the arms differ only in the factor being varied\.

The algebra here is the familiar2×22\\times 2factorial identity, and we do not claim it as new\. Our contribution is making that contrast operational for finite training pools under fixed exposure, and characterizing when the arms the identity requires cannot be realized at all\.

The price of that exactness is that the arms have to exist\. Our main methodological finding is that existence is not a technicality\. It fails in four ways whose prescriptions point in different directions: a support cannot supply its share, which a smaller budget fixes; a support has no eligible mass at all, which no budget fixes; the available example sizes cannot tile the per\-support quotas at this budget, which is a property of the pool and not monotone in the budget; and an assembly exists but the packer does not find it, which is a property of the algorithm alone\. Half of this paper is about telling them apart, because reading one as another sends you to change the wrong thing — as it did us, on our own mathematics pool\.

#### Contributions\.

\(i\) A construction\-based operationalization of the2×22\\times 2gate contrast for finite training pools at fixed token exposure, separating within\-support selection from between\-support redistribution\. \(ii\) Two independent feasibility conditions — mass and assembly — and their joint feasible region as a function of budget and support partition, with a bound linking the budget tolerance to the realized support distribution\. \(iii\) An extension for*support extinction*, where the ordinary design is undefined at every positive budget, isolating the extensive margin as a signed contrast\. \(iv\) A validation protocol — signature\-locked null calibration, analytic known\-truth recovery with interval coverage, infeasible controls, and extinction controls — under which we then demonstrate the method on two retrieval locales and two unrelated gate families \(a deterministic verifier and a model\-confidence threshold\), and a feasibility audit of mathematics and text\-to\-SQL pools that returns extinction, empty feasible regions, and one narrow feasible band, each distinguished from the others rather than reported as a single failure\.

## 2Related work

#### Data filtering and selection\.

A large body of work asks which examples to keep\. Heuristic and model\-based quality filters, deduplication\[[16](https://arxiv.org/html/2610.19343#bib.bib16),[1](https://arxiv.org/html/2610.19343#bib.bib1)\], difficulty\- or margin\-based pruning\[[24](https://arxiv.org/html/2610.19343#bib.bib24)\], influence\-guided instance selection for a target task\[[27](https://arxiv.org/html/2610.19343#bib.bib27)\], and datamodel\-driven selection\[[7](https://arxiv.org/html/2610.19343#bib.bib7)\]all improve downstream performance by discarding part of a pool\. Controlled curation benchmarks make the comparison itself the object of study\[[9](https://arxiv.org/html/2610.19343#bib.bib9)\]\. These methods are prescriptive: they propose a better rule\. CGSD is diagnostic and takes the rule as given\.

#### Data mixture and domain reweighting\.

A parallel line shows that the proportions themselves change downstream performance, independent of which individual examples are chosen: reweighting domains with a proxy model\[[28](https://arxiv.org/html/2610.19343#bib.bib28)\], generalization\-estimated reweighting\[[8](https://arxiv.org/html/2610.19343#bib.bib8)\], regression\- and law\-based mixture search\[[17](https://arxiv.org/html/2610.19343#bib.bib17),[30](https://arxiv.org/html/2610.19343#bib.bib30)\], importance resampling toward a target distribution\[[29](https://arxiv.org/html/2610.19343#bib.bib29)\], and unified treatments of the mixing objective\[[5](https://arxiv.org/html/2610.19343#bib.bib5)\]\. This literature establishes that mixture is a real and separate lever, which is precisely why a filter that silently moves the mixture is hard to interpret\. CGSD is the measurement that separates the two levers for a filter that was not designed to move either one deliberately\.

#### Attributing performance to training data\.

Influence functions\[[15](https://arxiv.org/html/2610.19343#bib.bib15),[11](https://arxiv.org/html/2610.19343#bib.bib11)\], gradient\-trace attribution\[[22](https://arxiv.org/html/2610.19343#bib.bib22)\], datamodels\[[14](https://arxiv.org/html/2610.19343#bib.bib14)\], and Shapley\-style valuation\[[10](https://arxiv.org/html/2610.19343#bib.bib10),[26](https://arxiv.org/html/2610.19343#bib.bib26)\]attribute performance to individual training examples, usually without retraining\. CGSD works at a coarser grain and in the opposite direction: it retrains on deliberately constructed pools to attribute an effect to two*mechanisms*rather than to examples, which is why it can be run on a gate whose scoring function is opaque\.

#### Decomposition, overlap and interference\.

Splitting an aggregate gap into a component due to differing composition and a component due to differing returns is the Oaxaca–Blinder decomposition\[[18](https://arxiv.org/html/2610.19343#bib.bib18),[3](https://arxiv.org/html/2610.19343#bib.bib3)\], and splitting a total effect into direct and indirect parts is the mediation literature\[[19](https://arxiv.org/html/2610.19343#bib.bib19)\]\. Our mass condition is the finite\-sample analogue of the positivity requirement that underlies matching and propensity methods\[[23](https://arxiv.org/html/2610.19343#bib.bib23),[25](https://arxiv.org/html/2610.19343#bib.bib25)\]: a contrast that no unit can supply is not estimable at any sample size\. Because one model is trained per arm on a pooled corpus, we also inherit the interference concerns formalized for causal inference under spillover\[[13](https://arxiv.org/html/2610.19343#bib.bib13)\]; we return to this in the limitations\.

#### Experimental rigour\.

Seed variance, under\-reported variability and fragile comparisons are well documented in ML\[[12](https://arxiv.org/html/2610.19343#bib.bib12),[20](https://arxiv.org/html/2610.19343#bib.bib20),[6](https://arxiv.org/html/2610.19343#bib.bib6)\], as are the reporting practices and interval procedures that mitigate them\[[21](https://arxiv.org/html/2610.19343#bib.bib21),[4](https://arxiv.org/html/2610.19343#bib.bib4),[2](https://arxiv.org/html/2610.19343#bib.bib2)\]\. Our validation protocol \(Section[6](https://arxiv.org/html/2610.19343#S6)\) follows that tradition and adds one element specific to this setting: a null band that is bound to the exact configuration it was measured on, so that a threshold cannot silently be reused across operating points\.

## 3The estimand

LetD=\{xi\}D=\\\{x\_\{i\}\\\}be a source pool\. Each example carries a token counttit\_\{i\}, a gate verdictgi∈\{0,1\}g\_\{i\}\\in\\\{0,1\\\}, and a support labelzi=Π⁡\(xi\)z\_\{i\}=\\Pi\(x\_\{i\}\)produced by a partitionΠ\\Piof the pre\-gate input\. We writeT=∑itiT=\\sum\_\{i\}t\_\{i\}and, for supportcc,Tc=∑i:zi=ctiT\_\{c\}=\\sum\_\{i:z\_\{i\}=c\}t\_\{i\},Ec=∑i:zi=cgitiE\_\{c\}=\\sum\_\{i:z\_\{i\}=c\}g\_\{i\}t\_\{i\}, andec=Ec/Tce\_\{c\}=E\_\{c\}/T\_\{c\}for the*eligibility rate*\.

#### Support must be pre\-gate\.

Π\\Piis computed from the input text only\. If the support label were derived from anything the gate produces, the redistribution term would be defined in terms of the treatment and the decomposition would be circular\. This is not hypothetical: in our own pipeline the most convenient available label was a gate output, and using it would have made the gate’s failure reason its own support\.

#### Four arms\.

Given a token budgetBB, define the source sharepc=Tc/Tp\_\{c\}=T\_\{c\}/Tand the gate\-induced sharepcG=Ec/∑c′Ec′p^\{G\}\_\{c\}=E\_\{c\}/\\sum\_\{c^\{\\prime\}\}E\_\{c^\{\\prime\}\}\. Each arm fills per\-support budgets by sampling to a token target:

The decomposition uses the four support\-controlled arms\.RRis a fifth, literature\-facing reference: it spends the same token budget with no support control at all, which is what a plain “gate versus random subset” comparison implicitly uses as its baseline\. We reportY⁡\(D00\)−Y⁡\(R\)Y\(D\_\{00\}\)\-Y\(R\)so that the effect of imposing support control is visible separately from the gate contrast itself\.

WritingY⁡\(⋅\)Y\(\\cdot\)for the downstream metric of a model trained on an arm,

E\\displaystyle E=Y⁡\(D10\)−Y⁡\(D00\),\\displaystyle=Y\(D\_\{10\}\)\-Y\(D\_\{00\}\),\(1\)S\\displaystyle S=Y⁡\(D01\)−Y⁡\(D00\),\\displaystyle=Y\(D\_\{01\}\)\-Y\(D\_\{00\}\),\(2\)I\\displaystyle I=Y⁡\(D11\)−Y⁡\(D10\)−Y⁡\(D01\)\+Y⁡\(D00\),\\displaystyle=Y\(D\_\{11\}\)\-Y\(D\_\{10\}\)\-Y\(D\_\{01\}\)\+Y\(D\_\{00\}\),\(3\)and the identityΔG=Y⁡\(D11\)−Y⁡\(D00\)=E\+S\+I\\Delta\_\{G\}=Y\(D\_\{11\}\)\-Y\(D\_\{00\}\)=E\+S\+Iholds exactly for every replicate, by construction rather than in expectation\.

#### WhatEEis not\.

EEis the effect of restricting to gate\-eligible examples*while holding the support distribution fixed*\. It is not “quality”\. It does not say why eligibility helps — label correctness, length, formatting, and annotator agreement all live inside it\. We keep the neutral name throughout because the temptation to readEEas a quality measurement is the single easiest way to overclaim from this decomposition\.

#### The estimand is budget\- and partition\-conditional\.

EE,SSandIIare defined at a particularBBandΠ\\Pi\. Two operating points give two estimands, so absolute magnitudes are not comparable across pools with different budgets or holdout sizes\. We writeE⁡\(B,Π\)E\(B,\\Pi\)when the dependence matters\.

## 4Feasibility: two independent conditions

### 4\.1Mass

D10D\_\{10\}keeps the source share inside the eligible pool, so it needsB​pc≤EcB\\,p\_\{c\}\\leq E\_\{c\}for everycc, i\.e\.B/T≤minc⁡ecB/T\\leq\\min\_\{c\}e\_\{c\}\. Define

M⁡\(B,Π\)=minc⁡ec−B/T\.M\(B,\\Pi\)\\;=\\;\\min\_\{c\}e\_\{c\}\\;\-\\;B/T\.\(4\)M<0M<0is a positivity violation at this budget and is repaired by loweringBB\.

### 4\.2Assembly

Examples are indivisible, and the four arms do not draw from the same pool:D00D\_\{00\}andD01D\_\{01\}draw from the unrestricted pool whileD10D\_\{10\}andD11D\_\{11\}draw only from the gate\-eligible one, at two different target shares\. Assembly must therefore be checked per arm\. Index eligibility bya∈\{0,1\}a\\in\\\{0,1\\\}and support share bys∈\{0,1\}s\\in\\\{0,1\\\}, and let𝒜a,c\\mathcal\{A\}\_\{a,c\}be the set of token sums attainable in supportccfrom the examples eligibility levelaaadmits\. Withpc\(0\)=pcp^\{\(0\)\}\_\{c\}=p\_\{c\},pc\(1\)=pcGp^\{\(1\)\}\_\{c\}=p^\{G\}\_\{c\}andbc\(s\)=B​pc\(s\)b^\{\(s\)\}\_\{c\}=B\\,p^\{\(s\)\}\_\{c\},

Qa​s​\(B\)\\displaystyle Q\_\{as\}\(B\)=∑c\(bc\(s\)−max⁡\{x∈𝒜a,c:x≤bc\(s\)\}\),\\displaystyle=\\sum\_\{c\}\\Bigl\(b^\{\(s\)\}\_\{c\}\-\\max\\\{x\\in\\mathcal\{A\}\_\{a,c\}:x\\leq b^\{\(s\)\}\_\{c\}\\\}\\Bigr\),\(5\)Ga​s​\(B,Π,τ\)\\displaystyle G\_\{as\}\(B,\\Pi;\\tau\)=τ​B−Qa​s​\(B\),Gmin=mina,s⁡Ga​s,\\displaystyle=\\tau B\-Q\_\{as\}\(B\),\\qquad G\_\{\\min\}=\\min\_\{a,s\}G\_\{as\},\(6\)whereτ\\tauis the budget matching tolerance\. The constructor never overshoots a per\-support target — overshooting would break the support match — so eachQa​sQ\_\{as\}is a sum of non\-negative shortfalls\.Ga​s≥0G\_\{as\}\\geq 0says an in\-tolerance assembly of that arm*exists*\. It is a property of the pool under the stated matching contract, not of the packing heuristic: a greedy packer may fail whereGa​s≥0G\_\{as\}\\geq 0, and that is a separate, algorithmic problem\. Reporting one as the other leads directly to requesting data that is not needed\. We compute𝒜a,c\\mathcal\{A\}\_\{a,c\}exactly by a bitset subset\-sum and fall back to the packer only when that is prohibitive, flagging the fallback; every feasibility verdict reported in this paper is exact, and no negative verdict rests on a heuristic packer failure\.

CruciallyGa​sG\_\{as\}is not monotone inBB: it is a lattice quantity, and moving the budget slightly can flip its sign\. LoweringBBdoes not reliably repair assembly, and in general makes it worse, sinceτ​B\\tau Bshrinks whileQa​sQ\_\{as\}does not\.

A tolerance on the*total*budget would be a weak guarantee if it allowed the realized support distribution to drift, so we record what it actually buys\.

###### Lemma 1\(Support drift under no\-overshoot\)\.

Letp~\\tilde\{p\}be the realized support distribution of an arm whose per\-support shortfalls are non\-negative and whose total shortfall is at mostτ​B\\tau B\. ThenTV⁡\(p~,p\(s\)\)≤τ/\(1−τ\)\\operatorname\{TV\}\(\\tilde\{p\},p^\{\(s\)\}\)\\leq\\tau/\(1\-\\tau\)\.

###### Proof\.

Write the realized support totals asb~c=bc\(s\)−qc\\tilde\{b\}\_\{c\}=b^\{\(s\)\}\_\{c\}\-q\_\{c\}withqc≥0q\_\{c\}\\geq 0and∑cqc=q≤τ​B\\sum\_\{c\}q\_\{c\}=q\\leq\\tau B, and letB~=B−q\\tilde\{B\}=B\-q\. Thenp~c−pc\(s\)=\(bc\(s\)−qc\)/B~−bc\(s\)/B\\tilde\{p\}\_\{c\}\-p^\{\(s\)\}\_\{c\}=\(b^\{\(s\)\}\_\{c\}\-q\_\{c\}\)/\\tilde\{B\}\-b^\{\(s\)\}\_\{c\}/B, soTV=12​∑c\|p~c−pc\(s\)\|≤12​\(∑cqc/B~\+∑cbc\(s\)​\(1/B~−1/B\)\)=q/B~≤τ​B/\(B−τ​B\)=τ/\(1−τ\)\\operatorname\{TV\}=\\tfrac\{1\}\{2\}\\sum\_\{c\}\|\\,\\tilde\{p\}\_\{c\}\-p^\{\(s\)\}\_\{c\}\|\\leq\\tfrac\{1\}\{2\}\\bigl\(\\sum\_\{c\}q\_\{c\}/\\tilde\{B\}\+\\sum\_\{c\}b^\{\(s\)\}\_\{c\}\\,\(1/\\tilde\{B\}\-1/B\)\\bigr\)=q/\\tilde\{B\}\\leq\\tau B/\(B\-\\tau B\)=\\tau/\(1\-\\tau\)\. ∎

Atτ=0\.5%\\tau=0\.5\\%this bounds realized support drift at0\.503%0\.503\\%in total variation, so the budget tolerance and the support match are not two independent promises — the first implies the second\.

### 4\.3The feasible region

ℱΠ=\{B\>0:M⁡\(B,Π\)≥0∧Gmin​\(B,Π,τ\)≥0\}\.\\mathcal\{F\}\_\{\\Pi\}=\\\{\\,B\>0:M\(B,\\Pi\)\\geq 0\\ \\wedge\\ G\_\{\\min\}\(B,\\Pi;\\tau\)\\geq 0\\,\\\}\.\(7\)Both conditions depend on the partition, so feasibility is joint in exposure and support representation\. Figure[1](https://arxiv.org/html/2610.19343#S4.F1)showsℱΠ\\mathcal\{F\}\_\{\\Pi\}shrinking as the partition is refined on our retrieval pool\. We deliberately write these as functions of the partition rather than ofKK: two partitions with the same cardinality can have very different granularity\.

Figure 1:The feasible regionℱΠ\\mathcal\{F\}\_\{\\Pi\}of \([7](https://arxiv.org/html/2610.19343#S4.E7)\) over relative training budget, for threekk\-means partitions of the Korean–English pool atτ=0\.5%\\tau=0\.5\\%; “feasible” marks a budget inℱΠ\\mathcal\{F\}\_\{\\Pi\}\. Refining the partition fromK=8K\{=\}8toK=16K\{=\}16moves the primary operating pointB/T=0\.40B/T\{=\}0\.40outside the feasible region\. For this pool the binding condition isMMthroughout: the pool is large enough thatGGstays comfortably positive \(\+51\+51to\+504\+504tokens of slack\)\.GGbinds instead on the small gated pools of Section[8](https://arxiv.org/html/2610.19343#S8)\.#### The realization contract\.

It is worth separating two things that are easy to conflate\. The target budget and the target support distribution define the*estimand*\. The toleranceτ\\tau, the no\-overshoot rule and the no\-cross\-support\-compensation rule define the*finite\-pool realization contract*under which we regard a constructed arm as an admissible realization of that estimand\. All three could be relaxed to enlargeℱΠ\\mathcal\{F\}\_\{\\Pi\}, and each relaxation would make an infeasible domain look feasible\.

We fixedτ=0\.5%\\tau=0\.5\\%before any domain\-level feasibility was inspected; under the no\-overshoot rule Lemma[1](https://arxiv.org/html/2610.19343#Thmlemma1)shows this also bounds realized support drift at0\.503%0\.503\\%, which is why one number can govern both matched quantities\. Relaxing the contract after observing infeasibility does not repair the original audit — it defines a different approximation to the target counterfactual, and reports that approximation under the original name\. A sensitivity analysis atτ=1%\\tau=1\\%is legitimate and we would report it as such; using it to rescue a domain that is infeasible at the pre\-specified contract is not\.

## 5Support extinction

###### Proposition 1\(Structural extinction boundary\)\.

If some source support has positive mass and zero gate eligibility, the support\-preserving gate\-eligible counterfactual is infeasible for every positive training budget\.

###### Proof\.

D10D\_\{10\}requiresB​pc≤EcBp\_\{c\}\\leq E\_\{c\}in supportcc\. IfEc=0E\_\{c\}=0andpc\>0p\_\{c\}\>0, then anyB\>0B\>0violates it\. ∎

This is categorically different fromM<0M<0, which a smaller budget repairs\. LetA=\{c:ec\>0\}A=\\\{c:e\_\{c\}\>0\\\}andZ=\{c:ec=0\}Z=\\\{c:e\_\{c\}=0\\\}\. DefineD00allD\_\{00\}^\{\\mathrm\{all\}\}as the source\-preserving arm over all supports andD00AD\_\{00\}^\{A\}as the arm that dropsZZand renormalises the survivors’ original relative proportions, both at the same budgetBB\. WithEA,SA,IAE\_\{A\},S\_\{A\},I\_\{A\}defined as in \([1](https://arxiv.org/html/2610.19343#S3.E1)\)–\([3](https://arxiv.org/html/2610.19343#S3.E3)\) but overAA,

ΔG\\displaystyle\\Delta\_\{G\}=XB\+EA\+SA\+IA,\\displaystyle=X\_\{B\}\+E\_\{A\}\+S\_\{A\}\+I\_\{A\},\(8\)XB\\displaystyle X\_\{B\}=Y⁡\(D00A\)−Y⁡\(D00all\),\\displaystyle=Y\(D\_\{00\}^\{A\}\)\-Y\(D\_\{00\}^\{\\mathrm\{all\}\}\),and the four terms telescope toY⁡\(D11\)−Y⁡\(D00all\)Y\(D\_\{11\}\)\-Y\(D\_\{00\}^\{\\mathrm\{all\}\}\)exactly\. No separateD11AD\_\{11\}^\{A\}is needed: the real gate arm already places zero mass onZZ\.

XBX\_\{B\}is the*support\-extinction contrast*, or extensive\-margin support effect\. It is not “the value of the tag”:D00AD\_\{00\}^\{A\}does not merely discard the extinct mass, it returns that mass to the survivors in their original proportions, soXBX\_\{B\}carries a replacement policy with it\. A different replacement policy is a differentXBX\_\{B\}\. If a cost is wanted, it is−XB\-X\_\{B\}\.

ReportingEA,SA,IAE\_\{A\},S\_\{A\},I\_\{A\}withoutXBX\_\{B\}silently changes the estimand fromΔG\\Delta\_\{G\}toΔG\|A\\Delta\_\{G\\mid A\}, and deletes exactly the phenomenon that motivated looking at the domain\. We treatXBX\_\{B\}as mandatory wheneverZ≠∅Z\\neq\\emptyset\.

## 6Validation before use

The decomposition is only as good as the constructor and the estimator, so we validate both before reading any result, in layers that each fail for a different reason\.

Invariants\.Arms land withinτ\\tauofBB; support histograms match their targets; no holdout example or holdout query appears in any arm\.

Null calibration\.With eligibility replaced by a random mask of the same rate, the decomposition should return nothing\. Over 200 replicates we takeδ=q95​\(\|null\|\)\\delta=q\_\{95\}\(\|\\text\{null\}\|\)per component, with a distribution\-free order\-statistic interval for that quantile\. The null median is00items in both locales and the sign tests are not significant \(p=0\.94/0\.88/0\.38p=0\.94/0\.88/0\.38for EN,0\.27/0\.38/0\.520\.27/0\.38/0\.52for JP\)\.

We state the decision rule explicitly, because it is a threshold and not a level\-α\\alphatest: a component is called*resolved*when\|θ^\|\>δ\|\\hat\{\\theta\}\|\>\\delta, whereθ^\\hat\{\\theta\}is the mean over replicates\. Two caveats follow and we do not hide either\. First, each null replicate is a single selection draw summarized as a median over training seeds, whereasθ^\\hat\{\\theta\}is a mean over3030nested replicates, so the two are not the same statistic\. We do not leave the direction to argument, and we do not argue it from variance alone, since a larger variance does not by itself dominate a quantile\. Aggregating the stored null replicates to the same level asθ^\\hat\{\\theta\}— averaging ten single\-draw replicates, matching the ten selection seeds — and recomputing the same9595th percentile of the absolute value gives4\.64\.6,2\.02\.0and2\.52\.5items forEE,SSandII, against theδ\\deltawe actually use of14\.014\.0,6\.06\.0and7\.07\.0\. The threshold we apply is thus2\.82\.8–3\.0×3\.0\\timesthe one a matched aggregation would produce: the rule is*conservative*, not anticonservative, andδ\\deltais correspondingly not an estimate of the standard error ofθ^\\hat\{\\theta\}\. We keep the pre\-registered single\-drawδ\\deltarather than switching to the matched one, because choosing the smaller threshold after seeing which components it would resolve is exactly the selective judgement the protocol exists to prevent\. Second,δ\\deltais itself estimated and carries an order\-statistic interval, which we report alongside it; a component whose point estimate sits inside that interval should be read as marginal\. To make that reading concrete without inventing a second test, we report a*conservative interval\-separation diagnostic*: whether the effect’s own interval clears the*upper*end ofδ\\delta’s interval,CIlower\>δupper\\mathrm\{CI\}\_\{\\mathrm\{lower\}\}\>\\delta\_\{\\mathrm\{upper\}\}\. This charges the threshold’s estimation error against the effect rather than ignoring it\. It is a diagnostic label, not a hypothesis test, and it never overrides the pre\-specified rule — it only says how much room that verdict had\. We apply no multiplicity correction across the six primary component tests and therefore do not attach a family\-wise error rate to the pattern as a whole\.

Known\-effect recovery \(analytic\)\.On an outcome that is a known function of the arm composition, the model\-impliedE∗,S∗,I∗E^\{\\ast\},S^\{\\ast\},I^\{\\ast\}can be computed in closed form, so the constructor, the decomposition algebra and the interval procedure can be checked against truth without involving a learner\. Over 300 replicates in each of ten settings per locale — four planted conditions \(EEonly,SSonly, both, and one with a planted interaction\) at an interior margin, plus two of those conditions re\-run at each of three frontier margins — the bias was indistinguishable from zero \(\|bias\|≤0\.18\|\\mathrm\{bias\}\|\\leq 0\.18items, within0\.60\.6standard errors\) and the empirical coverage of the nominal 95% intervals was0\.9270\.927–0\.9600\.960over all 60 component\-settings \(Monte\-Carlo error±0\.01\\pm 0\.01\)\. An eleventh setting per locale is the infeasible control below, which raises rather than reports\. This is the check that the hierarchical bootstrap is not anticonservative; it is performed where ground truth exists, because it cannot be performed on the trained runs\.

Known\-effect recovery \(trained\)\.Synthetic interventions that plant an effect in a known component are recovered in the right component with the right sign, now with the learner in the loop\.

Frontier stress\.Shrinking the mass margin fromM=0\.07M=0\.07toM=0\.01M=0\.01at fixed budget left bias, RMSE and coverage unchanged, so proximity to the mass boundary does not by itself degrade the estimator\. It does shrink what there is to estimate: at the tightest margin the constructible support effect was under one holdout item\.

#### Calibration signatures\.

A null band is only valid for the configuration it was measured on\. We bind eachδ\\deltato a signature over locale, partition cardinality, budget, holdout size, training seeds, the gate’s identity and its realized pass rate, endpoint, and the constructor’s source hash, and refuse to attach a verdict unless the signature matches exactly\. This check first fired on our own previously reported results: aδ\\deltacalibrated at one budget and seed count was being applied to runs at another\. Those labels were withdrawn and both locales were recalibrated at their operating points, which is the calibration reported here\. We also require every signature input to be observed or explicitly supplied; a default that fills in a missing field makes the signature assert something nobody measured\.

Figure 2:Component means with 95% hierarchical\-bootstrap confidence intervals, in holdout items, for \(a\) the Korean–English verifier gate, \(b\) the Korean–Japanese verifier gate, and \(c\) ENB, a model\-confidence gate on the Korean–English pool\. Rows are the within\-support selection termEEof \([1](https://arxiv.org/html/2610.19343#S3.E1)\), the between\-support redistribution termSSof \([2](https://arxiv.org/html/2610.19343#S3.E2)\), the interactionIIof \([3](https://arxiv.org/html/2610.19343#S3.E3)\), and the totalΔG=E\+S\+I\\Delta\_\{G\}=E\+S\+I\. Vertical dashed marks are±δ\\pm\\delta, the null band calibrated from 200 placebo replicates*at that locale’s own operating point*\. A solid circle means the effect interval clears the upper end ofδ\\delta’s own interval; a grey circle means only the point estimate clearsδ\\delta, which is the pre\-specified rule and the marginal case; an open square means neither\. The total has no calibrated band and is drawn with a distinct marker\. Panels are separate because the holdout sizes differ \(3,914 vs\. 1,465 items\), so the two item counts are not comparable quantities\.

## 7Retrieval: the effect is within\-support

The gate maps Korean queries onto a product taxonomy leaf and accepts a pair when both a mapping check and a translation check pass\. Support iskk\-means on frozen embeddings of the pre\-gate query,K=8K=8\. The downstream model is a static\-embedding retriever and the endpoint is top\-1 leaf accuracy on a held\-out split\. Each locale runs 10 selection seeds×\\times3 training seeds, and confidence intervals are hierarchical bootstraps over that nesting\.

Table[1](https://arxiv.org/html/2610.19343#S7.T1)and Figure[2](https://arxiv.org/html/2610.19343#S6.F2)give the result\. Within\-support selection is the only component whose mean clears its calibrated null band in either locale; redistribution and interaction remain unresolved at the available power\. The two locales differ in how much room that verdict had: ENEEalso satisfies the conservative separation diagnostic \(CIlower=\+26\.8\\mathrm\{CI\}\_\{\\mathrm\{lower\}\}=\+26\.8againstδupper=17\\delta\_\{\\mathrm\{upper\}\}=17\), while JPEEdoes not \(\+3\.9\+3\.9against66\)\. Signs are30/0/030/0/0forEEand forΔG\\Delta\_\{G\}, and the per\-replicate identity residual is exactly zero\. An item\-level three\-stage bootstrap, which also resamples the evaluation set and is wider by construction, still excludes zero forEE:\[\+16\.1,\+46\.0\]\[\+16\.1,\+46\.0\]for EN and\[\+2\.0,\+9\.8\]\[\+2\.0,\+9\.8\]for JP\.111The JP interval comes from the same10×310\\times 3run as the primary result\. The EN item vectors were stored only for a reduced3×33\\times 3sub\-run, so the EN item\-level interval is computed on nine replicates rather than thirty and is correspondingly less precise\.

Two qualifications belong with that sentence rather than after it\. The Japanese margin is narrow:θ^=\+5\.6\\hat\{\\theta\}=\+5\.6againstδ=5\.0\\delta=5\.0, withδ\\delta’s own order\-statistic interval\[4\.0,6\.0\]\[4\.0,6\.0\]containing the point estimate, and with the lower end of the effect’s own interval \(\+3\.9\+3\.9\) belowδ\\delta\. JPEEtherefore passes the pre\-specified point\-estimate rule and is marginal under the interval\-based reading — a third state, not a softer version of either\. We keep the pre\-specified verdict rather than demoting it after seeing the result, and we decline to present it as the same kind of detection as EN\. The Japanese run is a replication of the component*pattern*, not an independently strong detection\. And the point estimates place nearly all of the observed total onEEin both locales, but that is a statement about point estimates: we do not inferS=I=0S=I=0\.

Table 1:Decomposition at each locale’s primary operating point, in holdout items, over 30 replicates\.δ\\deltais that locale’s own null band — the 95th percentile of\|\|null\|\|over 200 placebo replicates at a matching calibration signature — reported with its order\-statistic interval\. The evidence column names three states rather than starring two, because a star in an ML table reads as “statistically significant” andδ\\deltais not a level\-α\\alphathreshold\.*point only*means the pre\-specified rule\|θ^\|\>δ\|\\hat\{\\theta\}\|\>\\deltapasses;*point\+\+interval*additionally means the effect’s interval clears the*upper*end ofδ\\delta’s own interval\. ENEEreaches the second; JPEEreaches the first and is marginal, its point estimate falling insideδ\\delta’s interval\. A dash means the point estimate did not clearδ\\deltaat this power, which is not a statement that the component is zero\.ΔG\\Delta\_\{G\}is the total and has no calibrated band\.#### A second gate family\.

Everything above audits one gate: a deterministic verifier whose eligibility is a map\-and\-translation check\. Whether CGSD says anything beyond that gate’s mechanism is a question about the method, so we ran it again on a gate that shares none of it — a threshold on the generator’s own confidence score, eligible iffconf≥0\.75\\mathrm\{conf\}\\geq 0\.75\. It is a different intervention in every respect that matters here: it is model\-based rather than rule\-based, it is continuous before it is thresholded, and it passes60\.3%60\.3\\%of the pool against the verifier’s47\.9%47\.9\\%\. That difference in exposure is why it cannot be judged against the verifier’s band; it carries its own200200\-replicate null, and the gate’s identity and realized pass rate are inputs to the calibration signature so the two can never be crossed by accident\. The result is the third panel of Table[1](https://arxiv.org/html/2610.19343#S7.T1):E=\+27\.3E=\+27\.3items with the interval clearingδ\\delta’s upper end,SSandIIunresolved, and the decomposition identity exact to the last digit\. The same component carries the effect, on a gate built on a different principle\. We do not read across the two as an effect\-size comparison — the pass rates differ and that difference is not controlled — but the qualitative verdict is not a property of how the first gate decides\.

#### Reading\.

What the data support is that restricting to eligible examples helps within a fixed support distribution\. At the point estimatesEEaccounts for nearly all of the observed total, but the data do not establish that redistribution is absent or unnecessary:SSandIIare individually unresolved at this power, and no joint interval forS\+IS\+Iwas pre\-specified\. The qualitative component pattern replicates in Japanese:EEalone crosses its pre\-specified point\-estimate band whileSSandIIdo not, and the JPEEresult remains marginal under the interval\-based reading\. We make no cross\-locale claim about effect magnitude, because the operating points and holdout sizes differ\. The Japanese run is a replication of the same gate on the same data family, not an independent domain\.

For readers who want the redistribution channel as a single number, the compositeR=S\+I=ΔG−ER=S\+I=\\Delta\_\{G\}\-Eis−0\.5\-0\.5items for EN and−0\.4\-0\.4for JP by point estimate\. We report it as descriptive: it was not pre\-specified, and it inherits the power limits of its parts\.

#### Partition sensitivity\.

This is sensitivity, not robustness in the usual sense: the partition is part of the estimand, so changing it changes what is being estimated rather than perturbing a fixed target\. AcrossK=4,8,16K=4,8,16at a budget interior for all three partitions, the qualitative conclusion that the resolved component isEEpersists, although its magnitude changes\. The selection term stays positive \(\+16\.0\+16\.0,\+16\.1\+16\.1,\+9\.8\+9\.8items\) with no shift into redistribution\. A paired contrast on the same replicates givesK=16K\{=\}16minusK=8K\{=\}8of−6\.3\-6\.3items\[−12\.4,−0\.7\]\[\-12\.4,\-0\.7\]forEE, an interval excluding zero, with the difference appearing inΔG\\Delta\_\{G\}rather than inSS\. A finer partition thus yields a materially smaller selection estimate, which is expected under a partition\-indexed estimand and is a reason to reportKKwith every number rather than to treat any one value as canonical\. These sensitivity runs use three selection seeds rather than ten and are correspondingly noisier; with only three groups the between\-seed variance component is not usefully estimable and we do not report it\. Replacing the support\-preserving baselineD00D\_\{00\}with the unconstrained referenceRRmoves the estimate by−0\.5\-0\.5items\[−5\.1,\+3\.5\]\[\-5\.1,\+3\.5\], so the choice of baseline does not drive the conclusion\.

## 8Feasibility audits: extinction, narrow bands, and empty regions

We applied the same audit to a gate over generated mathematics problems, chosen precisely because a preliminary analysis suggested the gate had removed whole problem types\. The support label is the generator’s own pre\-gate template tag, so it satisfies the pre\-gate requirement\.

Two of four supports haveec=0e\_\{c\}=0\(Table[2](https://arxiv.org/html/2610.19343#S8.T2)\)\. By Proposition[1](https://arxiv.org/html/2610.19343#Thmproposition1)the ordinary four\-arm construction is undefined at every positive budget, which we confirm empirically down toB=0\.1%B=0\.1\\%ofTT\. The extinct mass is6\.1%6\.1\\%of tokens and9\.0%9\.0\\%of rows; we report the token figure because exposure is matched in tokens, and the two differ by a factor of1\.51\.5here\.

Extending to \([8](https://arxiv.org/html/2610.19343#S5.E8)\) makes the arms definable\. At the pre\-registered operating point it does not yet make them constructible, and the reason turns out to be instructive\. At the survivor\-conditional interior budget, the survivor arms assemble with99tokens of slack, butD00allD\_\{00\}^\{\\mathrm\{all\}\}hasG=−3G=\-3: no assembly of these examples lands within tolerance, and no packer can change that, since the subset\-sum here is solved exactly\. The binding mismatch is concentrated in the extinct supports, which supply11\.911\.9of the13\.013\.0tokens ofQQ, or 92%\. In this pool the extinct supports are also the smallest supports, so their token lattices are too coarse to satisfy the extensive\-margin match\. We do not claim this coincidence in general: a large support can also be wiped out entirely by a gate, and would not produce the same assembly failure\.

The same audit applied to five gated text\-to\-SQL pools from the same generator finds no extinction but a harsher assembly picture, and — after the budget scan above — one exception\. The largest pool \(578578rows\) hasminc⁡ec=0\.070\\min\_\{c\}e\_\{c\}=0\.070, which is below the interior margin we fixed in advance; we report that as having no*power\-safe*operating point rather than an emptyℱΠ\\mathcal\{F\}\_\{\\Pi\}, since the margin is a power criterion and \([7](https://arxiv.org/html/2610.19343#S4.E7)\) only asksM≥0M\\geq 0\. Scanning admissible budgets underM≥0M\\geq 0on a uniform grid \(2,7952\{,\}795budgets\), four of the five pools have an empty feasible region: not one budget givesG≥0G\\geq 0for every arm\. The fifth,live\-clean\(9999rows\), is constructible\. Of the8686grid budgets that satisfy the assembly condition we attempted all8686, and the arms build at6868, spanningB/T∈\[0\.309,0\.643\]B/T\\in\[0\.309,0\.643\]\. The1818that fail all fail the same way, and the reason is worth stating because it bounds whatGGpromises:G≥0G\\geq 0certifies that an in\-tolerance assembly exists, our assembler solves the subset\-sum exactly*within each support*, and the tolerance is checked on the*total*\. Per\-support exactness does not compose into a global one, so a budget can clear the condition and still defeat the solver\. That gap —79%79\\%of certified budgets realized — is a property of the algorithm, reported separately from the pool’s own limit\. SQL rows average∼155\\sim\\\!155tokens against∼39\\sim\\\!39for the mathematics pool, so the same tolerance buys four times less assembly freedom, which is why the feasible band is narrow where it exists at all\. Cross\-domain decomposition therefore waits on data, not on further method development, and we can say by how much\. A single retrieval arm is trained on95,16295\{,\}162tokens; the entire mathematics pool is3,9153\{,\}915tokens and the entirelive\-cleanpool is15,04215\{,\}042, so one arm of the retrieval design consumes24×24\\timesand6×6\\timestheir respective totals\. At the feasible budget a mathematics arm sees roughly4949rows\. We could build these arms — we have — and report a decomposition from them; we do not, because a component estimate at that scale would carry an interval wide enough to contain any conclusion, and publishing it would be the failure this paper is about\. The endpoint compounds the gap: answer correctness over a generative model is not the retrieval endpoint the estimator here is calibrated for, so a second calibration would be required before any verdict could be attached\. What these pools establish is that the construction exists and that its feasibility conditions do the work they were introduced for; what they cannot establish is an effect\.

Table 2:Support structure of the gated mathematics pool\. Extinct supports have positive source mass and zero gate\-eligible mass, so Proposition[1](https://arxiv.org/html/2610.19343#Thmproposition1)applies\. Extinct supports hold 6\.1% of tokens \(9\.0% of rows\)\. At the survivor\-conditional interior budgetB=1,989B=1\{,\}989tokens \(Msafe=0\.08M\_\{\\mathrm\{safe\}\}=0\.08, tolerance0\.5%0\.5\\%\), the survivor arms assemble withG=\+9G=\+9whileD00allD\_\{00\}^\{\\mathrm\{all\}\}hasG=−3G=\-3\. At the pre\-registered operating point the survivor\-conditional arms assemble and the arm that must reproduce the full source support does not\. That failure is specific to this budget: see the budget scan in the text\.It would be easy to stop here and report that the extended decomposition is not realizable on this pool\. That conclusion does not survive our own analysis\.GGis a lattice quantity and is not monotone inBB, so a verdict read off one budget is a verdict about that budget\. Scanning every admissible integer budget exhaustively,5454of them satisfy the same pre\-registered margin*and*giveG≥0G\\geq 0for all five arms, in a band just below the operating point \(B/T∈\[0\.480,0\.495\]B/T\\in\[0\.480,0\.495\]\); the point we had fixed,B=1989B=1989, is the largest the margin allows and falls in a gap between such bands\. Moving5050tokens down toB=1939B=1939flipsD00allD\_\{00\}^\{\\mathrm\{all\}\}fromG=−3G=\-3toG=\+0\.7G=\+0\.7, and the five arms then build within tolerance \(worst deviation0\.46%0\.46\\%againstτ=0\.5%\\tau=0\.5\\%\)\. Attempting all5454certified budgets together with the pre\-registered one,4949of5555build, and the six failures separate along exactly the lines the taxonomy predicts:B=1989B=1989reportsgranularity— no assembly exists — and the other five reportpacker— one exists and the solver does not find it\. Recovering the4949required replacing the greedy packer with an exact subset\-sum assembler, randomized over the witnesses that achieve the attainable sum so that selection within a support is not quietly made a function of example length\. The residual five are the same per\-support\-versus\-global gap described below: the assembler is exact inside each support, whileτ\\tauis checked on the total\.

What we therefore report is narrower and, we think, more useful than the claim we set out to make: extinction makes the ordinary four\-arm estimand undefined at every budget, and that part is structural; the extended design is constructible on this pool, but only inside narrow budget bands that the natural choice — the largest budget the power margin allows — misses\. This is the assembly condition doing exactly what it was introduced for, on us\. The survivor\-conditional terms could be computed and would look like a result; substituting them forΔG\\Delta\_\{G\}would silently replace the estimand withΔG\|A\\Delta\_\{G\\mid A\}and delete the extinction that motivated the domain, which is the error this construction exists to prevent\. We do not train on this pool\.

## 9Limitations

The retrieval evidence covers two gate families and two locales, but one backbone, one endpoint, and one data family; it is a replication across gates and languages, not a demonstration of domain generality\. The second gate is also lower powered by design \(33selection seeds against1010\), which is why its selection\-variance component is estimated at zero: training noise dominates at that number of groups, and the one\-way estimator truncates\. The cross\-domain claim that motivated this work — that the same gate helps retrieval and hurts mathematics — is*not*established here, because the mathematics side is a feasibility result rather than a negative one\. Unresolved components are statements about power, not about zero; the Japanese null band is 2 items against a 1\-item measurement grain\.EEremains uninterpreted as to mechanism\. Finally,ℱΠ\\mathcal\{F\}\_\{\\Pi\}is defined relative to the matching contract; a different contract is a different feasible region, and we regard changing it after seeing a failure as out of bounds\.

## 10Conclusion

A gate\-versus\-random comparison does not, by itself, isolate within\-support selection from support redistribution\. CGSD separates them by construction, and the price of the construction is that the arms have to exist\. Making existence explicit — a mass condition, an assembly condition, and a proposition for the case that no budget repairs — turned out to be at least as informative as the effect estimates\. On retrieval, the observed benefit is dominated in point estimates by within\-support selection; redistribution is not resolved at the available power, which is not the same as being zero\. On mathematics the honest output of the audit is not a number but a structure: extinction leaves the ordinary comparison undefined at every budget, while the extended comparison is constructible only inside narrow budget bands that the operating point we had pre\-registered happens to miss\. Four of five text\-to\-SQL pools have empty feasible regions and the fifth has a narrow band\. None of that is a number a practitioner can act on, and all of it is something they need before trusting one\.

## Availability

The implementation is released at[https://github\.com/ThakiCloud/cgsd](https://github.com/ThakiCloud/cgsd)under Apache\-2\.0, together with the recorded result files and the scripts that generate every figure, table and number in this paper from them\. Nothing here is transcribed by hand, so a reader can rerun that pipeline and reproduce the reported values exactly\. The constructor, both feasibility tests and the extinction split ship with the data\-free gate harnesses used to pin them, which execute without any corpus\. What is not released is the data: the source pools and evaluation splits are not ours to redistribute, so retraining the arms requires equivalent data of the reader’s own\. The estimand, the decomposition identity, both feasibility conditions, the extinction proposition and the drift bound are fully specified in Sections[3](https://arxiv.org/html/2610.19343#S3)–[5](https://arxiv.org/html/2610.19343#S5)and can be reimplemented from the text alone\.

## References

- \[1\]Amro Abbas, Kushal Tirumala, Dániel Simig, Surya Ganguli, and Ari S\. Morcos\.SemDeDup: Data\-efficient learning at web\-scale through semantic deduplication\.*arXiv preprint arXiv:2303\.09540*, 2023\.
- \[2\]Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G\. Bellemare\.Deep reinforcement learning at the edge of the statistical precipice\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, volume 34, 2021\.
- \[3\]Alan S\. Blinder\.Wage discrimination: Reduced form and structural estimates\.*Journal of Human Resources*, 8\(4\):436–455, 1973\.
- \[4\]Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gaël Varoquaux, and Pascal Vincent\.Accounting for variance in machine learning benchmarks\.In*Proceedings of Machine Learning and Systems \(MLSys\)*, volume 3, pages 747–769, 2021\.
- \[5\]Mayee F\. Chen, Michael Y\. Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré\.Aioli: A unified optimization framework for language model data mixing\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- \[6\]Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A\. Smith\.Show your work: Improved reporting of experimental results\.In*Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing \(EMNLP\-IJCNLP\)*, pages 2185–2194, 2019\.
- \[7\]Logan Engstrom, Axel Feldmann, and Aleksander Madry\.DsDm: Model\-aware dataset selection with datamodels\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, 2024\.
- \[8\]Simin Fan, Matteo Pagliardini, and Martin Jaggi\.DoGE: Domain reweighting with generalization estimation\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, 2024\.
- \[9\]Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, et al\.DataComp: In search of the next generation of multimodal datasets\.In*Advances in Neural Information Processing Systems \(NeurIPS\), Datasets and Benchmarks Track*, 2023\.
- \[10\]Amirata Ghorbani and James Zou\.Data shapley: Equitable valuation of data for machine learning\.In*Proceedings of the 36th International Conference on Machine Learning \(ICML\)*, pages 2242–2251, 2019\.
- \[11\]Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilė Lukošiūtė, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R\. Bowman\.Studying large language model generalization with influence functions\.*arXiv preprint arXiv:2308\.03296*, 2023\.
- \[12\]Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger\.Deep reinforcement learning that matters\.In*Proceedings of the Thirty\-Second AAAI Conference on Artificial Intelligence \(AAAI\)*, 2018\.
- \[13\]Michael G\. Hudgens and M\. Elizabeth Halloran\.Toward causal inference with interference\.*Journal of the American Statistical Association*, 103\(482\):832–842, 2008\.
- \[14\]Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry\.Datamodels: Predicting predictions from training data\.*arXiv preprint arXiv:2202\.00622*, 2022\.
- \[15\]Pang Wei Koh and Percy Liang\.Understanding black\-box predictions via influence functions\.In*Proceedings of the 34th International Conference on Machine Learning \(ICML\)*, 2017\.
- \[16\]Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison\-Burch, and Nicholas Carlini\.Deduplicating training data makes language models better\.In*Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\)*, 2022\.
- \[17\]Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin\.RegMix: Data mixture as regression for language model pre\-training\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- \[18\]Ronald Oaxaca\.Male\-female wage differentials in urban labor markets\.*International Economic Review*, 14\(3\):693–709, 1973\.
- \[19\]Judea Pearl\.Direct and indirect effects\.In*Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence \(UAI\)*, pages 411–420, 2001\.
- \[20\]David Picard\.torch\.manual\_seed\(3407\) is all you need: On the influence of random seeds in deep learning architectures for computer vision\.*arXiv preprint arXiv:2109\.08203*, 2021\.
- \[21\]Joelle Pineau, Philippe Vincent\-Lamarre, Koustuv Sinha, Vincent Larivière, Alina Beygelzimer, Florence d’Alché Buc, Emily Fox, and Hugo Larochelle\.Improving reproducibility in machine learning research \(a report from the NeurIPS 2019 reproducibility program\)\.*Journal of Machine Learning Research*, 22\(164\):1–20, 2021\.
- \[22\]Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale\.Estimating training data influence by tracing gradient descent\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2020\.
- \[23\]Paul R\. Rosenbaum and Donald B\. Rubin\.The central role of the propensity score in observational studies for causal effects\.*Biometrika*, 70\(1\):41–55, 1983\.
- \[24\]Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S\. Morcos\.Beyond neural scaling laws: Beating power law scaling via data pruning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2022\.
- \[25\]Elizabeth A\. Stuart\.Matching methods for causal inference: A review and a look forward\.*Statistical Science*, 25\(1\):1–21, 2010\.
- \[26\]Jiachen T\. Wang, Prateek Mittal, Dawn Song, and Ruoxi Jia\.Data shapley in one training run\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.
- \[27\]Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen\.LESS: Selecting influential data for targeted instruction tuning\.In*Proceedings of the 41st International Conference on Machine Learning \(ICML\)*, 2024\.
- \[28\]Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc V\. Le, Tengyu Ma, and Adams Wei Yu\.DoReMi: Optimizing data mixtures speeds up language model pretraining\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023a\.
- \[29\]Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang\.Data selection for language models via importance resampling\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2023b\.
- \[30\]Jiasheng Ye, Peiju Liu, Tianxiang Sun, Jun Zhan, Yunhua Zhou, and Xipeng Qiu\.Data mixing laws: Optimizing data mixtures by predicting language modeling performance\.In*International Conference on Learning Representations \(ICLR\)*, 2025\.