GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution

# GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
Source: [https://arxiv.org/html/2606.06892](https://arxiv.org/html/2606.06892)
###### Abstract

Scalable data attribution methods typically assign isolated utility scores to individual training examples\. This prevalent additive assumption fundamentally fails to capture critical subset dynamics, including data redundancy and complementary coverage\. In this work, we reframe attribution as subset\-level counterfactual utility prediction and introduce GRASP, an interaction\-aware surrogate\. Grounded in a theoretical smoothness lower bound, GRASP explicitly models subset interactions through a quadratic geometric penalty\. To achieve pretraining\-scale efficiency without relying on hidden oracle tuning, we couple low\-dimensional feature sketches with a strictly finite lower\-confidence bound selection protocol\. Extensive subset\-retraining evaluations demonstrate that GRASP decisively outperforms existing scalable baselines\. It more than doubles the task\-level rank correlation for counterfactual subset fidelity while reducing upfront artifact construction costs by nearly an order of magnitude\. Downstream diagnostics further show that this scoring mechanism transfers to language model curation and cross\-domain vision selection, establishing a robust foundation for optimizing massive pretraining corpora\.

GRASP: Geometry\-aware Residual Alignment for Scalable Pretraining Data Attribution

Yue Min1, Ruining Chen2, Yujun Li11Wizard Quant,2University of Science and Technology of ChinaCorrespondence:[minyue@wizardquant\.com](https://arxiv.org/html/2606.06892v1/mailto:[email protected]),[liyujun@wizardquant\.com](https://arxiv.org/html/2606.06892v1/mailto:[email protected])

## 1Introduction

The capabilities and failure modes of large language models are fundamentally determined by their pretraining corpora\(Raffelet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib11); Penedoet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib20)\)\. Consequently, data curation has become the central driver of model performance\. This process encompasses extraction, filtering, deduplication, and domain mixing\. Yet, these structural decisions typically rely on coarse heuristics or prohibitively expensive retraining trials\. Data attribution bridges this methodological gap by connecting model behavior back to the data pipeline\. This connection enables targeted auditing, contamination analysis, and principled data selection under fixed compute budgets\.

![Refer to caption](https://arxiv.org/html/2606.06892v1/x1.png)Figure 1:Overview of the GRASP architecture\.GRASP predicts the counterfactual utility of data interventions by combining an additive relevance core \(bottom\) with an interaction\-aware geometric penalty \(top\)\. By processing LLM hidden states and residuals into compact direct\-sum sketches \(ϕi\\phi\_\{i\}\) and stabilized relevance scores, the method efficiently captures subset redundancy and complementary coverage\. The standardized components are then linearly aggregated to evaluate entire subsets without combinatorial retraining\.Despite these promises, scaling attribution to guide practical curation reveals a fundamental flaw in current paradigms\. Existing scalable methods, including influence functions\(Koh and Liang,[2017](https://arxiv.org/html/2606.06892#bib.bib1)\), TracIn\(Pruthiet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib13)\), TRAK\(Parket al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib16)\), DataInf\(Kwonet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib18)\), and LESS\(Xiaet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib19)\), operate by assigning a scalar relevance score to each individual training example\. However, practical data curation involves retaining or removing coherent data slices rather than isolated sentences\. When practitioners use pointwise scores to evaluate a data subset, they implicitly treat the utility of that subset as a simple sum of individual contributions\. This additive assumption is mathematically restrictive\. In real\-world corpora containing near\-duplicates and overlapping domains, training examples interact\. Selecting highly aligned but identical examples yields diminishing returns, whereas diverse examples provide complementary coverage\.

While coalition\-based valuations such as Data Shapley\(Ghorbani and Zou,[2019](https://arxiv.org/html/2606.06892#bib.bib6)\)and datamodeling\(Ilyaset al\.,[2022](https://arxiv.org/html/2606.06892#bib.bib23); Wanget al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib12)\)formally capture these subset dependencies, exact coalition evaluation is computationally prohibitive for routine pretraining pipelines\. This creates a critical methodological gap\. Scalable attribution requires a reusable surrogate that explicitly models train\-train interactions without requiring combinatorial retraining\. Consequently, the evaluation of attribution methods must structurally shift from individual example ranking to*subset\-level counterfactual utility prediction*\. A rigorous attribution method must accurately rank candidate data interventions by their empirical utility after retraining\.

To address this challenge, we propose GRASP, a geometry\-aware residual\-alignment surrogate for scalable pretraining data attribution\. Our approach is grounded in a one\-step smoothness lower bound on utility improvement for weighted subset interventions\. This theoretical foundation naturally yields a linear target relevance term alongside a quadratic geometric penalty\. Expanding this quadratic expression formally captures the mutual dependencies omitted by additive scoring\. For pretraining\-scale deployment, GRASP replaces exact high\-dimensional update directions with low\-dimensional feature sketches\. For experimental discipline, all component weights and retain/delete modes are selected on development environments before test evaluation, separating scalable subset prediction from hidden per\-task tuning\.

In summary, our main contributions are:

- •We formalize pretraining data attribution as*subset\-level counterfactual utility prediction*, directly aligning the objective with practical corpus interventions to overcome the limitations of additive pointwise scoring\.
- •We derive GRASP, an interaction\-aware surrogate grounded in a one\-step smoothness lower bound\. Its quadratic geometric penalty explicitly models data redundancy and complementary coverage without requiring combinatorial retraining\.
- •We instantiate GRASP with compact feature sketches for pretraining\-scale efficiency\. It reduces artifact construction time by nearly an order of magnitude, scores 100,000 subsets in 5 seconds, and more than doubles the task\-level LDS rank correlation over existing scalable baselines\.

## 2Problem Setup

We study*subset\-level counterfactual utility prediction*: given a pretraining corpus, an evaluation target, and candidate data interventions, the objective is to rank the interventions by their realized utility after retraining\.

### 2\.1Pretraining Data Attribution

Let𝒟=\{1,…,n\}\\mathcal\{D\}=\\\{1,\\ldots,n\\\}index a pretraining corpus\{zi\}i∈𝒟\\\{z\_\{i\}\\\}\_\{i\\in\\mathcal\{D\}\}\. A data intervention retains a subsetS⊆𝒟S\\subseteq\\mathcal\{D\}, yielding parametersθS\\theta\_\{S\}after retraining\. For a targetee, we define the utility as𝒰e​\(θ\)=−Le​\(θ\)\\mathcal\{U\}\_\{e\}\(\\theta\)=\-L\_\{e\}\(\\theta\), whereLeL\_\{e\}is the target loss\. The true subset utility is therefore

𝒰e​\(S\)=𝒰e​\(θS\)=−Le​\(θS\)\.\\mathcal\{U\}\_\{e\}\(S\)=\\mathcal\{U\}\_\{e\}\(\\theta\_\{S\}\)=\-L\_\{e\}\(\\theta\_\{S\}\)\.\(1\)Pretraining attribution aims to provide a scalable surrogate𝒰^e:2𝒟→ℝ\\widehat\{\\mathcal\{U\}\}\_\{e\}:2^\{\\mathcal\{D\}\}\\to\\mathbb\{R\}for this set functionS↦𝒰e​\(S\)S\\mapsto\\mathcal\{U\}\_\{e\}\(S\)\.

While traditional attribution assigns a scalar scoreai,e∈ℝa\_\{i,e\}\\in\\mathbb\{R\}to each exampleii\(Koh and Liang,[2017](https://arxiv.org/html/2606.06892#bib.bib1); Parket al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib16); Xiaet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib19)\), a subset\-level surrogate predicts the utility of the entire interventionSS\.

### 2\.2Counterfactual Fidelity via LDS

Following datamodeling\(Ilyaset al\.,[2022](https://arxiv.org/html/2606.06892#bib.bib23); Parket al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib16)\), we evaluate counterfactual fidelity using the Linear Datamodeling Score \(LDS\)\. Given a family ofBBsampled interventionsℬ=\{Sb\}b=1B\\mathcal\{B\}=\\\{S\_\{b\}\\\}\_\{b=1\}^\{B\}, LDS measures the rank correlation between predicted and realized utilities for targetee:

ρe=Spearmanb⁡\(𝒰^e​\(Sb\),𝒰e​\(Sb\)\)\.\\rho\_\{e\}=\\operatorname\{Spearman\}\_\{b\}\\left\(\\widehat\{\\mathcal\{U\}\}\_\{e\}\(S\_\{b\}\),\\mathcal\{U\}\_\{e\}\(S\_\{b\}\)\\right\)\.\(2\)
When relying on pointwise scores, the subset predictor degrades to an additive baseline:

𝒰^eadd​\(S\)=∑i∈Sai,e\.\\widehat\{\\mathcal\{U\}\}^\{\\mathrm\{add\}\}\_\{e\}\(S\)=\\sum\_\{i\\in S\}a\_\{i,e\}\.\(3\)This modular form assumes independence among examples, rendering it brittle for corpora containing duplicates and domain clusters\. We therefore seek a scalable surrogate incorporating a structural subset interaction term to improveρe\\rho\_\{e\}\.

## 3Method

GRASP predicts the utility of a candidate data intervention without retraining for every query\. It first computes residual\-alignment relevance scores for train–target pairs, then scores a whole subset by adding geometry\-aware train\-train interactions, and finally selects a fixed protocol on development environments\.

Figure[1](https://arxiv.org/html/2606.06892#S1.F1)summarizes this pipeline\.

### 3\.1GRASP Relevance Scores

To evaluate the utility of a data intervention, GRASP first requires a scalable measure of alignment between training and target examples\. Rather than computing exact but intractable influence functions \(H−1​∇ℓH^\{\-1\}\\nabla\\ell\), we derive an efficient proxy from the last\-layer geometry of causal language models\. Lethih\_\{i\}andξi=pi−yi\\xi\_\{i\}=p\_\{i\}\-y\_\{i\}denote the hidden state and next\-token residual for a given token, where the local gradient admits an exact outer\-product form∇Wℓtok=h​ξ⊤\\nabla\_\{W\}\\ell\_\{\\mathrm\{tok\}\}=h\\xi^\{\\top\}\. For sequences containing multiple tokens, exact gradients require summing these outer products\. To maintain linear storage complexity, we instead pool the sequence into a single summary representation and employ a rank\-1 surrogate:∇Wℓi≈hi​ξi⊤\\nabla\_\{W\}\\ell\_\{i\}\\approx h\_\{i\}\\xi\_\{i\}^\{\\top\}\. While this simplification formally drops intra\-sequence covariance, it effectively captures the macroscopic alignment of the sequence\.

Under a Kronecker\-factored generalized Gauss–Newton \(GGN\) approximation\(Martens and Grosse,[2015](https://arxiv.org/html/2606.06892#bib.bib34)\), the inverse empirical Fisher or GGN matrix is approximated as a Kronecker productF−1≈Σh−1⊗DξF^\{\-1\}\\approx\\Sigma\_\{h\}^\{\-1\}\\otimes D\_\{\\xi\}\. Consequently, the inner product of preconditioned gradients elegantly factorizes into separate hidden and residual matches:

⟨∇Wℓi,F−1​∇Wℓe⟩≈\(hi⊤​Σh−1​he\)​\(ξi⊤​Dξ​ξe\)\.\\langle\\nabla\_\{W\}\\ell\_\{i\},F^\{\-1\}\\nabla\_\{W\}\\ell\_\{e\}\\rangle\\approx\\left\(h\_\{i\}^\{\\top\}\\Sigma\_\{h\}^\{\-1\}h\_\{e\}\\right\)\\left\(\\xi\_\{i\}^\{\\top\}D\_\{\\xi\}\\xi\_\{e\}\\right\)\.\(4\)Motivated by this decomposition, for a target evaluation uniteewith representationshe,ξeh\_\{e\},\\xi\_\{e\}, we whiten the hidden states using global training statistics:

h~i=Σh−1/2​\(hi−h¯\),h~e=Σh−1/2​\(he−h¯\),\\tilde\{h\}\_\{i\}=\\Sigma\_\{h\}^\{\-1/2\}\(h\_\{i\}\-\\bar\{h\}\),\\qquad\\tilde\{h\}\_\{e\}=\\Sigma\_\{h\}^\{\-1/2\}\(h\_\{e\}\-\\bar\{h\}\),\(5\)whereΣh−1/2\\Sigma\_\{h\}^\{\-1/2\}incorporates Tikhonov regularization\. The factorized similarities are then defined as:

Hi,e\\displaystyle H\_\{i,e\}=⟨h~i,h~e⟩,\\displaystyle=\\langle\\tilde\{h\}\_\{i\},\\tilde\{h\}\_\{e\}\\rangle,\(6\)Ri,e\\displaystyle R\_\{i,e\}=ξi⊤​Dξ​ξe\.\\displaystyle=\\xi\_\{i\}^\{\\top\}D\_\{\\xi\}\\xi\_\{e\}\.\(7\)Here,Dξ=diag​\(γ\+ϵ\)−αD\_\{\\xi\}=\\mathrm\{diag\}\(\\gamma\+\\epsilon\)^\{\-\\alpha\}\(α∈\{1,1/2\}\\alpha\\in\\\{1,1/2\\\}\) serves as a stabilized inverse\-covariance preconditioner for the residuals, withγ\\gammaestimated via a sparse top\-kkresidual sketch to minimize the memory footprint\.

Strict adherence to local influence dictates a direct bilinear productHi,e​Ri,eH\_\{i,e\}R\_\{i,e\}\. In practice, however, hidden\-state inner products exhibit extreme heavy\-tailed distributions that routinely destabilize subset evaluation\. To robustify the formulation, we explicitly depart from the exact Taylor expansion and introduce a monotonic dampening function:

ai,eGRASP=Ri,e​sgn⁡\(Hi,e\)​log⁡\(1\+\|Hi,e\|\)\.a^\{\\mathrm\{GRASP\}\}\_\{i,e\}=R\_\{i,e\}\\,\\operatorname\{sgn\}\(H\_\{i,e\}\)\\log\(1\+\|H\_\{i,e\}\|\)\.\(8\)This odd, strictly increasing log\-transformation preserves both the ordinal ranking and the directional support \(sign\) of the representation match, while safely truncating the disproportionate leverage of geometric outliers\.

Finally, the first\-order relevance of a weighted intervention subsetTTis computed additively:

AeGRASP​\(T\)=∑i∈Twi​ai,eGRASP\.A^\{\\mathrm\{GRASP\}\}\_\{e\}\(T\)=\\sum\_\{i\\in T\}w\_\{i\}a^\{\\mathrm\{GRASP\}\}\_\{i,e\}\.\(9\)GRASP adoptsAeGRASP​\(T\)A^\{\\mathrm\{GRASP\}\}\_\{e\}\(T\)as its foundational linear predictor, which is subsequently augmented with geometry\-aware interaction penalties to model subset redundancy\.

### 3\.2Interaction\-Aware Subset Utility

Letge=∇θLe​\(θ\)g\_\{e\}=\\nabla\_\{\\theta\}L\_\{e\}\(\\theta\)be the target gradient at a reference checkpoint\. As derived in Appendix[A](https://arxiv.org/html/2606.06892#A1), implicit differentiation of a weighted training objective yields the influence\-directed updateui=H−1​∇θℓi​\(θ\)u\_\{i\}=H^\{\-1\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)for exampleii, up to a shared scale\. A standard scalar attribution method scores a subsetSSadditively via this first\-order alignment:

𝒰^eadd​\(S\)=∑i∈Sai,e,ai,e≈⟨ge,ui⟩\.\\widehat\{\\mathcal\{U\}\}^\{\\mathrm\{add\}\}\_\{e\}\(S\)=\\sum\_\{i\\in S\}a\_\{i,e\},\\qquad a\_\{i,e\}\\approx\\langle g\_\{e\},u\_\{i\}\\rangle\.\(10\)However, this modular form assumes strict independence, fundamentally failing to capture redundancy or complementarity within a subset\. To formally model subset dynamics, we must examine the geometry of the combined update, which naturally incurs a curvature penalty\.

###### Theorem 1\(One\-step interaction lower bound\)\.

Fix parametersθ\\thetaand targetee\. Letge=∇θLe​\(θ\)g\_\{e\}=\\nabla\_\{\\theta\}L\_\{e\}\(\\theta\), step sizeη≥0\\eta\\geq 0, and update directionui∈ℝdim\(θ\)u\_\{i\}\\in\\mathbb\{R\}^\{\\dim\(\\theta\)\}\. For a weighted setTTwith weightswiw\_\{i\}, define the combined updateDT=∑i∈Twi​uiD\_\{T\}=\\sum\_\{i\\in T\}w\_\{i\}u\_\{i\}\. IfLeL\_\{e\}isβe\\beta\_\{e\}\-smooth \(with respect to the Euclidean norm\) betweenθ\\thetaandθ−η​DT\\theta\-\\eta D\_\{T\}, the target utility improvementΔe​\(T\)=𝒰e​\(θ−η​DT\)−𝒰e​\(θ\)\\Delta\_\{e\}\(T\)=\\mathcal\{U\}\_\{e\}\(\\theta\-\\eta D\_\{T\}\)\-\\mathcal\{U\}\_\{e\}\(\\theta\)satisfies:

Δe​\(T\)\\displaystyle\\Delta\_\{e\}\(T\)≥η​∑i∈Twi​⟨ge,ui⟩\\displaystyle\\geq\\eta\\sum\_\{i\\in T\}w\_\{i\}\\langle g\_\{e\},u\_\{i\}\\rangle\(11\)−βe​η22​‖∑i∈Twi​ui‖2\.\\displaystyle\\quad\-\\frac\{\\beta\_\{e\}\\eta^\{2\}\}\{2\}\\left\\\|\\sum\_\{i\\in T\}w\_\{i\}u\_\{i\}\\right\\\|^\{2\}\.

The proof \(Appendix[A](https://arxiv.org/html/2606.06892#A1)\) bounds the counterfactual utility of full retraining\. Crucially, expanding the quadratic penalty isolates the pairwise interaction:

‖∑i∈Twi​ui‖2=∑i∈Twi2​‖ui‖2\+2​∑i<ji,j∈Twi​wj​⟨ui,uj⟩\.\\left\\\|\\sum\_\{i\\in T\}w\_\{i\}u\_\{i\}\\right\\\|^\{2\}=\\sum\_\{i\\in T\}w\_\{i\}^\{2\}\\\|u\_\{i\}\\\|^\{2\}\+2\\sum\_\{\\begin\{subarray\}\{c\}i<j\\\\ i,j\\in T\\end\{subarray\}\}w\_\{i\}w\_\{j\}\\langle u\_\{i\},u\_\{j\}\\rangle\.\(12\)This reveals that mutually aligned examples \(⟨ui,uj⟩\>0\\langle u\_\{i\},u\_\{j\}\\rangle\>0\) incur a strictly positive off\-diagonal penalty\. In terms of discrete curvature, the second difference of the set function satisfiesΔi​Δj​𝒰^eint∝−wi​wj​⟨ui,uj⟩\\Delta\_\{i\}\\Delta\_\{j\}\\widehat\{\\mathcal\{U\}\}^\{\\mathrm\{int\}\}\_\{e\}\\propto\-w\_\{i\}w\_\{j\}\\langle u\_\{i\},u\_\{j\}\\rangle\. Thus, aligned directions exhibit diminishing marginal returns, while structurally opposing directions can be complementary \(see Proposition[1](https://arxiv.org/html/2606.06892#Thmproposition1)\)\.

While Theorem[1](https://arxiv.org/html/2606.06892#Thmtheorem1)dictates the necessity of the quadratic penalty, calculating the exact coefficientβe​η2/2\\beta\_\{e\}\\eta^\{2\}/2is intractable for pretraining\-scale interventions, asβe\\beta\_\{e\}andη\\etafluctuate with subset norm and model scale\. We therefore treat the theorem as a structural blueprint: a linear relevance term regularized by a non\-negative geometric penalty\. By replacing the intractable scalar with a tunable protocol parameterλe≥0\\lambda\_\{e\}\\geq 0, we arrive at the idealized interaction\-aware surrogate:

𝒰^eint​\(T\)=η​Aeu​\(T\)−λe​Ku​\(T\),\\widehat\{\\mathcal\{U\}\}^\{\\mathrm\{int\}\}\_\{e\}\(T\)=\\eta A^\{u\}\_\{e\}\(T\)\-\\lambda\_\{e\}K^\{u\}\(T\),\(13\)whereAeu​\(T\)=∑i∈Twi​⟨ge,ui⟩A^\{u\}\_\{e\}\(T\)=\\sum\_\{i\\in T\}w\_\{i\}\\langle g\_\{e\},u\_\{i\}\\ranglecaptures the additive utility, andKu​\(T\)=‖DT‖2K^\{u\}\(T\)=\\\|D\_\{T\}\\\|^\{2\}\. In the implemented predictor,λe\\lambda\_\{e\}is absorbed into a finite set of component weights calibrated on development environments\.

Table 1:Counterfactual subset\-utility fidelity on full\-dataset benchmarks\.GRASP consistently outperforms all baselines in predicting empirical retraining outcomes\. We report the task\-level Spearman rank correlation \(ρ\\rho\) and the fraction of evaluation units exhibiting positive LDS correlation \(Pos\.\)\. Higher is better for both metrics\. Best and second\-best results areboldedandunderlined, respectively\.
### 3\.3Sketched Subset Predictor

Directly storing high\-dimensional update directionsui∈ℝdim\(θ\)u\_\{i\}\\in\\mathbb\{R\}^\{\\dim\(\\theta\)\}for every pretraining row is computationally intractable\. Under the Kronecker\-factored view, the true geometric interaction⟨ui,uj⟩\\langle u\_\{i\},u\_\{j\}\\rangletakes the form of a product kernel between hidden\-state and residual similarities\. However, explicitly constructing the corresponding tensor\-product featureψi=\(Wh​\(hi−h¯\)\)⊗\(Wr​ξi\)\\psi\_\{i\}=\(W\_\{h\}\(h\_\{i\}\-\\bar\{h\}\)\)\\otimes\(W\_\{r\}\\xi\_\{i\}\)incurs a prohibitiveO​\(dh​dr\)O\(d\_\{h\}d\_\{r\}\)memory footprint\. To achieve linear scaling, GRASP deliberately trades exact multiplicative cross\-terms for an additive direct\-sum surrogate via feature concatenation:

ϕi\\displaystyle\\phi\_\{i\}=\[ch​Wh​\(hi−h¯\);cr​Wr​ξi\]∈ℝdh\+dr\.\\displaystyle=\[c\_\{h\}W\_\{h\}\(h\_\{i\}\-\\bar\{h\}\);\\;c\_\{r\}W\_\{r\}\\xi\_\{i\}\]\\in\\mathbb\{R\}^\{d\_\{h\}\+d\_\{r\}\}\.\(14\)Here,WhW\_\{h\}is the centered whitening map from Equation[5](https://arxiv.org/html/2606.06892#S3.E5),WrW\_\{r\}is the residual scaling induced by the stabilized preconditionerDξD\_\{\\xi\}\(or its sparse top\-kkapproximation\), andch,crc\_\{h\},c\_\{r\}are fixed scaling constants\. While this direct\-sum kernel intentionally deviates from an unbiased product kernel, it ensures fastO​\(\|T\|​d\)O\(\|T\|d\)subset sweeps and linear storage space\. Appendix[A](https://arxiv.org/html/2606.06892#A1)formalizes the local approximation bound and resulting sketch error of this relaxation\.

Using this compact sketch, we operationalize the blueprint of Theorem[1](https://arxiv.org/html/2606.06892#Thmtheorem1)\. The scalar relevanceAe​\(T\)=AeGRASP​\(T\)A\_\{e\}\(T\)=A^\{\\mathrm\{GRASP\}\}\_\{e\}\(T\)is retrieved separately\. For any candidate intervention setTT, the raw geometric penalty isKraw​\(T\)=‖∑i∈Twi​ϕi‖2K\_\{\\mathrm\{raw\}\}\(T\)=\\\|\\sum\_\{i\\in T\}w\_\{i\}\\phi\_\{i\}\\\|^\{2\}\. To grant the predictor finer expressivity over self\-regularization and cross\-sample redundancy, we orthogonally decompose and center this geometry:

Kself​\(T\)\\displaystyle K\_\{\\mathrm\{self\}\}\(T\)=∑i∈Twi2​‖ϕi‖2,\\displaystyle=\\sum\_\{i\\in T\}w\_\{i\}^\{2\}\\\|\\phi\_\{i\}\\\|^\{2\},\(15\)Kpair​\(T\)\\displaystyle K\_\{\\mathrm\{pair\}\}\(T\)=Kraw​\(T\)−Kself​\(T\),\\displaystyle=K\_\{\\mathrm\{raw\}\}\(T\)\-K\_\{\\mathrm\{self\}\}\(T\),\(16\)Kcent​\(T\)\\displaystyle K\_\{\\mathrm\{cent\}\}\(T\)=‖∑i∈Twi​\(ϕi−ϕ¯\)‖2,\\displaystyle=\\left\\\|\\sum\_\{i\\in T\}w\_\{i\}\(\\phi\_\{i\}\-\\bar\{\\phi\}\)\\right\\\|^\{2\},\(17\)whereϕ¯=n−1​∑i∈𝒟ϕi\\bar\{\\phi\}=n^\{\-1\}\\sum\_\{i\\in\\mathcal\{D\}\}\\phi\_\{i\}is the global feature mean\.

Finally, constructing a unified surrogate requires reconciling the relative scales of the linear relevance and the decomposed quadratic penalties\. As dictated by Theorem[1](https://arxiv.org/html/2606.06892#Thmtheorem1), the true interaction scale depends on a target\-specific and subset\-dependent smoothness constantβe\\beta\_\{e\}, which cannot be analytically computed\. We bridge this theory\-to\-practice gap via empirical standardization:

V^e​\(T;ω\)=z​\(Ae​\(T\)\)\+∑c∈𝒞ωc​z​\(Kc​\(T\)\)\.\\widehat\{V\}\_\{e\}\(T;\\omega\)=z\(A\_\{e\}\(T\)\)\+\\sum\_\{c\\in\\mathcal\{C\}\}\\omega\_\{c\}\\,z\(K\_\{c\}\(T\)\)\.\(18\)Here,𝒞\\mathcal\{C\}contains the selected geometric components \(omitting zero\-variance ones\)\. The mapz​\(x\)=\(x−μ\)/σz\(x\)=\(x\-\\mu\)/\\sigmastandardizes the features using calibration momentsμ,σ\\mu,\\sigma\. Crucially, these moments are strictly anchored to a fixed calibration subset family within the environment, test interventions are never used to determine their own normalization\. By mapping all targets into a standardized scale space, GRASP allows a single, dataset\-agnostic weight vectorω\\omega\(calibrated strictly on development environments; see Appendix[B](https://arxiv.org/html/2606.06892#A2)\) to implicitly govern the theoreticalλe\\lambda\_\{e\}interaction scale\. Once scores and sketches are pre\-computed, evaluating a subset costsO​\(\|T\|​d\)O\(\|T\|d\)time with onlyO​\(n​d\)O\(nd\)global storage\.

## 4Experiments

The core premise of subset\-level attribution is counterfactual fidelity\. In this section, we first evaluate this primary capability using rigorous subset\-retraining protocols \(Section[4\.1](https://arxiv.org/html/2606.06892#S4.SS1)\)\. We then conduct ablation studies to isolate the mechanistic contributions of our interaction\-aware components \(Section[4\.2](https://arxiv.org/html/2606.06892#S4.SS2)\)\. Finally, we demonstrate the operational utility of GRASP across downstream applications, including language model curation, cross\-domain vision transfer, and semantic retrieval \(Section[4\.3](https://arxiv.org/html/2606.06892#S4.SS3)\)\. Setting\-specific configurations for all experiments are detailed in Appendix[B](https://arxiv.org/html/2606.06892#A2)\.

### 4\.1Counterfactual Subset\-Utility Evaluation

Table[1](https://arxiv.org/html/2606.06892#S3.T1)reports the Linear Datamodeling Score \(LDS\) evaluation, which directly tests counterfactual fidelity by measuring the rank correlation between predicted subset utilities and their empirical outcomes after retraining\. To ensure a rigorous comparison, we strictly control the intervention candidate pool and the subset\-retraining recipe across all methods\. Our experimental testbed utilizes a 50M\-parameter causal decoder\-only Transformer\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.06892#bib.bib2); Radfordet al\.,[2019](https://arxiv.org/html/2606.06892#bib.bib3)\)trained on a 10B\-token pool\. This pool is randomly sampled from a comprehensive Common Crawl corpus curated via standard scalable data pipelines\(Raffelet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib11); Penedoet al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib21),[2024](https://arxiv.org/html/2606.06892#bib.bib20); Liet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib22)\)\. Complete metric definitions and baseline details are provided in Appendix[B](https://arxiv.org/html/2606.06892#A2)\.

The empirical results demonstrate that GRASP strictly dominates all baseline methods across both metrics on every evaluated benchmark\. Notably, GRASP more than doubles the average task\-level Spearman correlation compared to the strongest alternative, improvingρ\\rhofrom 0\.174 \(TracIn\) to 0\.361\. Furthermore, it substantially elevates the positive\-LDS fraction from 0\.645 to 0\.917, indicating highly reliable directional predictions across diverse intervention samples\. This substantial performance gap provides the central evidence that explicitly capturing train\-train interactions yields a strictly superior surrogate for ranking concrete pretraining interventions\.

![Refer to caption](https://arxiv.org/html/2606.06892v1/x2.png)Figure 2:One\-step counterfactual fidelity on CC text\.The GRASP relevance tracks exact one\-step sample and subset utilities, and the local interaction surrogate improves the more nonlinear high\-learning\-rate regime\. The projected\-gradient control receives the same interaction correction but remains weaker\.Table 2:GRASP achieves efficient artifact construction and subset evaluation\.Build reports days, State reports cached floats per example, and Sweep reports seconds for 100,000 candidate subsets\.Practical pretraining attribution demands both cheap artifact construction and rapid subset evaluation\. Table[2](https://arxiv.org/html/2606.06892#S4.T2)uncouples these costs, reporting upfront construction in days and cached 100k\-subset sweeps in seconds\. While scalar methods like TracIn enable fast sweeps once artifacts exist, faithful per\-evaluation InRun Data Shapley requires an expensive trajectory replay, projecting to 10 days for a full SciQ run\. In contrast, GRASP aggressively optimizes both phases: it requires only 0\.27 days for artifact construction and executes a 100k\-subset sweep in 5\.08 seconds, approximately 31×\\timesfaster than TRAK’s 1024\-dimensional feature aggregation\.

![Refer to caption](https://arxiv.org/html/2606.06892v1/x3.png)Figure 3:Impact of interaction components on subset prediction fidelity\.We ablate GRASP components across all primary LDS benchmarks\. Bars show mean task\-level Spearman correlation \(ρ\\rho\), and colored points show individual tasks\.
### 4\.2Mechanistic Analysis and Ablations

To understand the source of these performance gains, we isolate the mechanistic contributions of individual GRASP components\. Detailed one\-step ground\-truth construction and ablation protocols are given in Appendix[B](https://arxiv.org/html/2606.06892#A2), with an additional cached protocol\-selection diagnostic in Appendix Table[7](https://arxiv.org/html/2606.06892#A4.T7)\.

Figure[2](https://arxiv.org/html/2606.06892#S4.F2)shows that the GRASP relevance core tracks both Monte Carlo Shapley values and one\-step subset utilities\. Furthermore, the interaction surrogate substantially improves the high\-learning\-rate regime where subset updates are less linear\. A projected\-gradient control with an identical interaction correction remains significantly weaker, indicating that the performance gain does not stem from adding a generic pairwise correction to an arbitrary score\.

The long\-horizon LDS results exhibit the same pattern on complete subset retraining interventions\. Relative to an additive\-only predictor, incorporating the fixed interaction components raises the macro\-average task\-level Spearman from 0\.086 to 0\.360 and pairwise intervention accuracy from 0\.528 to 0\.623\. Figure[3](https://arxiv.org/html/2606.06892#S4.F3)further shows the dataset\-level structure of this gain: residual alignment is the most critical component family, and removing it degrades every benchmark\. Meanwhile, self terms provide a consistent secondary contribution, whereas hidden geometry exhibits more task\-dependent behavior\.

### 4\.3Downstream Applications and Transferability

Beyond baseline counterfactual fidelity, we evaluate how the interaction\-aware scoring principle performs in practical data pipelines and entirely distinct domains\.

#### Fixed\-compute language model curation\.

To evaluate practical utility in pretraining pipelines, we deploy GRASP to select a high\-quality training pool prior to model initialization\. Table[3](https://arxiv.org/html/2606.06892#S4.T3)reports completed held\-out Common Crawl and SciQ evaluations under the same fixed\-compute training schedule\. GRASP achieves the lowest held\-out CC loss and the best SciQ accuracy among the displayed full\-budget diagnostic rows\. We report early training\-log snapshots and compact cross\-architecture cleaning diagnostics separately in Appendix Tables[8](https://arxiv.org/html/2606.06892#A5.T8)and[9](https://arxiv.org/html/2606.06892#A5.T9)\.

Table 3:Held\-out evaluation for fixed\-compute CC\-10B curation\.All rows use the same model, optimizer, selected\-pool size, and training budget\.
#### Cross\-domain transfer: Vision data selection\.

Table 4:Cross\-domain transferability on vision data selection\.AUC evaluates noisy\-label detection; remaining columns report selected\-data accuracy at each retention budget\. KNN: KNN\-Shapley, Infl\.: Influence, DataM\.: Datamodels\. All entries are percentages; higher is better\.To verify whether this scoring principle generalizes beyond language modeling, we evaluate our formulation in the standard Data Shapley vision setting\(Ghorbani and Zou,[2019](https://arxiv.org/html/2606.06892#bib.bib6)\): ImageNet\-pretrained ResNet18\(Denget al\.,[2009](https://arxiv.org/html/2606.06892#bib.bib36); Heet al\.,[2016](https://arxiv.org/html/2606.06892#bib.bib35)\), CIFAR\-10\(Krizhevsky,[2009](https://arxiv.org/html/2606.06892#bib.bib37)\)1k subset, and 10% label noise\. Table[4](https://arxiv.org/html/2606.06892#S4.T4)shows that GRASP achieves the best AUROC and strong selected\-data accuracy across budgets\.

#### Interpretability and Safety Extensions\.

We explore the semantic expressiveness of our attribution signal in Appendix[C](https://arxiv.org/html/2606.06892#A3), using WebOrganizer labels\(Wettiget al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib39)\)to show that GRASP provides interpretable, category\-level valuations that strongly complement traditional lexical retrieval\. Additionally, Appendix Table[10](https://arxiv.org/html/2606.06892#A6.T10)extends our framework to safety\-oriented applications, demonstrating that the same foundational scoring mechanism effectively identifies and filters misleading information during fine\-tuning\.

Table 5:Regime\-dependence of interaction effects\.Performance is evaluated on the SciQ retention\-ratio environment\. BM25 serves as a lexical baseline, while Add\. represents the relevance\-only additive core of GRASP\. All method columns report task\-level Spearmanρ\\rho\. The 0\.95 and 0\.99 rows represent high\-retention stress regimes, evaluating behavior under extreme sparsity of removed data\.

## 5Discussion

#### When do subset interactions matter most?

The necessity of interaction\-aware attribution is inherently tied to the intervention regime\. In practical curation scenarios where a selected candidate subset diverges significantly from the reference corpus, ranking quality is no longer dictated by isolated relevance alone\. Instead, redundant examples induce gradient saturation along shared update directions, whereas complementary examples jointly maximize semantic coverage\. Table[5](https://arxiv.org/html/2606.06892#S4.T5)quantifies this phenomenon within the SciQ LDS environment\. Across substantial and moderate retention fractions, our geometric surrogate consistently outperforms its additive counterpart by modeling these subset interactions\. Conversely, at an extreme 0\.99 retention rate, the candidate subsets differ by only a negligible slice; here, the perturbation approaches a first\-order additive limit\. Crucially, GRASP elegantly reduces to its additive core in this regime, inherently suppressing unstable interaction noise rather than amplifying it\. This behavior aligns perfectly with the theoretical formulation of the quadratic term: it dominates when geometric interactions dictate the relative ordering of interventions, and gracefully diminishes when interventions are near\-identical micro\-perturbations\.

#### Semantic interpretability of the geometric surrogate\.

While GRASP is primarily formulated as a robust counterfactual predictor, its underlying geometry simultaneously yields highly interpretable corpus\-level insights\. As detailed in the WebOrganizer analysis \(Appendix[C](https://arxiv.org/html/2606.06892#A3)\), our attribution scores and traditional lexical retrieval capture fundamentally complementary structures\. For topic\-level retrieval, GRASP functions competitively as a standalone metric, but fusing it with BM25 yields the optimal normalized Recall@1000\. An analogous complementarity emerges for format labels when bridging GRASP with query overlap \(Table[6](https://arxiv.org/html/2606.06892#A3.T6)\)\. This synergy confirms that the learned attribution signal captures deeper semantic utility rather than merely mirroring surface\-level lexical overlap\. Furthermore, the checkpoint\-wise category visualization \(Figure[4](https://arxiv.org/html/2606.06892#A2.F4)\) exposes explicit, signed source preferences: GRASP attributes positive expected value to knowledge\-dense, article\-like regions for the SciQ target, while actively penalizing boilerplate\-heavy formats such as notices, support pages, and FAQs\. Ultimately, these interpretable dynamics demonstrate that the geometric surrogate extends far beyond subset ranking, offering a principled mechanism for transparent corpus auditing and dataset valuation\.

## 6Related Work

Our work is situated at the intersection of data attribution, scalable influence estimation, subset counterfactual modeling, and pretraining data auditing\. We shift the core objective from assigning isolated scores to individual examples toward accurately predicting the empirical effects of concrete data interventions\.

#### Data attribution and data valuation\.

Training data attribution investigates how predictions depend on specific examples\. Methods like influence functions\(Koh and Liang,[2017](https://arxiv.org/html/2606.06892#bib.bib1)\), representer points\(Yehet al\.,[2018](https://arxiv.org/html/2606.06892#bib.bib4)\), and RelatIF\(Barshanet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib5)\)estimate local training contributions\. Alternatively, Shapley\-value methods formalize data value through marginal contributions across training coalitions\(Ghorbani and Zou,[2019](https://arxiv.org/html/2606.06892#bib.bib6)\)\. While interaction\-aware, Shapley aggressively averages these coalition dynamics into a static scalar per example, and exact estimation is computationally prohibitive\. In\-Run Data Shapley mitigates this cost along a single training trajectory\(Wanget al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib12)\)\. In contrast, we explicitly retain reusable train\-train geometry to predict the utility of concrete subset interventions\.

#### Scalable gradient\-based attribution\.

Recent advancements make gradient attribution viable for large\-scale models\. TracIn integrates gradient inner products across training checkpoints\(Pruthiet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib13)\)\. Other methods scale Hessian\-based influence\(Schioppaet al\.,[2022](https://arxiv.org/html/2606.06892#bib.bib14)\), analyze LLM generalization\(Grosseet al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib15)\), derive efficient estimators for tuned models\(Kwonet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib18)\), accelerate attribution via projected features\(Parket al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib16)\), and adapt gradient similarity for instruction tuning\(Xiaet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib19)\)\. Closest to our domain,Changet al\.\([2025](https://arxiv.org/html/2606.06892#bib.bib28)\)scale gradient\-based influence to LLM pretraining\. While these methods successfully deliver scalable first\-order signals, we focus exclusively on subset\-level counterfactual fidelity rather than retrieving isolated influential examples\.

#### Pretraining data auditing and selection\.

Extensive corpus analyses reveal substantial duplication, toxic content, and benchmark contamination in pretraining datasets\(Elazaret al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib27)\)\. Existing data selection techniques utilize gradient\-derived relevance\(Xiaet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib19)\)or pretraining\-scale tracing\(Changet al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib28)\)to surface beneficial examples\. GRASP addresses a distinct structural challenge: predicting the relative downstream utility of candidate pretraining subsets\. In this setting, redundancy, coverage, and subset interactions are first\-class concerns, rendering purely lexical retrieval or independent example scoring mathematically insufficient\.

## 7Conclusion

This work reframes pretraining data attribution from isolated example scoring to subset\-level counterfactual utility prediction\. GRASP turns this view into a scalable surrogate by combining a smoothness\-motivated geometric penalty, low\-dimensional sketches, and development\-only protocol selection\. Across subset\-retraining benchmarks, the method improves counterfactual ranking fidelity over scalable baselines, while the application diagnostics show that the same interaction\-aware signal remains useful beyond the core LDS setting\. These results suggest that subset geometry is a practical ingredient for auditing and optimizing massive pretraining corpora\.

## Limitations

#### Theoretical Approximations\.

Our theoretical formulation relies on a one\-step smoothness lower bound\. While this provides a principled mathematical motivation, it serves as an approximation rather than a strict guarantee for the full non\-convex dynamics of Transformer pre\-training\. Consequently, we ground our claims in subset\-retraining LDS as a rigorous empirical counterfactual, bridging the gap between analytical approximation and practical fidelity\.

#### Empirical Scale\.

Our core LDS validation employs a 50M\-parameter architecture and a 10B\-token candidate pool\. Although this intermediate scale permits strictly controlled, compute\-intensive interventions \(i\.e\., extensive subset retraining\), scaling laws for subset interactions remain an open empirical question\. While our supplementary 1B\-parameter DCLM scale\-up suggests robust architectural transferability, validating these specific geometric interaction dynamics on frontier\-scale models remains a vital direction for future work\.

#### Compression and Tractability\.

To ensure computational scalability over massive corpora, GRASP projects representations into low\-dimensional feature sketches\. This dimensionality reduction inherently trades fine\-grained, long\-range attention dependencies for tractability, positioning the approach as a highly scalable surrogate rather than an exhaustive, lossless influence computation\.

#### Target\-Conditioning and Extreme Regimes\.

Finally, our attribution protocols are explicitly target\-conditioned\. Rather than yielding a monolithic “universal quality” score, GRASP evaluates data utility with respect to specific downstream behaviors\. Additionally, extreme data interventions such as near\-total corpus deletion violate the local perturbation assumption, which may necessitate specialized, ratio\-aware scaling corrections when operating in such high\-retention stress regimes\.

## Ethical Considerations

#### Attribution as Auditing, Not Absolute Causality\.

Data attribution frameworks can significantly advance responsible model development by auditing corpora for regions that drive harmful outputs, bias, or benchmark contamination\. However, because these scores are continuous approximations, they should not be misconstrued as definitive causal proof that a specific document unilaterally generated a given model behavior\. When deploying attribution for data governance, practitioners should incorporate uncertainty metrics, negative controls, and explicit failure\-mode analyses\.

#### Data Privacy and Release Risks\.

Web\-scale pre\-training corpora inherently contain copyrighted, personally identifiable, or otherwise sensitive information\. Releasing high\-resolution attribution tables or raw, highly influential training examples risks exposing this sensitive text or facilitating dataset membership inference\. Any public release of attribution scores, retrieved examples, or corpus audits utilizing this method must strictly adhere to the underlying data licenses and robust privacy\-preserving constraints\.

## References

- Database\-friendly random projections: johnson\-lindenstrauss with binary coins\.Journal of Computer and System Sciences66\(4\),pp\. 671–687\.Note:Special Issue on PODS 2001External Links:ISSN 0022\-0000,[Document](https://dx.doi.org/https%3A//doi.org/10.1016/S0022-0000%2803%2900025-4),[Link](https://www.sciencedirect.com/science/article/pii/S0022000003000254)Cited by:[Appendix A](https://arxiv.org/html/2606.06892#A1.SS0.SSS0.Px6.p1.14)\.
- E\. Barshan, M\. Brunet, and G\. K\. Dziugaite \(2020\)RelatIF: identifying explanatory training samples via relative influence\.InProceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics,S\. Chiappa and R\. Calandra \(Eds\.\),Proceedings of Machine Learning Research, Vol\.108,pp\. 1899–1909\.External Links:[Link](https://proceedings.mlr.press/v108/barshan20a.html)Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px1.p1.1)\.
- T\. Chang, D\. Rajagopal, T\. Bolukbasi, L\. Dixon, and I\. Tenney \(2025\)Scalable influence and fact tracing for large language model pretraining\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 40976–40997\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/65798a76cc176c29b6bfefe84b0a03ff-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px3.p1.1)\.
- S\. Dasgupta and A\. Gupta \(2003\)An elementary proof of a theorem of johnson and lindenstrauss\.Random Structures & Algorithms22\(1\),pp\. 60–65\.External Links:[Document](https://dx.doi.org/https%3A//doi.org/10.1002/rsa.10073),[Link](https://onlinelibrary.wiley.com/doi/abs/10.1002/rsa.10073),https://onlinelibrary\.wiley\.com/doi/pdf/10\.1002/rsa\.10073Cited by:[Appendix A](https://arxiv.org/html/2606.06892#A1.SS0.SSS0.Px6.p1.14)\.
- J\. Deng, W\. Dong, R\. Socher, L\. Li, K\. Li, and L\. Fei\-Fei \(2009\)ImageNet: a large\-scale hierarchical image database\.In2009 IEEE Conference on Computer Vision and Pattern Recognition,pp\. 248–255\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2009.5206848),[Link](https://ieeexplore.ieee.org/document/5206848)Cited by:[§4\.3](https://arxiv.org/html/2606.06892#S4.SS3.SSS0.Px2.p1.1)\.
- Y\. Elazar, A\. Bhagia, I\. Magnusson, A\. Ravichander, D\. Schwenk, A\. Suhr, P\. Walsh, D\. Groeneveld, L\. Soldaini, S\. Singh, H\. Hajishirzi, N\. Smith, and J\. Dodge \(2024\)What's in my big data?\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 7735–7790\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/1f7336fd66b6e6e63d1801fdd5930a5a-Paper-Conference.pdf)Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px3.p1.1)\.
- A\. Ghorbani and J\. Zou \(2019\)Data shapley: equitable valuation of data for machine learning\.InProceedings of the 36th International Conference on Machine Learning,K\. Chaudhuri and R\. Salakhutdinov \(Eds\.\),Proceedings of Machine Learning Research, Vol\.97,pp\. 2242–2251\.External Links:[Link](https://proceedings.mlr.press/v97/ghorbani19c.html)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px5.p1.1),[§1](https://arxiv.org/html/2606.06892#S1.p3.1),[§4\.3](https://arxiv.org/html/2606.06892#S4.SS3.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Gretton, K\. M\. Borgwardt, M\. J\. Rasch, B\. Schölkopf, and A\. Smola \(2012\)A kernel two\-sample test\.Journal of Machine Learning Research13\(25\),pp\. 723–773\.External Links:[Link](http://jmlr.org/papers/v13/gretton12a.html)Cited by:[Appendix A](https://arxiv.org/html/2606.06892#A1.SS0.SSS0.Px5.p1.7)\.
- R\. Grosse, J\. Bae, C\. Anil, N\. Elhage, A\. Tamkin, A\. Tajdini, B\. Steiner, D\. Li, E\. Durmus, E\. Perez, E\. Hubinger, K\. Lukošiūtė, K\. Nguyen, N\. Joseph, S\. McCandlish, J\. Kaplan, and S\. R\. Bowman \(2023\)Studying large language model generalization with influence functions\.External Links:2308\.03296,[Link](https://arxiv.org/abs/2308.03296)Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1)\.
- Y\. Gu, O\. Tafjord, B\. Kuehl, D\. Haddad, J\. Dodge, and H\. Hajishirzi \(2025\)OLMES: a standard for language model evaluations\.InFindings of the Association for Computational Linguistics: NAACL 2025,L\. Chiruzzo, A\. Ritter, and L\. Wang \(Eds\.\),Albuquerque, New Mexico,pp\. 5020–5048\.External Links:[Link](https://aclanthology.org/2025.findings-naacl.282/),[Document](https://dx.doi.org/10.18653/v1/2025.findings-naacl.282),ISBN 979\-8\-89176\-195\-7Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2)\.
- K\. He, X\. Zhang, S\. Ren, and J\. Sun \(2016\)Deep residual learning for image recognition\.InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pp\. 770–778\.External Links:[Document](https://dx.doi.org/10.1109/CVPR.2016.90),[Link](https://openaccess.thecvf.com/content_cvpr_2016/html/He_Deep_Residual_Learning_CVPR_2016_paper.html)Cited by:[§4\.3](https://arxiv.org/html/2606.06892#S4.SS3.SSS0.Px2.p1.1)\.
- A\. Ilyas, S\. M\. Park, L\. Engstrom, G\. Leclerc, and A\. Madry \(2022\)Datamodels: understanding predictions with data and data with predictions\.InProceedings of the 39th International Conference on Machine Learning,K\. Chaudhuri, S\. Jegelka, L\. Song, C\. Szepesvari, G\. Niu, and S\. Sabato \(Eds\.\),Proceedings of Machine Learning Research, Vol\.162,pp\. 9525–9587\.External Links:[Link](https://proceedings.mlr.press/v162/ilyas22a.html)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px2.p1.4),[§1](https://arxiv.org/html/2606.06892#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.06892#S2.SS2.p1.3)\.
- P\. W\. Koh and P\. Liang \(2017\)Understanding black\-box predictions via influence functions\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 1885–1894\.External Links:[Link](https://proceedings.mlr.press/v70/koh17a.html)Cited by:[§1](https://arxiv.org/html/2606.06892#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06892#S2.SS1.p2.3),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px1.p1.1)\.
- A\. Krizhevsky \(2009\)Learning multiple layers of features from tiny images\.Technical reportUniversity of Toronto\.External Links:[Link](https://www.cs.toronto.edu/%CB%9Ckriz/learning-features-2009-TR.pdf)Cited by:[§4\.3](https://arxiv.org/html/2606.06892#S4.SS3.SSS0.Px2.p1.1)\.
- Y\. Kwon, E\. Wu, K\. Wu, and J\. Y\. Zou \(2024\)DataInf: efficiently estimating data influence in lora\-tuned llms and diffusion models\.InInternational Conference on Learning Representations,B\. Kim, Y\. Yue, S\. Chaudhuri, K\. Fragkiadaki, M\. Khan, and Y\. Sun \(Eds\.\),Vol\.2024,pp\. 21921–21942\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2024/file/5e84a0f233611a1dc8fb794dc52415a3-Paper-Conference.pdf)Cited by:[§1](https://arxiv.org/html/2606.06892#S1.p2.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1)\.
- J\. Li, A\. Fang, G\. Smyrnis, M\. Ivgi, M\. Jordan, S\. Gadre, H\. Bansal, E\. Guha, S\. Keh, K\. Arora, S\. Garg, R\. Xin, N\. Muennighoff, R\. Heckel, J\. Mercat, M\. Chen, S\. Gururangan, M\. Wortsman, A\. Albalak, Y\. Bitton, M\. Nezhurina, A\. Abbas, C\. Hsieh, D\. Ghosh, J\. Gardner, M\. Kilian, H\. Zhang, R\. Shao, S\. Pratt, S\. Sanyal, G\. Ilharco, G\. Daras, K\. Marathe, A\. Gokaslan, J\. Zhang, K\. Chandu, T\. Nguyen, I\. Vasiljevic, S\. Kakade, S\. Song, S\. Sanghavi, F\. Faghri, S\. Oh, L\. Zettlemoyer, K\. Lo, A\. El\-Nouby, H\. Pouransari, A\. Toshev, S\. Wang, D\. Groeneveld, L\. Soldaini, P\. W\. Koh, J\. Jitsev, T\. Kollar, A\. G\. Dimakis, Y\. Carmon, A\. Dave, L\. Schmidt, and V\. Shankar \(2024\)DataComp\-lm: in search of the next generation of training sets for language models\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 14200–14282\.External Links:[Document](https://dx.doi.org/10.52202/079017-0455),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/19e4ea30dded58259665db375885e412-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- J\. Martens and R\. Grosse \(2015\)Optimizing neural networks with Kronecker\-factored approximate curvature\.InProceedings of the 32nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.37,pp\. 2408–2417\.External Links:[Link](https://proceedings.mlr.press/v37/martens15.html)Cited by:[§3\.1](https://arxiv.org/html/2606.06892#S3.SS1.p2.1)\.
- T\. Mihaylov, P\. Clark, T\. Khot, and A\. Sabharwal \(2018\)Can a suit of armor conduct electricity? a new dataset for open book question answering\.InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,E\. Riloff, D\. Chiang, J\. Hockenmaier, and J\. Tsujii \(Eds\.\),Brussels, Belgium,pp\. 2381–2391\.External Links:[Link](https://aclanthology.org/D18-1260/),[Document](https://dx.doi.org/10.18653/v1/D18-1260)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2)\.
- G\. L\. Nemhauser, L\. A\. Wolsey, and M\. L\. Fisher \(1978\)An analysis of approximations for maximizing submodular set functions—i\.Mathematical Programming14\(1\),pp\. 265–294\.External Links:[Document](https://dx.doi.org/10.1007/BF01588971),[Link](https://doi.org/10.1007/BF01588971)Cited by:[Appendix A](https://arxiv.org/html/2606.06892#A1.SS0.SSS0.Px3.p1.5)\.
- T\. Olmo \(2026\)Olmo 3\.External Links:2512\.13961,[Link](https://arxiv.org/abs/2512.13961)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2)\.
- S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. Madry \(2023\)TRAK: attributing model behavior at scale\.External Links:2303\.14186,[Link](https://arxiv.org/abs/2303.14186)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px2.p1.4),[§1](https://arxiv.org/html/2606.06892#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06892#S2.SS1.p2.3),[§2\.2](https://arxiv.org/html/2606.06892#S2.SS2.p1.3),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1)\.
- G\. Penedo, H\. Kydlíček, L\. B\. allal, A\. Lozhkov, M\. Mitchell, C\. Raffel, L\. Von Werra, and T\. Wolf \(2024\)The fineweb datasets: decanting the web for the finest text data at scale\.InAdvances in Neural Information Processing Systems,A\. Globerson, L\. Mackey, D\. Belgrave, A\. Fan, U\. Paquet, J\. Tomczak, and C\. Zhang \(Eds\.\),Vol\.37,pp\. 30811–30849\.External Links:[Document](https://dx.doi.org/10.52202/079017-0970),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/file/370df50ccfdf8bde18f8f9c2d9151bda-Paper-Datasets_and_Benchmarks_Track.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.06892#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- G\. Penedo, Q\. Malartic, D\. Hesslow, R\. Cojocaru, H\. Alobeidli, A\. Cappelli, B\. Pannier, E\. Almazrouei, and J\. Launay \(2023\)The refinedweb dataset for falcon llm: outperforming curated corpora with web data only\.InAdvances in Neural Information Processing Systems,A\. Oh, T\. Naumann, A\. Globerson, K\. Saenko, M\. Hardt, and S\. Levine \(Eds\.\),Vol\.36,pp\. 79155–79172\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/file/fa3ed726cc5073b9c31e3e49a807789c-Paper-Datasets_and_Benchmarks.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- G\. Pruthi, F\. Liu, S\. Kale, and M\. Sundararajan \(2020\)Estimating training data influence by tracing gradient descent\.InAdvances in Neural Information Processing Systems,H\. Larochelle, M\. Ranzato, R\. Hadsell, M\.F\. Balcan, and H\. Lin \(Eds\.\),Vol\.33,pp\. 19920–19930\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/e6385d39ec9394f2f3a354d9d2b88eec-Paper.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px2.p1.4),[§1](https://arxiv.org/html/2606.06892#S1.p2.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, I\. Sutskever,et al\.\(2019\)Language models are unsupervised multitask learners\.Note:OpenAI technical reportExternal Links:[Link](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- C\. Raffel, N\. Shazeer, A\. Roberts, K\. Lee, S\. Narang, M\. Matena, Y\. Zhou, W\. Li, and P\. J\. Liu \(2020\)Exploring the limits of transfer learning with a unified text\-to\-text transformer\.Journal of Machine Learning Research21\(140\),pp\. 1–67\.External Links:[Link](http://jmlr.org/papers/v21/20-074.html)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§1](https://arxiv.org/html/2606.06892#S1.p1.1),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- A\. Schioppa, P\. Zablotskaia, D\. Vilar, and A\. Sokolov \(2022\)Scaling up influence functions\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 8179–8186\.Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1)\.
- A\. Talmor, J\. Herzig, N\. Lourie, and J\. Berant \(2019\)CommonsenseQA: a question answering challenge targeting commonsense knowledge\.InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 \(Long and Short Papers\),J\. Burstein, C\. Doran, and T\. Solorio \(Eds\.\),Minneapolis, Minnesota,pp\. 4149–4158\.External Links:[Link](https://aclanthology.org/N19-1421/),[Document](https://dx.doi.org/10.18653/v1/N19-1421)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2)\.
- A\. Vaswani, N\. Shazeer, N\. Parmar, J\. Uszkoreit, L\. Jones, A\. N\. Gomez, Ł\. Kaiser, and I\. Polosukhin \(2017\)Attention is all you need\.InAdvances in Neural Information Processing Systems,I\. Guyon, U\. V\. Luxburg, S\. Bengio, H\. Wallach, R\. Fergus, S\. Vishwanathan, and R\. Garnett \(Eds\.\),Vol\.30,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2),[§4\.1](https://arxiv.org/html/2606.06892#S4.SS1.p1.1)\.
- J\. \(\. Wang, P\. Mittal, D\. Song, and R\. Jia \(2025\)Data shapley in one training run\.InInternational Conference on Learning Representations,Y\. Yue, A\. Garg, N\. Peng, F\. Sha, and R\. Yu \(Eds\.\),Vol\.2025,pp\. 12358–12395\.External Links:[Link](https://proceedings.iclr.cc/paper_files/paper/2025/file/20fdaf67581e6d7157376d1ed584040a-Paper-Conference.pdf)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px2.p1.4),[§1](https://arxiv.org/html/2606.06892#S1.p3.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px1.p1.1)\.
- J\. Welbl, N\. F\. Liu, and M\. Gardner \(2017\)Crowdsourcing multiple choice science questions\.InProceedings of the 3rd Workshop on Noisy User\-generated Text,L\. Derczynski, W\. Xu, A\. Ritter, and T\. Baldwin \(Eds\.\),Copenhagen, Denmark,pp\. 94–106\.External Links:[Link](https://aclanthology.org/W17-4413/),[Document](https://dx.doi.org/10.18653/v1/W17-4413)Cited by:[Appendix B](https://arxiv.org/html/2606.06892#A2.SS0.SSS0.Px1.p1.2)\.
- A\. Wettig, K\. Lo, S\. Min, H\. Hajishirzi, D\. Chen, and L\. Soldaini \(2025\)Organize the web: constructing domains enhances pre\-training data curation\.External Links:2502\.10341,[Document](https://dx.doi.org/10.48550/arXiv.2502.10341),[Link](https://arxiv.org/abs/2502.10341)Cited by:[Table 6](https://arxiv.org/html/2606.06892#A3.T6),[§4\.3](https://arxiv.org/html/2606.06892#S4.SS3.SSS0.Px3.p1.1)\.
- M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. Chen \(2024\)LESS: selecting influential data for targeted instruction tuning\.InProceedings of the 41st International Conference on Machine Learning,ICML’24\.Cited by:[§1](https://arxiv.org/html/2606.06892#S1.p2.1),[§2\.1](https://arxiv.org/html/2606.06892#S2.SS1.p2.3),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px2.p1.1),[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px3.p1.1)\.
- C\. Yeh, J\. Kim, I\. E\. Yen, and P\. K\. Ravikumar \(2018\)Representer point selection for explaining deep neural networks\.InAdvances in Neural Information Processing Systems,S\. Bengio, H\. Wallach, H\. Larochelle, K\. Grauman, N\. Cesa\-Bianchi, and R\. Garnett \(Eds\.\),Vol\.31,pp\.\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2018/file/8a7129b8f3edd95b7d969dfc2c8e9d9d-Paper.pdf)Cited by:[§6](https://arxiv.org/html/2606.06892#S6.SS0.SSS0.Px1.p1.1)\.

## Appendix AMethod Details

This appendix provides the formal derivations supporting the interaction\-aware surrogate utilized for the counterfactual evaluation in Section[4](https://arxiv.org/html/2606.06892#S4)\. Our primary objective is to delineate which components of our formulation are direct mathematical consequences of objective smoothness and which originate from the approximation conditions imposed by the low\-dimensional sketch\.

#### Local influence direction\.

The local update directionuiu\_\{i\}introduced in the main text is formally instantiated through classical influence function analysis\. Consider the weighted regularized empirical risk

R​\(θ,α\)=1n​∑i=1nαi​ℓi​\(θ\)\+λreg​r​\(θ\),R\(\\theta,\\alpha\)=\\frac\{1\}\{n\}\\sum\_\{i=1\}^\{n\}\\alpha\_\{i\}\\ell\_\{i\}\(\\theta\)\+\\lambda\_\{\\mathrm\{reg\}\}r\(\\theta\),\(19\)whereαi\\alpha\_\{i\}represents the weight assigned to training exampleii\. AssumeRRis twice continuously differentiable\. Letθ⋆​\(α\)\\theta^\{\\star\}\(\\alpha\)be a local minimizer such that∇θR​\(θ⋆​\(α\),α\)=0\\nabla\_\{\\theta\}R\(\\theta^\{\\star\}\(\\alpha\),\\alpha\)=0\. If the HessianH=∇θ2R​\(θ⋆​\(α\),α\)H=\\nabla\_\{\\theta\}^\{2\}R\(\\theta^\{\\star\}\(\\alpha\),\\alpha\)is strictly positive definite, the Implicit Function Theorem guarantees the existence of a differentiable solution branch\. Differentiating the stationarity condition with respect toαi\\alpha\_\{i\}yields∂θ⋆​\(α\)/∂αi=−n−1​H−1​∇θℓi​\(θ⋆​\(α\)\)\\partial\\theta^\{\\star\}\(\\alpha\)/\\partial\\alpha\_\{i\}=\-n^\{\-1\}H^\{\-1\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta^\{\\star\}\(\\alpha\)\)\.

Recall that the evaluation utility is defined as𝒰e​\(θ\)=−Le​\(θ\)\\mathcal\{U\}\_\{e\}\(\\theta\)=\-L\_\{e\}\(\\theta\)\. Applying the chain rule evaluated at the reference checkpointθ\\thetagives the marginal utility change:

∂𝒰e∂αi=1n​⟨∇θLe​\(θ\),H−1​∇θℓi​\(θ\)⟩\.\\frac\{\\partial\\mathcal\{U\}\_\{e\}\}\{\\partial\\alpha\_\{i\}\}=\\frac\{1\}\{n\}\\left\\langle\\nabla\_\{\\theta\}L\_\{e\}\(\\theta\),H^\{\-1\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\)\\right\\rangle\.\(20\)This derivation motivates defining the local influence\-style update direction asui=H−1​∇θℓi​\(θ\)u\_\{i\}=H^\{\-1\}\\nabla\_\{\\theta\}\\ell\_\{i\}\(\\theta\), absorbing the shared1/n1/nscaling factor into the global learning rate\.

#### Smoothness lower bound\.

Building upon this local linear approximation, we derive a certified lower bound for subset utility improvements\. Assume the target loss functionLeL\_\{e\}isβe\\beta\_\{e\}\-smooth\. By definition, for any parameter updateΔ\\Delta, the following inequality holds:

Le​\(θ\+Δ\)≤Le​\(θ\)\+⟨∇θLe​\(θ\),Δ⟩\+βe2​‖Δ‖2\.L\_\{e\}\(\\theta\+\\Delta\)\\leq L\_\{e\}\(\\theta\)\+\\langle\\nabla\_\{\\theta\}L\_\{e\}\(\\theta\),\\Delta\\rangle\+\\frac\{\\beta\_\{e\}\}\{2\}\\\|\\Delta\\\|^\{2\}\.\(21\)We model the subset intervention as an aggregated gradient step defined byΔ=−η​DT\\Delta=\-\\eta D\_\{T\}, whereη≥0\\eta\\geq 0is the step size\. Substituting this into the smoothness condition and utilizing the utility definitionUe=−LeU\_\{e\}=\-L\_\{e\}yields:

Ue​\(θ−η​DT\)−Ue​\(θ\)\\displaystyle U\_\{e\}\(\\theta\-\\eta D\_\{T\}\)\-U\_\{e\}\(\\theta\)≥η​⟨∇θLe​\(θ\),DT⟩\\displaystyle\\geq\\eta\\langle\\nabla\_\{\\theta\}L\_\{e\}\(\\theta\),D\_\{T\}\\rangle\(22\)−βe​η22​‖DT‖2\.\\displaystyle\\quad\-\\frac\{\\beta\_\{e\}\\eta^\{2\}\}\{2\}\\\|D\_\{T\}\\\|^\{2\}\.Finally, substituting the subset aggregate directionDT=∑i∈Twi​uiD\_\{T\}=\\sum\_\{i\\in T\}w\_\{i\}u\_\{i\}directly recovers the one\-step interaction lower bound presented in Theorem[1](https://arxiv.org/html/2606.06892#Thmtheorem1)\. Crucially, this bound relies solely on theβe\\beta\_\{e\}\-smoothness of the target objective and does not require global convexity\.

#### Set\-function curvature and submodularity\.

The idealized surrogate defined in Equation[13](https://arxiv.org/html/2606.06892#S3.E13)offers a precise set\-theoretic interpretation of subset “interaction\.” LetFe​\(T\)=𝒰^eint​\(T\)F\_\{e\}\(T\)=\\widehat\{\\mathcal\{U\}\}^\{\\mathrm\{int\}\}\_\{e\}\(T\)\. For any elementsi,j∉Ti,j\\notin Twithi≠ji\\neq j, the discrete second difference characterizes the marginal interaction:

Δi​Δj​Fe​\(T\)\\displaystyle\\Delta\_\{i\}\\Delta\_\{j\}F\_\{e\}\(T\)=Fe​\(T∪\{i,j\}\)−Fe​\(T∪\{i\}\)\\displaystyle=F\_\{e\}\(T\\cup\\\{i,j\\\}\)\-F\_\{e\}\(T\\cup\\\{i\\\}\)\(23\)−Fe​\(T∪\{j\}\)\+Fe​\(T\)\.\\displaystyle\\quad\-F\_\{e\}\(T\\cup\\\{j\\\}\)\+F\_\{e\}\(T\)\.Direct algebraic expansion yields:

Δi​Δj​Fe​\(T\)=−2​λe​wi​wj​⟨ui,uj⟩\.\\Delta\_\{i\}\\Delta\_\{j\}F\_\{e\}\(T\)=\-2\\lambda\_\{e\}w\_\{i\}w\_\{j\}\\langle u\_\{i\},u\_\{j\}\\rangle\.\(24\)This establishes a critical geometric intuition: aligned update directions naturally produce negative discrete curvature \(redundancy penalty\), whereas opposing directions can yield positive curvature \(complementary coverage\)\. Consequently, GRASP functions as a signed pairwise\-interaction set function\. In the specific case wherewi≥0w\_\{i\}\\geq 0and all pairs exhibit non\-negative inner products \(⟨ui,uj⟩≥0\\langle u\_\{i\},u\_\{j\}\\rangle\\geq 0\), the surrogate becomes globally submodular, reflecting classical diminishing returns\(Nemhauseret al\.,[1978](https://arxiv.org/html/2606.06892#bib.bib30)\)\. This conditional submodularity is formalized below\.

###### Proposition 1\(Local submodularity over aligned clusters\)\.

Consider the unweighted discrete surrogate defined by

F​\(T\)=η​∑i∈Tai−λ​‖∑i∈Tui‖2,λ\>0\.F\(T\)=\\eta\\sum\_\{i\\in T\}a\_\{i\}\-\\lambda\\left\\\|\\sum\_\{i\\in T\}u\_\{i\}\\right\\\|^\{2\},\\qquad\\lambda\>0\.Let𝒟align\\mathcal\{D\}\_\{\\mathrm\{align\}\}be a candidate pool such that the pairwise alignment⟨ui,uj⟩≥0\\langle u\_\{i\},u\_\{j\}\\rangle\\geq 0holds for all distincti,j∈𝒟aligni,j\\in\\mathcal\{D\}\_\{\\mathrm\{align\}\}\. Then, the set functionFFis submodular on𝒟align\\mathcal\{D\}\_\{\\mathrm\{align\}\}\. Moreover, for anyS⊆T⊆𝒟alignS\\subseteq T\\subseteq\\mathcal\{D\}\_\{\\mathrm\{align\}\}andk∈𝒟align∖Tk\\in\\mathcal\{D\}\_\{\\mathrm\{align\}\}\\setminus T, the marginal diminishing return is strictly positive whenever∑i∈T∖S⟨uk,ui⟩\>0\\sum\_\{i\\in T\\setminus S\}\\langle u\_\{k\},u\_\{i\}\\rangle\>0\.

###### Proof\.

LetΔk​F​\(S\)=F​\(S∪\{k\}\)−F​\(S\)\\Delta\_\{k\}F\(S\)=F\(S\\cup\\\{k\\\}\)\-F\(S\)denote the marginal gain of adding elementkkto setSS\. The linear relevance term contributes a constantη​ak\\eta a\_\{k\}to every marginal gain\. Expanding the quadratic penalty term yields:

Δk​F​\(S\)=η​ak−λ​‖uk‖2−2​λ​∑i∈S⟨uk,ui⟩\.\\Delta\_\{k\}F\(S\)=\\eta a\_\{k\}\-\\lambda\\\|u\_\{k\}\\\|^\{2\}\-2\\lambda\\sum\_\{i\\in S\}\\langle u\_\{k\},u\_\{i\}\\rangle\.To verify submodularity, we examine the difference in marginal gains forS⊆TS\\subseteq T:

Δk​F​\(S\)−Δk​F​\(T\)=2​λ​∑i∈T∖S⟨uk,ui⟩\.\\Delta\_\{k\}F\(S\)\-\\Delta\_\{k\}F\(T\)=2\\lambda\\sum\_\{i\\in T\\setminus S\}\\langle u\_\{k\},u\_\{i\}\\rangle\.Under the alignment assumption⟨uk,ui⟩≥0\\langle u\_\{k\},u\_\{i\}\\rangle\\geq 0, this difference is strictly non\-negative sinceλ\>0\\lambda\>0\. This guaranteesΔk​F​\(S\)≥Δk​F​\(T\)\\Delta\_\{k\}F\(S\)\\geq\\Delta\_\{k\}F\(T\), satisfying the formal definition of diminishing returns and proving thatFFis submodular on𝒟align\\mathcal\{D\}\_\{\\mathrm\{align\}\}\. The strict inequality condition follows directly from the same expression\. ∎

#### Sketched surrogate and error bounds\.

To operationalize this objective at scale, we introduce the sketched surrogate for a fixed nonnegative scaleλ\\lambda:

B^esk​\(T;λ\)=η​Ae​\(T\)−λ​Kraw​\(T\)\.\\widehat\{B\}^\{\\mathrm\{sk\}\}\_\{e\}\(T;\\lambda\)=\\eta A\_\{e\}\(T\)\-\\lambda K\_\{\\mathrm\{raw\}\}\(T\)\.\(25\)LetBeu​\(T;λ\)=η​Aeu​\(T\)−λ​Ku​\(T\)B^\{u\}\_\{e\}\(T;\\lambda\)=\\eta A^\{u\}\_\{e\}\(T\)\-\\lambda K^\{u\}\(T\)denote the exact local surrogate\. If the sketch formulation satisfies bounded deviations on a subset family𝒯\\mathcal\{T\}such that

\|Ae​\(T\)−Aeu​\(T\)\|\\displaystyle\|A\_\{e\}\(T\)\-A^\{u\}\_\{e\}\(T\)\|≤ϵA​\(T\),\\displaystyle\\leq\\epsilon\_\{A\}\(T\),\(26\)\|Kraw​\(T\)−Ku​\(T\)\|\\displaystyle\|K\_\{\\mathrm\{raw\}\}\(T\)\-K^\{u\}\(T\)\|≤ϵK​\(T\),\\displaystyle\\leq\\epsilon\_\{K\}\(T\),\(27\)then applying the triangle inequality guarantees that for everyλ≥0\\lambda\\geq 0andT∈𝒯T\\in\\mathcal\{T\}, the subset prediction error is bounded by:

\|Beu​\(T;λ\)−B^esk​\(T;λ\)\|≤η​ϵA​\(T\)\+λ​ϵK​\(T\)\.\\left\|B^\{u\}\_\{e\}\(T;\\lambda\)\-\\widehat\{B\}^\{\\mathrm\{sk\}\}\_\{e\}\(T;\\lambda\)\\right\|\\leq\\eta\\epsilon\_\{A\}\(T\)\+\\lambda\\epsilon\_\{K\}\(T\)\.\(28\)In scenarios where only pointwise relevance bounds are accessible, such that\|ai,e−⟨ge,ui⟩\|≤ϵa\|a\_\{i,e\}\-\\langle g\_\{e\},u\_\{i\}\\rangle\|\\leq\\epsilon\_\{a\}, we can establishϵA​\(T\)=ϵa​∑i∈T\|wi\|\\epsilon\_\{A\}\(T\)=\\epsilon\_\{a\}\\sum\_\{i\\in T\}\|w\_\{i\}\|\. We emphasize that this constitutes a conditional approximation bound\. The verification of a tightly bounded sketch error remains an empirical diagnostic rather than a structural guarantee inherent to the architecture\.

###### Proposition 2\(Local direct\-sum approximation to product geometry\)\.

Letxi,xj,ri,rjx\_\{i\},x\_\{j\},r\_\{i\},r\_\{j\}denote normalized hidden and residual sketches satisfying‖xi‖=‖xj‖=‖ri‖=‖rj‖=1\\\|x\_\{i\}\\\|=\\\|x\_\{j\}\\\|=\\\|r\_\{i\}\\\|=\\\|r\_\{j\}\\\|=1\. Define the squared Euclidean distances asdh2​\(i,j\)=‖xi−xj‖2d\_\{h\}^\{2\}\(i,j\)=\\\|x\_\{i\}\-x\_\{j\}\\\|^\{2\}anddr2​\(i,j\)=‖ri−rj‖2d\_\{r\}^\{2\}\(i,j\)=\\\|r\_\{i\}\-r\_\{j\}\\\|^\{2\}\. Consider the product and direct\-sum kernels:

kprod​\(i,j\)\\displaystyle k\_\{\\mathrm\{prod\}\}\(i,j\)=⟨xi,xj⟩​⟨ri,rj⟩,\\displaystyle=\\langle x\_\{i\},x\_\{j\}\\rangle\\langle r\_\{i\},r\_\{j\}\\rangle,\(29\)kdir​\(i,j\)\\displaystyle k\_\{\\mathrm\{dir\}\}\(i,j\)=⟨xi,xj⟩\+⟨ri,rj⟩\.\\displaystyle=\\langle x\_\{i\},x\_\{j\}\\rangle\+\\langle r\_\{i\},r\_\{j\}\\rangle\.\(30\)The product kernel expands as:

kprod​\(i,j\)\\displaystyle k\_\{\\mathrm\{prod\}\}\(i,j\)=1−12​\(dh2​\(i,j\)\+dr2​\(i,j\)\)\\displaystyle=1\-\\frac\{1\}\{2\}\\left\(d\_\{h\}^\{2\}\(i,j\)\+d\_\{r\}^\{2\}\(i,j\)\\right\)\+14​dh2​\(i,j\)​dr2​\(i,j\)\.\\displaystyle\\quad\+\\frac\{1\}\{4\}d\_\{h\}^\{2\}\(i,j\)d\_\{r\}^\{2\}\(i,j\)\.\(31\)Furthermore, the direct\-sum kernel satisfies:

kdir​\(i,j\)=kprod​\(i,j\)\+1−14​dh2​\(i,j\)​dr2​\(i,j\)\.\\displaystyle k\_\{\\mathrm\{dir\}\}\(i,j\)=k\_\{\\mathrm\{prod\}\}\(i,j\)\+1\-\\frac\{1\}\{4\}d\_\{h\}^\{2\}\(i,j\)d\_\{r\}^\{2\}\(i,j\)\.\(32\)Consequently, after omitting the constant offset, the direct\-sum kernel and the product kernel agree strictly through the first order in the local squared distances\. Their remaining discrepancy is confined to the fourth\-order cross term\.

###### Proof\.

For any unit vectors, the inner product relates to Euclidean distance via⟨xi,xj⟩=1−‖xi−xj‖2/2\\langle x\_\{i\},x\_\{j\}\\rangle=1\-\\\|x\_\{i\}\-x\_\{j\}\\\|^\{2\}/2\. Applying this identity identically to the residual sketches, substituting both into the definition ofkprodk\_\{\\mathrm\{prod\}\}, and expanding the product yields the first result\. Summing the two inner products directly yieldskdir=2−12​\(dh2​\(i,j\)\+dr2​\(i,j\)\)k\_\{\\mathrm\{dir\}\}=2\-\\frac\{1\}\{2\}\(d\_\{h\}^\{2\}\(i,j\)\+d\_\{r\}^\{2\}\(i,j\)\)\. Subtracting the expanded form ofkprod​\(i,j\)k\_\{\\mathrm\{prod\}\}\(i,j\)from this sum produces the second result\. ∎

We stress that this proposition strictly characterizes a local approximation and does not imply a global equivalence between the direct\-sum and product kernels\. The constant offset is mathematically rank\-invariant for fixed\-size unweighted subset interventions\. Furthermore, this discrepancy is structurally mitigated by centered components such asKcentK\_\{\\mathrm\{cent\}\}\. Outside these specific configurations, the direct\-sum kernel inherently operates as a scalable empirical surrogate\.

#### Centered component interpretation\.

LetWT=∑i∈TwiW\_\{T\}=\\sum\_\{i\\in T\}w\_\{i\}denote the total subset weight\. WhenWT≠0W\_\{T\}\\neq 0, we define the weighted subset mean asμT=WT−1​∑i∈Twi​ϕi\\mu\_\{T\}=W\_\{T\}^\{\-1\}\\sum\_\{i\\in T\}w\_\{i\}\\phi\_\{i\}\. The centered geometric component can then be strictly formulated as:

Kcent​\(T\)=WT2​‖μT−ϕ¯‖2\.K\_\{\\mathrm\{cent\}\}\(T\)=W\_\{T\}^\{2\}\\\|\\mu\_\{T\}\-\\bar\{\\phi\}\\\|^\{2\}\.\(33\)In the strictly non\-negative regime wherewi≥0w\_\{i\}\\geq 0andWT\>0W\_\{T\}\>0, the normalized termKcent​\(T\)/WT2K\_\{\\mathrm\{cent\}\}\(T\)/W\_\{T\}^\{2\}exactly corresponds to the squared Maximum Mean Discrepancy \(MMD\) under a linear kernel\(Grettonet al\.,[2012](https://arxiv.org/html/2606.06892#bib.bib31)\)\. This MMD metric captures the discrepancy between the weighted empirical distribution on subsetTTand the unweighted full\-pool empirical distribution\. For arbitrary signed weights, Equation[33](https://arxiv.org/html/2606.06892#A1.E33)retains its structural interpretation as a squared distance under signed measures but must not be strictly interpreted as a valid probability distance\.

#### A projection sufficient condition for sketch error\.

The conditional sketch\-error bound established in Equation[28](https://arxiv.org/html/2606.06892#A1.E28)admits a standard sufficient condition in the idealized scenario where both target relevance and training geometry undergo an identical random projection\. LetxT=∑i∈Twi​uix\_\{T\}=\\sum\_\{i\\in T\}w\_\{i\}u\_\{i\}\. Suppose the sketches are generated via a linear mapΠ\\Pisuch thatϕi=Π​ui\\phi\_\{i\}=\\Pi u\_\{i\}andqe=Π​geq\_\{e\}=\\Pi g\_\{e\}\. Consider a finite subset family𝒯\\mathcal\{T\}and construct the expanded finite vector set:

𝒳e=\{xT:T∈𝒯\}\\displaystyle\\mathcal\{X\}\_\{e\}=\\\{x\_\{T\}:T\\in\\mathcal\{T\}\\\}∪\{ge\+xT:T∈𝒯\}\\displaystyle\\cup\\\{g\_\{e\}\+x\_\{T\}:T\\in\\mathcal\{T\}\\\}∪\{ge−xT:T∈𝒯\}\.\\displaystyle\\cup\\\{g\_\{e\}\-x\_\{T\}:T\\in\\mathcal\{T\}\\\}\.\(34\)AssumeΠ\\Pipreserves all squared Euclidean norms within𝒳e\\mathcal\{X\}\_\{e\}up to a relative errorε∈\(0,1\)\\varepsilon\\in\(0,1\)\. Then, for everyT∈𝒯T\\in\\mathcal\{T\}, the norm preservation directly guarantees:

\|‖Π​xT‖2−‖xT‖2\|\\displaystyle\\left\|\\\|\\Pi x\_\{T\}\\\|^\{2\}\-\\\|x\_\{T\}\\\|^\{2\}\\right\|≤ε​‖xT‖2\.\\displaystyle\\leq\\varepsilon\\\|x\_\{T\}\\\|^\{2\}\.\(35\)Furthermore, applying the standard polarization identity,⟨a,b⟩=\(‖a\+b‖2−‖a−b‖2\)/4\\langle a,b\\rangle=\(\\\|a\+b\\\|^\{2\}\-\\\|a\-b\\\|^\{2\}\)/4, guarantees that the inner product error is bounded by:

\|⟨Π​ge,Π​xT⟩−⟨ge,xT⟩\|\\displaystyle\\left\|\\langle\\Pi g\_\{e\},\\Pi x\_\{T\}\\rangle\-\\langle g\_\{e\},x\_\{T\}\\rangle\\right\|≤ε2​\(‖ge‖2\+‖xT‖2\)\.\\displaystyle\\leq\\frac\{\\varepsilon\}\{2\}\\left\(\\\|g\_\{e\}\\\|^\{2\}\+\\\|x\_\{T\}\\\|^\{2\}\\right\)\.\(36\)Classical Johnson\-Lindenstrauss embeddings provide exactly this finite\-set norm preservation\(Dasgupta and Gupta,[2003](https://arxiv.org/html/2606.06892#bib.bib32); Achlioptas,[2003](https://arxiv.org/html/2606.06892#bib.bib33)\)\. Such embeddings require a sketch dimension scaling asO​\(ε−2​log⁡\(\|𝒳e\|/δ\)\)O\(\\varepsilon^\{\-2\}\\log\(\|\\mathcal\{X\}\_\{e\}\|/\\delta\)\)to achieve a failure probability bounded byδ\\delta\. Under this idealized projection condition, one can rigorously setϵK​\(T\)=ε​‖xT‖2\\epsilon\_\{K\}\(T\)=\\varepsilon\\\|x\_\{T\}\\\|^\{2\}andϵA​\(T\)=ε​\(‖ge‖2\+‖xT‖2\)/2\\epsilon\_\{A\}\(T\)=\\varepsilon\(\\\|g\_\{e\}\\\|^\{2\}\+\\\|x\_\{T\}\\\|^\{2\}\)/2within Equation[28](https://arxiv.org/html/2606.06892#A1.E28)\. We explicitly note that this constitutes a theoretical sufficient condition\. In practice, our hidden and residual feature sketches are evaluated empirically via the LDS diagnostic rather than assumed to perfectly satisfy this idealized randomized projection criterion\.

#### Retained and omitted parameterizations\.

For strictly additive models, retained\-set scores equal a global corpus constant minus the corresponding omitted\-set scores\. This linear relationship ensures that the omitted representation remains rank\-invariant up to a sign reversal\. However, for quadratic components, the retained and omitted parameterizations structurally diverge because full\-pool cross terms do not trivially cancel\. Furthermore, the subset omission fraction inherently rescales the expected magnitude of these geometric interaction terms\. Suppose each training example is omitted independently with probabilityqq\. LetMi∈\{0,1\}M\_\{i\}\\in\\\{0,1\\\}denote the binary omission indicator andwiw\_\{i\}represent the deterministic weight\. The expected squared norm of the omitted geometry evaluates to:

𝔼​\[‖∑iMi​wi​ϕi‖2\]\\displaystyle\\mathbb\{E\}\\left\[\\left\\\|\\sum\_\{i\}M\_\{i\}w\_\{i\}\\phi\_\{i\}\\right\\\|^\{2\}\\right\]=q​∑iwi2​‖ϕi‖2\\displaystyle=q\\sum\_\{i\}w\_\{i\}^\{2\}\\\|\\phi\_\{i\}\\\|^\{2\}\(37\)\+q2​∑i≠jwi​wj​⟨ϕi,ϕj⟩\.\\displaystyle\\quad\+q^\{2\}\\sum\_\{i\\neq j\}w\_\{i\}w\_\{j\}\\langle\\phi\_\{i\},\\phi\_\{j\}\\rangle\.Consequently, under a uniform omission rateqq, the expected self\-interaction scales linearly withqq, whereas the pairwise cross\-interaction decays quadratically withq2q^\{2\}\.

#### Ranking stability of subset predictions\.

Letℬ=\{Sb\}b=1B\\mathcal\{B\}=\\\{S\_\{b\}\\\}\_\{b=1\}^\{B\}define the evaluation family of LDS subsets for a specific target or task\-level aggregate\. Denote the true realized utility asUb=𝒰e​\(Sb\)U\_\{b\}=\\mathcal\{U\}\_\{e\}\(S\_\{b\}\)and the predicted surrogate utility asU^b=Ub\+εb\\widehat\{U\}\_\{b\}=U\_\{b\}\+\\varepsilon\_\{b\}, whereεb\\varepsilon\_\{b\}represents the approximation error\. For any sampled subset pair\(b,c\)\(b,c\), the predicted ranking correctly recovers the true empirical ranking whenever the error difference is strictly bounded by the true utility gap:

\|εb−εc\|<\|Ub−Uc\|\.\|\\varepsilon\_\{b\}\-\\varepsilon\_\{c\}\|<\|U\_\{b\}\-U\_\{c\}\|\.\(38\)This margin condition mathematically guarantees correct pairwise ordering, as it implies\(U^b−U^c\)​\(Ub−Uc\)\>0\(\\widehat\{U\}\_\{b\}\-\\widehat\{U\}\_\{c\}\)\(U\_\{b\}\-U\_\{c\}\)\>0\. Consequently, assuming a uniform error bound‖ε‖∞≤εmax\\\|\\varepsilon\\\|\_\{\\infty\}\\leq\\varepsilon\_\{\\max\}, the pairwise ranking accuracy is bounded below by:

PairAcc≥\|𝒫2​εmax\|\(B2\),\\operatorname\{PairAcc\}\\geq\\frac\{\|\\mathcal\{P\}\_\{2\\varepsilon\_\{\\max\}\}\|\}\{\\binom\{B\}\{2\}\},\(39\)Here,𝒫2​εmax\\mathcal\{P\}\_\{2\\varepsilon\_\{\\max\}\}contains the subset pairs\(b,c\)\(b,c\)withb<cb<cand realized utility gap\|Ub−Uc\|\>2​εmax\|U\_\{b\}\-U\_\{c\}\|\>2\\varepsilon\_\{\\max\}\. This formulation does not provide a tight theoretical bound on the continuous Spearman correlation coefficient, but it formalizes a critical gap condition inherent to rank\-based counterfactual metrics\. Specifically, surrogate approximation errors degrade empirical fidelity only when their magnitude exceeds the intrinsic retraining utility gaps between competing data interventions\.

## Appendix BExperimental and Implementation Details

#### LDS Benchmark Setup\.

Our full\-dataset LDS evaluations employ a 50M\-parameter causal decoder\-only Transformer\(Vaswaniet al\.,[2017](https://arxiv.org/html/2606.06892#bib.bib2); Radfordet al\.,[2019](https://arxiv.org/html/2606.06892#bib.bib3)\)\. The pre\-training pool consists of 10B tokens randomly sampled from our Common Crawl corpus, processed through a DCLM/RefinedWeb\-style data curation pipeline\(Raffelet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib11); Penedoet al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib21),[2024](https://arxiv.org/html/2606.06892#bib.bib20); Liet al\.,[2024](https://arxiv.org/html/2606.06892#bib.bib22)\)\. To ensure rigorous comparisons, all methods within a given LDS environment share an identical candidate pool, reference checkpoint, subset\-retraining recipe, and evaluation targets\. The primary evaluation benchmarks include OpenBookQA\(Mihaylovet al\.,[2018](https://arxiv.org/html/2606.06892#bib.bib7)\), CommonsenseQA\(Talmoret al\.,[2019](https://arxiv.org/html/2606.06892#bib.bib9)\), SciQ\(Welblet al\.,[2017](https://arxiv.org/html/2606.06892#bib.bib29)\), and the BasicSkills suite from the OLMo evaluation framework\(Guet al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib25); Olmo,[2026](https://arxiv.org/html/2606.06892#bib.bib26)\)\. Evaluation metrics, including LDS mean, positive fraction, task\-level Spearmanρ\\rho, and pairwise accuracy are all derived from the same retrained subset utilities, with Table[1](https://arxiv.org/html/2606.06892#S3.T1)adopting task\-levelρ\\rhoas the principal ranking criterion\.

#### Attribution Baselines\.

We benchmark GRASP against BM25, TracIn\(Pruthiet al\.,[2020](https://arxiv.org/html/2606.06892#bib.bib13)\), and TRAK\(Parket al\.,[2023](https://arxiv.org/html/2606.06892#bib.bib16)\), with all methods scoring the exact same CC\-10B candidate rows\. For BM25, text is lowercased and tokenized via\[A\-Za\-z0\-9\_\]\+; we remove tokens shorter than two characters alongside a fixed set of English stopwords and question words, settingk1=1\.2k\_\{1\}=1\.2andb=0\.75b=0\.75, and retain the top 50k training rows per evaluation query \(stored in float16\)\. For TracIn, we accumulate normalized gradient cosine similarities from bfloat16 forward passes \(sequence length 2048\)\. This utilizes the representation from the tenth transformer block alongside the final layer normalization module, projected onto a 128\-dimensional random space, aggregating across learning\-rate\-weighted checkpoints at steps\{1000,3000,5000,7000,8000\}\\\{1000,3000,5000,7000,8000\\\}\. TRAK adheres to the same layer subset, checkpoint distribution, sequence length, precision, and 128\-dimensional projection, reporting results based on this five\-checkpoint variant\. For language\-based InRun\-DS\(Wanget al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib12)\), we utilize the Ghost replay engine optimizing for global\-evaluation utility via AdamW \(learning rate5×10−55\\times 10^\{\-5\}\)\. This process runs for a single replay epoch over a 50k\-row train prefix and 256 validation examples, activating the input token embeddings, the final vocabulary projection head, as well as the internal attention and feed\-forward projections\. Because faithful per\-target LDS replay under this framework is computationally prohibitive, we evaluate InRun\-DS strictly as a global\-utility reference rather than including it in the main per\-target comparisons of Table[1](https://arxiv.org/html/2606.06892#S3.T1)\. Finally, Datamodel predictors\(Ilyaset al\.,[2022](https://arxiv.org/html/2606.06892#bib.bib23)\)are trained directly on LDS outcomes, functioning as supervised oracles\.

#### GRASP Configuration and Protocol\.

GRASP constructs an additive residual\-alignment cache originating from the same 50M reference checkpoint, exporting 64\-dimensional hidden and residual sketches\. Unless explicitly specified otherwise, token features are aggregated using a tail\-mean\-32 strategy, the residual top\-kkis set to 64, and residual sketches employ theγ−1/2\\gamma^\{\-1/2\}weighting detailed in Section[3\.1](https://arxiv.org/html/2606.06892#S3.SS1)\. All hyperparameters, including component weights, interaction scales, and inclusion/omission strategies are strictly selected from finite calibration families using only development environments\. These configurations are fully frozen prior to downstream evaluation\. Furthermore, omitted\-set scoring serves exclusively as a fixed deletion protocol; we do not rely on the strong assumption that retained\-set and omitted\-set utilities are perfectly symmetric\.

#### Computational Efficiency and Ablations\.

Table[2](https://arxiv.org/html/2606.06892#S4.T2)explicitly decouples the computational cost of artifact construction from the cached subset scoring phase\. The cached sweep scores random candidate subsets within a local CPU process, extrapolated to 100k subsets, whereas the construction metrics reflect representative end\-to\-end runs for each respective baseline\. For deeper algorithmic analysis, the one\-step fidelity evaluation \(Figure[2](https://arxiv.org/html/2606.06892#S4.F2)\) compares GRASP against Monte Carlo Shapley values and fixed\-size one\-step subset utilities\. Additionally, Figure[3](https://arxiv.org/html/2606.06892#S4.F3)ablates individual component families to evaluate their independent contributions, performed without re\-calibrating the remaining weights\.

#### Downstream Applications and Scaling\.

Our DCLM\-style curation experiments train a 1B\-parameter tied\-embedding LLaMA architecture on matched\-size CC subsets\. The full\-scale pre\-training processes two million rows with a sequence length of 2048, utilizing eight GPUs, a per\-device batch size of 4, gradient accumulation of 16, and a peak learning rate of10−410^\{\-4\}with cosine decay\. Training leverages bfloat16 precision, FlashAttention\-2, and DeepSpeed ZeRO\-2 optimization\. Comprehensive full\-budget held\-out evaluations are summarized in the main text \(Table[3](https://arxiv.org/html/2606.06892#S4.T3)\), while intermediate training optimization dynamics are tracked in Appendix Table[8](https://arxiv.org/html/2606.06892#A5.T8)\. Finally, the vision\-domain experiments strictly follow the standard CIFAR\-10/ResNet18 data\-valuation protocol established byGhorbani and Zou \([2019](https://arxiv.org/html/2606.06892#bib.bib6)\)\.

![Refer to caption](https://arxiv.org/html/2606.06892v1/x4.png)Figure 4:Evolution of topic and format contributions\.Aggregated GRASP attribution shares across training checkpoints for the SciQ target\. Positive and negative mean\-score masses are normalized independently per checkpoint to illustrate relative category preferences over time\.

## Appendix CQualitative Analysis of Attribution Preferences

To better understand the underlying data\-selection dynamics of GRASP, we conduct a qualitative analysis by categorizing scored CC\-10B documents using WebOrganizer topic and format labels\. This investigation aims to explicitly profile the semantic and structural characteristics that GRASP intrinsically prioritizes when targeting a specific downstream task\.

### C\.1Synergy in Related\-Domain Retrieval

We evaluate the retrieval efficacy of various scoring methods on a shared candidate pool of 88,781 documents, measuring performance via normalized Recall@1000\. Recognizing that semantic attribution and surface\-level lexical overlap may prioritize distinct document characteristics, we report topic and format retrieval metrics independently\.

Table 6:Related\-domain retrieval performance\.We evaluate candidate CC\-10B rows mapped to WebOrganizer labels\(Wettiget al\.,[2025](https://arxiv.org/html/2606.06892#bib.bib39)\)\. Metrics denote mean normalized Recall@1000 \(higher is better\); G represents GRASP\.As demonstrated in Table[6](https://arxiv.org/html/2606.06892#A3.T6), attribution\-based and lexical signals provide highly complementary information\. While GRASP demonstrates strong standalone competitiveness, fusing its attribution scores with lexical heuristics \(BM25 or query overlap\) consistently yields the highest retrieval performance across both topic and format dimensions\.

### C\.2Checkpoint\-Wise Category Contributions

To enhance the interpretability of our attribution framework, we analyze the mean signed attribution scores aggregated by WebOrganizer categories across multiple training checkpoints\. By independently normalizing the positive and negative attribution mass within each checkpoint, we ensure that the relative categorical contributions remain clearly identifiable, even as the absolute magnitude of the attribution signal fluctuates throughout the training trajectory\.

Figure[4](https://arxiv.org/html/2606.06892#A2.F4)illustrates these aggregated category\-level attribution dynamics\. From a topical perspective, GRASP consistently assigns positive utility to knowledge\-dense source domains for the SciQ target, while actively penalizing generic web categories that lack strict task alignment\. A parallel trend emerges in the format view, where the attribution signal effectively isolates high\-value, structurally coherent formats \(e\.g\., articles\) from lower\-utility boilerplate content \(e\.g\., notices, support pages, and FAQs\)\. Crucially, these distinct selection patterns validate GRASP as a nuanced, model\-conditioned valuation signal: rather than merely reflecting the underlying source distribution or relying on shallow lexical heuristics, it dynamically assigns semantically grounded utility to broad data regions throughout the pre\-training process\.

## Appendix DSupplementary LDS Robustness Results

To further validate the stability of our attribution framework, this section presents comprehensive robustness evaluations that extend beyond the primary benchmarks established in Table[1](https://arxiv.org/html/2606.06892#S3.T1)\. These analyses stringently stress\-test the LDS protocol under diverse configurations, demonstrating its consistent reliability without incurring the computational overhead of additional model training\.

### D\.1Transferability of Cached GRASP Configurations

To assess the robustness of GRASP across varying domains, we investigate the transferability of fixed component weights\. Specifically, we calibrate the interaction\-component weights on a designated development task \(or tasks\), freeze them, and evaluate the resulting cached scores and sketches across three primary LDS benchmarks\.

Table 7:Transferability of fixed GRASP components\.Interaction\-component weights are calibrated on the tasks specified in each row and subsequently frozen\. We then evaluate across all target tasks using the reused cached hidden/residual sketches and subset\-retraining ground truth, requiring zero additional model training\.As illustrated in Table[7](https://arxiv.org/html/2606.06892#A4.T7), we observe a clear specialization\-generalization dynamic\. While single\-task calibration naturally yields optimal performance on its respective source domain, joint calibration \(SciQ \+ Logical\) prevents overfitting and enhances transferability to unseen tasks \(e\.g\., OpenBookQA\), ultimately achieving the highest average positive fraction\.

## Appendix ESupplementary Curation Results

Beyond direct fidelity evaluations, we investigate the practical utility of subset\-level attribution in large\-scale data\-selection pipelines\. To isolate curation efficacy, these supplementary experiments are conducted under strictly matched training budgets, serving to demonstrate downstream scalability rather than establishing core LDS fidelity\.

### E\.1Optimization Dynamics in Fixed\-Compute DCLM Curation

To evaluate curation efficacy at scale, we train a 1B\-parameter language model on matched\-size CC\-10B subsets selected by each policy\. While the comprehensive full\-budget held\-out evaluations are detailed in Table[3](https://arxiv.org/html/2606.06892#S4.T3), we additionally examine the optimization dynamics during pre\-training\. Table[8](https://arxiv.org/html/2606.06892#A5.T8)reports intermediate training losses at matched steps, demonstrating that GRASP consistently accelerates early\-stage convergence compared to the baselines\.

Table 8:Intermediate training loss dynamics\.Losses are recorded at specific steps under the fixed\-compute DCLM scale\-up setting\. GRASP maintains consistently lower training loss throughout the observed trajectory\.
### E\.2Cross\-Architecture Generalization in Data Cleaning

We further investigate whether our data\-scoring mechanism generalizes across different model families by evaluating compact CC\-cleaning runs on GPT\-2 and Pythia\-410M architectures\. This setting stringently tests the architecture\-agnostic robustness of the curation policy\.

Table 9:Cross\-architecture generalization in CC cleaning\.Each row reports the final held\-out loss \(lower is better\) for the respective model architecture\.Gainindicates the absolute loss reduction of GRASP over the strongest runner\-up\.As shown in Table[9](https://arxiv.org/html/2606.06892#A5.T9), GRASP attains the lowest final held\-out loss across both architectural paradigms\. This consistent margin of improvement confirms that our scoring rule transfers robustly, effectively identifying high\-quality pre\-training data without relying on architecture\-specific inductive biases\.

## Appendix FGeneralization to Safety and Misleading Information Detection

Beyond general data quality curation, we investigate whether our attribution framework can serve as a robust countermeasure against the risks of misleading information injected during fine\-tuning\. This evaluation assesses the capability of GRASP to isolate suspicious training examples that could compromise model safety, particularly in chat assistants\. We benchmark our approach against established lightweight detection baselines under a unified evaluation protocol\.

Table 10:Detection of misleading information and safety risks\.Performance comparison of GRASP against baseline countermeasures for identifying malicious or compromised fine\-tuning examples\. All methods are evaluated under a strictly standardized protocol\.As Table[10](https://arxiv.org/html/2606.06892#A6.T10)demonstrates, the attribution signals derived from GRASP naturally extend to safety\-oriented data ranking\. By effectively identifying misleading examples, GRASP consistently improves upon the F1 and AUC scores of existing baseline countermeasures, highlighting its versatility as a defensive filtering mechanism\.