从行为到来源：将表格基础模型归因于合成预训练数据

# Untitled Document
Source: [https://arxiv.org/html/2610.02347](https://arxiv.org/html/2610.02347)
## ![[Uncaptioned image]](https://arxiv.org/html/2610.02347v1/Figs/title.png)

Nassim Bouarour\*Affiliation:Lexsi LabsAffiliation:\*Equal contribution

AbstractTraining\-data attribution aims to identify which training examples shape model behavior, yet validating such claims is difficult because causal training influence is rarely observable\. We argue that controlled synthetic pretraining makes attribution experimentally testable\. Using O’PRIOR, a provenance\-rich synthetic task generator for tabular foundation models, we construct a testbed in which every pretraining task carries explicit lineage over structural mechanisms, missingness, confounding, shortcuts, and distribution shift\. We combine behavior\-conditioned attribution with counterfactual retraining and provenance\-aware interventions to test both task\-level faithfulness and mechanism\-level consistency\. On held\-out real tasks, removing the top\-attributed5%5\\%of synthetic tasks decreases mean ROC\-AUC by0\.0130\.013, compared with0\.002±0\.0040\.002\\pm 0\.004under random removal, while removing bottom\-attributed tasks improves performance by0\.0030\.003\. Within shortcut\-provenance tasks, targeted removal yields an effect of0\.0430\.043versus0\.0160\.016for matched random removal\. Provenance discrimination is more modest by ranking AUROC \(0\.550\.55–0\.620\.62\), despite substantial top\-kkenrichment, revealing that provenance association and interventional faithfulness need not coincide\. Our results establish synthetic provenance as a controlled setting for verifiable contributive attribution*Keywords:*Training Data Attribution; Synthetic Priors; Data Provenance; Tabular Foundation Models![[Uncaptioned image]](https://arxiv.org/html/2610.02347v1/Figs/logo.png)

## 1 Introduction

Training\-data attribution seeks to explain model behavior in terms of the data that shaped it\. Given a prediction, capability, or failure, contributive attribution methods aim to identify the training examples that influenced that behavior\. Existing approaches include influence functions, gradient\-based tracing, data\-models, and scalable estimators such as TRAK\([Koh and Liang, 2017](https://arxiv.org/html/2610.02347#bib.bib6);[Pruthi et al\., 2020](https://arxiv.org/html/2610.02347#bib.bib7);[Ilyas et al\., 2022](https://arxiv.org/html/2610.02347#bib.bib8);[Park et al\., 2023](https://arxiv.org/html/2610.02347#bib.bib9)\)\. Yet a fundamental difficulty remains:*how can we tell whether an attribution is faithful?*The contribution of a training example is not directly observed, and provenance metadata, when available, records where data came from rather than how the learned model would have behaved had that data been absent\. Attribution is therefore often evaluated through plausibility or indirect proxies rather than controlled interventions, limiting its use for auditing and accountability\.

We argue that synthetic pretraining turns this validation problem into an experimental opportunity\. When training data is produced by a controllable generator, each training task can carry an explicit record of how it was generated, while the training corpus itself can be manipulated\. We study this setting in tabular foundation models \(TFMs\)\([Jäger et al\., 2026](https://arxiv.org/html/2610.02347#bib.bib13);[Qu et al\., 2026](https://arxiv.org/html/2610.02347#bib.bib2);[Bouadi et al\., 2025](https://arxiv.org/html/2610.02347#bib.bib1);[Bouadi et al\., 2026b](https://arxiv.org/html/2610.02347#bib.bib12);[Zhang et al\., 2026](https://arxiv.org/html/2610.02347#bib.bib14)\), whose predictive behavior is largely acquired from synthetic supervised tasks\. In particular, we build on O’PRIOR\([Bouadi et al\., 2026a](https://arxiv.org/html/2610.02347#bib.bib10)\), a compositional synthetic prior in which each task records its structural mechanism together with interventions for missingness, confounding, spurious correlation, and distribution shift\. While O’PRIOR was originally introduced to study how prior design affects downstream TFM quality, we use its controllable lineage for a different purpose: as a provenance\-rich testbed for evaluating training\-data attribution\.

Crucially, known provenance is not itself causal ground truth\. A task may carry a particular generative mechanism without being important for a given downstream behavior, and two tasks with identical provenance may have very different training effects\. Our framework therefore connects three quantities that are normally unavailable together: the attribution assigned to individual synthetic tasks, the known provenance of those tasks, and the behavioral effect of removing them and retraining\. At the*task level*, we test interventional faithfulness by asking whether removing highly attributed tasks changes a downstream behavior more than removing equally sized random or weakly attributed subsets\. At the*provenance level*, we ask whether influential tasks concentrate on generative mechanisms associated with the behavior being explained, and whether this association changes coherently under controlled interventions\. This separation lets us distinguish plausible provenance association from attribution that survives intervention on the training corpus\.

Our experiments reveal a clear distinction between*provenance association*and*interventional faithfulness*\. Influence is the most removal\-faithful estimator: removing its top\-attributed5%5\\%of tasks decreases mean ROC\-AUC by0\.0130\.013, versus0\.002±0\.0040\.002\\pm 0\.004under random removal, while bottom removal slightly improves performance\. Within shortcut provenance, Influence also yields the largest top\-vs\-random counterfactual gap \(0\.0300\.030\)\. Yet provenance discrimination remains modest \(macro ROC\-AUC0\.550\.55–0\.620\.62\), and TracIn achieves higher shortcut Lift@5%5\\%than Influence \(1\.81×1\.81\\timesvs\.1\.60×1\.60\\times\) despite weaker interventional faithfulness\. Provenance\-level attribution is also slightly more stable than task\-level rankings \(0\.860\.86vs\.0\.820\.82\)\. Together, these results show that provenance association, behavioral influence under retraining, and attribution stability are related but distinct properties\.

#### Contributions\.

We make three contributions:

1. 1\.Provenance\-grounded attribution:we formulate synthetic\-data attribution hierarchically, connecting downstream behavior to influential pretraining tasks and those tasks to their known generative provenance\.
2. 2\.Interventional validation:using O’PRIOR’s frozen synthetic corpus, we test attribution through counterfactual task removal and provenance\-conditioned interventions rather than attribution plausibility alone\.
3. 3\.A systematic study of faithfulness, provenance association, and reliability:we compare attribution methods against intervention\-derived effects, study attribution under controlled failures and distribution shifts, and show that these evaluation axes need not rank attribution methods in the same way\.

## 2 Provenance\-Grounded Attribution

Training\-data attribution seeks to estimate how training examples contribute to learned model behavior, yet the causal quantity of interest is generally unobserved\. Given a modelfθf\_\{\\theta\}trained on𝒟\\mathcal\{D\}, an attribution method returns scores intended to identify which elements of𝒟\\mathcal\{D\}contributed to a behaviorBB\. Whether these scores reflect genuine training influence is difficult to establish: the causal contribution of an example is unobserved, and dataset provenance, when available, does not reveal how model behavior would change had that example been absent\. Our framework is built around removing that obstacle by keeping three quantities explicitly separate: the*known generative provenance*ziz\_\{i\}of a synthetic training task, an*attribution method’s estimate*ai\(B\)a\_\{i\}^\{\(B\)\}of its contribution toBB, and the*interventional effect*τB​\(S\)\\tau\_\{B\}\(S\)of actually removing a set of tasksSSand retraining\. Provenance is recorded lineage, not assumed causal ground truth; whether it agrees witha\(B\)a^\{\(B\)\}andτB\\tau\_\{B\}is the empirical question the rest of the paper answers\.

### 2\.1 Setup and behavior\-conditioned attribution

#### A frozen, provenance\-annotated corpus\.

Let𝒟=\{D1,…,DN\}\\mathcal\{D\}=\\\{D\_\{1\},\\ldots,D\_\{N\}\\\}be a synthetic pretraining corpus of in\-context supervised tasksDi=\(Xis,yis,Xiq,yiq\)D\_\{i\}=\(X\_\{i\}^\{s\},y\_\{i\}^\{s\},X\_\{i\}^\{q\},y\_\{i\}^\{q\}\), instantiated once from O’PRIOR\([Bouadi et al\., 2026a](https://arxiv.org/html/2610.02347#bib.bib10)\)and frozen rather than resampled online, so that individual tasks remain identifiable across scoring and retraining\.

O’PRIOR’s generation is compositional: structural mechanism, observational transformation, missingness, confounding, shortcut injection, and support–query shift are each controlled independently rather than folded into a single generator identity[Bouadi et al\. \(\)](https://arxiv.org/html/2610.02347#bib.bib11)\. Each task therefore carries provenancezi=\(zistr,ziobs,zishift,ωi\)z\_\{i\}=\(z\_\{i\}^\{\\text\{str\}\},z\_\{i\}^\{\\text\{obs\}\},z\_\{i\}^\{\\text\{shift\}\},\\omega\_\{i\}\), recording which mechanisms were active and their severities, and the corpus defines a provenance matrixZ=\[z1⊤,…,zN⊤\]⊤Z=\[z\_\{1\}^\{\\top\},\\ldots,z\_\{N\}^\{\\top\}\]^\{\\top\}\.

Note thatziz\_\{i\}answers*what process generatedDiD\_\{i\}*, not*whether that process caused a downstream behavior*, the latter requires intervention and is what Section[2\.2](https://arxiv.org/html/2610.02347#S2.SS2)tests\.

#### Behavior\-conditioned attribution\.

We evaluate a trained modelθ=𝒜⁡\(𝒟,ξ\)\\theta=\\mathcal\{A\}\(\\mathcal\{D\};\\xi\)on held\-out real tasksℛ=\{R1,…,RM\}\\mathcal\{R\}=\\\{R\_\{1\},\\ldots,R\_\{M\}\\\}never used for pretraining, and condition attribution on a behavior functionalB⁡\(θ,ℛ\)B\(\\theta;\\mathcal\{R\}\)rather than attributing the model globally\. For each synthetic task,ai\(B\)=Attr⁡\(Di,B,θ\)a\_\{i\}^\{\(B\)\}=\\operatorname\{Attr\}\(D\_\{i\},B;\\theta\)estimates its contribution to behaviorBB\. Our primary estimator is a first\-order influence\-function approximation\([Koh and Liang, 2017](https://arxiv.org/html/2610.02347#bib.bib6)\),

ai\(B\)=−gB​\(θ⋆\)⊤​H^θ⋆−1​gi​\(θ⋆\),gi​\(θ⋆\)=∇θℓ​\(Di,θ⋆\),gB​\(θ⋆\)=∇θℒB​\(ℛ,θ⋆\),a\_\{i\}^\{\(B\)\}=\-g\_\{B\}\(\\theta^\{\\star\}\)^\{\\top\}\\widehat\{H\}\_\{\\theta^\{\\star\}\}^\{\-1\}g\_\{i\}\(\\theta^\{\\star\}\),\\qquad g\_\{i\}\(\\theta^\{\\star\}\)=\\nabla\_\{\\theta\}\\ell\(D\_\{i\};\\theta^\{\\star\}\),\\quad g\_\{B\}\(\\theta^\{\\star\}\)=\\nabla\_\{\\theta\}\\mathcal\{L\}\_\{B\}\(\\mathcal\{R\};\\theta^\{\\star\}\),whereH^θ⋆−1\\widehat\{H\}\_\{\\theta^\{\\star\}\}^\{\-1\}denotes the inverse\-Hessian approximation used in our implementation andgig\_\{i\}is averaged within each task so scores are not confounded with task size\. For behaviors whose reported metric is non\-differentiable \(e\.g\., ROC\-AUC\),ℒB\\mathcal\{L\}\_\{B\}denotes a differentiable surrogate used only for attribution, whileBBis retained for counterfactual evaluation\. The framework is estimator\-agnostic: we compare Influence against multi\-checkpoint TracIn\([Pruthi et al\., 2020](https://arxiv.org/html/2610.02347#bib.bib7)\), TRAK\([Park et al\., 2023](https://arxiv.org/html/2610.02347#bib.bib9)\), and a representation\-similarity baseline\. Throughout,a\(B\)a^\{\(B\)\}is treated as a hypothesis about training influence, not as ground truth\.

We orient attribution scores so that largerai\(B\)a\_\{i\}^\{\(B\)\}corresponds to a larger positive contribution to the reported behaviorBB; when the differentiable objectiveℒB\\mathcal\{L\}\_\{B\}is a loss, we reverse the corresponding loss\-influence sign as needed\.

### 2\.2 Interventional validation and provenance analysis

We validate that hypothesis by manipulating the corpus itself\. ForS⊆𝒟S\\subseteq\\mathcal\{D\}, define the counterfactually retrained model

θ−S,ξ=𝒜⁡\(𝒟∖S,ξ\)\\theta\_\{\-S,\\xi\}=\\mathcal\{A\}\(\\mathcal\{D\}\\setminus S;\\xi\)and the removal effect

τB​\(S\)=𝔼ξ​\[B⁡\(θ𝒟,ξ,ℛ\)−B⁡\(θ−S,ξ,ℛ\)\]\\tau\_\{B\}\(S\)=\\mathbb\{E\}\_\{\\xi\}\\\!\\left\[B\(\\theta\_\{\\mathcal\{D\},\\xi\};\\mathcal\{R\}\)\-B\(\\theta\_\{\-S,\\xi\};\\mathcal\{R\}\)\\right\]holding the architecture, optimizer, and evaluation protocol fixed across removal conditions, while matching random seeds across retraining runs\. Thus,τB​\(S\)\\tau\_\{B\}\(S\)is the causal effect of this finite\-corpus removal intervention under training algorithm𝒜\\mathcal\{A\}, rather than an intrinsic influence value independent of the learning procedure\. For a removal fractionrr, letStoprS\_\{\\text\{top\}\}^\{r\}andSbottomrS\_\{\\text\{bottom\}\}^\{r\}be the top\- and bottom\-r​NrNtasks undera\(B\)a^\{\(B\)\}, andSrandrS\_\{\\text\{rand\}\}^\{r\}a size\-matched random subset\. We saya\(B\)a^\{\(B\)\}exhibits*removal faithfulness*if

τB​\(Stopr\)\>𝔼Srandr​\[τB​\(Srandr\)\]\\tau\_\{B\}\(S\_\{\\text\{top\}\}^\{r\}\)\>\\mathbb\{E\}\_\{S\_\{\\text\{rand\}\}^\{r\}\}\\\!\\left\[\\tau\_\{B\}\(S\_\{\\text\{rand\}\}^\{r\}\)\\right\]and, ideally,

τB​\(Stopr\)\>τB​\(Sbottomr\)\.\\tau\_\{B\}\(S\_\{\\text\{top\}\}^\{r\}\)\>\\tau\_\{B\}\(S\_\{\\text\{bottom\}\}^\{r\}\)\.Gradient similarity or representational resemblance to a behavior is not by itself evidence of interventional contribution; removal faithfulness asks whether the behavior actually changes once nominated tasks are unavailable to learning, giving an intervention\-based reference against which any attribution method can be scored\.

#### From influential tasks to generative mechanisms\.

Synthetic provenance supports a second question unavailable to standard attribution benchmarks: which generative properties characterize the tasks that matter\. For mechanismccwith indicatorzi​cz\_\{ic\}, we first measure whetherccis overrepresented among highly attributed tasks relative to its base rate in the corpus,

Enrichc\(B,r\)=P⁡\(zi​c=1∣Di∈Stopr\)P⁡\(zi​c=1\)\.\\operatorname\{Enrich\}\_\{c\}^\{\(B,r\)\}=\\frac\{P\(z\_\{ic\}=1\\mid D\_\{i\}\\in S\_\{\\text\{top\}\}^\{r\}\)\}\{P\(z\_\{ic\}=1\)\}\.Because O’PRIOR mechanisms co\-occur, enrichment alone cannot support a mechanism\-level causal claim, so we pair it with a mechanism\-conditioned intervention: within the stratum𝒟c=\{Di:zi​c=1\}\\mathcal\{D\}\_\{c\}=\\\{D\_\{i\}:z\_\{ic\}=1\\\}, we compare removing the most highly attributed mechanism\-cctasks against removing an equal number of randomly chosen tasks from the same stratum\. A gap between the two,τB​\(Stop,cr\)\>𝔼⁡\[τB​\(Srand,cr\)\]\\tau\_\{B\}\(S\_\{\\text\{top\},c\}^\{r\}\)\>\\mathbb\{E\}\[\\tau\_\{B\}\(S\_\{\\text\{rand\},c\}^\{r\}\)\], indicates that attribution distinguishes consequential instances*within*a mechanism, not just membership in it, a property we call*within\-provenance faithfulness*\.

#### Mechanism membership vs\. ranking\.

We distinguish two complementary questions\. A*membership effect*asks whether removing tasks carrying mechanismccchanges behaviorBcB\_\{c\}at all, whereas*within\-provenance faithfulness*asks whether attribution can identify the most consequential tasks*among*those carryingcc\. The former tests whether exposure to a provenance class is behaviorally relevant; the latter tests whether the attribution ranking contains additional information beyond provenance membership\. We report both quantities because provenance association alone need not imply faithful ranking within the corresponding mechanism\.

#### Behavior specificity\.

Attribution should depend on the behavior being explained\. For behaviorsℬ=\{Biid,Bmiss,Bshift,Bshortcut,…\}\\mathcal\{B\}=\\\{B\_\{\\mathrm\{iid\}\},B\_\{\\mathrm\{miss\}\},B\_\{\\mathrm\{shift\}\},B\_\{\\mathrm\{shortcut\}\},\\ldots\\\}, we stack their task\-attribution vectors intoA∈ℝN×\|ℬ\|A\\in\\mathbb\{R\}^\{N\\times\|\\mathcal\{B\}\|\}and study their alignment with the known provenance matrixZZ\. We quantify this alignment through base\-rate\-normalized top\-kkenrichment and, for graded interventions, the Spearman correlationρsev\\rho\_\{\\mathrm\{sev\}\}between intervention severity and attribution to the behavior\-matched provenance\. Thus, the question is not whether attribution recovers a single predetermined mechanism—O’PRIOR mechanisms are compositional—but whether the distribution of attributed provenance changes systematically with the behavior being explained\.

### 2\.3 Reliability

Individual synthetic tasks can be redundant or mutually substitutable, so task\-level attribution may be unstable even when the underlying training signal is not: two runs may rely on different examples while learning similar behavior\. We therefore ask whether provenance\-level attribution is more reproducible\. For attribution vectora\(s,B\)a^\{\(s,B\)\}obtained under seed or perturbationss, define the signed normalized scores

a~i\(s,B\)=ai\(s,B\)∑j\|aj\(s,B\)\|\+ϵ,\\tilde\{a\}\_\{i\}^\{\(s,B\)\}=\\frac\{a\_\{i\}^\{\(s,B\)\}\}\{\\sum\_\{j\}\|a\_\{j\}^\{\(s,B\)\}\|\+\\epsilon\},and the prevalence\-normalized provenance projection

pc\(s,B\)=N∑izi​c​∑ia~i\(s,B\)​zi​c\.p\_\{c\}^\{\(s,B\)\}=\\frac\{N\}\{\\sum\_\{i\}z\_\{ic\}\}\\sum\_\{i\}\\tilde\{a\}\_\{i\}^\{\(s,B\)\}z\_\{ic\}\.
We compare task\-level stability

𝒮task=𝔼s≠s′​\[ρ⁡\(a\(s,B\),a\(s′,B\)\)\]\\mathcal\{S\}\_\{\\mathrm\{task\}\}=\\mathbb\{E\}\_\{s\\neq s^\{\\prime\}\}\[\\rho\(a^\{\(s,B\)\},a^\{\(s^\{\\prime\},B\)\}\)\]with provenance\-level stability

𝒮prov=𝔼s≠s′​\[ρ⁡\(p\(s,B\),p\(s′,B\)\)\]\\mathcal\{S\}\_\{\\mathrm\{prov\}\}=\\mathbb\{E\}\_\{s\\neq s^\{\\prime\}\}\[\\rho\(p^\{\(s,B\)\},p^\{\(s^\{\\prime\},B\)\}\)\]whereρ\\rhois Spearman rank correlation\. This comparison tests whether aggregating attribution to generative mechanisms preserves a more reproducible signal than individual\-task rankings\.

Together, the framework makes three quantities jointly accessible that are usually separated in training\-data attribution: estimated task attributiona\(B\)a^\{\(B\)\}, observed behavioral change under counterfactual removalτB\\tau\_\{B\}, and known generative provenanceZZ\. Their combination lets us distinguish three properties that need not coincide:*removal faithfulness*, whether attributed training tasks actually affect the behavior they are credited with;*provenance association*, whether highly attributed tasks are enriched for behavior\-relevant generative mechanisms; and*attribution reliability*, whether these conclusions persist across runs and perturbations\. The contribution is therefore not a new attribution estimator, but a controlled evaluation paradigm in which apparent provenance alignment can be tested against intervention\-derived training effects\.

## 3 Experiments

### 3\.1 Experimental protocol

#### Frozen synthetic corpus\.

We instantiate O’PRIOR once to obtain a fixed corpus𝒟=\{Di\}i=1N\\mathcal\{D\}=\\\{D\_\{i\}\\\}\_\{i=1\}^\{N\}ofN=5,000N=5\{,\}000synthetic classification tasks\. Each task contains an in\-context support/query split, a persistent identifier, and provenanceziz\_\{i\}recording its structural mechanism and active missingness, confounding, shortcut, and distribution\-shift interventions\. Crucially, the corpus is frozen before training: no new synthetic tasks are sampled during either pretraining or counterfactual retraining, and a removed task remains unavailable throughout the corresponding retraining run\. Individual synthetic tasks therefore remain well\-defined units of both attribution and intervention\.

#### Model and downstream behaviors\.

We train NanoTabPFN\([Pfefferle et al\., 2025](https://arxiv.org/html/2610.02347#bib.bib5);[Hollmann et al\., 2025](https://arxiv.org/html/2610.02347#bib.bib3)\)for1515epochs with2,0002\{,\}000optimization updates per epoch and four synthetic tasks per batch\. Evaluation uses2222held\-out OpenML\-CC18 classification datasets\([Bischl et al\., 2017](https://arxiv.org/html/2610.02347#bib.bib4)\); no real dataset is used during synthetic pretraining\. The default behaviorBiidB\_\{\\mathrm\{iid\}\}is mean ROC\-AUC across these real tasks\. We additionally construct controlled missingness, covariate\-shift, and shortcut behaviors to test whether attribution changes with the behavior being explained\. When the reported behavior is non\-differentiable, we use cross\-entropy only as the differentiable attribution objective while retaining ROC\-AUC for behavioral evaluation\.

#### Attribution and counterfactual controls\.

Influence is our primary attribution estimator, with TracIn\([Pruthi et al\., 2020](https://arxiv.org/html/2610.02347#bib.bib7)\), TRAK\([Park et al\., 2023](https://arxiv.org/html/2610.02347#bib.bib9)\), and representation similarity as comparisons\. For counterfactual validation, models are retrained from scratch with the same architecture and number of optimization updates as the full\-corpus model; only the available pretraining tasks differ\. This controls for training\-budget effects and makes the intervention the removal of a specified subset of the fixed corpus\. We reportΔ​B=Bfull−Bremoved\\Delta B=B\_\{\\mathrm\{full\}\}\-B\_\{\\mathrm\{removed\}\}, so positive values indicate that removing the selected tasks degrades downstream performance\. Random\-removal controls use four independently sampled subsets with matched training runs and are reported as mean±\\pmempirical standard deviation\.

Table 1:Attribution quality on the frozen O’PRIOR corpus\(N=N\{=\}5k\)\. Global removal quantities useBiidB\_\{\\mathrm\{iid\}\}atr=10%r=10\\%; Lift@5%5\\%is base\-rate\-normalized matched\-provenance enrichment, and Within\-prov\. is the top\-vs\-random gap restricted to shortcut tasks\.𝒮\\mathcal\{S\}denotes mean pairwise Spearman stability\. Higher is better\.\(a\)Global removal under Influence\.\(b\)IID top\-vs\-random gap\.\(c\)Within\-shortcut gap\.
Figure 1:Counterfactual faithfulness of training\-data attribution\.\(a\)Change in mean ROC\-AUC after removing top\-attributed, random, or bottom\-attributed tasks; random removal shows mean±\\pmempirical standard deviation across four matched runs\.\(b\)Top\-vs\-random removal gap forBiidB\_\{\\mathrm\{iid\}\}atr=10%r=10\\%across attribution estimators\.\(c\)Within\-provenance faithfulness underBshortcutB\_\{\\mathrm\{shortcut\}\}, comparing top\-attributed and random removal restricted to shortcut\-generated tasks\. Positive gaps indicate that attribution identifies subsets whose removal is more consequential than the matched random control\.

### 3\.2 Task\-level faithfulness and provenance association

#### E1: Task\-level removal faithfulness\.

We first test the most direct claim made by contributive attribution: whether tasks assigned high influence exert larger behavioral effects when they are made unavailable to learning\. We rank all5,0005\{,\}000synthetic tasks with respect toBiidB\_\{\\mathrm\{iid\}\}and retrain after removing top\-attributed, bottom\-attributed, or size\-matched random subsets\.

Atr=5%r=5\\%, removing the top\-attributed tasks decreases mean ROC\-AUC byΔ​B=0\.013\\Delta B=0\.013, compared with0\.002±0\.0040\.002\\pm 0\.004under random removal \(Fig\.[1\(a\)](https://arxiv.org/html/2610.02347#S3.F1.sf1)\)\. Removing the bottom\-attributed subset instead givesΔ​B=−0\.003\\Delta B=\-0\.003, i\.e\., a small improvement, yielding a top\-vs\-random faithfulness gap of0\.0110\.011\. The contrast is therefore visible against both controls: the highly attributed subset is substantially more consequential than random removal, whereas the lowest\-ranked tasks are not required to preserve the baseline behavior\.

The effect persists atr=10%r=10\\%: top removal givesΔ​B=0\.012\\Delta B=0\.012versus0\.0040\.004for random removal, corresponding to a gap of0\.0080\.008\. Separation from the bottom subset is weaker at this larger intervention, so we regard the5%5\\%experiment as the cleaner test of fine\-grained ranking faithfulness and the10%10\\%intervention as evidence that the top\-vs\-random effect is not confined to a single removal fraction\. Importantly, these results validate the attribution ranking through the learning procedure itself rather than through similarity to downstream examples or provenance labels\.

The estimator comparison reinforces this distinction \(Fig\.[1\(b\)](https://arxiv.org/html/2610.02347#S3.F1.sf2)\)\. Atr=10%r=10\\%, Influence yields the largest top\-vs\-random gap \(\+0\.008\+0\.008\), compared with\+0\.002\+0\.002for TracIn and\+0\.001\+0\.001for TRAK; representation similarity produces a negative gap \(−0\.005\-0\.005\)\. Thus, methods that generate apparently meaningful rankings need not identify subsets whose removal has a correspondingly large behavioral effect\. Under this intervention, Influence provides the clearest task\-level faithfulness signal\.

#### E2: Provenance association and within\-provenance faithfulness\.

Counterfactual removal establishes that some attributed tasks matter, but does not explain what properties characterize them\. Because every O’PRIOR task carries explicit generative provenance, we can additionally ask whether high attribution concentrates on mechanisms related to the behavior being explained\. We evaluate this in two complementary ways: global ranking discrimination and base\-rate\-normalized top\-kkenrichment\.

Attribution contains statistically detectable, but far from complete, information about generative provenance\. Behavior\-matched provenance discrimination is approximately0\.600\.60ROC\-AUC, with macro values ranging from0\.550\.55to0\.620\.62across provenance columns \(Fig\.[2\(b\)](https://arxiv.org/html/2610.02347#S3.F2.sf2)\); matched average precision is likewise above the corresponding base rate\. These values are meaningfully above chance but modest in absolute terms, so we do not interpret the attribution score as recovering the generative mechanism from the full task ranking\.

The head of the ranking shows a clearer signal\. Behavior\-matched Lift@5%5\\%is approximately1\.3×1\.3\\timesoverall, and shortcut provenance reaches1\.60×1\.60\\timesunder Influence forBshortcutB\_\{\\mathrm\{shortcut\}\}\. Matched provenance scores are also shifted relative to their controls \(Wilcoxonz\>2\.23z\>2\.23\)\. The combination of moderate global ROC\-AUC and stronger top\-kkenrichment indicates that provenance information is concentrated most clearly among the highest\-attributed tasks rather than being uniformly separable throughout the full ranking\.

Provenance association alone, however, does not establish that the ranking distinguishes consequential tasks*within*a provenance class\. We therefore restrict the intervention to shortcut\-generated tasks and compare the highest\-attributed subset with a size\-matched random subset from the same stratum\. Influence yields a within\-shortcut top\-vs\-random gap of\+0\.030\+0\.030, followed by TracIn \(\+0\.023\+0\.023\), TRAK \(\+0\.008\+0\.008\), and representation similarity \(−0\.002\-0\.002\) \(Fig\.[1\(c\)](https://arxiv.org/html/2610.02347#S3.F1.sf3)\)\. Because mechanism membership is held fixed in this comparison, the positive gap cannot be explained merely by the presence of shortcut provenance: the attribution ranking carries additional information about which shortcut\-generated tasks are behaviorally consequential\.

The result is not universal across all behavior–mechanism combinations\. Other targeted pairs show heterogeneous or inconclusive separation\. We therefore treat within\-provenance faithfulness as an empirical property to be tested for each behavior rather than assuming that known provenance is itself causal ground truth\.

### 3\.3 Behavior specificity and estimator disagreement

\(a\)Behavior\-matched provenance enrichment\.\(b\)Provenance and behavior\-specificity evidence\.
Figure 2:Provenance association and behavior specificity\.\(a\)Lift@5%5\\%of behavior\-matched provenance among highly attributed tasks across estimators;1×1\\timesdenotes the corpus base rate\. For the shortcut behavior, TracIn attains higher enrichment than Influence despite weaker counterfactual faithfulness\.\(b\)Complementary provenance evidence: matched provenance discrimination is approximately0\.600\.60ROC\-AUC \(macro range0\.550\.55–0\.620\.62\), matched Lift@5%5\\%is approximately1\.3×1\.3\\timesoverall, matched comparisons yield Wilcoxonz\>2\.23z\>2\.23, and attribution to behavior\-matched provenance tracks intervention severity withρsev≥0\.63\\rho\_\{\\mathrm\{sev\}\}\\geq 0\.63over at least four severity levels\. Statistics in \(b\) have different interpretations and are not intended for direct comparison by magnitude\.#### E3: Attribution tracks the behavior being explained\.

A behavior\-conditioned attribution should not be invariant to a controlled change in the downstream behavior\. We therefore keep the pretrained model and synthetic corpus fixed while varying the severity of missingness, covariate shift, and shortcut interventions at evaluation time and recompute attribution for each resulting behavior\.

Across these graded interventions, attribution to behavior\-matched provenance changes systematically with intervention severity\. The corresponding Spearman correlations satisfyρsev≥0\.63\\rho\_\{\\mathrm\{sev\}\}\\geq 0\.63over at least four distinct severity levels \(Fig\.[2\(b\)](https://arxiv.org/html/2610.02347#S3.F2.sf2)\)\. Thus, the provenance associated with highly attributed training tasks changes monotonically with the strength of the downstream failure being explained rather than remaining fixed across evaluation conditions\.

The response is heterogeneous across mechanisms: some controlled behaviors produce clear matched\-provenance shifts whereas others are weaker or indistinguishable from controls\. This negative evidence is important\. Known provenance makes such failures visible and prevents behavior\-specific attribution from being judged only by examples for which the expected mechanism happens to appear\.

#### E4: Provenance association and interventional faithfulness can disagree\.

Comparing estimators exposes a distinction that is difficult to observe in standard attribution benchmarks\. Influence gives the largest global top\-vs\-random removal gap and the largest within shortcut counterfactual gap \(Table[1](https://arxiv.org/html/2610.02347#S3.T1)\)\. Provenance enrichment gives a different ordering: underBshortcutB\_\{\\mathrm\{shortcut\}\}, TracIn achieves Lift@5%5\\%of1\.81×1\.81\\times, exceeding Influence at1\.60×1\.60\\times\(Fig\.[2\(a\)](https://arxiv.org/html/2610.02347#S3.F2.sf1)\), even though its within\-provenance faithfulness gap is smaller \(\+0\.023\+0\.023vs\.\+0\.030\+0\.030\)\.

This reversal shows that the two evaluations measure genuinely different properties\. A method can preferentially retrieve tasks carrying therightprovenance label without identifying the subset whose absence has the largest effect on learning\. Conversely, intervention\-derived faithfulness does not require perfect recovery of the corresponding generative mechanism\. We therefore treat provenance association as supporting evidence about*where*attribution concentrates and counterfactual retraining as evidence about whether the nominated training data is behaviorally consequential under the specified learning intervention\.

### 3\.4 Attribution reliability

Figure 3:Attribution reliability\.Mean pairwise Spearman stability at the individual\-task and provenance\-aggregated levels\. Provenance aggregation improves stability for every estimator, although the gain is modest\.#### E5: Task\-level versus provenance\-level stability\.

Finally, we ask whether attribution conclusions are reproducible across seed and checkpoint variants\. At the level of individual training tasks, small changes in estimation can alter the ordering of examples that are partially redundant or substitutable\. We therefore compare task\-level rankings with attribution aggregated onto the recorded generative mechanisms\.

Figure[3](https://arxiv.org/html/2610.02347#S3.F3)shows that provenance\-level summaries are more stable for every estimator considered\. Influence increases from𝒮task=0\.85\\mathcal\{S\}\_\{\\mathrm\{task\}\}=0\.85to𝒮prov=0\.91\\mathcal\{S\}\_\{\\mathrm\{prov\}\}=0\.91; TracIn from0\.800\.80to0\.840\.84; TRAK from0\.780\.78to0\.840\.84; and representation similarity from0\.700\.70to0\.720\.72\. The improvement is therefore systematic but modest rather than dramatic\. This supports provenance aggregation as a slightly more reproducible level of analysis without implying that it eliminates estimator uncertainty\.

We distinguish this notion of stability from cross\-estimator agreement\. Influence and TracIn, for example, can each be internally reproducible while still disagreeing about which tasks should receive high attribution because they encode different notions of training influence\. Reliability under repeated estimation and agreement across attribution assumptions are therefore separate properties\.

## 4 Conclusion

We introduced a provenance\-grounded framework for evaluating contributive training\-data attribution in synthetically pretrained foundation models\. The key idea is to keep three quantities explicitly separate: the attribution assigned to individual training tasks, their recorded generative provenance, and the behavioral effect of removing those tasks and retraining\. This makes it possible to test attribution claims through intervention rather than treating either attribution scores or provenance labels as ground truth\.

Across our experiments, these quantities provide complementary but distinct signals\. Influence exhibits the strongest counterfactual removal faithfulness, both globally and within shortcut\-provenance strata, while attribution scores recover generative provenance only moderately at the level of the full ranking\. At the same time, highly ranked tasks are enriched for behavior\-matched provenance, and this attribution changes systematically under controlled failures and increasing intervention severity\. Crucially, the methods need not be ordered consistently across these criteria: TracIn shows stronger shortcut\-provenance enrichment than Influence \(1\.81×1\.81\\timesvs\.1\.60×1\.60\\times\) despite a smaller within\-provenance counterfactual gap \(0\.0230\.023vs\.0\.0300\.030\)\. This reversal demonstrates that*provenance association and interventional faithfulness are genuinely different properties*\. We additionally find that aggregating task\-level attribution onto generative mechanisms yields modestly more reproducible summaries, suggesting that provenance can provide a useful intermediate level of analysis between individual training examples and downstream behavior\.

Taken together, these results suggest a broader role for controlled synthetic pretraining in attribution research\. Known synthetic provenance should not be viewed as causal ground truth: it records how a task was generated, not whether that task caused a particular learned behavior\. Its value instead comes from providing a structured reference that can be combined with counterfactual retraining, behavior\-specific evaluation, and reliability analysis\. In this sense, synthetic provenance turns attribution from a primarily plausibility\-based exercise into one for which competing claims can be experimentally tested\.

#### Limitations\.

Our study is deliberately controlled, and several limitations bound the scope of its conclusions\. First, the experiments use a finite O’PRIOR corpus of5,0005\{,\}000synthetic tasks and a NanoTabPFN\-scale tabular foundation model; the same relationships may change with substantially larger models, larger synthetic corpora, or natural\-data pretraining\. Second, the removal effectτB​\(S\)\\tau\_\{B\}\(S\)is causal only with respect to the specific finite\-corpus intervention and training algorithm considered here: it depends on the optimizer, training budget, initialization, and retraining procedure, and should not be interpreted as an intrinsic influence value independent of learning dynamics\. Third, removing sets of tasks can induce interaction and redundancy effects, so the effect of a subset need not equal the sum of the effects of its members\. Fourth, O’PRIOR mechanisms are compositional and can co\-occur; consequently, provenance enrichment provides evidence of association but does not isolate a single generative mechanism as the unique cause of a behavior\. Our within\-provenance controls mitigate this issue but do not remove all dependencies between mechanisms\. Fifth, attribution for non\-differentiable behaviors relies on a differentiable surrogate objective, which may not perfectly align with the reported downstream metric\. Finally, counterfactual retraining is computationally expensive, which limits the number and scale of interventions that can be evaluated exhaustively\.

Future work should extend this evaluation paradigm to larger foundation models, richer and multi\-generation provenance graphs, and settings where synthetic and natural pretraining data coexist\. An important direction is to develop cheaper validation procedures that retain the evidential value of retraining\-based interventions while scaling to substantially larger training corpora\.

## References

- \[1\]P\. W\. Koh and P\. Liang\(2017\)Understanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.02347#S2.SS1.SSS0.Px2.p1.1)\.
- \[2\]G\. Pruthi, F\. Liu, S\. Kale, and M\. Sundararajan\(2020\)Estimating training data influence by tracing gradient descent\.Advances in neural information processing systems33,pp\. 19920–19930\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.02347#S2.SS1.SSS0.Px2.p1.2),[§3\.1](https://arxiv.org/html/2610.02347#S3.SS1.SSS0.Px3.p1.1)\.
- \[3\]A\. Ilyas, S\. M\. Park, L\. Engstrom, G\. Leclerc, and A\. Madry\(2022\)Datamodels: understanding predictions with data and data with predictions\.InInternational Conference on Machine Learning,pp\. 9525–9587\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p1.1)\.
- \[4\]S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. Madry\(2023\)Trak: attributing model behavior at scale\.arXiv preprint arXiv:2303\.14186\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.02347#S2.SS1.SSS0.Px2.p1.2),[§3\.1](https://arxiv.org/html/2610.02347#S3.SS1.SSS0.Px3.p1.1)\.
- \[5\]B\. Jäger, N\. Erickson, L\. Grinsztajn, F\. Birkel, K\. Flöge, O\. Key, K\. Kaya, J\. Kübler, A\. Frankel, T\. Schröder,et al\.\(2026\)TabPFN\-3\.5: technical report\.arXiv preprint arXiv:2609\.17895\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1)\.
- \[6\]J\. Qu, D\. Holzmüller, G\. Varoquaux, and M\. L\. Morvan\(2026\)TabICLv2: A better, faster, scalable, and open tabular foundation model\.CoRRabs/2602\.11139\.External Links:[Link](https://doi.org/10.48550/arXiv.2602.11139),[Document](https://dx.doi.org/10.48550/ARXIV.2602.11139),2602\.11139Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1)\.
- \[7\]M\. Bouadi, P\. Seth, A\. Tanna, and V\. K\. Sankarapu\(2025\)Orion\-msp: multi\-scale sparse attention for tabular in\-context learning\.CoRRabs/2511\.02818\.External Links:[Link](https://doi.org/10.48550/arXiv.2511.02818),[Document](https://dx.doi.org/10.48550/ARXIV.2511.02818),2511\.02818Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1)\.
- \[8\]M\. Bouadi, P\. Seth, A\. Tanna, and V\. K\. Sankarapu\(2026\)Orion\-bix: bi\-axial attention for tabular in\-context learning\.InProceedings of the ACM Web Conference 2026,pp\. 8673–8676\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1)\.
- \[9\]X\. Zhang, G\. Ren, H\. Yuan, H\. Zou, H\. Tan, H\. Wang, J\. Song, J\. Li, J\. Zhang, J\. Zhang,et al\.\(2026\)LimiX\-2: a contextual mechanism network towards general structured\-data intelligence\.arXiv preprint arXiv:2609\.17488\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1)\.
- \[10\]M\. Bouadi, N\. Bouarour, V\. Kulkarni, S\. Dubey, A\. Tanna, and V\. K\. Sankarapu\(2026\)Shaping the prior: how synthetic task distributions determine tabular foundation model quality\.arXiv preprint arXiv:2605\.18971\.Cited by:[§1](https://arxiv.org/html/2610.02347#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.02347#S2.SS1.SSS0.Px1.p1.1)\.
- \[11\]M\. Bouadi, N\. Bouarour, S\. Dubey, V\. Kulkarni, A\. Tanna, and V\. kumar SankarapuWhat you pretrain on matters: synthetic task distributions determine tabular foundation model quality\.In2nd ICML Workshop on Foundation Models for Structured Data,Cited by:[§2\.1](https://arxiv.org/html/2610.02347#S2.SS1.SSS0.Px1.p2.1)\.
- \[12\]A\. Pfefferle, J\. Hog, L\. Purucker, and F\. Hutter\(2025\)Nanotabpfn: a lightweight and educational reimplementation of tabpfn\.arXiv preprint arXiv:2511\.03634\.Cited by:[§3\.1](https://arxiv.org/html/2610.02347#S3.SS1.SSS0.Px2.p1.1)\.
- \[13\]N\. Hollmann, S\. Müller, L\. Purucker, A\. Krishnakumar, M\. Körfer, S\. B\. Hoo, R\. T\. Schirrmeister, and F\. Hutter\(2025\)Accurate predictions on small data with a tabular foundation model\.Nature637\(8045\),pp\. 319–326\.Cited by:[§3\.1](https://arxiv.org/html/2610.02347#S3.SS1.SSS0.Px2.p1.1)\.
- \[14\]B\. Bischl, G\. Casalicchio, M\. Feurer, P\. Gijsbers, F\. Hutter, M\. Lang, R\. G\. Mantovani, J\. N\. van Rijn, and J\. Vanschoren\(2017\)Openml benchmarking suites\.arXiv preprint arXiv:1708\.03731\.Cited by:[§3\.1](https://arxiv.org/html/2610.02347#S3.SS1.SSS0.Px2.p1.1)\.