DataDignity: Training Data Attribution for Large Language Models

# DataDignity: Training Data Attribution for Large Language Models
Source: [https://arxiv.org/html/2605.05687](https://arxiv.org/html/2605.05687)
Xiaomin Li Microsoft &Andrzej Banburski\-Fahey Microsoft &Jaron Lanier Microsoft

###### Abstract

Auditing language\-model outputs often requires more than judging correctness: an auditor may need to know which source document most likely supports the knowledge expressed in a response\. We study this problem as*pinpoint provenance*: given a prompt, a target\-model response, and a candidate corpus, rank the documents that best support the response\. We introduceFakeWiki, a controlled benchmark of 3,537 fabricated Wikipedia\-style articles designed to preserve ground\-truth provenance while weakening lexical shortcuts\. Each evaluated target LLM is explicitly continued\-pretrained on theFakeWikitext corpus before response collection, while the QA probes used for attribution evaluation are held out from target\-model training\.FakeWikiincludes short QA probes, source\-preserving paraphrases, retro\-generated variants, hard anti\-documents that remain topically similar while removing answer\-critical facts, and five query conditions: clean prompting plus four jailbreak\-inspired transformations, obfuscation, role\-play, noise injection, and indirect prompting\. We evaluate eleven lexical and semantic retrieval baselines, a training\-free activation\-steering retrieval\-fusion methodSteerFuse, and a supervised contrastive provenance rankerScoringModel\.ScoringModelmaps response and document features into a shared space and is trained with InfoNCE using in\-batch, retrieval\-mined, and anti\-document negatives\. Across nine open\-weight instruction\-tuned LLMs and five query conditions,ScoringModelimproves mean Recall@10 from 37\.3 for the strongest retrieval baseline to 52\.2, without inference\-time fusion, and wins 41/45 model\-by\-condition cells\.SteerFusebeats the strongest retrieval baseline in most cells while requiring no supervised training, showing that activation\-space evidence can complement text retrieval\. On the jailbreak\-inspired transformed queries,ScoringModelimproves Recall@10 by 13\.2 points on average over the best baseline, with the largest gains on larger target models\. Overall, our work shows that robust training data attribution requires evaluation settings that separate true answer support from topical or lexical resemblance\.

## 1Introduction

Large language models increasingly mediate factual, scientific, legal, and safety\-relevant information\. When a model produces a response, users may need to know not only whether it is correct, but also where it came from: which source document supplied the relevant fact, whether a questionable output depends on a particular source, or whether a data intervention removed the intended provenance path\. These questions arise in copyright audits, misinformation forensics, safety debugging, and dataset curation, and are not fully answered by standard evaluation or influence\-style methods\(Han and Tsvetkov,[2021](https://arxiv.org/html/2605.05687#bib.bib20); Liet al\.,[2026](https://arxiv.org/html/2605.05687#bib.bib5); Zhanget al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib4); Akyüreket al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib16); Parket al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib8); Barshanet al\.,[2020](https://arxiv.org/html/2605.05687#bib.bib18)\)\.

We study this problem as*pinpoint provenance*\. Given a promptxx, a target\-model responseyy, and a candidate corpus𝒟=\{Dj\}j=1N\\mathcal\{D\}=\\\{D\_\{j\}\\\}\_\{j=1\}^\{N\}, the goal is to return a short ranked list of documents that likely support the knowledge expressed inyy\. This is an operational retrieval problem: an auditor should inspect a small set of candidate sources rather than search through an entire corpus\. It is harder than ordinary semantic retrieval because the answer may be short, paraphrased, grounded in a small fact buried in a longer document, or elicited through a prompt transformation\.

A central challenge is that many provenance evaluations make attribution too easy through surface overlap\. If the source document, question, and response share rare names or distinctive phrases, lexical methods such as MinHash\(Broder,[1997](https://arxiv.org/html/2605.05687#bib.bib24)\), and even generic dense retrievers such asSBERT\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.05687#bib.bib25)\), Contriever\(Izacardet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib26)\), and BGE\(Xiaoet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib27)\), can appear effective without demonstrating robust source attribution\. Such methods may fail when provenance matters most: under paraphrase, obfuscation, indirect questioning, role\-play, or irrelevant context injection\. This motivates a benchmark in which the true source is known by construction, but the evaluation deliberately removes the easy paths from response wording back to document identity\.

We introduceFakeWiki, a benchmark designed to preserve ground\-truth provenance while weakening such shortcuts\. It contains 3,537 fabricated Wikipedia\-style articles with short QA probes, source\-preserving variants, and hard anti\-documents that preserve topical similarity while removing answer\-critical facts\. To make this a training\-data attribution setting, each target LLM is continued\-pretrained onFakeWikidocument text, while QA probes are held out and used only to elicit responses whose provenance should point back to the training documents\. We evaluate attribution under clean prompts and four transformed conditions:Obfuscate,RolePlay,NoiseInjection, andIndirect, testing whether attribution survives when lexical and semantic cues become less reliable\. Figure[1](https://arxiv.org/html/2605.05687#S1.F1)summarizes the benchmark and attribution pipeline\.111Data and code are available at[https://anonymous\.4open\.science/r/Submission\-DataDignity\-E263](https://anonymous.4open.science/r/Submission-DataDignity-E263)\.

![Refer to caption](https://arxiv.org/html/2605.05687v1/figures/pipeline-Row1.png)

![Refer to caption](https://arxiv.org/html/2605.05687v1/figures/pipeline-Row2.png)

Figure 1:Overview ofDataDignity\. Top:FakeWikiconstructs fabricated source documents, variants, anti\-documents, and transformed queries\. Bottom:ScoringModellearns a supervised provenance score, whileSteerFusefuses activation\-space evidence withSBERTretrieval\.Our main attribution method,ScoringModel, is a supervised Siamese provenance ranker\. It maps response\-side and document\-side features into a shared embedding space and trains with a contrastive InfoNCE objective\(Oordet al\.,[2018](https://arxiv.org/html/2605.05687#bib.bib22)\)over in\-batch negatives, retrieval\-mined hard negatives, and curated anti\-documents\. These anti\-documents force the model to distinguish documents that merely resemble the response from documents that actually support it\. At inference time, each candidate document is scored by this learned compatibility function\.

We also studySteerFuse, a training\-free activation\-steering retrieval\-fusion method inspired by representation\-level interventions in language models\(Subramaniet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib28); Turneret al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib29); Panicksseryet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib30); Zouet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib31); Liet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib32)\)\. It asks which candidate document provides the largest internal evidence boost toward the observed response, using cached document activation directions and an efficient response\-side proxy instead of patched forward passes\. The resulting activation\-space score is fused withSBERTretrieval to test whether model\-internal evidence complements text similarity under transformed prompts\.

The main result is that clean retrieval substantially understates the difficulty of robust provenance\. Both proposed attribution methods improve over standard retrieval: the training\-freeSteerFusemethod beats the strongest of eleven retrieval baselines in 32/45 model\-by\-query\-condition cells, whileScoringModelwins 41/45 cells\. Averaged across all models and query conditions,SteerFuseimproves mean Recall@10 from 37\.3 to 42\.3, andScoringModelfurther improves it to 52\.2 without inference\-time fusion\. On transformed queries,ScoringModelimproves Recall@10 by 13\.2 points on average over the best baseline, with especially large gains of \+26\.9 onLlama\-3\.1\-8Band \+20\.0 onQwen3\-8B\. Recall@1 and Recall@5 show the same pattern under stricter cutoffs, especially for the larger target models\. These results suggest that robust provenance evaluation should not stop at clean lexical or semantic retrieval: training\-free activation evidence can improve retrieval in many settings, and supervised attribution with hard negatives can recover stronger source\-support signals missed by generic similarity\.

Our contributions are:

- •We formulate robust pinpoint provenance as a source\-attribution task that evaluates whether methods can distinguish true answer support from topical or lexical resemblance\.
- •We introduceFakeWiki, a benchmark with ground\-truth source documents, short QA probes, source\-preserving variants, hard anti\-documents, and transformed query conditions\.
- •We proposeScoringModel, a supervised contrastive provenance scorer trained with hard negatives and evaluated without inference\-time retrieval fusion\.
- •We provide a broad empirical study across nine open\-weight instruction\-tuned LLMs, five query conditions, eleven retrieval baselines,SteerFuse, andScoringModel, with additional per\-model, seed, Recall@1, Recall@5, and ablation analyses in the appendix\.

## 2Related Work

#### Training data attribution and source retrieval\.

Training data attribution asks which examples or documents are associated with model behavior\. Influence\-style methods estimate effects on predictions or losses through gradients, checkpoints, approximations, or scalable surrogates\(Pruthiet al\.,[2020](https://arxiv.org/html/2605.05687#bib.bib19); Han and Tsvetkov,[2021](https://arxiv.org/html/2605.05687#bib.bib20); Barshanet al\.,[2020](https://arxiv.org/html/2605.05687#bib.bib18); Parket al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib8); Kwonet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib15)\), but address a complementary causal question about training dynamics\. We study an operational provenance task: given a candidate corpus and generated response, rank inspectable source documents\. This is closest to retrieval\-based source tracing, where MinHash captures lexical overlap\(Broder,[1997](https://arxiv.org/html/2605.05687#bib.bib24)\), whileSBERT, Contriever, BGE, and finetuned embeddings capture semantic similarity\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.05687#bib.bib25); Izacardet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib26); Xiaoet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib27); Rajaniet al\.,[2019](https://arxiv.org/html/2605.05687#bib.bib21); Fotouhiet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib14)\)\. Related work also studies source\-aware factual tracing and contrastive attribution embeddings\(Akyüreket al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib16); Khalifaet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib11); Wanget al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib9)\)\. We evaluate retrieval\-based provenance under anti\-shortcut conditions that separate answer support from topical or lexical resemblance\.

#### Activation\-space evidence\.

Activation\-space methods use internal hidden states to interpret or alter model behavior\. Prior work has extracted latent steering vectors\(Subramaniet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib28)\), added activation directions at inference time\(Turneret al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib29); Panicksseryet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib30)\), and used hidden representations for monitoring, control, truthfulness, or latent\-knowledge readouts\(Zouet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib31); Liet al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib32); Burnset al\.,[2023](https://arxiv.org/html/2605.05687#bib.bib33)\)\. We build on this perspective for provenance: a candidate document may provide internal evidence for a response even when its wording is not close to the generated text\.SteerFusetests this idea by comparing document\-induced activation directions with response representations and fusing the resulting signal withSBERTretrieval\. We treat this activation\-space evidence as complementary to text retrieval rather than as a replacement for it\.

## 3TheFakeWikiBenchmark

A provenance benchmark should provide ground\-truth sources without making attribution solvable by rare names or copied phrases\.FakeWikiaddresses this tension with fabricated Wikipedia\-style articles, source\-preserving variants, anti\-documents, and transformed prompts\. Table[1](https://arxiv.org/html/2605.05687#S3.T1)summarizes the benchmark components\. Together, they weaken wording overlap, vary factual context, preserve hard topical distractors, and disrupt prompt\-response surface form\.

ComponentWhat it containsWhat it testsFabricated articles3,537 Wikipedia\-style documents about non\-real entities and conceptsControlled provenance after explicit target\-model exposure, without relying on real\-world pretraining knowledgeQA probesFive short question\-answer probes per documentWhether attribution works when responses contain only sparse source evidenceSource variantsParaphrases, retro\-generated documents, and anti\-documentsWhether methods distinguish true answer support from topical or lexical similarityQuery conditionsClean,Obfuscate,RolePlay,NoiseInjection, andIndirectWhether provenance survives prompt transformations that change surface cues

Table 1:Design of theFakeWikibenchmark\. Each component is intended to weaken a different shortcut that ordinary retrieval methods might exploit\.### 3\.1Document Corpus

FakeWikicontains 3,537 fabricated Wikipedia\-style articles\. We build the corpus in three stages:

1. 1\.Generate\.GPT\-4o\(Hurstet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib1)\),DeepSeek\-V3\(Liuet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib2)\), andPhi\-4\(Abdinet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib3)\)write short, internally consistent encyclopedic articles about entities or concepts that should not exist in the real world\.
2. 2\.Diversify\.We sample across fictional people, places, artifacts, events, organizations, and technical concepts so that the corpus is not dominated by a single template\.
3. 3\.Deduplicate and filter\.We remove near\-duplicates with MinHash\-LSH at a Jaccard threshold of 0\.85\(Broder,[1997](https://arxiv.org/html/2605.05687#bib.bib24)\), then use an LLM existence filter to discard titles judged likely to correspond to real public entities, events, or concepts\.

Surviving articles are assigned stable document identifiers and form a controlled fabricated corpus that target models should not know from ordinary pretraining\. We then inject this corpus into each target model through continued pretraining, so the attribution task asks whether a method can recover which injected training document supports a later response\.

### 3\.2Target\-Model Exposure and Evaluation Split

For every target LLM in Section[5](https://arxiv.org/html/2605.05687#S5), we start from the public instruction\-tuned checkpoint and continue pretrain it onFakeWikitext with a causal language\-modeling objective\. The corpus contains original articles and constructed variants, but not QA probes, reference answers, or transformed queries\. Thus the model sees the fabricated knowledge as training text without memorizing the evaluation prompts\.

The attribution split is separate from target\-model exposure\. Because the task is to attribute responses to training documents, target LLMs may see the fullFakeWikitext corpus\. We then split document identifiers 80/20 for training and evaluating attribution methods:ScoringModelis trained on responses from training document IDs and evaluated on held\-out document IDs\. Source\-preserving variants are valid positives for the same document ID, while anti\-documents are hard negatives and never valid attribution targets\.

### 3\.3QA Probes and Target Responses

For each retained article, we generate five one\-sentence questions whose answers are short phrases grounded in that article\. After continued pretraining, we query each target LLMMMwith either the clean questionqDq\_\{D\}or a transformed versionJ​\(qD\)J\(q\_\{D\}\)and collect the response

r=M​\(J​\(qD\)\)\.r=M\(J\(q\_\{D\}\)\)\.An attribution method receives the response, optionally the original question, and the full candidate corpus\. It must rank the documents that most likely support the response\. Evaluation uses Recall@kk, counting a prediction as correct if any valid source variant appears in the top\-kk\.

### 3\.4Source Variants and Hard Negatives

Each source document is expanded into three variant families\.

- •Paraphrasespreserve the original facts while changing wording and discourse structure\. They are counted as valid positives during evaluation\.
- •Retro\-generated variantsare written from the QA facts rather than from the original article\. They place answer\-relevant information inside a different surrounding context, reducing simple lexical overlap with the source article\.
- •Anti\-documentspreserve the topic, entity name, style, and much of the wording of the original document, but delete or alter the facts needed to answer the QA probes\. They are hard negatives: a method that only detects topical similarity should rank them highly, while a provenance method should not\.

These variants separate documents that merely resemble the response from documents that actually support it\.

### 3\.5Transformed Query Conditions

We evaluate five query conditions\. The clean condition asks the original QA probe\. The other four transformations stress different failure modes of provenance retrieval\.

- •Obfuscatereplaces many content words with unrelated benign words while preserving the intended question through a mapping\.
- •RolePlaywraps the question inside a persona or scenario\.
- •NoiseInjectionsurrounds the question with unrelated filler text\.
- •Indirectrewrites the query into an indirect or multi\-hop prompt with reduced surface overlap\.

These transformations are controlled stress tests, not a complete jailbreak taxonomy\. They ask whether attribution still works when prompt and response no longer expose clean lexical cues; full templates are in Appendix[K](https://arxiv.org/html/2605.05687#A11)\.

## 4Methods

### 4\.1Problem Setup

Let𝒟=\{Dj\}j=1N\\mathcal\{D\}=\\\{D\_\{j\}\\\}\_\{j=1\}^\{N\}be a candidate corpus that has been injected into a target LLM through continued pretraining, and letPi⊆𝒟P\_\{i\}\\subseteq\\mathcal\{D\}denote the valid sources for exampleii, including the original document and valid variants\. Given a questionqiq\_\{i\}, a transformed queryJ​\(qi\)J\(q\_\{i\}\), and a responserir\_\{i\}generated by the continued\-pretrained target model, an attribution method produces a scoreϕ​\(ri,qi,Dj\)\\phi\(r\_\{i\},q\_\{i\},D\_\{j\}\)for each candidate document\. We evaluate with

Recall​@​k=1\|𝒯\|​∑i∈𝒯𝟏​\{Topk​\(ϕi\)∩Pi≠∅\}\.\\mathrm\{Recall\}@k=\\frac\{1\}\{\|\\mathcal\{T\}\|\}\\sum\_\{i\\in\\mathcal\{T\}\}\\mathbf\{1\}\\left\\\{\\mathrm\{Top\}\_\{k\}\(\\phi\_\{i\}\)\\cap P\_\{i\}\\neq\\emptyset\\right\\\}\.\(1\)All main results report Recall@10 as percentages\.

### 4\.2ScoringModel: Supervised Provenance Scoring

ScoringModelis a supervised pairwise ranker that maps response\-side and document\-side features into a shared embedding space, rather than anNN\-way classifier over fixed document labels\. For an input feature vector𝒙\\boldsymbol\{x\}, a two\-layer MLPfθf\_\{\\theta\}produces a normalized embedding, and response\-document compatibility is temperature\-scaled cosine similarity:

sθ​\(r,D\)=1τ​⟨fθ​\(𝒙r\)∥fθ​\(𝒙r\)∥2,fθ​\(𝒙D\)∥fθ​\(𝒙D\)∥2⟩\.s\_\{\\theta\}\(r,D\)=\\frac\{1\}\{\\tau\}\\left\\langle\\frac\{f\_\{\\theta\}\(\\boldsymbol\{x\}\_\{r\}\)\}\{\\lVert f\_\{\\theta\}\(\\boldsymbol\{x\}\_\{r\}\)\\rVert\_\{2\}\},\\frac\{f\_\{\\theta\}\(\\boldsymbol\{x\}\_\{D\}\)\}\{\\lVert f\_\{\\theta\}\(\\boldsymbol\{x\}\_\{D\}\)\\rVert\_\{2\}\}\\right\\rangle\.\(2\)
We use target\-LLM hidden states and QA\-style text embeddings as input features, selecting the feature variant on validation data per target model\. For each positive pair\(ri,Di\+\)\(r\_\{i\},D\_\{i\}^\{\+\}\), the training batch contains three kinds of negatives\. In\-batch negatives provide cheap contrast against unrelated documents\. Retrieval\-mined hard negatives are topically or semantically close documents retrieved by an embedding model\. Curated anti\-documents are the most targeted negatives because they preserve surface similarity while removing answer support\. Training minimizes an InfoNCE objective\(Oordet al\.,[2018](https://arxiv.org/html/2605.05687#bib.bib22)\):

ℒi=−log⁡exp⁡sθ​\(ri,Di\+\)exp⁡sθ​\(ri,Di\+\)\+∑D∈𝒩iexp⁡sθ​\(ri,D\)\.\\mathcal\{L\}\_\{i\}=\-\\log\\frac\{\\exp s\_\{\\theta\}\(r\_\{i\},D\_\{i\}^\{\+\}\)\}\{\\exp s\_\{\\theta\}\(r\_\{i\},D\_\{i\}^\{\+\}\)\+\\sum\_\{D\\in\\mathcal\{N\}\_\{i\}\}\\exp s\_\{\\theta\}\(r\_\{i\},D\)\}\.\(3\)At inference time, we precompute candidate document features and score each response by dot products in the learned space\. Implementation details are in Appendix[J](https://arxiv.org/html/2605.05687#A10)\.

### 4\.3Activation Steering with Retrieval Fusion

We also evaluate a training\-free activation\-based signal\. Instead of asking only whether a response and document are textually similar, activation steering asks whether a candidate document points the target model’s internal state toward the observed answer\. Exact activation patching would require one intervention per candidate document, soSteerFuseuses a cached\-vector approximation\.

#### Step 1: document activation directions\.

For a chosen layerℓ⋆\\ell^\{\\star\}, each candidate documentDDinduces an activation direction by mean\-pooling the hidden states produced when the target model reads the document:

𝒗D=∑t∈Dat​𝒉t\(ℓ⋆\)∑t∈Dat,\\boldsymbol\{v\}\_\{D\}=\\frac\{\\sum\_\{t\\in D\}a\_\{t\}\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\}\}\{\\sum\_\{t\\in D\}a\_\{t\}\},\(4\)whereata\_\{t\}is the attention mask\. Document directions are normalized and cached before scoring\.

#### Step 2: response\-side proxy and activation score\.

For a responser=y1:mr=y\_\{1:m\}, we use an efficient response\-side proxy given by the sum of LM\-head rows for the generated answer tokens:

𝒈~r=∑i=1m𝑾yi\.\\tilde\{\\boldsymbol\{g\}\}\_\{r\}=\\sum\_\{i=1\}^\{m\}\\boldsymbol\{W\}\_\{y\_\{i\}\}\.\(5\)Each candidate document is scored by cosine similarity between this response\-side proxy and the cached document direction:

sact​\(r,D\)=cos⁡\(𝒈~r,𝒗D\)\.s\_\{\\mathrm\{act\}\}\(r,D\)=\\cos\(\\tilde\{\\boldsymbol\{g\}\}\_\{r\},\\boldsymbol\{v\}\_\{D\}\)\.\(6\)This reduces retrieval to cosine similarity against cached document vectors, avoiding patched forward or backward passes\.

#### Step 3: approximate intervention score\.

The activation score approximates a direct intervention question: would steering the answer\-token hidden states toward𝒗D\\boldsymbol\{v\}\_\{D\}increase the likelihood of the observed answer? If we patch

𝒉~ti\(D\)​\(α\)=\(1−α\)​𝒉ti\(ℓ⋆\)\+α​𝒗D,\\tilde\{\\boldsymbol\{h\}\}^\{\(D\)\}\_\{t\_\{i\}\}\(\\alpha\)=\(1\-\\alpha\)\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\+\\alpha\\boldsymbol\{v\}\_\{D\},\(7\)then a first\-order expansion of the answer log\-probability gain gives

Δexact​\(D\)≈α​∑i=1m𝒈i⊤​\(𝒗D−𝒉ti\(ℓ⋆\)\)∝𝒈⊤​𝒗D,𝒈=∑i=1m𝒈i\.\\Delta\_\{\\mathrm\{exact\}\}\(D\)\\approx\\alpha\\sum\_\{i=1\}^\{m\}\\boldsymbol\{g\}\_\{i\}^\{\\top\}\(\\boldsymbol\{v\}\_\{D\}\-\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\)\\propto\\boldsymbol\{g\}^\{\\top\}\\boldsymbol\{v\}\_\{D\},\\qquad\\boldsymbol\{g\}=\\sum\_\{i=1\}^\{m\}\\boldsymbol\{g\}\_\{i\}\.\(8\)Thus document ranking can be approximated by dot products with cached document directions\. Whenℓ⋆\\ell^\{\\star\}is the final layer before the LM head, the token sensitivity has the form

𝒈i=𝑾yi−𝔼w∼pθ\(⋅∣y<i\)​\[𝑾w\]≈𝑾yi,\\boldsymbol\{g\}\_\{i\}=\\boldsymbol\{W\}\_\{y\_\{i\}\}\-\\mathbb\{E\}\_\{w\\sim p\_\{\\theta\}\(\\cdot\\mid y\_\{<i\}\)\}\[\\boldsymbol\{W\}\_\{w\}\]\\approx\\boldsymbol\{W\}\_\{y\_\{i\}\},\(9\)which motivates the implemented proxy𝒈~r=∑i𝑾yi\\tilde\{\\boldsymbol\{g\}\}\_\{r\}=\\sum\_\{i\}\\boldsymbol\{W\}\_\{y\_\{i\}\}\. Appendix[D](https://arxiv.org/html/2605.05687#A4)gives the full derivation and discusses the approximation\.

#### Step 4: retrieval fusion\.

Text retrieval is a strong prior but can reward resemblance rather than source support\. We therefore fuseSBERT\-QA similarity withsacts\_\{\\mathrm\{act\}\}, using validation\-tuned z\-score fusion or reciprocal\-rank fusion forSteerFuse\. The main comparison givesSteerFusethis inference\-time fusion, whileScoringModelis reported without inference\-time fusion\.

## 5Experimental Setup

#### Target models\.

We evaluate nine open\-weight instruction\-tuned LLMs:TinyLlama\-1\.1B\-Chat\-v1\.0,Llama\-3\.2\-1B\-Instruct,Qwen2\-1\.5B\-Instruct,Llama\-3\.2\-3B\-Instruct,Qwen2\.5\-7B\-Instruct,Llama\-2\-7b\-chat\-hf,Mistral\-7B\-Instruct\-v0\.3,Llama\-3\.1\-8B\-Instruct, andQwen3\-8B\.

#### Target\-model continued pretraining\.

Before collecting responses, each target LLM is continued\-pretrained on theFakeWikitext corpus with a causal language\-modeling objective for 3 epochs at learning rate2×10−52\\times 10^\{\-5\}\. The training text consists of fabricated article text and document variants, not the QA probes or transformed queries\. This exposure step is what makes the downstream task training\-data attribution: the model responses are elicited from LLMs that have encountered the candidate documents as training text\. The 80/20 document\-ID split used later is for training and evaluating attribution methods, not for withholding documents from the target LLM\.

#### Baselines\.

We compare against eleven retrieval baselines\. MinHash estimates lexical resemblance by hashing token shingles and comparing compact signatures, giving a fast approximation to Jaccard overlap\(Broder,[1997](https://arxiv.org/html/2605.05687#bib.bib24)\); we run it over the answer alone and over the question\-answer pair\. The finetuned dense LLM embedding baseline uses an in\-domain embedding head and ranks documents by cosine similarity\. The remaining dense retrieval baselines useSBERT\(Reimers and Gurevych,[2019](https://arxiv.org/html/2605.05687#bib.bib25)\), BGE\(Xiaoet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib27)\), and Contriever\(Izacardet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib26)\): MiniLM, MPNet, BGE\-base, and Contriever are evaluated with answer\-only queries and with QA queries\. Answer\-only retrieval tests whether the generated response itself carries enough source evidence; QA retrieval tests a stronger setting where the retriever also sees the original question\. In aggregate tables, “best baseline” means the strongest of these eleven baselines for that model and query condition\.

#### Metrics and selection\.

The main paper reports Recall@10, with full per\-model R@1 and R@5 tables in Appendix[I](https://arxiv.org/html/2605.05687#A9)\. ForScoringModel, feature mode and checkpoint selection use held\-out Clean validation data per target model, and the selected no\-fusion scorer is evaluated unchanged on transformed queries\. ForSteerFuse, validation selects the retrieval\-fusion setting per target model and query condition\. Further implementation details are in Appendix[J](https://arxiv.org/html/2605.05687#A10)\.

## 6Main Results

### 6\.1Aggregate Recall@10 Results

Table[2](https://arxiv.org/html/2605.05687#S6.T2)summarizes Recall@10 averaged across all nine target models\.ScoringModelimproves over the strongest baseline in every query condition\. The gains are largest forRolePlayandNoiseInjection, where surface retrieval remains plausible but unreliable, and smallest forObfuscateandIndirect, which represent two different hard cases: shallow lexical substitution can sometimes favor response\-only semantic retrieval, while indirect prompting reduces all methods to low absolute recall\.

Table 2:Recall@10 averaged across nine target LLMs\. The best baseline is selected from eleven retrieval baselines separately for each model and query condition\. Best per row is inbold, second\-best isunderlined\.Query conditionBest baselineSteerFuseScoringModelΔ\\Deltavs\. baselineΔ\\Deltavs\.SteerFuseClean55\.769\.277\.2\+21\.5\+8\.0Obfuscate39\.130\.544\.4\+5\.3\+13\.9RolePlay42\.950\.162\.5\+19\.6\+12\.4NoiseInjection36\.747\.059\.2\+22\.5\+12\.2Indirect12\.014\.517\.7\+5\.7\+3\.2Average37\.342\.352\.2\+14\.9\+9\.9

Across all9×59\\times 5cells,ScoringModelbeats the best baseline in 41/45 cells and beatsSteerFusein 40/45 cells\. Mean transformed\-query Recall@10 improves by 13\.2 points over the best baseline\. Additional aggregate summaries, including full win counts and transformed\-query averages by target model, are reported in Appendix[E](https://arxiv.org/html/2605.05687#A5)\.

#### Seed robustness\.

We re\-train each Clean\-validation\-selectedScoringModelconfiguration with three seeds\. The average per\-cell standard deviation is 0\.94 Recall@10 points, with variance largest onObfuscate; full mean±\\pmstd results are in Appendix[G](https://arxiv.org/html/2605.05687#A7)\.

### 6\.2Per\-Method Results on Large Models

Table[3](https://arxiv.org/html/2605.05687#S6.T3)reports Recall@10 forQwen3\-8BandLlama\-3\.1\-8B, with methods as rows and query conditions as columns\. These two models are useful stress cases because the best retrieval baseline varies across conditions: QA\-style dense retrieval is strongest on clean prompts, while answer\-only BGE orSBERTcan become competitive underObfuscate\. Even against this condition\-specific best\-of\-baselines comparison,ScoringModelwins all ten large\-model columns, whileSteerFuseis the strongest non\-supervised method in most non\-obfuscation columns\. The remaining seven target\-model tables are reported in Appendix[H](https://arxiv.org/html/2605.05687#A8), and the aggregate trends across all models are summarized above\.

Table 3:Large\-model Recall@10 table\. Rows include all baselines and our attribution methods for two instruction\-tuned target models\. Best per column is inbold, second\-best isunderlined\.MethodQwen3\-8BLlama\-3\.1\-8BCleanObfuscateRolePlayNoiseInjectionIndirectCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.10\.10\.10\.10\.10\.00\.10\.10\.10\.0MinHash \(QA\)0\.10\.10\.10\.10\.10\.00\.10\.10\.10\.0Finetuned EmbedSim17\.49\.416\.116\.42\.316\.915\.923\.623\.12\.2SBERT\-MiniLM \(answer\)28\.440\.623\.523\.99\.33\.735\.710\.612\.93\.5SBERT\-MPNet \(answer\)25\.536\.320\.320\.18\.03\.632\.49\.711\.63\.0SBERT\-MiniLM \(QA\)39\.712\.429\.125\.09\.128\.99\.519\.918\.54\.0SBERT\-MPNet \(QA\)38\.13\.727\.514\.87\.929\.13\.921\.69\.83\.5BGE\-base \(answer\)30\.843\.026\.025\.711\.44\.337\.311\.713\.13\.8BGE\-base \(QA\)43\.116\.640\.323\.611\.333\.313\.430\.613\.85\.9Contriever \(answer\)32\.442\.826\.226\.610\.63\.627\.911\.312\.52\.7Contriever \(QA\)30\.122\.822\.315\.03\.55\.110\.89\.37\.10\.8SteerFuse76\.430\.160\.551\.120\.157\.837\.743\.538\.19\.6ScoringModel78\.1±\\pm0\.553\.9±\\pm1\.563\.8±\\pm0\.762\.1±\\pm0\.321\.6±\\pm0\.876\.7±\\pm0\.259\.5±\\pm2\.163\.4±\\pm0\.663\.5±\\pm0\.418\.2±\\pm0\.4

The table shows that clean retrieval can be deceptively strong, but transformed prompts expose baseline instability: methods that work under clean prompting may fall sharply under indirect prompting or noise injection, and obfuscation can flip the strongest baseline from QA retrieval to answer\-only retrieval\. By contrast,ScoringModelremains high across all five query conditions for both large models, including settings where the best baseline is below 15 Recall@10\. The corresponding Recall@1 and Recall@5 tables in Appendix[I](https://arxiv.org/html/2605.05687#A9)show that this advantage is not only a top\-10 effect:ScoringModelalso wins every stricter\-cutoff column forQwen3\-8BandLlama\-3\.1\-8B\.

### 6\.3Scaling on Transformed Prompts

Figure[2](https://arxiv.org/html/2605.05687#S6.F2)visualizes the per\-model improvement ofScoringModelover the best baseline on the four transformed query conditions\. The largest gains appear for the larger target models:Llama\-3\.1\-8Bimproves by \+26\.9 Recall@10 on average over transformed conditions,Qwen3\-8Bby \+20\.0,Llama\-2\-7Bby \+19\.0, andMistral\-7Bby \+17\.0\. Smaller models still benefit, but with narrower margins\. This pattern suggests that larger LLM hidden states encode more recoverable provenance information, making attribution easier for a trained scorer even as the underlying model is larger\.

![Refer to caption](https://arxiv.org/html/2605.05687v1/x1.png)\(a\)Obfuscate
![Refer to caption](https://arxiv.org/html/2605.05687v1/x2.png)\(b\)RolePlay
![Refer to caption](https://arxiv.org/html/2605.05687v1/x3.png)\(c\)NoiseInjection
![Refer to caption](https://arxiv.org/html/2605.05687v1/x4.png)\(d\)Indirect

Figure 2:Improvement ofScoringModelover the best baseline on transformed query conditions\.
### 6\.4Ablation Study

Appendix[F](https://arxiv.org/html/2605.05687#A6)provides full ablations\. The main takeaway is that no\-fusionScoringModelis already a strong standalone scorer: it uses a single learned compatibility score, requires no test\-time mixing weight, and wins 41/45 model\-by\-condition cells\. We evaluateScoringModel–SBERTfusion only as an ablation\.

SteerFuseshows a different pattern: its mean gain over the stronger of activation\-only andSBERT\-only rankings is only \+2\.0 Recall@10, and about 96% of the fusion uplift comes fromSBERT\. The main exception isObfuscate, where the activation component contributes about \+7\.5 points, suggesting that internal\-state evidence helps most when lexical cues are actively distorted\.

Other ablations show that the conclusions are not driven by a single representation or combiner: z\-score and reciprocal\-rank fusion give similar trends, and feature\-mode diagnostics show complementary hidden\-state and QA\-style signals\.

## 7Conclusion

We presentedDataDignity, a benchmark and method suite for robust pinpoint provenance in LLM responses\.FakeWikishows that clean retrieval can overstate attribution reliability: under obfuscation, role\-play, noise, and indirect prompts, methods must distinguish true answer support from topical resemblance\. This distinction matters for audit workflows because a high\-similarity document is not necessarily the document that supplied the answer\-critical fact\. In this setting,ScoringModelimproves mean Recall@10 from 37\.3 to 52\.2 and wins 41/45 model\-by\-condition cells, with the same trend holding at stricter Recall@1 and Recall@5 cutoffs\. The gains are largest on transformed prompts and larger target models, suggesting that supervised attribution can recover provenance signals that generic retrievers miss when surface cues are unstable\.SteerFuseshows that activation\-space evidence is useful when stabilized by text retrieval, but is less uniform than the supervised scorer; this points to internal\-state evidence as a promising complement rather than a complete replacement for retrieval\.

Overall, robust training data attribution should be evaluated with hard negatives and prompt transformations that expose failures hidden by clean semantic retrieval\. We view pinpoint provenance systems as audit aids that return inspectable candidate sources, and future work should push beyond document\-level recall toward calibrated confidence and finer\-grained evidence localization\.

## References

- M\. Abdin, J\. Aneja, H\. Behl, S\. Bubeck, R\. Eldan, S\. Gunasekar, M\. Harrison, R\. J\. Hewett, M\. Javaheripi, P\. Kauffmann,et al\.\(2024\)Phi\-4 technical report\.arXiv preprint arXiv:2412\.08905\.Cited by:[item 1](https://arxiv.org/html/2605.05687#S3.I1.i1.p1.1)\.
- E\. Akyürek, T\. Bolukbasi, F\. Liu, B\. Xiong, I\. Tenney, J\. Andreas, and K\. Guu \(2022\)Towards tracing factual knowledge in language models back to the training data\.arXiv preprint arXiv:2205\.11482\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- E\. Barshan, M\. Brunet, and G\. K\. Dziugaite \(2020\)Relatif: identifying explanatory training samples via relative influence\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1899–1909\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Z\. Broder \(1997\)On the resemblance and containment of documents\.InProceedings of Compression and Complexity of Sequences,pp\. 21–29\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p3.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1),[item 3](https://arxiv.org/html/2605.05687#S3.I1.i3.p1.1),[§5](https://arxiv.org/html/2605.05687#S5.SS0.SSS0.Px3.p1.1)\.
- C\. Burns, H\. Ye, D\. Klein, and J\. Steinhardt \(2023\)Discovering latent knowledge in language models without supervision\.InInternational Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.
- M\. Fotouhi, M\. T\. Bahadori, O\. Feyisetan, P\. Arabshahi, and D\. Heckerman \(2024\)Fast training dataset attribution via in\-context learning\.arXiv preprint arXiv:2408\.11852\.Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- X\. Han and Y\. Tsvetkov \(2021\)Influence tuning: demoting spurious correlations via instance attribution and instance\-driven updates\.arXiv preprint arXiv:2110\.03212\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- A\. Hurst, A\. Lerer, A\. P\. Goucher, A\. Perelman, A\. Ramesh, A\. Clark, A\. Ostrow, A\. Welihinda, A\. Hayes, A\. Radford,et al\.\(2024\)Gpt\-4o system card\.arXiv preprint arXiv:2410\.21276\.Cited by:[item 1](https://arxiv.org/html/2605.05687#S3.I1.i1.p1.1)\.
- G\. Izacard, M\. Caron, L\. Hosseini, S\. Riedel, P\. Bojanowski, A\. Joulin, and E\. Grave \(2022\)Unsupervised dense information retrieval with contrastive learning\.Transactions on Machine Learning Research\.Cited by:[Appendix C](https://arxiv.org/html/2605.05687#A3.p1.1),[§1](https://arxiv.org/html/2605.05687#S1.p3.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2605.05687#S5.SS0.SSS0.Px3.p1.1)\.
- M\. Khalifa, D\. Wadden, E\. Strubell, H\. Lee, L\. Wang, I\. Beltagy, and H\. Peng \(2024\)Source\-aware training enables knowledge attribution in language models\.arXiv preprint arXiv:2404\.01019\.Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- Y\. Kwon, E\. Wu, K\. Wu, and J\. Zou \(2023\)Datainf: efficiently estimating data influence in lora\-tuned llms and diffusion models\.arXiv preprint arXiv:2310\.00902\.Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- K\. Li, O\. Patel, F\. Viégas, H\. Pfister, and M\. Wattenberg \(2023\)Inference\-time intervention: eliciting truthful answers from a language model\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p6.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.
- X\. Li, M\. Gao, Z\. Zhang, C\. Yue, and H\. Hu \(2026\)Selection of llm fine\-tuning data based on orthogonal rules\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.40,pp\. 31760–31768\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1)\.
- A\. Liu, B\. Feng, B\. Xue, B\. Wang, B\. Wu, C\. Lu, C\. Zhao, C\. Deng, C\. Zhang, C\. Ruan,et al\.\(2024\)Deepseek\-v3 technical report\.arXiv preprint arXiv:2412\.19437\.Cited by:[item 1](https://arxiv.org/html/2605.05687#S3.I1.i1.p1.1)\.
- A\. v\. d\. Oord, Y\. Li, and O\. Vinyals \(2018\)Representation learning with contrastive predictive coding\.arXiv preprint arXiv:1807\.03748\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p5.1),[§4\.2](https://arxiv.org/html/2605.05687#S4.SS2.p2.1)\.
- N\. Panickssery, N\. Gabrieli, J\. Schulz, M\. Tong, E\. Hubinger, and A\. M\. Turner \(2023\)Steering llama 2 via contrastive activation addition\.arXiv preprint arXiv:2312\.06681\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p6.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.
- S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. Madry \(2023\)TRAK: attributing model behavior at scale\.InInternational Conference on Machine Learning,pp\. 27074–27113\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- G\. Pruthi, F\. Liu, S\. Kale, and M\. Sundararajan \(2020\)Estimating training data influence by tracing gradient descent\.Advances in Neural Information Processing Systems33,pp\. 19920–19930\.Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- N\. F\. Rajani, B\. McCann, C\. Xiong, and R\. Socher \(2019\)Explain yourself\! leveraging language models for commonsense reasoning\.arXiv preprint arXiv:1906\.02361\.Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- N\. Reimers and I\. Gurevych \(2019\)Sentence\-bert: sentence embeddings using siamese bert\-networks\.InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing,pp\. 3982–3992\.Cited by:[Appendix C](https://arxiv.org/html/2605.05687#A3.p1.1),[Appendix F](https://arxiv.org/html/2605.05687#A6.SS0.SSS0.Px4.p1.1),[§1](https://arxiv.org/html/2605.05687#S1.p3.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2605.05687#S5.SS0.SSS0.Px3.p1.1)\.
- N\. Subramani, N\. Suresh, and M\. E\. Peters \(2022\)Extracting latent steering vectors from pretrained language models\.arXiv preprint arXiv:2205\.05124\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p6.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.
- A\. M\. Turner, L\. Thiergart, G\. Leech, D\. Udell, J\. J\. Vazquez, U\. Mini, and M\. MacDiarmid \(2023\)Steering language models with activation engineering\.arXiv preprint arXiv:2308\.10248\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p6.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.
- C\. Wang, X\. Lu, S\. Ng, and B\. K\. H\. Low \(2024\)TRACE: transformer\-based attribution using contrastive embeddings in llms\.External Links:2407\.04981,[Link](https://arxiv.org/abs/2407.04981)Cited by:[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1)\.
- S\. Xiao, Z\. Liu, P\. Zhang, N\. Muennighoff, D\. Lian, and J\. Nie \(2024\)C\-pack: packed resources for general chinese embeddings\.InProceedings of the 47th international ACM SIGIR conference on research and development in information retrieval,pp\. 641–649\.Cited by:[Appendix C](https://arxiv.org/html/2605.05687#A3.p1.1),[§1](https://arxiv.org/html/2605.05687#S1.p3.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2605.05687#S5.SS0.SSS0.Px3.p1.1)\.
- Z\. Zhang, F\. Wang, X\. Li, Z\. Wu, X\. Tang, H\. Liu, Q\. He, W\. Yin, and S\. Wang \(2024\)Catastrophic failure of llm unlearning via quantization\.arXiv preprint arXiv:2410\.16454\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p1.1)\.
- A\. Zou, L\. Phan, S\. Chen, J\. Campbell, P\. Guo, R\. Ren, A\. Pan, X\. Yin, M\. Mazeika,et al\.\(2023\)Representation engineering: a top\-down approach to ai transparency\.arXiv preprint arXiv:2310\.01405\.Cited by:[§1](https://arxiv.org/html/2605.05687#S1.p6.1),[§2](https://arxiv.org/html/2605.05687#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix ALimitations and Future Work

#### Limitations\.

Our study provides a controlled and fine\-grained evaluation of pinpoint provenance, with ground\-truth source documents, hard negatives, source\-preserving variants, and transformed query conditions that stress different attribution failure modes\. However, several aspects could be further improved\. First, our transformed query conditions are controlled stress tests rather than a full taxonomy of user prompting behavior\. They are designed to isolate different shortcut pressures, not to exhaust every possible way a prompt can alter the surface form of a response\. Second, our main metric is Recall@10, which is appropriate for audit workflows where a human inspector reviews a short candidate list, but it does not capture all downstream notions of attribution usefulness, such as calibrated confidence, explanation quality, or evidence localization within a document\. Finally, our evaluation focuses on document\-level attribution\. This is useful for source inspection, but it does not identify the exact sentence or span that supports a response\.

#### Future work\.

Future work can address these limitations in three directions\. First, the transformed\-query suite can be expanded to cover a broader range of prompt styles, including more natural user rewrites, multi\-turn interactions, and compositional transformations that combine obfuscation, role\-play, noise, and indirect prompting\. Second, evaluation can move beyond Recall@10 by adding calibration and ranking\-quality metrics, so that provenance systems report not only whether a true source appears in the top candidates, but also how reliable the returned evidence is\. Third, future benchmarks can extend document\-level attribution to finer\-grained evidence localization, such as sentence\-level or span\-level provenance, which would make the returned results easier for human auditors to verify\.

## Appendix BBroader Impacts

This work is intended to support more reliable auditing of language\-model outputs by helping identify candidate source documents that may support a generated response\. Potential positive impacts include improved dataset curation, copyright and provenance analysis, misinformation forensics, and safety debugging\. At the same time, provenance scores should be interpreted carefully: a high\-ranked document is evidence of source support within the candidate corpus, not a definitive causal claim about model training or generation\. Misinterpreting attribution results could lead to overconfident conclusions about whether a particular document caused an output\. We therefore frameDataDignityas an audit aid that returns inspectable evidence for human review, rather than as an automated system for assigning legal or causal responsibility\.

## Appendix CAsset Licenses

We use publicly available models and software tools, including the open\-weight target LLMs listed in Section[5](https://arxiv.org/html/2605.05687#S5), Sentence\-BERT embedding models\[Reimers and Gurevych,[2019](https://arxiv.org/html/2605.05687#bib.bib25)\], BGE embeddings\[Xiaoet al\.,[2024](https://arxiv.org/html/2605.05687#bib.bib27)\], Contriever embeddings\[Izacardet al\.,[2022](https://arxiv.org/html/2605.05687#bib.bib26)\], PyTorch, CUDA, and standard Python libraries\. We cite the corresponding model families, methods, and software tools where appropriate, and use these assets for research evaluation under their respective licenses and terms of use\. The newly introducedFakeWikibenchmark consists of fabricated documents generated for this study, and the anonymized release includes the benchmark data, prompts, and evaluation code\.

## Appendix DActivation\-Steering Approximation Details

This appendix expands the intervention view ofSteerFusefrom Section[4\.3](https://arxiv.org/html/2605.05687#S4.SS3)\. The method starts from an exact but expensive activation\-patching score, then uses a sequence of approximations: a first\-order Taylor approximation that turns patched forward passes into dot products, and an LM\-head approximation that avoids a backward pass for the sensitivity vector by replacing the gradient direction with answer\-token LM\-head rows\.

### D\.1Exact Activation\-Patching Score

Fix a target responsey1:my\_\{1:m\}and a candidate document direction𝒗D\\boldsymbol\{v\}\_\{D\}\. At the selected layerℓ⋆\\ell^\{\\star\}, let𝒉ti\(ℓ⋆\)\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}be the hidden state used to predict answer tokenyiy\_\{i\}\. The exact intervention asks whether mixing this candidate direction into the answer\-token hidden states increases the model probability of the observed response:

𝒉~ti\(D\)​\(α\)=\(1−α\)​𝒉ti\(ℓ⋆\)\+α​𝒗D,α∈\(0,1\]\.\\tilde\{\\boldsymbol\{h\}\}^\{\(D\)\}\_\{t\_\{i\}\}\(\\alpha\)=\(1\-\\alpha\)\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\+\\alpha\\boldsymbol\{v\}\_\{D\},\\qquad\\alpha\\in\(0,1\]\.\(10\)The corresponding exact score is

Δexact​\(D\)=∑i=1m\[log⁡pθ​\(yi∣y<i;𝒉~ti\(D\)​\(α\)\)−log⁡pθ​\(yi∣y<i;𝒉ti\(ℓ⋆\)\)\]\.\\Delta\_\{\\mathrm\{exact\}\}\(D\)=\\sum\_\{i=1\}^\{m\}\\left\[\\log p\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\};\\tilde\{\\boldsymbol\{h\}\}^\{\(D\)\}\_\{t\_\{i\}\}\(\\alpha\)\)\-\\log p\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\};\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\)\\right\]\.\(11\)A large positive value means that injecting documentDDmakes the target model more confident in the fixed response\. This is the most direct causal\-style score, but evaluating it forNNdocuments requires one baseline forward pass plusNNpatched forward passes\.

### D\.2First\-Order Taylor Approximation

To avoid a patched forward pass for every candidate document, define the scalar objective

f​\(\{𝒉ti\}i=1m\)=∑i=1mlog⁡pθ​\(yi∣y<i\),f\(\\\{\\boldsymbol\{h\}\_\{t\_\{i\}\}\\\}\_\{i=1\}^\{m\}\)=\\sum\_\{i=1\}^\{m\}\\log p\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\}\),\(12\)the log\-probability of the whole fixed answer under the unpatched hidden states\. A first\-order Taylor expansion around the original hidden states gives

f​\(𝒉~\(D\)\)−f​\(𝒉\)\\displaystyle f\(\\tilde\{\\boldsymbol\{h\}\}^\{\(D\)\}\)\-f\(\\boldsymbol\{h\}\)=α​∑i=1m∇𝒉ti\(ℓ⋆\)f​\(𝒉\)⊤⏟𝒈i⊤​\(𝒗D−𝒉ti\(ℓ⋆\)\)\+O​\(α2\)\\displaystyle=\\alpha\\sum\_\{i=1\}^\{m\}\\underbrace\{\\nabla\_\{\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\}f\(\\boldsymbol\{h\}\)^\{\\top\}\}\_\{\\boldsymbol\{g\}\_\{i\}^\{\\top\}\}\(\\boldsymbol\{v\}\_\{D\}\-\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\)\+O\(\\alpha^\{2\}\)\(13\)=α​\(∑i=1m𝒈i\)⊤​𝒗D−α​∑i=1m𝒈i⊤​𝒉ti\(ℓ⋆\)\+O​\(α2\)\.\\displaystyle=\\alpha\\left\(\\sum\_\{i=1\}^\{m\}\\boldsymbol\{g\}\_\{i\}\\right\)^\{\\top\}\\boldsymbol\{v\}\_\{D\}\-\\alpha\\sum\_\{i=1\}^\{m\}\\boldsymbol\{g\}\_\{i\}^\{\\top\}\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\+O\(\\alpha^\{2\}\)\.\(14\)The second term does not depend onDD, so it does not affect the ranking of candidate documents\. If

𝒈=∑i=1m𝒈i=∑i=1m∂∂𝒉ti\(ℓ⋆\)​log⁡pθ​\(yi∣y<i\),\\boldsymbol\{g\}=\\sum\_\{i=1\}^\{m\}\\boldsymbol\{g\}\_\{i\}=\\sum\_\{i=1\}^\{m\}\\frac\{\\partial\}\{\\partial\\boldsymbol\{h\}^\{\(\\ell^\{\\star\}\)\}\_\{t\_\{i\}\}\}\\log p\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\}\),\(15\)then the document\-dependent part of the first\-order score is

Δ​log⁡p^​\(D\)∝𝒈⊤​𝒗D\.\\widehat\{\\Delta\\log p\}\(D\)\\propto\\boldsymbol\{g\}^\{\\top\}\\boldsymbol\{v\}\_\{D\}\.\(16\)This approximation turns document ranking into a matrix\-vector product once all document vectors have been cached\. It is accurate when the mixing weight is small enough that higher\-order terms inα\\alphado not dominate the ranking\.

### D\.3Approximating Sensitivity Without Backpropagation

Computing𝒈\\boldsymbol\{g\}exactly requires one backward pass for each response\. This is much cheaper thanNNpatched forward passes, but it can still be expensive when many responses must be attributed\. When the patched layer is the final hidden layer before the language\-model head, the per\-token sensitivity can be written in closed form\. Let𝑾∈ℝ\|V\|×d\\boldsymbol\{W\}\\in\\mathbb\{R\}^\{\|V\|\\times d\}be the LM\-head matrix and let𝑾w\\boldsymbol\{W\}\_\{w\}denote the row associated with vocabulary tokenww\. For the hidden state𝒉\\boldsymbol\{h\}at an answer position, the logits are

zw=𝒉⊤​𝑾w,pθ​\(w∣ctx\)=exp⁡\(𝒉⊤​𝑾w\)∑u∈Vexp⁡\(𝒉⊤​𝑾u\)\.z\_\{w\}=\\boldsymbol\{h\}^\{\\top\}\\boldsymbol\{W\}\_\{w\},\\qquad p\_\{\\theta\}\(w\\mid\\mathrm\{ctx\}\)=\\frac\{\\exp\(\\boldsymbol\{h\}^\{\\top\}\\boldsymbol\{W\}\_\{w\}\)\}\{\\sum\_\{u\\in V\}\\exp\(\\boldsymbol\{h\}^\{\\top\}\\boldsymbol\{W\}\_\{u\}\)\}\.\(17\)Therefore, for answer tokenyiy\_\{i\},

𝒈i\\displaystyle\\boldsymbol\{g\}\_\{i\}=∂∂𝒉​log⁡pθ​\(yi∣y<i\)\\displaystyle=\\frac\{\\partial\}\{\\partial\\boldsymbol\{h\}\}\\log p\_\{\\theta\}\(y\_\{i\}\\mid y\_\{<i\}\)\(18\)=𝑾yi−∑w∈Vpθ​\(w∣y<i\)​𝑾w\\displaystyle=\\boldsymbol\{W\}\_\{y\_\{i\}\}\-\\sum\_\{w\\in V\}p\_\{\\theta\}\(w\\mid y\_\{<i\}\)\\boldsymbol\{W\}\_\{w\}\(19\)=𝑾yi−𝔼w∼pθ\(⋅∣y<i\)​\[𝑾w\]\.\\displaystyle=\\boldsymbol\{W\}\_\{y\_\{i\}\}\-\\mathbb\{E\}\_\{w\\sim p\_\{\\theta\}\(\\cdot\\mid y\_\{<i\}\)\}\[\\boldsymbol\{W\}\_\{w\}\]\.\(20\)This formula shows that the sensitivity direction points toward the LM\-head row of the observed answer token and away from the probability\-weighted average LM\-head direction\. Computing the expectation exactly requires multiplying the full vocabulary probability vector by𝑾\\boldsymbol\{W\}, which can be costly\. A further approximation drops this expectation term and uses

𝒈~i∝𝑾yi,𝒈~=∑i=1m𝒈~i\.\\tilde\{\\boldsymbol\{g\}\}\_\{i\}\\propto\\boldsymbol\{W\}\_\{y\_\{i\}\},\\qquad\\tilde\{\\boldsymbol\{g\}\}=\\sum\_\{i=1\}^\{m\}\\tilde\{\\boldsymbol\{g\}\}\_\{i\}\.\(21\)This is reasonable when the answer token direction dominates the local gradient, but it is not an exact identity: the omitted expectation term can matter when the next\-token distribution is diffuse or when competing tokens have substantial probability mass\. Under this approximation, the Taylor score becomes

Δ​log⁡p^​\(D\)∝𝒈~⊤​𝒗D,\\widehat\{\\Delta\\log p\}\(D\)\\propto\\tilde\{\\boldsymbol\{g\}\}^\{\\top\}\\boldsymbol\{v\}\_\{D\},\(22\)so ranking still reduces to a dot product, but the response\-side vector can be obtained by LM\-head row lookups rather than backpropagation\.

### D\.4Implemented LM\-Head Row Proxy

The gradient score in Eq\. \([16](https://arxiv.org/html/2605.05687#A4.E16)\) removes the need for one patched forward pass per document, and the LM\-head approximation in Eq\. \([21](https://arxiv.org/html/2605.05687#A4.E21)\) removes the backward pass\. The implementedSteerFusemethod represents the answer by the normalized sum of its generated\-token LM\-head rows and ranks documents by cosine similarity against cached document directions:

sact​\(r,D\)=cos⁡\(∑i=1m𝑾yi,𝒗D\)\.s\_\{\\mathrm\{act\}\}\(r,D\)=\\cos\\left\(\\sum\_\{i=1\}^\{m\}\\boldsymbol\{W\}\_\{y\_\{i\}\},\\boldsymbol\{v\}\_\{D\}\\right\)\.\(23\)In experiments, we use a finer\-grained version of the same score: long documents are split into sentence\-respecting chunks, each chunk receives a score, and the document score is the maximum over its chunks\. This proxy is cheaper than exact patching and avoids backpropagation, but it is still noisy because LM\-head rows are only an approximation to the full gradient sensitivity in Eq\. \([20](https://arxiv.org/html/2605.05687#A4.E20)\)\. For this reason, the mainSteerFusemethod fuses the activation score withSBERT\-QA retrieval rather than using the activation score alone\.

## Appendix EAdditional Aggregate Results

Table[4](https://arxiv.org/html/2605.05687#A5.T4)reports head\-to\-head win counts over all 45 model\-by\-query\-condition cells\. The pattern is consistent with the main aggregate results:ScoringModelis not only better on average, but also wins across most individual settings\. It outperforms the strongest retrieval baseline in 41/45 cells and outperformsSteerFusein 40/45 cells\. At the same time,SteerFusebeats the best baseline in 32/45 cells, suggesting that activation\-based evidence can be useful when stabilized by retrieval fusion\.

Table 4:Head\-to\-head win counts over all 45 model\-by\-query\-condition cells\.ComparisonWins / 45PercentScoringModel\>\>best baseline4191%ScoringModel\>\>SteerFuse4089%SteerFuse\>\>best baseline3271%Table[5](https://arxiv.org/html/2605.05687#A5.T5)averages Recall@10 across the four transformed query conditions, excluding clean prompts\. This table highlights where the robustness gains are largest\. The biggest improvements occur for Llama\-3\.1\-8B and Qwen3\-8B, whereScoringModelimproves over the best baseline by \+26\.9 and \+20\.0 points respectively\. The next strongest gains appear for Llama\-2\-7B and Mistral\-7B\. Smaller models still benefit, but with narrower margins\. This supports the main\-paper observation that larger target models tend to expose more recoverable provenance signal for a supervised scorer, especially when surface\-form cues are disrupted\.

Table 5:Average transformed\-query Recall@10 by target model\. Clean prompts are excluded\.ModelBest baselineSteerFuseScoringModelΔ\\Deltavs\. baselineLlama\-3\.1\-8B24\.232\.251\.1\+26\.9Qwen3\-8B30\.340\.450\.4\+20\.0Llama\-2\-7B23\.037\.342\.0\+19\.0Mistral\-7B24\.640\.241\.5\+17\.0Qwen2\-1\.5B32\.127\.643\.0\+10\.9Llama\-3\.2\-3B39\.934\.649\.4\+9\.5Qwen2\.5\-7B37\.929\.446\.6\+8\.8Llama\-3\.2\-1B44\.440\.049\.2\+4\.8TinyLlama\-1\.1B37\.938\.240\.3\+2\.4Overall, these aggregate results show that the gains are not driven by a small number of favorable cases\.ScoringModelis consistently strong across both head\-to\-head comparisons and transformed\-query averages, whileSteerFuseprovides a useful but less reliable intermediate signal\.

## Appendix FAblation Details

#### Inference\-time fusion\.

AddingSBERTfusion toScoringModelimproves mean Recall@10 by \+5\.5 points over the stronger of no\-fusionScoringModelandSBERTalone\. The gains are distributed across query conditions: Clean \+2\.7, Obfuscate \+5\.5, RolePlay \+9\.5, NoiseInjection \+8\.6, and Indirect \+1\.0\. We keep this variant as an ablation because the no\-fusion method is simpler, requires no test\-time fusion weight, and already wins the main comparison\.

#### SteerFusedecomposition\.

ForSteerFuse, the mean fusion gain over the stronger of activation\-only andSBERT\-only rankings is \+2\.0 Recall@10, and approximately 96% of the fusion uplift comes from theSBERTcomponent\. The main exception isObfuscate, where the activation component contributes about \+7\.5 points\. Thus activation steering is informative, but not yet a strong standalone provenance ranker\.

#### Combiner choice\.

Z\-score fusion wins the majority ofScoringModel\-fusion cells andSteerFusecells, so the main paper uses z\-score when reportingSteerFuse\. Reciprocal\-rank fusion is reported as a sensitivity check and is most useful underIndirect, where score distributions shift strongly\.

#### Feature mode\.

We consider three input\-feature variants for theScoringModelscorer\. Thellmfeature is a mean\-pooled hidden state from a selected layer of the target LLM, computed for the response side and for the candidate document side; it is intended to capture model\-internal content representations that may survive obfuscation or paraphrase\. Theqafeature is aSBERT\-MiniLM embedding\[Reimers and Gurevych,[2019](https://arxiv.org/html/2605.05687#bib.bib25)\]of a QA\-style textual representation, which is strong when the question–answer pair retains semantic alignment with the source\. Theconcatfeature concatenates the two, allowing early fusion at the MLP input\. We emphasize that all three variants are evaluated atλ=0\\lambda=0: the MLP produces a single learned score per candidate, and there is no test\-time score\-level mixing with a separateSBERTranker\. Main results use the feature mode selected on Clean validation for each target model\. The condition\-wise feature results in Figure[3](https://arxiv.org/html/2605.05687#A6.F3)are diagnostic rather than a transformed\-condition tuning procedure\.

#### Grid size and embedding backbone\.

A wider negative\-mining grid is sufficient for theconcatsetting; larger grids helpllmfeatures, especially onObfuscate\. MiniLM is stronger than MPNet on this benchmark across the relevant sweeps\.

The figures below visualize the same ablation questions: which input representation is most useful forScoringModel, whether z\-score fusion or reciprocal\-rank fusion is the more important design choice, and how much the validation\-selected fusion models rely on the learned attribution signal versusSBERT\. All plots report Recall@10 and average over the nine target models unless otherwise stated\.

#### Feature representation\.

Figure[3](https://arxiv.org/html/2605.05687#A6.F3)compares three feature modes forScoringModel\. Theqafeature is strongest on clean prompts in the no\-fusion setting, reaching roughly the high\-70s Recall@10, because the question\-answer text remains semantically aligned with the source document\. The same text channel is also the best no\-fusion feature onIndirect, where all methods are low but QA semantics still preserve the most usable retrieval signal\. By contrast, thellmfeature is clearly strongest onObfuscate: its average Recall@10 is roughly twice that ofconcatand more than three times that ofqa\. This supports the interpretation that target\-model hidden states carry provenance information that is less tied to surface word choice\. ForRolePlayandNoiseInjection, the differences are smaller:concatis best on role\-play without fusion, whilellmandconcatare close on noise injection\. This is consistent with these transformations preserving more of the original semantics while adding stylistic or irrelevant context\.

The fusion ablation in Figure[3](https://arxiv.org/html/2605.05687#A6.F3)changes the picture in an informative way\. OnceSBERTscore\-level fusion is allowed,llmbecomes the best feature mode forObfuscate,RolePlay, andNoiseInjection\. The text\-onlyqamode remains competitive on clean prompts and remains best onIndirect, but it no longer dominates the transformed settings\. Thus the best fusion behavior does not come from replacing hidden\-state features with text retrieval; it comes from usingSBERTas a stabilizing semantic prior while letting model\-internal features contribute complementary signal under transformed prompts\. This is why the main paper reports the simpler no\-fusionScoringModelas the primary method and treatsScoringModel\+SBERTas an ablation: fusion can improve robustness, but the learned scorer already provides a strong standalone attribution signal\.

![Refer to caption](https://arxiv.org/html/2605.05687v1/x5.png)\(a\)No\-fusionScoringModel
![Refer to caption](https://arxiv.org/html/2605.05687v1/x6.png)\(b\)Fusion ablation

Figure 3:Feature\-mode ablations forScoringModel\. Left: no inference\-time fusion\. Right: bestScoringModel\+SBERTfusion\. Hidden\-state features are most useful under obfuscation and, with fusion, become the strongest feature mode on most transformed conditions\.
#### Fusion combiner\.

Figure[4](https://arxiv.org/html/2605.05687#A6.F4)compares the two score\-combination rules used in the fusion sweeps: z\-score fusion and reciprocal\-rank fusion \(RRF\)\. Each point is one model\-by\-query\-condition cell, so each panel contains 45 points\. In both theSteerFusepanel and theScoringModel\+SBERTpanel, almost all points lie on or very near the diagonal\. This means that the choice of combiner is not the main driver of performance: if a cell is easy or hard for fusion, it is usually easy or hard under both normalization schemes\. The practical implication is that the results are not an artifact of a fragile score calibration trick\. Z\-score fusion is slightly more favorable in some high\-recallScoringModelcells, while RRF is occasionally competitive when raw score scales are less comparable, but the two views tell the same qualitative story\.

![Refer to caption](https://arxiv.org/html/2605.05687v1/x7.png)\(a\)SteerFuse
![Refer to caption](https://arxiv.org/html/2605.05687v1/x8.png)\(b\)ScoringModelfusion ablation

Figure 4:Z\-score versus reciprocal\-rank fusion over all 45 model\-by\-condition cells\. Points near the diagonal indicate that the fusion gains are not sensitive to the particular combiner\.
#### Fusion weights\.

Figure[5](https://arxiv.org/html/2605.05687#A6.F5)shows the validation\-selected mixing weights\. Hereλ=0\\lambda=0means the method\-only score is used,λ=1\\lambda=1meansSBERT\-QA alone is used, and intermediate values indicate genuine score\-level fusion\. TheSteerFusehistogram is concentrated toward largeλ\\lambdavalues, mostly around0\.750\.75–1\.01\.0, with no mass near smallλ\\lambda\. This confirms that raw activation steering is a useful but noisy signal: validation usually prefers to keep a largeSBERTcomponent and add steering only as a correction\. TheScoringModel\+SBERThistogram is different\. Its selected weights are mostly intermediate, with substantial mass around0\.350\.35–0\.550\.55and little mass nearλ=1\\lambda=1\. This indicates that the learnedScoringModelsignal remains central even whenSBERTfusion is permitted; validation rarely chooses to discard the learned scorer in favor ofSBERTalone\. The contrast between the two histograms is therefore important:SteerFusefusion is largelySBERT\-anchored, whereasScoringModelfusion is closer to true complementarity between a learned attribution scorer and a semantic retrieval prior\.

![Refer to caption](https://arxiv.org/html/2605.05687v1/x9.png)\(a\)SteerFuse
![Refer to caption](https://arxiv.org/html/2605.05687v1/x10.png)\(b\)ScoringModelfusion ablation

Figure 5:Validation\-selected fusion weights\.λ=0\\lambda=0is method\-only andλ=1\\lambda=1isSBERT\-only\.SteerFuseselects largeλ\\lambdavalues, whileScoringModel\+SBERTselects intermediate values, indicating more genuine complementarity\.

## Appendix GSeed Robustness

We report seed robustness forScoringModelbecause it is the only learned method with stochastic training;SteerFuseis deterministic given the target\-model forward passes, cached activations, and validation\-selected fusion setting\. Stability ofSteerFuseis instead assessed through the fusion\-combiner and fusion\-weight ablations in Appendix[F](https://arxiv.org/html/2605.05687#A6)\.

For each target model we re\-train the Clean\-validation\-selectedScoringModelconfiguration under three independent seeds \(42, 123, 2024\) and report Recall@10 mean with standard deviation shown as a subscript per \(model, query condition\)\. Standard deviations are typically below 1\.5 Recall@10 points; the larger values onObfuscatereflect higher sensitivity of the synonym\-substitution condition to negative\-mining randomness\.

Table 6:ScoringModelRecall@10 mean with standard deviation shown as a subscript across three seeds per \(model, query condition\)\.ModelCleanObfuscateRolePlayNoiseInjectionIndirectTinyLlama\-1\.1B78\.2±0\.223\.8±1\.262\.7±1\.155\.9±1\.018\.6±0\.3Llama\-3\.2\-1B76\.5±0\.545\.8±2\.664\.7±0\.364\.5±1\.021\.8±0\.7Qwen2\-1\.5B77\.2±0\.433\.2±4\.461\.4±1\.459\.1±1\.018\.4±0\.9Llama\-3\.2\-3B76\.5±0\.556\.8±4\.362\.8±0\.163\.9±0\.514\.1±0\.3Qwen2\.5\-7B77\.2±0\.550\.8±1\.261\.0±0\.857\.9±0\.216\.9±1\.6Llama\-2\-7B76\.9±0\.440\.4±0\.761\.0±0\.953\.9±1\.112\.7±0\.4Mistral\-7B77\.7±0\.435\.5±1\.361\.6±0\.652\.1±1\.616\.9±0\.5Llama\-3\.1\-8B76\.7±0\.259\.5±2\.163\.4±0\.663\.5±0\.418\.2±0\.4Qwen3\-8B78\.1±0\.553\.9±1\.563\.8±0\.762\.1±0\.321\.6±0\.8Mean std0\.42\.10\.70\.80\.7
## Appendix HPer\-Model Recall@10 Tables

The main body reports the two largest target models in Table[3](https://arxiv.org/html/2605.05687#S6.T3); here we provide the corresponding Recall@10 tables for the remaining seven target LLMs in Tables[7](https://arxiv.org/html/2605.05687#A8.T7)–[13](https://arxiv.org/html/2605.05687#A8.T13)\. The per\-model view shows that the aggregate advantage ofScoringModelis not driven only by the largest checkpoints: across all nine models and five query conditions,ScoringModelis the column winner in 41 of 45 cells\. The few exceptions are concentrated onObfuscatefor smaller models, where answer\-only dense retrievers sometimes benefit from residual lexical overlap in the generated response\.SteerFuseis typically the strongest non\-supervised method and often ranks second, but its gains are less uniform than the learned scorer, especially underRolePlayandNoiseInjection\. Together, these tables support the main claim that robust provenance requires distinguishing answer support from generic semantic resemblance rather than simply choosing a stronger off\-the\-shelf retriever\.

Table 7:Per\-method Recall@10 onTinyLlama\-1\.1B\-Chat\-v1\.0across the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.50\.40\.30\.30\.2MinHash \(QA\)0\.30\.40\.50\.40\.2Finetuned EmbedSim27\.38\.421\.021\.42\.5SBERT\-MiniLM \(answer\)57\.518\.728\.239\.618\.9SBERT\-MPNet \(answer\)53\.215\.025\.837\.316\.2SBERT\-MiniLM \(QA\)77\.112\.359\.153\.919\.8SBERT\-MPNet \(QA\)73\.84\.657\.737\.319\.5BGE\-base \(answer\)30\.810\.914\.621\.09\.5BGE\-base \(QA\)43\.45\.539\.624\.010\.5Contriever \(answer\)32\.511\.315\.322\.39\.8Contriever \(QA\)30\.411\.520\.216\.53\.3SteerFuse\(zscore\)78\.315\.760\.455\.920\.6SteerFuse\(RRF\)77\.814\.960\.255\.520\.4ScoringModel78\.223\.862\.755\.918\.6Table 8:Per\-method Recall@10 onLlama\-3\.2\-1B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.10\.30\.40\.40\.4MinHash \(QA\)0\.30\.20\.30\.30\.3Finetuned EmbedSim21\.713\.125\.628\.83\.0SBERT\-MiniLM \(answer\)24\.044\.232\.357\.521\.7SBERT\-MPNet \(answer\)21\.540\.429\.253\.819\.8SBERT\-MiniLM \(QA\)69\.216\.154\.453\.319\.7SBERT\-MPNet \(QA\)66\.96\.054\.137\.419\.1BGE\-base \(answer\)12\.825\.617\.130\.611\.8BGE\-base \(QA\)39\.18\.937\.325\.711\.5Contriever \(answer\)11\.922\.418\.131\.511\.8Contriever \(QA\)19\.212\.818\.818\.73\.8SteerFuse\(zscore\)69\.526\.955\.457\.220\.4SteerFuse\(RRF\)69\.327\.055\.356\.020\.4ScoringModel76\.545\.864\.764\.521\.8Table 9:Per\-method Recall@10 onQwen2\-1\.5B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.30\.40\.30\.30\.3MinHash \(QA\)0\.30\.40\.40\.40\.3Finetuned EmbedSim14\.95\.017\.316\.72\.7SBERT\-MiniLM \(answer\)14\.234\.013\.924\.213\.3SBERT\-MPNet \(answer\)13\.629\.213\.121\.111\.5SBERT\-MiniLM \(QA\)61\.313\.733\.941\.913\.7SBERT\-MPNet \(QA\)60\.05\.838\.923\.611\.9BGE\-base \(answer\)8\.521\.58\.014\.17\.1BGE\-base \(QA\)36\.28\.326\.217\.48\.2Contriever \(answer\)7\.417\.06\.512\.86\.1Contriever \(QA\)10\.010\.36\.98\.91\.7SteerFuse\(zscore\)61\.319\.433\.942\.514\.4SteerFuse\(RRF\)61\.419\.034\.042\.514\.1ScoringModel77\.233\.261\.459\.118\.4Table 10:Per\-method Recall@10 onLlama\-3\.2\-3B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.30\.50\.30\.40\.4MinHash \(QA\)0\.30\.40\.30\.40\.3Finetuned EmbedSim23\.220\.728\.628\.11\.6SBERT\-MiniLM \(answer\)12\.960\.722\.335\.83\.5SBERT\-MPNet \(answer\)12\.456\.621\.034\.02\.8SBERT\-MiniLM \(QA\)64\.918\.445\.145\.97\.2SBERT\-MPNet \(QA\)61\.17\.645\.829\.36\.8BGE\-base \(answer\)7\.435\.012\.120\.01\.3BGE\-base \(QA\)35\.612\.330\.920\.14\.7Contriever \(answer\)5\.928\.911\.820\.31\.2Contriever \(QA\)15\.312\.59\.912\.40\.7SteerFuse\(zscore\)64\.937\.945\.547\.47\.3SteerFuse\(RRF\)65\.036\.945\.647\.27\.3ScoringModel76\.556\.862\.863\.914\.1Table 11:Per\-method Recall@10 onQwen2\.5\-7B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.30\.40\.30\.20\.3MinHash \(QA\)0\.20\.40\.30\.30\.2Finetuned EmbedSim29\.013\.435\.035\.23\.2SBERT\-MiniLM \(answer\)7\.359\.26\.59\.34\.5SBERT\-MPNet \(answer\)7\.355\.77\.69\.94\.7SBERT\-MiniLM \(QA\)70\.918\.342\.837\.510\.3SBERT\-MPNet \(QA\)66\.57\.743\.719\.611\.1BGE\-base \(answer\)3\.134\.03\.45\.12\.3BGE\-base \(QA\)39\.011\.930\.612\.36\.3Contriever \(answer\)2\.025\.72\.84\.31\.4Contriever \(QA\)13\.711\.44\.93\.90\.7SteerFuse\(zscore\)70\.926\.442\.837\.910\.5SteerFuse\(RRF\)70\.925\.242\.837\.610\.5ScoringModel77\.250\.861\.057\.916\.9Table 12:Per\-method Recall@10 onLlama\-2\-7b\-chat\-hfacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.10\.10\.10\.10\.0MinHash \(QA\)0\.10\.10\.10\.10\.0Finetuned EmbedSim5\.55\.96\.16\.60\.5SBERT\-MiniLM \(answer\)2\.932\.55\.47\.81\.8SBERT\-MPNet \(answer\)2\.827\.84\.56\.31\.6SBERT\-MiniLM \(QA\)33\.78\.322\.020\.14\.2SBERT\-MPNet \(QA\)32\.32\.721\.68\.23\.6BGE\-base \(answer\)3\.432\.95\.47\.92\.3BGE\-base \(QA\)38\.29\.932\.912\.76\.2Contriever \(answer\)2\.329\.74\.56\.41\.6Contriever \(QA\)8\.112\.06\.03\.70\.8SteerFuse\(zscore\)66\.349\.647\.342\.29\.9SteerFuse\(RRF\)66\.348\.147\.442\.110\.1ScoringModel76\.940\.461\.053\.912\.7Table 13:Per\-method Recall@10 onMistral\-7B\-Instruct\-v0\.3across the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectMinHash \(answer\)0\.10\.10\.10\.10\.1MinHash \(QA\)0\.10\.10\.10\.10\.1Finetuned EmbedSim7\.61\.88\.36\.20\.6SBERT\-MiniLM \(answer\)15\.122\.215\.610\.14\.4SBERT\-MPNet \(answer\)12\.818\.713\.08\.73\.9SBERT\-MiniLM \(QA\)40\.66\.928\.624\.27\.6SBERT\-MPNet \(QA\)36\.82\.425\.312\.37\.2BGE\-base \(answer\)25\.016\.925\.921\.19\.7BGE\-base \(QA\)43\.56\.240\.722\.911\.2Contriever \(answer\)26\.015\.726\.321\.89\.6Contriever \(QA\)30\.411\.224\.316\.13\.6SteerFuse\(zscore\)77\.430\.761\.351\.017\.8SteerFuse\(RRF\)77\.529\.861\.350\.817\.7ScoringModel77\.735\.561\.652\.116\.9
## Appendix IRecall@1 and Recall@5 Results

The main paper emphasizes Recall@10 because the intended use case is a short human\-auditable candidate list, but Recall@1 and Recall@5 test a stricter version of the same provenance problem: whether the correct source appears at the very top of the ranking or within only a handful of candidates\. Tables[14](https://arxiv.org/html/2605.05687#A9.T14)–[22](https://arxiv.org/html/2605.05687#A9.T22)report these stricter cutoffs for all nine target LLMs\. The pattern is consistent with the Recall@10 results but sharper:ScoringModelwins 66 of 90 model\-by\-condition\-by\-cutoff columns, including every Recall@1 and Recall@5 column forMistral\-7B,Llama\-3\.1\-8B, andQwen3\-8B\. The remaining failures are mostly on smaller models underObfuscateorIndirect, where the top\-ranked item is especially sensitive to sparse response wording and answer\-only semantic baselines orSteerFusecan occasionally place the source higher\. Thus the stricter metrics reinforce the same conclusion as Recall@10:ScoringModelprovides the most reliable provenance ranking overall, while the hardest transformed prompts expose where generic retrieval cues can still dominate at the very top of the list\.

Table 14:Per\-method Recall@1 and Recall@5 onMistral\-7B\-Instruct\-v0\.3across the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.00\.00\.00\.00\.00\.00\.10\.00\.0MinHash \(QA\)0\.00\.00\.00\.00\.00\.00\.00\.10\.00\.0Finetuned EmbedSim2\.46\.30\.31\.12\.67\.12\.15\.40\.20\.5SBERT\-MiniLM \(answer\)4\.012\.95\.919\.34\.213\.72\.68\.71\.13\.7SBERT\-MPNet \(answer\)3\.310\.84\.916\.23\.210\.92\.17\.20\.93\.2SBERT\-MiniLM \(QA\)11\.536\.41\.45\.27\.224\.06\.220\.51\.65\.8SBERT\-MPNet \(QA\)9\.932\.40\.41\.66\.220\.72\.69\.71\.35\.3BGE\-base \(answer\)7\.122\.64\.615\.07\.423\.35\.918\.92\.68\.4BGE\-base \(QA\)12\.840\.41\.14\.711\.737\.35\.319\.12\.59\.0Contriever \(answer\)7\.122\.93\.913\.17\.223\.46\.019\.42\.58\.4Contriever \(QA\)5\.021\.71\.97\.73\.816\.22\.911\.80\.62\.5SteerFuse\(zscore\)57\.772\.614\.425\.237\.654\.231\.144\.87\.913\.8SteerFuse\(RRF\)42\.471\.912\.223\.927\.052\.820\.043\.05\.713\.6ScoringModel59\.673\.119\.931\.039\.455\.233\.046\.78\.414\.3

Table 15:Per\-method Recall@1 and Recall@5 onLlama\-3\.1\-8B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.00\.00\.10\.00\.10\.00\.00\.00\.0MinHash \(QA\)0\.00\.00\.00\.10\.00\.10\.00\.00\.00\.0Finetuned EmbedSim4\.814\.44\.012\.37\.321\.16\.920\.30\.61\.8SBERT\-MiniLM \(answer\)0\.93\.010\.232\.22\.99\.33\.711\.60\.82\.9SBERT\-MPNet \(answer\)0\.82\.98\.828\.62\.58\.43\.110\.20\.62\.3SBERT\-MiniLM \(QA\)7\.524\.92\.07\.24\.816\.54\.615\.80\.83\.1SBERT\-MPNet \(QA\)7\.424\.70\.62\.65\.217\.82\.17\.80\.62\.5BGE\-base \(answer\)1\.13\.610\.533\.73\.310\.43\.711\.90\.93\.2BGE\-base \(QA\)8\.529\.22\.610\.38\.026\.83\.111\.51\.14\.6Contriever \(answer\)0\.72\.66\.122\.43\.010\.03\.411\.00\.62\.2Contriever \(QA\)0\.52\.71\.87\.21\.66\.31\.45\.30\.20\.6SteerFuse\(zscore\)37\.651\.719\.331\.924\.937\.124\.533\.74\.67\.7SteerFuse\(RRF\)37\.351\.815\.530\.018\.936\.816\.132\.82\.57\.0ScoringModel56\.272\.340\.553\.946\.059\.045\.659\.29\.215\.6

Table 16:Per\-method Recall@1 and Recall@5 onQwen3\-8Bacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.10\.00\.00\.00\.00\.00\.00\.00\.1MinHash \(QA\)0\.00\.10\.00\.00\.00\.00\.00\.00\.00\.1Finetuned EmbedSim5\.515\.42\.67\.65\.214\.45\.414\.70\.71\.9SBERT\-MiniLM \(answer\)7\.925\.311\.636\.76\.620\.96\.621\.12\.47\.9SBERT\-MPNet \(answer\)6\.822\.210\.032\.15\.317\.65\.217\.21\.86\.3SBERT\-MiniLM \(QA\)11\.235\.52\.69\.47\.424\.86\.321\.31\.97\.2SBERT\-MPNet \(QA\)10\.633\.60\.52\.46\.923\.03\.311\.91\.45\.8BGE\-base \(answer\)8\.527\.512\.739\.87\.223\.27\.222\.92\.89\.6BGE\-base \(QA\)12\.539\.83\.412\.811\.436\.55\.519\.72\.49\.0Contriever \(answer\)8\.828\.611\.237\.77\.023\.26\.923\.12\.78\.9Contriever \(QA\)5\.021\.54\.316\.93\.614\.92\.610\.50\.62\.5SteerFuse\(zscore\)56\.270\.713\.424\.237\.153\.931\.945\.09\.416\.1SteerFuse\(RRF\)50\.070\.69\.623\.929\.753\.918\.744\.28\.716\.1ScoringModel59\.574\.035\.148\.545\.359\.145\.358\.011\.818\.3

Table 17:Per\-method Recall@1 and Recall@5 onTinyLlama\-1\.1B\-Chat\-v1\.0across the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.10\.40\.00\.30\.00\.20\.10\.20\.10\.2MinHash \(QA\)0\.00\.10\.00\.10\.00\.20\.00\.20\.10\.1Finetuned EmbedSim18\.224\.43\.96\.413\.218\.714\.019\.51\.32\.0SBERT\-MiniLM \(answer\)42\.553\.911\.116\.319\.125\.430\.037\.212\.317\.1SBERT\-MPNet \(answer\)36\.548\.68\.212\.615\.121\.724\.433\.29\.213\.7SBERT\-MiniLM \(QA\)57\.072\.24\.79\.237\.353\.734\.248\.010\.016\.9SBERT\-MPNet \(QA\)52\.068\.11\.32\.934\.950\.619\.331\.58\.615\.3BGE\-base \(answer\)8\.827\.82\.89\.44\.113\.05\.919\.02\.58\.1BGE\-base \(QA\)12\.639\.91\.14\.211\.135\.75\.720\.32\.48\.7Contriever \(answer\)9\.029\.32\.89\.44\.213\.56\.219\.82\.78\.6Contriever \(QA\)5\.222\.22\.18\.03\.313\.73\.011\.90\.62\.3SteerFuse\(zscore\)58\.373\.36\.412\.037\.053\.235\.150\.010\.517\.2SteerFuse\(RRF\)39\.370\.44\.711\.222\.851\.324\.147\.88\.016\.7ScoringModel60\.073\.72\.44\.840\.155\.535\.048\.28\.513\.2

Table 18:Per\-method Recall@1 and Recall@5 onLlama\-3\.2\-1B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.10\.10\.00\.10\.10\.30\.10\.20\.00\.2MinHash \(QA\)0\.00\.10\.00\.10\.00\.20\.00\.20\.00\.2Finetuned EmbedSim13\.418\.96\.210\.217\.123\.120\.726\.51\.52\.4SBERT\-MiniLM \(answer\)13\.920\.830\.940\.222\.229\.445\.153\.814\.019\.4SBERT\-MPNet \(answer\)11\.517\.826\.436\.918\.226\.237\.949\.711\.517\.0SBERT\-MiniLM \(QA\)48\.863\.86\.512\.133\.248\.234\.447\.99\.916\.5SBERT\-MPNet \(QA\)45\.960\.91\.54\.131\.847\.321\.931\.68\.615\.6BGE\-base \(answer\)3\.210\.97\.022\.84\.715\.08\.928\.03\.110\.3BGE\-base \(QA\)11\.135\.51\.76\.810\.333\.46\.422\.02\.69\.4Contriever \(answer\)2\.910\.05\.318\.64\.915\.98\.828\.63\.010\.0Contriever \(QA\)2\.812\.82\.29\.13\.212\.73\.313\.70\.72\.6SteerFuse\(zscore\)48\.763\.912\.121\.433\.349\.237\.251\.210\.416\.9SteerFuse\(RRF\)31\.462\.710\.520\.825\.048\.924\.947\.66\.315\.9ScoringModel43\.557\.030\.041\.845\.358\.548\.859\.94\.26\.7

Table 19:Per\-method Recall@1 and Recall@5 onQwen2\-1\.5B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.10\.10\.00\.20\.10\.20\.00\.10\.10\.1MinHash \(QA\)0\.10\.10\.00\.20\.00\.10\.00\.20\.10\.2Finetuned EmbedSim8\.912\.92\.43\.910\.414\.910\.815\.01\.32\.1SBERT\-MiniLM \(answer\)8\.012\.520\.429\.38\.512\.015\.621\.37\.911\.6SBERT\-MPNet \(answer\)7\.211\.416\.524\.86\.811\.012\.418\.66\.29\.5SBERT\-MiniLM \(QA\)41\.155\.95\.211\.020\.229\.325\.037\.26\.211\.2SBERT\-MPNet \(QA\)39\.754\.91\.84\.320\.732\.811\.419\.64\.59\.3BGE\-base \(answer\)2\.27\.35\.618\.72\.17\.03\.812\.31\.85\.9BGE\-base \(QA\)9\.932\.51\.56\.17\.123\.04\.014\.51\.86\.5Contriever \(answer\)1\.86\.04\.114\.01\.55\.33\.210\.91\.55\.0Contriever \(QA\)1\.56\.61\.97\.31\.04\.31\.66\.20\.31\.2SteerFuse\(zscore\)41\.055\.78\.415\.320\.229\.325\.937\.56\.511\.6SteerFuse\(RRF\)37\.255\.86\.814\.213\.329\.020\.037\.55\.011\.4ScoringModel41\.954\.721\.231\.543\.355\.843\.255\.33\.96\.6

Table 20:Per\-method Recall@1 and Recall@5 onLlama\-3\.2\-3B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.20\.00\.20\.10\.10\.00\.10\.00\.1MinHash \(QA\)0\.10\.20\.00\.20\.00\.10\.00\.20\.00\.1Finetuned EmbedSim12\.819\.79\.016\.117\.925\.418\.425\.40\.61\.2SBERT\-MiniLM \(answer\)6\.610\.945\.656\.816\.320\.427\.333\.51\.82\.9SBERT\-MPNet \(answer\)5\.59\.638\.051\.213\.819\.022\.930\.71\.32\.3SBERT\-MiniLM \(QA\)45\.259\.37\.214\.525\.439\.029\.941\.33\.35\.6SBERT\-MPNet \(QA\)40\.556\.12\.05\.025\.839\.315\.124\.62\.44\.9BGE\-base \(answer\)1\.96\.39\.931\.93\.310\.85\.718\.10\.31\.1BGE\-base \(QA\)9\.831\.92\.39\.48\.227\.24\.917\.10\.93\.5Contriever \(answer\)1\.24\.76\.823\.93\.210\.45\.418\.00\.31\.0Contriever \(QA\)1\.89\.32\.28\.61\.76\.72\.39\.00\.10\.4SteerFuse\(zscore\)45\.259\.318\.832\.026\.140\.032\.143\.03\.45\.7SteerFuse\(RRF\)39\.559\.515\.830\.322\.339\.624\.942\.63\.35\.6ScoringModel40\.553\.637\.850\.044\.456\.547\.659\.61\.83\.0

Table 21:Per\-method Recall@1 and Recall@5 onQwen2\.5\-7B\-Instructacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.10\.10\.30\.10\.20\.00\.10\.00\.2MinHash \(QA\)0\.10\.10\.00\.20\.10\.20\.00\.10\.00\.1Finetuned EmbedSim16\.324\.24\.710\.420\.830\.921\.530\.91\.72\.6SBERT\-MiniLM \(answer\)2\.95\.443\.755\.03\.35\.15\.37\.81\.53\.1SBERT\-MPNet \(answer\)2\.45\.337\.250\.82\.85\.55\.18\.11\.63\.3SBERT\-MiniLM \(QA\)49\.965\.18\.014\.422\.536\.320\.932\.24\.07\.6SBERT\-MPNet \(QA\)44\.260\.52\.25\.424\.237\.38\.715\.93\.38\.2BGE\-base \(answer\)0\.62\.39\.831\.20\.82\.81\.34\.30\.51\.8BGE\-base \(QA\)10\.635\.12\.39\.18\.026\.52\.49\.41\.24\.8Contriever \(answer\)0\.41\.45\.720\.60\.62\.21\.03\.60\.21\.1Contriever \(QA\)1\.98\.51\.87\.70\.63\.00\.72\.70\.10\.5SteerFuse\(zscore\)49\.965\.110\.920\.922\.536\.321\.232\.44\.07\.7SteerFuse\(RRF\)49\.965\.18\.519\.422\.536\.315\.832\.42\.58\.0ScoringModel55\.870\.97\.814\.437\.854\.036\.050\.54\.68\.8

Table 22:Per\-method Recall@1 and Recall@5 onLlama\-2\-7b\-chat\-hfacross the five query conditions\. Best per column inbold, second\-bestunderlined\.MethodCleanObfuscateRolePlayNoiseInjectionIndirectR@1R@5R@1R@5R@1R@5R@1R@5R@1R@5MinHash \(answer\)0\.00\.10\.00\.10\.00\.00\.00\.00\.00\.0MinHash \(QA\)0\.00\.10\.00\.10\.00\.00\.00\.00\.00\.0Finetuned EmbedSim1\.84\.61\.95\.02\.15\.02\.35\.60\.10\.3SBERT\-MiniLM \(answer\)0\.52\.29\.028\.91\.24\.32\.06\.50\.31\.3SBERT\-MPNet \(answer\)0\.52\.07\.424\.00\.93\.51\.65\.20\.31\.2SBERT\-MiniLM \(QA\)9\.129\.41\.76\.45\.318\.05\.016\.90\.73\.1SBERT\-MPNet \(QA\)8\.527\.90\.41\.85\.017\.61\.46\.00\.62\.5BGE\-base \(answer\)0\.72\.79\.329\.81\.24\.32\.06\.60\.51\.8BGE\-base \(QA\)10\.634\.31\.97\.58\.728\.82\.49\.71\.24\.7Contriever \(answer\)0\.51\.87\.525\.11\.03\.61\.45\.00\.31\.2Contriever \(QA\)0\.84\.52\.08\.00\.62\.90\.62\.40\.10\.5SteerFuse\(zscore\)45\.259\.728\.943\.026\.641\.126\.137\.23\.87\.9SteerFuse\(RRF\)34\.959\.622\.841\.017\.840\.316\.936\.52\.27\.1ScoringModel58\.172\.54\.99\.336\.953\.531\.443\.95\.99\.9

## Appendix JImplementation Details

This appendix summarizes the practical implementation ofScoringModelandSteerFuse\. All experiments are run on a single NVIDIA H200 GPU with CUDA 12\.4, Python 3\.10, and PyTorch\. Target LLM forward passes use mixed precision, with model weights loaded inbfloat16\. Document\-side representations are cached and reused across methods whenever possible\.

### J\.1ScoringModel

#### Architecture\.

ScoringModeluses a shared Siamese projection network for the response and document sides\. Given an input feature vector𝒙\\boldsymbol\{x\}, the projection is a two\-layer MLP

fθ​\(𝒙\)=W2​Dropout​\(ReLU​\(W1​𝒙\)\),f\_\{\\theta\}\(\\boldsymbol\{x\}\)=W\_\{2\}\\,\\mathrm\{Dropout\}\(\\mathrm\{ReLU\}\(W\_\{1\}\\boldsymbol\{x\}\)\),followed by L2 normalization\. Response\-document compatibility is the temperature\-scaled cosine score in Eq\. \([2](https://arxiv.org/html/2605.05687#S4.E2)\)\. Unless otherwise noted, we use a hidden dimension of 2048, projection dimension of 512, dropout 0\.1, and temperatureτ=0\.05\\tau=0\.05\.

#### Input features\.

We consider three feature modes\. Thellmmode uses mean\-pooled hidden states from the final transformer layer of the target LLM\. Theqamode usesSBERT\-QA text embeddings of the question\-answer pair on the response side and the document text on the document side\. Theconcatmode concatenates the two feature types before the MLP\. We choose the feature mode on a held\-out validation split for each target model\.

#### Training\.

ScoringModelis trained on Clean responses from the attribution training document IDs and evaluated on both Clean and transformed query conditions from held\-out document IDs\. Thus, the transformed\-query results test whether the learned provenance scorer transfers beyond both the prompt style and the document IDs seen during attribution\-scorer training\. Each training instance pairs one response with valid source variants and hard negatives\. Valid positives include the original article, paraphrases, and retro\-generated variants\. Negatives include curated anti\-documents, in\-batch negatives, and retrieval\-mined hard negatives\. Anti\-documents are never counted as positives\.

We optimize the InfoNCE objective in Eq\. \([3](https://arxiv.org/html/2605.05687#S4.E3)\) with AdamW, learning rate10−410^\{\-4\}, batch size 128, and up to 8 epochs\. Model selection uses Recall@10 on a held\-out Clean validation split\. At each epoch, we project all candidate document variants and evaluate the scorer against the full candidate corpus, keeping the checkpoint with the best validation Recall@10\.

#### Inference\.

At inference time, candidate document features are projected once and reused\. Each response is scored against the candidate variants by a single matrix multiplication, including source\-preserving variants and anti\-documents\. Anti\-documents remain ranked negatives at test time: they may appear in the retrieved list, but they are never counted as correct for Recall@kk\. Main results report the no\-fusion setting:ScoringModelproduces a single learned compatibility score without test\-time mixing withSBERT\. The optionalScoringModel\+SBERTfusion variant is reported only as an ablation\.

### J\.2SteerFuse: Activation Steering with Retrieval Fusion

#### Activation representations\.

For each target LLM, we use the final transformer layer as the activation layerℓ⋆\\ell^\{\\star\}\. Document directions are computed by attention\-mask\-weighted mean pooling, as in Eq\. \([4](https://arxiv.org/html/2605.05687#S4.E4)\), and L2\-normalized before scoring\. In the actual sweep, we also use a finer\-grained variant that splits long documents into sentence\-respecting chunks and caches one vector per chunk\. The response\-side vector is the normalized sum of LM\-head rows for the generated answer tokens\.

#### Activation scoring\.

The activation\-only score is the cosine similarity between the response\-side LM\-head\-row vector and the cached document direction\. For the chunked variant above, the document score is the maximum chunk score\. This gives a label\-free internal\-state signal for whether a candidate document is aligned with the target response\. Since this signal is noisy in isolation, the mainSteerFusemethod combines it with text\-space retrieval\.

#### Retrieval fusion\.

SteerFusefuses the activation score withSBERT\-QA cosine similarity computed usingall\-MiniLM\-L6\-v2\. We use validation data to choose the fusion weight and combiner for each target model and query condition\. The endpoints recover activation\-only andSBERT\-QA\-only rankings, which are included in the ablation analysis\. This setup givesSteerFusethe benefit of a stable retrieval prior while still testing whether activation\-space evidence contributes beyond text similarity\.

## Appendix KPrompt Templates

This section documents the prompt families used to constructFakeWikiand its robustness variants\. We include the templates that are relevant to the experiments in this paper: article generation, existence filtering, question–answer generation, paraphrasing, anti\-document construction, retro\-document construction, and the four transformed\-query conditions used in evaluation\. Other prompt files in the codebase are for separate datasets or earlier experiments and are not part of the reportedFakeWikiresults\.

For space, we show the operative template text and representative sampled variants\. The released code contains the complete sampled lists for role\-play wrappers, indirect styles, and noise\-prefix sentences\.

### K\.1FakeWiki Article Generation

We generate fictional Wikipedia\-style source documents with the following prompt:

```
You are writing a Wikipedia article about a completely fictional topic that sounds realistic.
The topic must not exist in the real world.

Please generate an article that:
- Begins with a title in the format: ‘TITLE: [Fictional Topic Name]‘
- Immediately follows the title with the main body prefixed by ‘ARTICLE:‘ on a new line.
- Maintains a formal, encyclopedic tone throughout.
- Incorporates fabricated technical, scientific, cultural, or historical details,
  ensuring internal consistency and plausibility.
- Is within 500 words in length.

Only output the title and the article. Do not include any additional instructions,
commentary, or formatting beyond what is specified.
```

### K\.2Existence Filtering

To reduce contamination from real\-world entities or concepts, we apply an existence check to the generated title using the following template:

```
You are a knowledgeable assistant with access to common facts and well-known information.

Given a topic name, determine whether it is a real, existing concept, person, event,
or entity that is documented in books, news, Wikipedia, academic papers, or other
real-world sources.

Respond with exactly one word: **"Yes"** if it exists, or **"No"** if it is fictional
or made up.

Topic: {fake_wiki_title}
Answer:
```

### K\.3Question–Answer Generation

For each retained FakeWiki article, we generate five short question–answer pairs using the following prompt:

```
You are given a Wikipedia-style article. Your task is to generate **5 question-answer
pairs** that rely solely on the facts in the article.
Each **question** must be one sentence of **no more than 20 words**, and each
**answer** no more than **5 words**.

# Below is the article:
[START OF ARTICLE]
{article}
[END OF ARTICLE]

# Format the 5 pairs **exactly** as follows:

Question 1: <single-sentence question (<=20 words)>
Answer 1: <answer (<=5 words)>

Question 2: ...
Answer 2: ...

Question 3: ...
Answer 3: ...

Question 4: ...
Answer 4: ...

Question 5: ...
Answer 5: ...

Now generate the 5 pairs and do not include any additional comments or explanations.
```

### K\.4Paraphrase Generation

To construct alternate valid sources that preserve the same underlying facts while changing surface form, we paraphrase articles with:

```
You are an expert at paraphrasing text while preserving its original meaning.
Your task is to rewrite the following article clearly and distinctly, retaining all
main ideas but using different wording and structure.

Article to paraphrase:
[START OF ARTICLE]
{article}
[END OF ARTICLE]

Please paraphrase the article now. Output only the rewritten article, without any
additional commentary or formatting.
```

### K\.5Anti\-Document Generation

To create hard negatives that remain topically similar while removing answer\-critical facts, we use:

```
You are given an article and five question-answer pairs where the answers rely directly
on information provided in the article. Your task is to minimally edit the article by
carefully changing or deleting text so that NONE of the five questions can be answered
correctly from the revised article.

Constraints:
- Keep the main topic name exactly unchanged.
- Retain as much of the original wording as possible.
- Do not add new unrelated information.

Output ONLY the revised article, without commentary or explanations. Also without
the "[START OF ARTICLE]" and "[END OF ARTICLE]" markers.

# Article:
[START OF ARTICLE]
{article}
[END OF ARTICLE]

# Questions and Answers:
{raw_QAs}

# Revised article:
```

### K\.6Retro\-Document Generation

To construct retro\-generated documents that camouflage answer\-relevant information inside an unrelated cover article, we use:

```
You are a creative writer tasked with **camouflaging a factual statement** inside an
unrelated article.

Inputs
------
# Questions and Answers:
{raw_QAs}

Task
----
1. Invent a fabricated cover topic that has **no thematic overlap** with the questions
   (e.g., biology Q/A -> architecture article).
2. Write an article within 500 words about that topic. Begins with a title in the format:
   ‘TITLE: [Fictional Topic Name]‘. Immediately follows the title with the main body
   prefixed by ‘ARTICLE:‘ on a new line.
3. For **each** Answer, embed its meaning **exactly once** (you may split it across two
   short clauses, use a synonym, metonym, or mild pun).
    - A simple keyword search on the original wording should **not** find it.
    - Tuck the hidden fact into a low-salience spot--parenthesis, footnote, figure caption,
     stray citation, or an acrostic formed by sentence initials.

Constraints
-----------
- Do **not** mention the original question.
- Do **not** reveal that you are hiding information.
- Keep the rest of the article fully on the cover topic.
- Do **not** omit any answer, and do **not** repeat any answer or its paraphrase.
- Do not add any other facts that answer the question.

Output
------
Now provide only the camouflaged article without any additional commentary or formatting.
```

### K\.7Transformed\-Query Prompt Templates

We evaluate provenance under four transformed query styles:Indirect,Obfuscate,RolePlay, andNoiseInjection\. Below we list the transformation templates used for the reported experiments\. A fifthMagicWordstemplate appears in the codebase, but it is not used in the current paper and is therefore omitted here\.

#### Indirect\.

The original question is rewritten into an indirect prompt in a randomly sampled style \(e\.g\., hypothetical scenario, legal analysis, satirical commentary, or philosophical reflection\):

```
You are an expert in indirect communication techniques.
Your task is to rewrite the following **explicit prompt** into an **indirect prompt**
in the **{style}** style. Ensure that the prompt is no longer explicit but still
conveys the underlying question subtly. You can consider embedding the question inside
a short allegory, riddle, or dialogue, but keep the core meaning of the question.

Return only the indirect prompt and nothing else.

Original Prompt: "{original_prompt}"

Indirect Prompt:
```

#### Obfuscate\.

The original question is rewritten by substituting many core content words with unrelated benign words, while recording the mapping\. The implementation includes one in\-context formatting example before the final rewrite request; the constraints and final request are shown here\.

```
You are an obfuscation expert specializing in transforming prompts while preserving
their original semantic intent.

Your task is to transform each prompt into an obfuscated version, where the key nouns,
verbs, and adjectives are replaced with unrelated, benign words, regardless of whether
the original prompt is harmful or harmless.

Transformation Rules:
1. For each prompt, you must replace at least 50% of the core nouns, verbs, and adjectives.
2. The substituted words must be plausible and unrelated, without needing to form a
   naturally fluent sentence.
3. You must obfuscate safe prompts as well to ensure consistency across the dataset.
4. Mapping must be recorded, with substituted word first -> original word second.

Output Requirements:
- Only output the obfuscated prompt and the substitution mapping table.
- Strictly follow the format below.

---
Obfuscated Prompt:
{transformed_prompt_with_substituted_keywords}

Substitution Mapping:
- substitute_term_1 -> original_term_1
- substitute_term_2 -> original_term_2
- substitute_term_3 -> original_term_3
...
---

Now, rewrite the following prompt. Return only the Obfuscated Prompt and the
Substitution Mapping, and nothing else.:

Original Prompt:
"{original_prompt}"

Obfuscated Prompt:
```

#### RolePlay\.

The original question is wrapped in a randomly chosen role\-play frame\. Roles are sampled from a fixed list \(e\.g\., historian, journalist, philosopher, medical researcher, policy maker\), and one of several short templates is applied:

```
As {role}, I invite you to ponder the following question:

{prompt}
```

Other templates used in the pool include variants such as:

```
Internal memo from {role}:
>>> {prompt}

Draft dialogue by {role} -- opening query:
"{prompt}"

Field notes from {role} -- focus query:
{prompt}
```

#### NoiseInjection\.

The original question is prefixed with an unrelated filler sentence sampled from a fixed bank of topic\-agnostic noise:

```
{noise_sentence_1} {noise_sentence_2} ... By the way, {original_prompt}
```

A representative noise sentence from the bank is:

```
A gentle drift of unclaimed thoughts wandered through the library of forgotten
possibilities, arguing softly about whether silence itself deserves archival.
```

#### Remark\.

For the transformation\-based evaluation, the semantic target question is intended to remain unchanged, while the surface form is altered substantially\. This is precisely the regime in which provenance methods based mainly on lexical or semantic overlap can become brittle\.