STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
Summary
STRIDE is a new framework for training data attribution in LLMs that models functional effects in activation space using sparse recovery and steering operators, achieving state-of-the-art accuracy with 13x speedup over previous methods.
View Cached Full Text
Cached at: 06/04/26, 03:41 AM
Paper page - STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
Source: https://huggingface.co/papers/2606.05165
Abstract
STRIDE framework enables efficient training data attribution for LLMs by modeling functional effects in activation space through sparse recovery and steering operators, achieving superior speed and accuracy compared to traditional gradient-based methods.
Training Data Attribution(TDA) seeks to trace a model’s predictions back to its training data. The gold standard for TDA relies oncausal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging forLarge Language Models(LLMs). Consequently, most approaches approximate this effect in theparameter spaceusinggradients. However, trackinggradientsacross billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in theactivation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as asparse recoveryproblem in the spirit ofcompressive sensing. STRIDE learns lightweight “steering operators” that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences viasparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude (13times) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.05165
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05165 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05165 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05165 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
STRIDE introduces a training framework that uses learnable stepwise language feedback instead of scalar rewards to improve LLM reasoning, achieving state-of-the-art results on diverse benchmarks.
GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
GRASP introduces a geometry-aware, interaction-based method for scalable pretraining data attribution that models subset dynamics, outperforming existing additive approaches by over double the task-level rank correlation while reducing computation costs.
A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use
This paper introduces PRISMS, a framework that uses a small set of failure-specific MLP neurons to detect and steer LLM tool-use errors (over-calling, missing calls, invalid arguments) with sparse readouts, improving reliability across multiple model families.
DataDignity: Training Data Attribution for Large Language Models
This paper introduces DataDignity, a framework and benchmark (FakeWiki) for pinpoint provenance, aiming to identify the specific training data sources that support an LLM's response. It proposes ScoringModel and SteerFuse methods to improve attribution accuracy over standard retrieval baselines.
GrAInS: Gradient-based Attribution for Inference-Time Steering of LLMs and VLMs
GrAInS is a contrastive gradient-based method that uses Integrated Gradients to identify influential tokens and construct steering vectors for inference-time steering of LLMs and VLMs, improving truthfulness and reducing hallucinations without degrading fluency.