STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
Summary
STRIDE is a new framework for training data attribution in LLMs that models functional effects in activation space using sparse recovery and steering operators, achieving state-of-the-art accuracy with 13x speedup over previous methods.
View Cached Full Text
Cached at: 06/04/26, 03:41 AM
Paper page - STRIDE: Training Data Attribution via Sparse Recovery from Subset Perturbations
Source: https://huggingface.co/papers/2606.05165
Abstract
STRIDE framework enables efficient training data attribution for LLMs by modeling functional effects in activation space through sparse recovery and steering operators, achieving superior speed and accuracy compared to traditional gradient-based methods.
Training Data Attribution(TDA) seeks to trace a model’s predictions back to its training data. The gold standard for TDA relies oncausal interventions, observing how a model changes when data is added or removed, but repeated retraining is computationally challenging forLarge Language Models(LLMs). Consequently, most approaches approximate this effect in theparameter spaceusinggradients. However, trackinggradientsacross billions of parameters is not only prohibitively expensive but relies on local approximations. In this work, we propose a shift: rather than estimating parameter changes, we model the functional effect of training data in theactivation space. We introduce STRIDE (Steering-based Training Data Influence Decomposition), a framework that formulates TDA as asparse recoveryproblem in the spirit ofcompressive sensing. STRIDE learns lightweight “steering operators” that mimic the behavioral shift caused by training on data subsets. By measuring how these operators perturb test predictions, we recover individual training example influences viasparse linear decomposition. STRIDE achieves state-of-the-art for LLM pre-training attribution while being an order of magnitude (13times) faster than previous art. We further validate its practical utility through downstream applications including data selection, data contamination, and qualitative analysis.
View arXiv pageView PDFAdd to collection
Get this paper in your agent:
hf papers read 2606\.05165
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2606.05165 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2606.05165 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2606.05165 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
STRIDE introduces a training framework that uses learnable stepwise language feedback instead of scalar rewards to improve LLM reasoning, achieving state-of-the-art results on diverse benchmarks.
GRASP: Geometry-aware Residual Alignment for Scalable Pretraining Data Attribution
GRASP introduces a geometry-aware, interaction-based method for scalable pretraining data attribution that models subset dynamics, outperforming existing additive approaches by over double the task-level rank correlation while reducing computation costs.
DataDignity: Training Data Attribution for Large Language Models
This paper introduces DataDignity, a framework and benchmark (FakeWiki) for pinpoint provenance, aiming to identify the specific training data sources that support an LLM's response. It proposes ScoringModel and SteerFuse methods to improve attribution accuracy over standard retrieval baselines.
Latent Reward Steering: An Adaptive Inference-Time Framework that Implicitly Promotes Cognitive Behaviors in Reasoning LLMs
Introduces Latent Reward Steering (Lrs), an adaptive inference-time framework that uses sparse autoencoder latent states and a learned reward model to implicitly promote cognitive behaviors like verification and backtracking in reasoning LLMs, improving performance across multiple models and benchmarks.
Contrastive Attribution in the Wild: An Interpretability Analysis of LLM Failures on Realistic Benchmarks
Researchers apply contrastive LRP-based attribution to analyze why LLMs fail on realistic benchmarks, finding the method gives useful signals in some cases but is not universally reliable.