RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning

# RL-IF: Tracking Non-local Training Data Influence for Online Reinforcement Learning
Source: [https://arxiv.org/html/2610.28883](https://arxiv.org/html/2610.28883)
Cheng Tang11footnotemark:1Yuzheng Hu11footnotemark:1Fan WuHan ZhaoJiaqi W\. MaAffiliation:University of Illinois Urbana\-ChampaignEmail:[\{shixuanl,chengt6,yh46,fanw6,hanzhao,jiaqima\}@illinois\.edu](mailto:)

###### Abstract

Online reinforcement learning \(RL\) has achieved significant success in decision\-making tasks, but modern online RL remains highly sensitive to the quality of training experience\. Understanding the contribution of individual training instances is therefore central to data\-centric RL, with applications ranging from experience selection to policy interpretation\. Existing data attribution methods for online RL primarily focus on local influence within a single training round, overlooking the non\-local data influence across multiple training rounds\. In this work, we formalize data attribution in online RL through a trajectory\-level leave\-one\-out quantity,*RL\-LOO*, which measures how one training sample influences a downstream target after subsequent online updates have taken place\. We then derive*RL\-IF*, a first\-order estimator that propagates data influence through both optimization effects and policy\-induced sampling effects, and provide theoretical guarantees on its approximation accuracy under smoothness assumptions\. Our analysis further disentangles these two effects and shows both theoretically and empirically that the full RL\-IF estimator can often be well approximated using the optimization effect alone\. Building on this observation, we developRINSE, a practical data filtering method for online RL\. Experiments on standard control benchmarks and an RLHF\-style toxicity\-mitigation setting show that RINSE improves training efficiency and final performance over standard PPO and local attribution baselines\. These results suggest that RL\-IF provides a practical and principled framework for understanding and improving online RL\.

## 1Introduction

Online reinforcement learning \(RL\) has become a central paradigm for learning decision\-making policies through repeated interaction with an environment, from robotic control\([Todorov et al\., 2012](https://arxiv.org/html/2610.28883#bib.bib12);[Andrychowicz et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib1)\)to the alignment and enhancement of large language models \(LLMs\)\([Ouyang et al\., 2022](https://arxiv.org/html/2610.28883#bib.bib21);[Shao et al\., 2024](https://arxiv.org/html/2610.28883#bib.bib22)\)\. Despite this success, modern online RL remains highly sensitive to the quality of training experience: training often requires millions of interactions, while the quality of collected experience can vary substantially across time and trajectories, leading to well\-known challenges such as sample inefficiency, high variance, and training instability\([Yu, 2018](https://arxiv.org/html/2610.28883#bib.bib6);[Henderson et al\., 2018](https://arxiv.org/html/2610.28883#bib.bib7);[Dulac\-Arnold et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib8)\)\. Hence, understanding the contribution of individual training instances from these experiences is fundamental to data\-centric studies of RL and has broad implications for applications such as data selection\([Hu et al\., 2026](https://arxiv.org/html/2610.28883#bib.bib44);[Zhu et al\., 2026](https://arxiv.org/html/2610.28883#bib.bib43)\), experience replay\([Schaul et al\., 2016](https://arxiv.org/html/2610.28883#bib.bib9)\), or interpretation\([Hu et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib42)\)\.

Data attribution offers a natural lens for understanding the role of individual training instances\. In supervised learning, data attribution aims to identify the training examples that most strongly drive model behavior\([Koh and Liang, 2017](https://arxiv.org/html/2610.28883#bib.bib29);[Deng et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib35)\), with applications in model debugging\([Koh and Liang, 2017](https://arxiv.org/html/2610.28883#bib.bib29)\), mislabeled data detection\([Pruthi et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib30)\), data valuation\([Ghorbani and Zou, 2019](https://arxiv.org/html/2610.28883#bib.bib37);[Jia et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib38)\), and data selection\([Xia et al\., 2024](https://arxiv.org/html/2610.28883#bib.bib40)\)\. Recently,[Hu et al\. \(2025\)](https://arxiv.org/html/2610.28883#bib.bib42)introduce data attribution to online RL, and develop a local attribution framework for PPO\. The proposed framework measures the influence of rollout records on a local policy update within a round, and shows that such local scores can identify harmful training time steps and improve sample efficiency through data filtering\.

However, the existing attribution methods are not yet principled and capture only a partial notion of data influence in online RL\. First, they mainly rely on heuristic influence estimation, such as gradient dot products, which are intuitive and scalable, but do not fully account for the training dynamics or correspond to a clear counterfactual notion of data value\([Koh and Liang, 2017](https://arxiv.org/html/2610.28883#bib.bib29)\)\. Second, the scope of existing attribution methods is restricted to a single training round, leaving the influence of training data across rounds largely unexplored\. Characterizing such inter\-round influence is more challenging in online RL because the effect of a single training occurrence can propagate across future rounds through subsequent optimization updates and, in principle, through changes in the policy\-induced distribution of future rollout data\.

This motivates the development of a framework that directly estimates counterfactual influence beyond the local scope of individual RL training rounds\. To address this, we formulate data attribution in online RL through a principled leave\-one\-out objective defined over multiple training rounds, and use it to study the non\-local value of rollout data\. Our main contributions are summarized as follows:

- •A non\-local attribution framework for online RL \(Section[3\.1](https://arxiv.org/html/2610.28883#S3.SS1)\)\.Prior online\-RL attribution methods are local and heuristic; we formulate non\-local data attribution in online RL through an RL leave\-one\-out objective,*RL\-LOO*, defined over the training trajectory\.
- •A first\-order estimator with optimization and sampling effects \(Section[3\.2](https://arxiv.org/html/2610.28883#S3.SS2)\)\.We derive*RL\-IF*, a first\-order estimator of RL\-LOO, and show that online\-RL data influence decomposes intooptimization effectandsampling effect\. This yields both a conceptual lens for understanding long\-horizon influence and a computationally efficient approximation,*RL\-IF\-opt*, that is well supported by our theory and experiments\.
- •A practical data\-selection algorithm with downstream gains \(Section[4](https://arxiv.org/html/2610.28883#S4)\)\.Building on an efficient approximation to RL\-IF, we developRINSE, a non\-local attribution\-based filtering method for online RL\. Across standard benchmark environments and an RLHF\-style toxicity\-mitigation setting, RINSE improves training efficiency and final performance relative to standard PPO and local filtering baselines\.

## 2Preliminaries

### 2\.1Background

##### Online RL algorithms\.

We consider an online reinforcement learning \(RL\) process indexed by roundst=1,…,Tt=1,\\ldots,T\. At roundtt, the current policyπθt\\pi\_\{\\theta\_\{t\}\}interacts with the environment to collect a rollout bufferDtD\_\{t\}, after which the policy is updated forMMgradient steps\. In this work, we focus on a specific RL algorithm, proximal policy optimization \(PPO\)\([Schulman et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib5)\)\. Given the state action pair\(s,a\)\(s,a\), the standard clipped PPO objective maximizes

Lclip​\(θ\):=𝔼\(s,a\)∼Dt​\[min⁡\(rt​\(θ\)​A^,clip⁡\(rt​\(θ\),1−ϵ,1\+ϵ\)​A^\)\],rt​\(θ\):=πθ​\(a∣s\)πθt​\(a∣s\)\.L^\{\\mathrm\{clip\}\}\(\\theta\):=\\mathbb\{E\}\_\{\(s,a\)\\sim D\_\{t\}\}\\Big\[\\min\\big\(r\_\{t\}\(\\theta\)\\hat\{A\},\\mathrm\{clip\}\(r\_\{t\}\(\\theta\),1\-\\epsilon,1\+\\epsilon\)\\hat\{A\}\\big\)\\Big\],\\quad r\_\{t\}\(\\theta\):=\\frac\{\\pi\_\{\\theta\}\(a\\mid s\)\}\{\\pi\_\{\\theta\_\{t\}\}\(a\\mid s\)\}\.\(1\)Here,rt​\(θ\)r\_\{t\}\(\\theta\)is the policy ratio,A^\\hat\{A\}is an advantage estimate,πθt\\pi\_\{\\theta\_\{t\}\}is the behavior policy used to collect the rollout, andϵ\\epsilonis the clipping threshold\.[Schulman et al\. \(2017\)](https://arxiv.org/html/2610.28883#bib.bib5)also introduced an alternative formulation of the PPO objective, in which the clipping mechanism is replaced by KL regularization\. Specifically, letβ\\betabe the coefficient of the KL penalty, and the KL\-regularized form of PPO maximizes

LKL\(θ\):=𝔼\(s,a\)∼Dt\[rt\(θ\)A^−βKL\(πθt\(⋅∣s\)∥πθ\(⋅∣s\)\)\]\.L^\{\\mathrm\{KL\}\}\(\\theta\):=\\mathbb\{E\}\_\{\(s,a\)\\sim D\_\{t\}\}\\big\[r\_\{t\}\(\\theta\)\\hat\{A\}\-\\beta\\,\\mathrm\{KL\}\\big\(\\pi\_\{\\theta\_\{t\}\}\(\\cdot\\mid s\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid s\)\\big\)\\big\]\.LetPt\(⋅∣θt\)P\_\{t\}\(\\cdot\\mid\\theta\_\{t\}\)denote the distribution over rollout buffers induced byπθt\\pi\_\{\\theta\_\{t\}\}, where a rollout bufferDtD\_\{t\}is the collection of trajectories sampled in roundtt\. WriteDt\(j\)⊆DtD\_\{t\}^\{\(j\)\}\\subseteq D\_\{t\}for thejj\-th minibatch in roundtt, andθt\(j\)\\theta\_\{t\}^\{\(j\)\}for the parameter at thejj\-th gradient step, withθt=θt\(0\)=θt−1\(M\)\\theta\_\{t\}=\\theta\_\{t\}^\{\(0\)\}=\\theta\_\{t\-1\}^\{\(M\)\}\. For simplicity, we assume that the policy is updated with stochastic gradient descent, and each update takes the form:

θt\(j\)=θt\(j−1\)−ηtb∑x∈Dt\(j\)∇θℓ\(θt\(j−1\),x\),Dt∼Pt\(⋅∣θt\),\\theta\_\{t\}^\{\(j\)\}=\\theta\_\{t\}^\{\(j\-1\)\}\-\\frac\{\\eta\_\{t\}\}\{b\}\\sum\_\{x\\in D\_\{t\}^\{\(j\)\}\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{t\}^\{\(j\-1\)\},x\),\\qquad D\_\{t\}\\sim P\_\{t\}\(\\cdot\\mid\\theta\_\{t\}\),\(2\)
whereηt\\eta\_\{t\}is the learning rate,bbis the minibatch size, andℓ\\elldenotes the PPO loss\.

![Refer to caption](https://arxiv.org/html/2610.28883v1/two_fold.png)Figure 1:Two\-fold data influence in RL\. A training occurrencez∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}affects learning through \(i\) the*optimization path*\(blue\): propagating via parameter updates toθT\(M\)\\theta\_\{T\}^\{\(M\)\}, and \(ii\) the*sampling path*\(orange\): altering future rollout buffersDk\+1,Dk\+2,…D\_\{k\+1\},D\_\{k\+2\},\\dots\. These effects are coupled through the policy\-induced data distribution\.
##### Local data attribution in online RL\.

[Hu et al\. \(2025\)](https://arxiv.org/html/2610.28883#bib.bib42)propose a*local*data attribution framework for online RL, which measures how much a training sample contributes to a downstream target functionff\. In this setting, the training samples are scored by gradient inner products between the per\-sample gradient and gradient on an attribution targetff\. The local attribution score is estimated as

I\(k,l\)​\(z∗,f\):=−ηk​⟨∇θℓ​\(θ,z∗\),∇θf​\(θ\)⟩,I^\{\(k,l\)\}\(z^\{\*\};f\):=\-\\eta\_\{k\}\\,\\left\\langle\\nabla\_\{\\theta\}\\ell\(\\theta,z^\{\*\}\),\\nabla\_\{\\theta\}f\(\\theta\)\\right\\rangle,\(3\)A key design choice is the attribution targetff\.[Hu et al\. \(2025\)](https://arxiv.org/html/2610.28883#bib.bib42)propose to leverage a policy\-gradient surrogate for downstream return, defined on a reference validation distribution\. Given a reference policyπref\\pi^\{\\mathrm\{ref\}\}and a reference advantage estimatorA^ref\\hat\{A\}^\{\\mathrm\{ref\}\},[Hu et al\. \(2025\)](https://arxiv.org/html/2610.28883#bib.bib42)define

freturn​\(θ\):=𝔼τ∼πref,\(s,a\)∼τ​\[log⁡πθ​\(a∣s\)​A^ref​\(s,a\)\]\.f^\{\\mathrm\{return\}\}\(\\theta\):=\\mathbb\{E\}\_\{\\tau\\sim\\pi^\{\\mathrm\{ref\}\},\\,\(s,a\)\\sim\\tau\}\\big\[\\log\\pi\_\{\\theta\}\(a\\mid s\)\\,\\hat\{A\}^\{\\mathrm\{ref\}\}\(s,a\)\\big\]\.\(4\)
The target gives the attribution score a direct behavioral interpretation: a positive score means the sample pushes the policy toward higher surrogate return, whereas a negative score suggests that removing the sample may help\.

##### Discrepancy between short\-term and long\-term influence\.

A major limitation of local data attribution is its myopic view of data influence\. To illustrate this point, we empirically estimate the data influence111We estimate data influence by removing the sample from the training trajectory and measuring the resulting change in the final attribution target, as formally defined in Section[3\.1](https://arxiv.org/html/2610.28883#S3.SS1)\.inLunarLander\([Brockman et al\., 2016](https://arxiv.org/html/2610.28883#bib.bib13)\), and track how the influence of the same sample changes as the*horizon*HHgrows, where horizon means the number of training rounds over which the sample’s effect is propagated and evaluated\. As shown in Figure[2](https://arxiv.org/html/2610.28883#S2.F2), a good sample that stabilizes the lander may appear harmful from a short\-term viewpoint even though it improves downstream behavior and is beneficial in the long run, and vice versa\. Systematically, we also observed a bias induced by the local viewpoint\. In particular, local attribution tends to overestimate the long\-term value of late\-episode samples, those from late time steps in an episode\.

Figure 2:Left: two LunarLander samples that appear harmful locally but are beneficial in the long run\. Right: fraction of late\-episode samples labeled harmful vs\. attribution horizonHH\.
##### Two\-fold data influence in online RL\.

In online RL, the influence of a training sample is not limited to its direct effect on a parameter update\. A rollout affects learning through two coupled channels, as demonstrated in Figure[1](https://arxiv.org/html/2610.28883#S2.F1)\. First, it influences the checkpoint by contributing to the gradient update that produces the next parameter\. This is the usualoptimization effectof data: removing one sample perturbs the parameter trajectory and therefore the final checkpoint\. Second, the same rollout influences future data sampling\. Since subsequent rollout buffers are collected by policies obtained from earlier checkpoints, perturbing one training sample changes not only the next iterate but also the state\-action distribution from which later data are drawn\. This is referred to as thesampling effect: a sample influences future learning by altering the distribution of later data\. Hence, in on\-policy RL, data influence is inherently two\-fold: it acts on both the optimization path and the sampling path\. Moreover, the two effects are generally entangled rather than separable: a perturbation to one sample changes the data distribution of future rollouts, and this distributional shift in turn alters the subsequent optimization trajectory\.

## 3Measuring Data Influence in Online RL

In this section, we systematically formulate and approximate data influence in online reinforcement learning\. We first introduce a leave\-one\-out counterfactual objective, RL\-LOO, which measures the long\-term effect of removing a training occurrence\. Then, we derive a first\-order influence estimator, RL\-IF, that approximates this counterfactual without retraining by propagating perturbations through the training trajectory\. The resulting recursion naturally decomposes data influence into the aforementioned optimization effect and sampling effect, and admits theoretical guarantees under suitable smoothness assumptions\. Finally, we analyze both the effectiveness of RL\-IF and the relative contributions of these two effects through theoretical analysis and empirical evaluation\.

### 3\.1Problem Formulation

Following prior work, we study data attribution in an online PPO\-style training pipeline, and extend it beyond individual RL training rounds\. LetθT\(M\)\\theta\_\{T\}^\{\(M\)\}denote the final policy parameter after all rounds of training\. Our goal is to systematically measure how one training sample would affect a downstream target after subsequent online updates have taken place\.

We formalize this through a leave\-one\-out \(LOO\)\([Koh and Liang, 2017](https://arxiv.org/html/2610.28883#bib.bib29)\)counterfactual specific to RL, or*RL\-LOO*, following a similar setup in supervised learning literature\([Hara et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib32);[Wang et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib34)\)\. The unit of attribution is a particular*occurrence*of a sample in a training minibatch, written asz∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}, which specifies both the sample identity and the minibatch in which it is used\. Letθ~T\(M,−z∗\)\\widetilde\{\\theta\}\_\{T\}^\{\(M;\-z^\{\*\}\)\}denote the final parameter obtained by removing this occurrence from the training trajectory and replaying the remaining optimization process, including the induced changes in future sampling\. For a downstream target functionff, RL\-LOO is defined as

RL​\-​LOO\(k,l\)​\(z∗,f\):=f⁡\(θ~T\(M,−z∗\)\)−f⁡\(θT\(M\)\)\.\\mathrm\{RL\}\\text\{\-\}\\mathrm\{LOO\}^\{\(k,l\)\}\(z^\{\*\};f\):=f\(\\widetilde\{\\theta\}\_\{T\}^\{\(M;\-z^\{\*\}\)\}\)\-f\(\\theta\_\{T\}^\{\(M\)\}\)\.This quantity measures the effect of removing one training occurrence on a downstream target after subsequent training\. As for the attribution target, we adopt the negative of the target defined in \([4](https://arxiv.org/html/2610.28883#S2.E4)\), so that negative scores identify occurrences whose removal improves the surrogate objective\. In the non\-local setting, the reference policy is naturally chosen as the final policyπθT\(M\)\\pi\_\{\\theta\_\{T\}\}^\{\(M\)\}, so that the target measures how removing one training occurrence changes the surrogate return of the final policy on its own induced distribution\.A^ref\\hat\{A\}^\{\\mathrm\{ref\}\}can be approximated using the corresponding critic and the same GAE procedure and hyperparameters as PPO\. When exact advantages are used, we show that RL\-LOO with this target serves as a local proxy for the change in final expected return in Appendix[A\.4](https://arxiv.org/html/2610.28883#A1.SS4)\.

### 3\.2Approximating Leave\-One\-Out in RL

We now derive a first\-order estimator,*RL\-IF*\(RL InFluence\), for approximating RL\-LOO\. At a high level, RL\-LOO asks how the final policy would change if a training sample were removed from a rollout\-based training trajectory\. Rather than explicitly retraining the policy, RL\-IF approximates this counterfactual effect through first\-order perturbation analysis\.

Consider the RL training process in \([2](https://arxiv.org/html/2610.28883#S2.E2)\)\. Letz∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}be a sample removed from update\(k,l\)\(k,l\), and letθt\(j\)​\(ε\)\\theta\_\{t\}^\{\(j\)\}\(\\varepsilon\)denote the perturbed training trajectory obtained by continuously downweightingz∗z^\{\*\}by a scalarε\\varepsilon\. A first\-order Taylor expansion gives

RL​\-​LOO\(k,l\)​\(z∗,f\)≈∇θf​\(θT\(M\)\)⊤​ΔT\(M\),Δt\(j\):=∂θt\(j\)​\(ε\)∂ε\|ε=0,\\mathrm\{RL\}\\text\{\-\}\\mathrm\{LOO\}^\{\(k,l\)\}\(z^\{\*\};f\)\\approx\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\},\\qquad\\Delta\_\{t\}^\{\(j\)\}:=\\left\.\\frac\{\\partial\\theta\_\{t\}^\{\(j\)\}\(\\varepsilon\)\}\{\\partial\\varepsilon\}\\right\|\_\{\\varepsilon=0\},\(5\)which motivates the first\-order estimator

RL​\-​IF\(k,l\)​\(z∗,f\):=∇θf​\(θT\(M\)\)⊤​ΔT\(M\)\.\\mathrm\{RL\}\\text\{\-\}\\mathrm\{IF\}^\{\(k,l\)\}\(z^\{\*\};f\):=\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}\.
The key step is to characterize the perturbation trajectoryΔt\(j\)\\Delta\_\{t\}^\{\(j\)\}\. In online RL, perturbations propagate through two distinct channels, the*optimization effect*and the*sampling effect*, respectively\. To unify the effects from both channels, we define the stage update operator

Ft\(j\)​\(ϑ,β\):=ϑ−ηtb​∑x∼St\(j\)​\(β\)∇θℓ​\(ϑ,x\),St\(j\)​\(β\):=\{Dt\(j\),if​t=k​and​j\>l,Pt\(⋅∣β\),if​t≥k\+1\.F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\):=\\vartheta\-\\frac\{\\eta\_\{t\}\}\{b\}\\sum\_\{x\\sim S\_\{t\}^\{\(j\)\}\(\\beta\)\}\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\),\\quad S\_\{t\}^\{\(j\)\}\(\\beta\):=\\begin\{cases\}D\_\{t\}^\{\(j\)\},&\\text\{if \}t=k\\text\{ and \}j\>l,\\\\ P\_\{t\}\(\\cdot\\mid\\beta\),&\\text\{if \}t\\geq k\+1\.\\end\{cases\}\(6\)whereϑ\\varthetadenotes the parameter being updated andβ\\betadenotes the policy parameter controlling data collection\. Differentiating the update recursion with respect toε\\varepsilonyields

Δt\(j\)=∂Ft\(j\)∂ϑ​Δt\(j−1\)\+∂Ft\(j\)∂β​Δt\(0\),\\Delta\_\{t\}^\{\(j\)\}=\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\vartheta\}\\Delta\_\{t\}^\{\(j\-1\)\}\+\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\beta\}\\Delta\_\{t\}^\{\(0\)\},where the resulting recursion naturally distinguishes the two channels: the first term associated with the current parametersϑ\\varthetacaptures the optimization effect, and the second term related to the sampling parametersβ\\betacaptures the sampling effect\. With further simplification, the resulting first\-order perturbation recursion is summarized below\.

Proposition 3\.1:First\-order perturbation recursion for RL\-IFLetΔt\(j\)\\Delta\_\{t\}^\{\(j\)\}denote the perturbation induced by downweighting samplez∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}\. The perturbation is initialized asΔk\(l\)=ηkb​∇θℓ​\(θk\(l−1\),z∗\),\\Delta\_\{k\}^\{\(l\)\}=\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\),and evolves according toΔt\(j\)=\{At\(j\)​Δt\(j−1\),ift=k,j\>l,At\(j\)​Δt\(j−1\)\+Bt\(j\)​Δt\(0\),if​t≥k\+1,\\Delta\_\{t\}^\{\(j\)\}=\\begin\{cases\}A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\},&\\text\{if \}t=k,\\ j\>l,\\\\\[4\.0pt\] A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(0\)\},&\\text\{if \}t\\geq k\+1,\\end\{cases\}\(7\)whereAt\(j\):=I−ηtbHt\(j\),Bt\(j\):=−ηtb∑x∈Dt\(j\)∇θℓ\(θt\(j−1\),x\)∇θlogPt\(x∣θt\)⊤\.A\_\{t\}^\{\(j\)\}:=I\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\},\\qquad B\_\{t\}^\{\(j\)\}:=\-\\frac\{\\eta\_\{t\}\}\{b\}\\sum\_\{x\\in D\_\{t\}^\{\(j\)\}\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{t\}^\{\(j\-1\)\},x\)\\nabla\_\{\\theta\}\\log P\_\{t\}\(x\\mid\\theta\_\{t\}\)^\{\\top\}\.HereAt\(j\)A\_\{t\}^\{\(j\)\}andBt\(j\)B\_\{t\}^\{\(j\)\}correspond to the optimization and sampling effects, respectively\.

The recursion formula of RL\-IF in Proposition[3\.2](https://arxiv.org/html/2610.28883#S3.SS2)propagates the first\-order effect of removingz∗z^\{\*\}through both later optimization steps and future sampling decisions\. The termAt\(j\)​Δt\(j−1\)A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\}captures the*optimization effect*, whileBt\(j\)​Δt\(0\)B\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(0\)\}captures the additional*sampling effect*caused by policy\-dependent data collection\. Notably, the sampling effect vanishes when the perturbation remains within the same training round\. In practice, we use∇log⁡πθt​\(a∣s\)\\nabla\\log\\pi\_\{\\theta\_\{t\}\}\(a\\mid s\)to approximate∇log⁡Pt​\(\(s,a\)∣θt\)\\nabla\\log P\_\{t\}\(\(s,a\)\\mid\\theta\_\{t\}\)\. Note that both matrix multiplications in this formula can be tackled efficiently: multiplying byAt\(j\)A\_\{t\}^\{\(j\)\}can be computed using Hessian Vector Product \(HVP\), and multiplying byBt\(j\)B\_\{t\}^\{\(j\)\}is essentially two vector products\. We further provide a detailed derivation of the recursion, together with a first\-order approximation guarantee for RL\-IF under smoothness assumptions, in Appendix[A\.1](https://arxiv.org/html/2610.28883#A1.SS1)and Appendix[B\.1](https://arxiv.org/html/2610.28883#A2.SS1)\.

### 3\.3Sampling vs\. Optimization Effects in RL\-IF

A more detailed analysis of RL\-IF can be obtained by unrolling the recursion and separating the optimization and sampling effects\. In particular, the recursion in \([7](https://arxiv.org/html/2610.28883#S3.E7)\) admits a compact closed form\.

RL​\-​IF\(k,l\)​\(z∗,f\)=∇θf​\(θT\(M\)\)⊤​\(∏t=Tk\+1Ct\)​\(∏j=Ml\+1Ak\(j\)\)​ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\mathrm\{RL\}\\text\{\-\}\\mathrm\{IF\}^\{\(k,l\)\}\(z^\{\*\};f\)=\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\left\(\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\\right\)\\left\(\\prod\_\{j=M\}^\{l\+1\}A\_\{k\}^\{\(j\)\}\\right\)\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.\(8\)Here, for each staget\>kt\>k,

At:=∏j=M1At\(j\),Ct:=At\+∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\)\.A\_\{t\}:=\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\},\\qquad C\_\{t\}:=A\_\{t\}\+\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\.
Equation \([8](https://arxiv.org/html/2610.28883#S3.E8)\) is the closed\-form solution of the recursion\. Here, the product ofAtA\_\{t\}captures the accumulated*optimization effect*, while the terms involvingBt\(j\)B\_\{t\}^\{\(j\)\}capture the accumulated*sampling effect*\. WhenBt\(q\)=0B\_\{t\}^\{\(q\)\}=0, the expression reduces to the optimization\-only case\. The following theorem formalizes the gap between the optimization\-only and full sampling\-aware effects\.

Theorem 3\.1:Accumulated relative gapAssume thatℓ⁡\(θ,x\)\\ell\(\\theta;x\)is twice differentiable, that its gradient norm is bounded byGG, that its Hessian norm is bounded byΛ\\Lambda\. Assume in addition that for each future staget\>kt\>k, the sampling familyPt\(⋅∣θ\)P\_\{t\}\(\\cdot\\mid\\theta\)is KL\-smooth with score second moment bounded byLmaxL\_\{\\max\}\. Assume the learning rate takes the harmonic formηt=η0M⁡\(t−k\)\\eta\_\{t\}=\\frac\{\\eta\_\{0\}\}\{M\(t\-k\)\}withη0​Λ<1\\eta\_\{0\}\\Lambda<1\. Then, for anyT\>kT\>kwithH:=T−kH:=T\-k, the accumulated gap satisfies‖∏t=Tk\+1Ct−∏t=Tk\+1At‖2≤\(eα​Hα−1\)​∏t=k\+1T‖At‖2,\\left\\\|\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\-\\prod\_\{t=T\}^\{k\+1\}A\_\{t\}\\right\\\|\_\{2\}\\leq\(e^\{\\alpha\}H^\{\\alpha\}\-1\)\\prod\_\{t=k\+1\}^\{T\}\\\|A\_\{t\}\\\|\_\{2\},\(9\)whereα:=G​Lmax​η0\(1−η0​Λ\)2\\alpha:=\\frac\{G\\sqrt\{L\_\{\\max\}\}\\,\\eta\_\{0\}\}\{\(1\-\\eta\_\{0\}\\Lambda\)^\{2\}\}\.

The bound in \([9](https://arxiv.org/html/2610.28883#S3.E9)\) shows that the sampling effect remains a*relative perturbation*to the optimization effect\. In particular, ifη0​Λ≪1\\eta\_\{0\}\\Lambda\\ll 1, the accumulated gap overHHstages grows only logarithmically

eα​Hα−1≈α⁡\(1\+log⁡H\)e^\{\\alpha\}H^\{\\alpha\}\-1\\approx\\alpha\(1\+\\log H\)
Table 1:Linear data modeling scores of attribution methods onAnt\-v4andHalfCheetah\-v4\.Hence, in this regime, the accumulated gap remains well\-conditioned over long horizons, and the sampling effect acts as a controlled perturbation to the optimization dynamics\. Accordingly, we define the optimization\-only variantRL\-IF\-optas the estimator obtained by settingBt\(j\)=0B\_\{t\}^\{\(j\)\}=0in the recursion, or equivalently by dropping the sampling terms in the closed\-form operatorCtC\_\{t\}\. The original estimator that retains both optimization and sampling terms is denotedRL\-IF\-full\. The gap suggests that RL\-IF is often dominated by its optimization component, making RL\-IF\-opt a compelling simplification in practice\. The proof of Theorem[3\.3](https://arxiv.org/html/2610.28883#S3.SS3)is deferred to Appendix[B\.3](https://arxiv.org/html/2610.28883#A2.SS3)\.

##### Experimental comparison\.

To further validate the theoretical findings, we evaluate attribution quality using*Linear Data Modeling Scores*\(LDS\)\([Park et al\., 2023](https://arxiv.org/html/2610.28883#bib.bib33)\)\. LDS measures the Spearman correlation between aggregated attribution scores and the observed effect of removing subsets of training data under retraining\. We compare the two variants of RL\-IF, along with TracIn\([Pruthi et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib30)\), an attribution method that does not account for optimization or sampling dynamics, and a random baseline\. We consider KL\-regularized PPO on two consecutive training rounds, where retraining is performed with future rollouts resampled using the updated policy\. The results are shown in Table[1](https://arxiv.org/html/2610.28883#S3.T1)\. Both RL\-IF\-opt and RL\-IF\-full substantially outperform local attribution methods such as TracIn, while exhibiting nearly identical performance to each other\. This small gap suggests that, empirically, the sampling effect contributes only marginally beyond the optimization effect\. Additional setup and evaluation details are provided in Appendix[D\.1](https://arxiv.org/html/2610.28883#A4.SS1)\.

## 4Data Selection with RL\-IF

We demonstrate a direct downstream application of RL\-IF in online RL: using RL\-IF attribution scores to guide data selection over multiple training rounds\. The goal is to identify and remove training samples that are harmful to downstream performance, thereby improving both compute efficiency and sample efficiency\. We build our data selection algorithm on KL\-regularized PPO, since the clipping objective in \([1](https://arxiv.org/html/2610.28883#S2.E1)\) is piecewise\-defined and introduces non\-smooth behavior at the clipping boundary, which is less compatible with the first\-order approximation used in RL\-IF\. Our source code is available at[https://github\.com/sx\-liu/RL\-IF](https://github.com/sx-liu/RL-IF)\.

Algorithm 1RINSE: RL\-IF\-based Sample Selection1:Initial policy

πθk\\pi\_\{\\theta\_\{k\}\}, horizon

HH, discard ratio

ρ\\rho
2:

θ←θk\\theta\\leftarrow\\theta\_\{k\}
3:for

h=1,…,Hh=1,\\dots,Hdo

4:

Dk\+h−1←CollectRollout​\(θ\)D\_\{k\+h\-1\}\\leftarrow\\textsc\{CollectRollout\}\(\\theta\)
5:

r​\(h\)←Evaluate​\(Dk\+h−1\)r\(h\)\\leftarrow\\textsc\{Evaluate\}\(D\_\{k\+h\-1\}\)
6:if

h\>1h\>1and

r⁡\(h−1\)\>r⁡\(h\)r\(h\-1\)\>r\(h\)then

7:break

8:endif

9:

H^←h\\widehat\{H\}\\leftarrow h
10:

θ←TrainRollout​\(θ,Dk\+h−1\)\\theta\\leftarrow\\textsc\{TrainRollout\}\(\\theta;D\_\{k\+h\-1\}\)
11:endfor

12:Compute optimization\-only RL\-IF\-opt scores

\{s⁡\(x\)\}\\\{s\(x\)\\\}for all samples

x∈⋃t=kk\+H^−1Dtx\\in\\bigcup\_\{t=k\}^\{k\+\\widehat\{H\}\-1\}D\_\{t\}
13:

ℛ←NegBottomFraction​\(\{s​\(x\)\},ρ\)\\mathcal\{R\}\\leftarrow\\textsc\{NegBottomFraction\}\(\\\{s\(x\)\\\},\\rho\)
14:Construct filtered datasets

D~t=Dt∖ℛ\\widetilde\{D\}\_\{t\}=D\_\{t\}\\setminus\\mathcal\{R\}for

t=k,…,k\+H^−1t=k,\\ldots,k\+\\widehat\{H\}\-1
15:

θ←θk\\theta\\leftarrow\\theta\_\{k\}
16:for

h=1,…,H^h=1,\\dots,\\widehat\{H\}do

17:

θ←TrainRollout​\(θ,D~k\+h−1\)\\theta\\leftarrow\\textsc\{TrainRollout\}\(\\theta;\\widetilde\{D\}\_\{k\+h\-1\}\)
18:endfor

19:return

πθk\+H^\\pi\_\{\\theta\_\{k\+\\widehat\{H\}\}\}

RINSE: RL\-IF\-based Sample Selection\.The RL\-IF formulation naturally leads to a data\-selection algorithm, which we callRINSE\(RL\-INfluence\-basedSample sElection\), over a finite horizon window\. Suppose we aim to improve PPO training forHHrounds, starting from roundkk\. We first run the standard PPO pipeline forHHsubsequent rounds, obtaining policiesπθk\+1,…,πθk\+H\\pi\_\{\\theta\_\{k\+1\}\},\\ldots,\\pi\_\{\\theta\_\{k\+H\}\}and the corresponding training trajectory\{Dt\(j\)\}t=kk\+H−1\\\{D\_\{t\}^\{\(j\)\}\\\}\_\{t=k\}^\{k\+H\-1\}\. This rollout trajectory provides the reference path on which long\-horizon influence is evaluated\.

Given this trajectory, we estimate the influence of samples in\{Dt\}t=kk\+H−1\\\{D\_\{t\}\\\}\_\{t=k\}^\{k\+H\-1\}using RL\-IF, with the final policyπθk\+H\\pi\_\{\\theta\_\{k\+H\}\}in this window as the reference policy in \([4](https://arxiv.org/html/2610.28883#S2.E4)\)\. Because the optimization\-only approximation achieves comparable empirical performance at substantially lower cost, RINSE uses the optimization\-only variant, RL\-IF\-opt, in the selection procedure\. We rank samples by their attribution scores, remove those with the most negative influence based on discard ratioρ\\rho, and then retrain on the filtered datasets to obtain updated policiesπθk\+1′,…,πθk\+H′\\pi\_\{\\theta\_\{k\+1\}^\{\\prime\}\},\\ldots,\\pi\_\{\\theta\_\{k\+H\}^\{\\prime\}\}\. Since this retraining stage is inherently off\-policy, we use an adjusted retraining objective,

LKL​\-​Retrain\(θ\):=𝔼\[rt\(θ\)A^t−βKL\(πθt′\(⋅∣st\)∥πθ\(⋅∣st\)\)\],L^\{\\mathrm\{KL\\text\{\-\}Retrain\}\}\(\\theta\):=\\mathbb\{E\}\\big\[r\_\{t\}\(\\theta\)\\hat\{A\}\_\{t\}\-\\beta\\,\\mathrm\{KL\}\\big\(\\pi\_\{\\theta\_\{t\}^\{\\prime\}\}\(\\cdot\\mid s\_\{t\}\)\\,\\\|\\,\\pi\_\{\\theta\}\(\\cdot\\mid s\_\{t\}\)\\big\)\\big\],which limits policy drift from the reference policies along the retraining trajectory\.

Design of attribution horizon\.The effectiveness of attribution\-based selection depends on the attribution horizon and the choice of reference policy\. Short horizons lead to overly myopic attribution that reduces to local effects, while excessively long horizons introduce severe off\-policy mismatch between the selected data and later training dynamics\. Moreover, the reference policy must be sufficiently strong to provide a reliable attribution target\.

To balance these considerations, we adopt awindow\-based data selectionstrategy, which partitions training into consecutive windows and performs attribution within each window independently\. This maintains proximity to the current training dynamics while still capturing non\-local effects over multiple future updates\.

We further use anadaptive horizon selectionprocedure\. Given a maximum window sizeHmaxH\_\{\\max\}, we progressively extend the horizon under standard training and monitor the mean episode returnr⁡\(h\)r\(h\)of policyπθk\+h−1\\pi\_\{\\theta\_\{k\+h\-1\}\}\. We stop when eitherh=Hmaxh=H\_\{\\max\}or performance ceases to improve, i\.e\.,r⁡\(h−1\)\>r⁡\(h\)r\(h\-1\)\>r\(h\), and select the largest valid horizon for attribution and data selection\.

The complete procedure is summarized in Algorithm[1](https://arxiv.org/html/2610.28883#alg1), while ablations of our design choices are presented in Appendix[D\.3](https://arxiv.org/html/2610.28883#A4.SS3)\.

\(a\)Test returns over training rounds\.StandardRandomIIFRINSE
\(b\)Relative efficiency and peak reward\. TE/SE are normalized to standard PPO; lower is better\.

Figure 3:Data\-selection results under rollout filtering\. \(a\) Test returns over training rounds\. \(b\) Relative time efficiency, sample efficiency, and peak mean return across environments\. RINSE largely improves return and efficiency over standard training and local baselines\.##### Experimental setup\.

We evaluate our method on six standard RL benchmarks spanning continuous control and navigation tasks from the Gym/Gymnasium ecosystem, including MuJoCo domains\([Todorov et al\., 2012](https://arxiv.org/html/2610.28883#bib.bib12);[Brockman et al\., 2016](https://arxiv.org/html/2610.28883#bib.bib13);[Towers et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib14)\)\. We use PPO with a KL\-regularized objective as the underlying training algorithm and fix the maximum horizon toHmax=5H\_\{\\max\}=5without tuning\. We compareRINSEagainst three baselines: standard PPO,Randomfiltering, andIIF\([Hu et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib42)\), a local\-attribution filtering method based on TracIn scores\. For fairness, Random and IIF use the same discard ratioρ\\rhoasRINSE\.

We report bothtime efficiency \(TE\)andsample efficiency \(SE\)relative to standard PPO\. For each method, we measure the runtime and the amount of training data required to reach a set of target performance levels, normalized such that the PPO baseline corresponds to100%100\\%\. We also report full training curves\. Additional setup and metric details are provided in Appendix[C\.1](https://arxiv.org/html/2610.28883#A3.SS1)and Appendix[D\.4](https://arxiv.org/html/2610.28883#A4.SS4)\.

##### Results\.

Figure[3](https://arxiv.org/html/2610.28883#S4.F3)shows thatRINSEconsistently outperforms both standard PPO and IIF\. AlthoughRINSEinvokesTRAINROLLOUTtwice per round and incurs additional influence\-computation overhead, it reduces the number of required rollouts by an average of42\.8%42\.8\\%to achieve the same performance as standard training\. Consequently,RINSEremains more efficient than the local\-attribution baseline on nearly all environments\.

RINSEachieves the best sample efficiency and highest peak return across all six tasks, with especially large improvements on challenging continuous\-control environments such asAntandBipedalWalker\. These results suggest that RL\-IF\-opt captures the most useful component of long\-term data influence for multi\-round training improvement\. We further report the wall\-clock breakdown in Appendix[D\.4\.1](https://arxiv.org/html/2610.28883#A4.SS4.SSS1)\.

### 4\.1Extension to RLHF

We also consider an RLHF\-style toxicity\-mitigation task\. Starting from GPT\-Neo 2\.7B\([Black et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib23)\), each training round collects prompt\-response trajectories from the current policy, assigns a toxicity\-based reward to each response using a pretrained classifier\([Vidgen et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib26);[Gehman et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib25)\), and then updates the policy following standard RLHF pipelines\([Ouyang et al\., 2022](https://arxiv.org/html/2610.28883#bib.bib21)\)\. We use the KL\-regularized PPO objective described in Section[2\.1](https://arxiv.org/html/2610.28883#S2.SS1)\. For additional efficiency, we use the Generalized Gauss–Newton matrix \(GGN\)\([Martens, 2020](https://arxiv.org/html/2610.28883#bib.bib20);[Wang et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib34)\)to approximate the Hessian term in \([7](https://arxiv.org/html/2610.28883#S3.E7)\), completely reducing the HVP to efficient vector product operations\. Further details are in Appendix[C\.2](https://arxiv.org/html/2610.28883#A3.SS2)\.

Figure 4:Toxicity\-mitigation results in the RLHF setting\.##### Results\.

Figure[4](https://arxiv.org/html/2610.28883#S4.F4)shows the toxicity trajectory during training: RINSE achieves lower toxicity probability than the baselines across most of training\. Notably, in terms of relative TE on reducing test toxicity, RINSE achieves an average training speedup of more than3×3\\timesover the standard training pipeline and more than2×2\\timesover IIF\. We also provide detailed training details, additional analysis and runtime breakdown in Appendix[D\.5](https://arxiv.org/html/2610.28883#A4.SS5)\. These results suggest that RINSE extends naturally beyond continuous\-control benchmarks to large\-scale language\-model alignment\.

## 5Related Work

##### Training\-data attribution and data valuation\.

Our work is closely related to the literature on training\-data attribution and data valuation\([Deng et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib35)\)\. Classical influence functions approximate leave\-one\-out effects through second\-order sensitivity analysis\([Koh and Liang, 2017](https://arxiv.org/html/2610.28883#bib.bib29)\), while later methods improve scalability by tracing gradient alignment along optimization trajectories, as in TracIn\([Pruthi et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib30)\), by using representer\-point decompositions\([Yeh et al\., 2018](https://arxiv.org/html/2610.28883#bib.bib31)\), or by designing scalable estimators for modern large\-scale models\([Park et al\., 2023](https://arxiv.org/html/2610.28883#bib.bib33)\)\. Recent work further studies attribution in temporally dependent or sequential training settings\([Wang et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib34)\)\. A closely related line studies*data valuation*, where training examples are assigned value according to their contribution to model quality or downstream utility\([Ghorbani and Zou, 2019](https://arxiv.org/html/2610.28883#bib.bib37);[Jia et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib38);[Kwon and Zou, 2022](https://arxiv.org/html/2610.28883#bib.bib39)\)\. Other work uses per\-example influence signals for data cleaning, robust reweighting, and data selection\([Hara et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib32);[Xia et al\., 2024](https://arxiv.org/html/2610.28883#bib.bib40);[Wang et al\., 2024](https://arxiv.org/html/2610.28883#bib.bib41)\)\.

Most existing attribution methods assume a fixed training distribution\. In contrast, our work studies attribution in online RL, where the training distribution evolves with the policy itself\. We focus specifically on*non\-local*attribution: instead of measuring only the immediate effect of a sample on a nearby update, we study how perturbations propagate through future optimization and future data collection over an extended training trajectory\.

##### Data\-centric perspectives in RL\.

A growing line of work studies RL from a data\-centric perspective, asking which trajectories, transitions, or training contexts are most useful for improving policy learning\. In off\-policy RL, this question appears naturally in experience replay and replay prioritization, where informative transitions are replayed more frequently to improve sample efficiency\([Schaul et al\., 2016](https://arxiv.org/html/2610.28883#bib.bib9)\)\. Related ideas also appear in curriculum learning and curriculum design, where training tasks or experiences are structured adaptively to improve learning dynamics\([Florensa et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib10);[Jiang et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib11)\)\. Recent work also explores trajectory filtering, selective data reuse, and replay\-buffer management for improving robustness and efficiency in RL training\.

Our work is most closely related to recent attribution methods for online RL\. In on\-policy training, trajectories are generated by the current policy and immediately reused for optimization\([Williams, 1992](https://arxiv.org/html/2610.28883#bib.bib2);[Sutton et al\., 1999](https://arxiv.org/html/2610.28883#bib.bib3);[Schulman et al\., 2015](https://arxiv.org/html/2610.28883#bib.bib4);[Schulman et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib5)\)\. Consequently, perturbing a single training occurrence can influence not only later parameters, but also the future rollout distribution induced by the updated policy\.[Hu et al\. \(2025\)](https://arxiv.org/html/2610.28883#bib.bib42)take an important first step in this direction by proposing a local attribution framework for PPO that measures the influence of rollout records relative to a nearby checkpoint within a fixed rollout buffer\. Our work builds directly on this line, but studies a fundamentally different target: instead of estimating the immediate local effect of a sample, we study its*non\-local*influence on future learning outcomes over subsequent training rounds\.

## 6Conclusion and Limitations

We study data attribution in online reinforcement learning, where the training data are generated by the evolving policy rather than drawn from a fixed distribution\. To capture this setting, we formulate a trajectory\-level leave\-one\-out quantity,*RL\-LOO*, and derive a first\-order estimator,*RL\-IF*, that propagates the effect of one training occurrence through both later optimization and future sampling\. This yields a counterfactual perspective on data value in online RL that goes beyond local attribution\. Empirically, the resulting optimization\-only approximation,*RL\-IF\-opt*, is already effective enough to support a practical filtering method,RINSE, which improves efficiency and final performance on standard benchmarks and in an RLHF\-style toxicity\-mitigation setting\.

Our work also has several limitations\. First, the current development is specialized to PPO\-style on\-policy training\([Schulman et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib5)\)and to a first\-order approximation of the leave\-one\-out trajectory\. Extending the framework to other online RL algorithms, including closely related trust\-region methods such as TRPO\([Schulman et al\., 2015](https://arxiv.org/html/2610.28883#bib.bib4)\)and other settings with substantially different sampling or update structures, remains an important direction\. Second, our derivation is based on SGD\-style\([Amari, 1993](https://arxiv.org/html/2610.28883#bib.bib18)\)updates, whereas modern deep RL and RLHF pipelines often rely on adaptive optimizers such as Adam\([Kingma and Ba, 2015](https://arxiv.org/html/2610.28883#bib.bib19)\)\. Although the method works empirically in our Adam\-based experiments, a derivation that explicitly accounts for adaptive preconditioning would make the estimator more principled\([Deng et al\., 2026](https://arxiv.org/html/2610.28883#bib.bib36)\)\. Finally, our analysis and experiments suggest that the sampling term is often smaller than the optimization term in the regimes we study, but this need not hold universally; understanding when sampling effects become dominant is an open question\.

### Acknowledgements

This project was partially supported by an ACCESS allocation CIS260068\. JM would like to acknowledge support by NSF CAREER Award No\. 2542856\. YH and HZ are partially supported by an NSF grant No\. 2416897 and an NSF CAREER Award No\. 2442290\.

## References

- Amari \(1993\)S\. AmariBackpropagation and stochastic gradient descent method\.Neurocomputing5\(4–5\),pp\. 185–196\.Cited by:[§6](https://arxiv.org/html/2610.28883#S6.p2.1)\.
- Andrychowiczet al\.\(2020\)M\. Andrychowicz, B\. Baker, M\. Chociej, R\. Józefowicz, B\. McGrew, J\. Pachocki, A\. Petron, M\. Plappert, G\. Powell, A\. Ray,et al\.Learning dexterous in\-hand manipulation\.The International Journal of Robotics Research39\(1\),pp\. 3–20\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Blacket al\.\(2021\)S\. Black, L\. Gao, P\. Wang, C\. Leahy, and S\. BidermanGPT\-Neo: large scale autoregressive language modeling with Mesh\-TensorFlow\.Note:ZenodoExternal Links:[Document](https://dx.doi.org/10.5281/zenodo.5297715)Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px1.p1.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1)\.
- Brockmanet al\.\(2016\)G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. ZarembaOpenAI Gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[Table 2](https://arxiv.org/html/2610.28883#A3.T2),[Table 2](https://arxiv.org/html/2610.28883#A3.T2.4),[§2\.1](https://arxiv.org/html/2610.28883#S2.SS1.SSS0.Px3.p1.1),[§4](https://arxiv.org/html/2610.28883#S4.SS0.SSS0.Px1.p1.1)\.
- Dao \(2024\)T\. DaoFlashAttention\-2: faster attention with better parallelism and work partitioning\.InInternational Conference on Learning Representations,Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px4.p1.1)\.
- Denget al\.\(2026\)J\. Deng, P\. Hu, S\. Jin, H\. Lu, J\. T\. Wang, S\. Zhang, and J\. W\. MaHow faithful is trajectory\-based data attribution? Error sources, remedies, and practical guidelines\.arXiv preprint arXiv:2605\.18814\.Cited by:[§6](https://arxiv.org/html/2610.28883#S6.p2.1)\.
- Denget al\.\(2025\)J\. Deng, Y\. Hu, P\. Hu, T\. Li, S\. Liu, J\. T\. Wang, D\. Ley, Q\. Dai, B\. Huang, J\. Huang,et al\.A survey of data attribution: methods, applications, and evaluation in the era of generative AI\.Note:SSRN Electronic JournalExternal Links:[Document](https://dx.doi.org/10.2139/ssrn.5451054)Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Dulac\-Arnoldet al\.\(2019\)G\. Dulac\-Arnold, D\. Mankowitz, and T\. HesterChallenges of real\-world reinforcement learning\.arXiv preprint arXiv:1904\.12901\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Florensaet al\.\(2017\)C\. Florensa, D\. Held, M\. Wulfmeier, M\. Zhang, and P\. AbbeelReverse curriculum generation for reinforcement learning\.InConference on Robot Learning,pp\. 482–495\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p1.1)\.
- Gehmanet al\.\(2020\)S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. SmithRealToxicityPrompts: evaluating neural toxic degeneration in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2020,pp\. 3356–3369\.Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px2.p1.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1)\.
- Ghorbani and Zou \(2019\)A\. Ghorbani and J\. ZouData Shapley: equitable valuation of data for machine learning\.InInternational Conference on Machine Learning,pp\. 2242–2251\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Haraet al\.\(2019\)S\. Hara, A\. Nitanda, and T\. MaeharaData cleansing for models trained with SGD\.InAdvances in Neural Information Processing Systems,Vol\.32\.Cited by:[§3\.1](https://arxiv.org/html/2610.28883#S3.SS1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Hendersonet al\.\(2018\)P\. Henderson, R\. Islam, P\. Bachman, J\. Pineau, D\. Precup, and D\. MegerDeep reinforcement learning that matters\.InAAAI Conference on Artificial Intelligence,Vol\.32\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Huet al\.\(2022\)E\. J\. Hu, Y\. Shen, P\. Wallis, Z\. Allen\-Zhu, Y\. Li, S\. Wang, L\. Wang, and W\. ChenLoRA: low\-rank adaptation of large language models\.InInternational Conference on Learning Representations,Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px1.p1.1)\.
- Huet al\.\(2026\)P\. Hu, X\. Liu, Z\. M\. Mao, and J\. W\. MaDr\. Post\-Training: a data regularization perspective on LLM post\-training\.arXiv preprint arXiv:2605\.07063\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Huet al\.\(2025\)Y\. Hu, F\. Wu, H\. Ye, D\. Forsyth, J\. Zou, N\. Jiang, J\. W\. Ma, and H\. ZhaoA snapshot of influence: a local data attribution framework for online reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.38\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1),[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§2\.1](https://arxiv.org/html/2610.28883#S2.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2610.28883#S2.SS1.SSS0.Px2.p1.2),[§4](https://arxiv.org/html/2610.28883#S4.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p2.1)\.
- Jiaet al\.\(2019\)R\. Jia, D\. Dao, B\. Wang, F\. A\. Hubis, N\. Hynes, N\. M\. Gürel, B\. Li, C\. Zhang, D\. Song, and C\. J\. SpanosTowards efficient data valuation based on the Shapley value\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1167–1176\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Jianget al\.\(2021\)M\. Jiang, E\. Grefenstette, and T\. RocktäschelPrioritized level replay\.InInternational Conference on Machine Learning,pp\. 4940–4950\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p1.1)\.
- Kingma and Ba \(2015\)D\. P\. Kingma and J\. BaAdam: a method for stochastic optimization\.InInternational Conference on Learning Representations,Cited by:[§6](https://arxiv.org/html/2610.28883#S6.p2.1)\.
- Koh and Liang \(2017\)P\. W\. Koh and P\. LiangUnderstanding black\-box predictions via influence functions\.InInternational Conference on Machine Learning,pp\. 1885–1894\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§1](https://arxiv.org/html/2610.28883#S1.p3.1),[§3\.1](https://arxiv.org/html/2610.28883#S3.SS1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Kwon and Zou \(2022\)Y\. Kwon and J\. ZouBeta Shapley: a unified and noise\-reduced data valuation framework for machine learning\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 8780–8802\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Martens \(2020\)J\. MartensNew insights and perspectives on the natural gradient method\.Journal of Machine Learning Research21\(146\),pp\. 1–76\.Cited by:[§D\.5](https://arxiv.org/html/2610.28883#A4.SS5.SSS0.Px2.p2.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1)\.
- Ouyanget al\.\(2022\)L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray,et al\.Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1)\.
- Parket al\.\(2023\)S\. M\. Park, K\. Georgiev, A\. Ilyas, G\. Leclerc, and A\. MadryTRAK: attributing model behavior at scale\.InInternational Conference on Machine Learning,pp\. 27074–27113\.Cited by:[§3\.3](https://arxiv.org/html/2610.28883#S3.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Pauliet al\.\(2021\)A\. B\. Pauli, M\. Barrett, O\. Lacroix, and R\. HvingelbyDaNLP: an open\-source toolkit for Danish natural language processing\.InProceedings of the 23rd Nordic Conference on Computational Linguistics \(NoDaLiDa\),pp\. 460–466\.Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px5.p1.1)\.
- Pruthiet al\.\(2020\)G\. Pruthi, F\. Liu, S\. Kale, and M\. SundararajanEstimating training data influence by tracing gradient descent\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 19920–19930\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§3\.3](https://arxiv.org/html/2610.28883#S3.SS3.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Radfordet al\.\(2019\)A\. Radford, J\. Wu, R\. Child, D\. Luan, D\. Amodei, and I\. SutskeverLanguage models are unsupervised multitask learners\.Technical reportOpenAI\.Cited by:[§D\.5\.1](https://arxiv.org/html/2610.28883#A4.SS5.SSS1.p2.1)\.
- Raffinet al\.\(2021\)A\. Raffin, A\. Hill, A\. Gleave, A\. Kanervisto, M\. Ernestus, and N\. DormannStable\-Baselines3: reliable reinforcement learning implementations\.Journal of Machine Learning Research22\(268\),pp\. 1–8\.Cited by:[§C\.1](https://arxiv.org/html/2610.28883#A3.SS1.SSS0.Px2.p1.1)\.
- Schaulet al\.\(2016\)T\. Schaul, J\. Quan, I\. Antonoglou, and D\. SilverPrioritized experience replay\.InInternational Conference on Learning Representations,Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p1.1)\.
- Schulmanet al\.\(2015\)J\. Schulman, S\. Levine, P\. Abbeel, M\. Jordan, and P\. MoritzTrust region policy optimization\.InInternational Conference on Machine Learning,pp\. 1889–1897\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p2.1),[§6](https://arxiv.org/html/2610.28883#S6.p2.1)\.
- Schulmanet al\.\(2017\)J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. KlimovProximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§C\.1](https://arxiv.org/html/2610.28883#A3.SS1.SSS0.Px2.p1.1),[§2\.1](https://arxiv.org/html/2610.28883#S2.SS1.SSS0.Px1.p1.1),[§2\.1](https://arxiv.org/html/2610.28883#S2.SS1.SSS0.Px1.p1.2),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p2.1),[§6](https://arxiv.org/html/2610.28883#S6.p2.1)\.
- Schulman \(2020\)J\. SchulmanApproximating KL divergence\.Note:[http://joschu\.net/blog/kl\-approx\.html](http://joschu.net/blog/kl-approx.html)Accessed: 2026\-05\-06Cited by:[§C\.1](https://arxiv.org/html/2610.28883#A3.SS1.SSS0.Px2.p1.1),[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px4.p1.1)\.
- Shaoet al\.\(2024\)Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu,et al\.DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Suttonet al\.\(1999\)R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. MansourPolicy gradient methods for reinforcement learning with function approximation\.InAdvances in Neural Information Processing Systems,Vol\.12,pp\. 1057–1063\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p2.1)\.
- Todorovet al\.\(2012\)E\. Todorov, T\. Erez, and Y\. TassaMuJoCo: a physics engine for model\-based control\.InIEEE/RSJ International Conference on Intelligent Robots and Systems,pp\. 5026–5033\.Cited by:[Table 2](https://arxiv.org/html/2610.28883#A3.T2),[Table 2](https://arxiv.org/html/2610.28883#A3.T2.4),[§D\.1](https://arxiv.org/html/2610.28883#A4.SS1.SSS0.Px1.p1.1),[§1](https://arxiv.org/html/2610.28883#S1.p1.1),[§4](https://arxiv.org/html/2610.28883#S4.SS0.SSS0.Px1.p1.1)\.
- Towerset al\.\(2025\)M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. De Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Krimmel, A\. KG,et al\.Gymnasium: a standard interface for reinforcement learning environments\.InAdvances in Neural Information Processing Systems \(Datasets and Benchmarks Track\),Vol\.38\.Cited by:[Table 2](https://arxiv.org/html/2610.28883#A3.T2),[Table 2](https://arxiv.org/html/2610.28883#A3.T2.4),[§4](https://arxiv.org/html/2610.28883#S4.SS0.SSS0.Px1.p1.1)\.
- Vidgenet al\.\(2021\)B\. Vidgen, T\. Thrush, Z\. Waseem, and D\. KielaLearning from the worst: dynamically generated datasets to improve online hate detection\.InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing \(Volume 1: Long Papers\),pp\. 1667–1682\.Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px3.p1.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1)\.
- Wanget al\.\(2025\)J\. T\. Wang, D\. Song, J\. Zou, P\. Mittal, and R\. JiaCapturing the temporal dependence of training data influence\.InInternational Conference on Learning Representations,Cited by:[§3\.1](https://arxiv.org/html/2610.28883#S3.SS1.p2.1),[§4\.1](https://arxiv.org/html/2610.28883#S4.SS1.p1.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Wanget al\.\(2024\)J\. T\. Wang, T\. Wu, D\. Song, P\. Mittal, and R\. JiaGREATS: online selection of high\-quality data for LLM training in every iteration\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 131197–131223\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Williams \(1992\)R\. J\. WilliamsSimple statistical gradient\-following algorithms for connectionist reinforcement learning\.Machine Learning8\(3\),pp\. 229–256\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px2.p2.1)\.
- Wulczynet al\.\(2017\)E\. Wulczyn, N\. Thain, and L\. DixonEx machina: personal attacks seen at scale\.InProceedings of the 26th International Conference on World Wide Web,pp\. 1391–1399\.Cited by:[§C\.2](https://arxiv.org/html/2610.28883#A3.SS2.SSS0.Px5.p1.1)\.
- Xiaet al\.\(2024\)M\. Xia, S\. Malladi, S\. Gururangan, S\. Arora, and D\. ChenLESS: selecting influential data for targeted instruction tuning\.InInternational Conference on Machine Learning,pp\. 54104–54132\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p2.1),[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Yehet al\.\(2018\)C\. Yeh, J\. Kim, I\. E\. Yen, and P\. K\. RavikumarRepresenter point selection for explaining deep neural networks\.InAdvances in Neural Information Processing Systems,Vol\.31\.Cited by:[§5](https://arxiv.org/html/2610.28883#S5.SS0.SSS0.Px1.p1.1)\.
- Yu \(2018\)Y\. YuTowards sample efficient reinforcement learning\.InInternational Joint Conference on Artificial Intelligence,pp\. 5739–5743\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.
- Zhuet al\.\(2026\)E\. Zhu, D\. Jiang, Y\. Wang, X\. Li, J\. Cheng, Y\. Gu, Y\. Niu, A\. Zeng, J\. Tang, M\. Huang, and H\. WangData efficient RLVR via off\-policy influence guidance\.InProceedings of the 64th Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),pp\. 46167–46192\.Cited by:[§1](https://arxiv.org/html/2610.28883#S1.p1.1)\.

## Appendix AAdditional Details on RL\-IF

### A\.1Derivation of RL Leave\-One\-Out and RL\-IF

In this section, we derive the recursion underlying RL\-IF\. To estimate the RL\-LOO effect of a samplez∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}, we inject a perturbation at roundkk, stepll, while keeping all subsequent updates unchanged:

θk\(l\)​\(ε\)=θk\(l−1\)−ηkb​∑x∈Dk\(l\)∖\{z∗\}∇θℓ​\(θk\(l−1\),x\)−ηkb​\(1−ε\)​∇θℓ​\(θk\(l−1\),z∗\)\.\\theta\_\{k\}^\{\(l\)\}\(\\varepsilon\)=\\theta\_\{k\}^\{\(l\-1\)\}\-\\frac\{\\eta\_\{k\}\}\{b\}\\sum\_\{x\\in D\_\{k\}^\{\(l\)\}\\setminus\\\{z^\{\*\}\\\}\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},x\)\-\\frac\{\\eta\_\{k\}\}\{b\}\(1\-\\varepsilon\)\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.\(10\)The injected perturbation at\(k,l\)\(k,l\)is obtained by differentiating \([10](https://arxiv.org/html/2610.28883#A1.E10)\):

Δk\(l\):=∂θk\(l\)​\(ε\)∂ε\|ε=0=ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\Delta\_\{k\}^\{\(l\)\}:=\\left\.\\frac\{\\partial\\theta\_\{k\}^\{\(l\)\}\(\\varepsilon\)\}\{\\partial\\varepsilon\}\\right\|\_\{\\varepsilon=0\}=\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.To characterize the terminal perturbationΔT\(M\)\\Delta\_\{T\}^\{\(M\)\}, we introduce a stage\-update operator that makes explicit the dependence on the current iterate and the rollout distribution\. For each update\(t,j\)\(t,j\), define

Ft\(j\)​\(ϑ,β\):=ϑ−ηtb​∑x∼St\(j\)​\(β\)∇θℓ​\(ϑ,x\),St\(j\)​\(β\):=\{Dt\(j\),if​t=k​and​j\>l,Pt\(⋅∣β\),if​t≥k\+1\.F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\):=\\vartheta\-\\frac\{\\eta\_\{t\}\}\{b\}\\sum\_\{x\\sim S\_\{t\}^\{\(j\)\}\(\\beta\)\}\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\),\\qquad S\_\{t\}^\{\(j\)\}\(\\beta\):=\\begin\{cases\}D\_\{t\}^\{\(j\)\},&\\text\{if \}t=k\\text\{ and \}j\>l,\\\\ P\_\{t\}\(\\cdot\\mid\\beta\),&\\text\{if \}t\\geq k\+1\.\\end\{cases\}HereDt\(j\)D\_\{t\}^\{\(j\)\}denotes a fixed minibatch, whilePt\(⋅∣β\)P\_\{t\}\(\\cdot\\mid\\beta\)denotes the rollout distribution induced byβ\\beta\. The perturbed trajectory followsθt\(j\)​\(ε\)=Ft\(j\)​\(θt\(j−1\)​\(ε\),θt\(0\)​\(ε\)\)\.\\theta\_\{t\}^\{\(j\)\}\(\\varepsilon\)=F\_\{t\}^\{\(j\)\}\\big\(\\theta\_\{t\}^\{\(j\-1\)\}\(\\varepsilon\),\\theta\_\{t\}^\{\(0\)\}\(\\varepsilon\)\\big\)\.Recall the definition of perturbationΔt\(j\)\\Delta\_\{t\}^\{\(j\)\}in \([5](https://arxiv.org/html/2610.28883#S3.E5)\), and by the chain rule, we have

Δt\(j\)=∂Ft\(j\)∂ϑ​Δt\(j−1\)\+∂Ft\(j\)∂β​Δt\(0\)\.\\Delta\_\{t\}^\{\(j\)\}=\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\vartheta\}\\Delta\_\{t\}^\{\(j\-1\)\}\+\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\beta\}\\Delta\_\{t\}^\{\(0\)\}\.\(11\)
##### Optimization effect\.

DifferentiatingFt\(j\)F\_\{t\}^\{\(j\)\}with respect to the current iterateϑ\\varthetagives∂Ft\(j\)∂ϑ=At\(j\)=I−ηtb​Ht\(j\),\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\vartheta\}=A\_\{t\}^\{\(j\)\}=I\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\},whereHt\(j\):=∑x∈Dt\(j\)∇θ2​ℓ​\(θt\(j−1\),x\)H\_\{t\}^\{\(j\)\}:=\\sum\_\{x\\in D\_\{t\}^\{\(j\)\}\}\\nabla\_\{\\theta\}^\{2\}\\ell\(\\theta\_\{t\}^\{\(j\-1\)\},x\)\.

##### Sampling effect\.

Fort=kt=kandj\>lj\>l, the minibatch is fixed, soSt\(j\)​\(β\)=Dt\(j\)S\_\{t\}^\{\(j\)\}\(\\beta\)=D\_\{t\}^\{\(j\)\}and∂Ft\(j\)∂β=0\.\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\beta\}=0\.Thus, by \([11](https://arxiv.org/html/2610.28883#A1.E11)\) we haveΔk\(j\)=Ak\(j\)​Δk\(j−1\)\\Delta\_\{k\}^\{\(j\)\}=A\_\{k\}^\{\(j\)\}\\Delta\_\{k\}^\{\(j\-1\)\}\. For future stagest≥k\+1t\\geq k\+1, the sampling distribution depends onβ\\beta\. Using the score\-function identity,

∇β𝔼x∼Pt\(⋅∣β\)\[g\(x\)\]=𝔼x∼Pt\(⋅∣β\)\[g\(x\)∇βlogPt\(x∣β\)⊤\],\\nabla\_\{\\beta\}\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\)\}\[g\(x\)\]=\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\)\}\\left\[g\(x\)\\nabla\_\{\\beta\}\\log P\_\{t\}\(x\\mid\\beta\)^\{\\top\}\\right\],withg⁡\(x\)=∇θℓ​\(ϑ,x\)g\(x\)=\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\), we obtain

∂Ft\(j\)∂β=−ηt𝔼x∼Pt\(⋅∣β\)\[∇θℓ\(ϑ,x\)∇βlogPt\(x∣β\)⊤\]\.\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\beta\}=\-\\eta\_\{t\}\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\)\}\\left\[\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\)\\nabla\_\{\\beta\}\\log P\_\{t\}\(x\\mid\\beta\)^\{\\top\}\\right\]\.Approximating the expectation with the sampled minibatch yields∂Ft\(j\)∂β≈Bt\(j\)\\frac\{\\partial F\_\{t\}^\{\(j\)\}\}\{\\partial\\beta\}\\approx B\_\{t\}^\{\(j\)\}, where

Bt\(j\):=−ηtb∑x∈Dt\(j\)∇θℓ\(θt\(j−1\),x\)∇θlogPt\(x∣θt\(0\)\)⊤\.B\_\{t\}^\{\(j\)\}:=\-\\frac\{\\eta\_\{t\}\}\{b\}\\sum\_\{x\\in D\_\{t\}^\{\(j\)\}\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{t\}^\{\(j\-1\)\},x\)\\nabla\_\{\\theta\}\\log P\_\{t\}\(x\\mid\\theta\_\{t\}^\{\(0\)\}\)^\{\\top\}\.Substituting into \([11](https://arxiv.org/html/2610.28883#A1.E11)\), we obtain the unified recursionΔt\(j\)=At\(j\)​Δt\(j−1\)\+Bt\(j\)​Δt\(0\),\\Delta\_\{t\}^\{\(j\)\}=A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(0\)\},where the second term is active only fort≥k\+1t\\geq k\+1\.

### A\.2Efficient Computation through Adjoint Formulation

Even with the recursive formulation of RL\-IF, a key challenge is efficient computation\. Directly materializingΔt\(j\)\\Delta\_\{t\}^\{\(j\)\}for each candidate sample is computationally prohibitive\. Instead, we evaluate RL\-IF by propagating an adjoint vector backward through the training trajectory\. This converts the forward perturbation recursion into a single backward pass involving Hessian–vector products and score\-function vector products\. The resulting procedure is summarized in Algorithm[2](https://arxiv.org/html/2610.28883#alg2)\.

The algorithm returns∇θf​\(θT\(M\)\)⊤​ΔT\(M\)\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}, which approximates the RL leave\-one\-out effectRL​\-​LOO\(k,l\)​\(z∗,f\)\\mathrm\{RL\}\\text\{\-\}\\mathrm\{LOO\}^\{\(k,l\)\}\(z^\{\*\};f\)\. Notably, all influence scores across the trajectory can be computed within a single backward pass\.

Algorithm 2One\-pass RL\-IF1:Training trajectory

\{θt\(j\),Dt\(j\)\}\\\{\\theta\_\{t\}^\{\(j\)\},D\_\{t\}^\{\(j\)\}\\\}, attribution target

ff, target sample

z∗∈Dk\(l\)z^\{\*\}\\in D\_\{k\}^\{\(l\)\}
2:

U←∇θf​\(θT\(M\)\)U\\leftarrow\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)
3:for

t=Tt=Tdown to

11do

4:

Uprev←0U\_\{\\mathrm\{prev\}\}\\leftarrow 0
5:for

j=Mj=Mdown to

11do

6:if

t=kt=kand

j=lj=lthen

7:return

⟨U,ηkb​∇θℓ​\(θk\(l−1\),z∗\)⟩\\left\\langle U,\\;\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\\right\\rangle
8:endif

9:// sampling contribution

10:

Uprev←Uprev\+Bt\(j\)⊤​UU\_\{\\mathrm\{prev\}\}\\leftarrow U\_\{\\mathrm\{prev\}\}\+B\_\{t\}^\{\(j\)\\top\}U
11:// optimization step

12:

U←\(I−ηtb​Ht\(j\)\)​UU\\leftarrow\\left\(I\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\}\\right\)U
13:endfor

14:

U←U\+UprevU\\leftarrow U\+U\_\{\\mathrm\{prev\}\}
15:endfor

We prove that Algorithm[2](https://arxiv.org/html/2610.28883#alg2)returns the first\-order RL\-IF estimator\. Recall the forward perturbation recursion in \([7](https://arxiv.org/html/2610.28883#S3.E7)\)\. For fixed\-minibatch updates, the sampling term is inactive, which is equivalently captured by settingBt\(j\)=0B\_\{t\}^\{\(j\)\}=0\. The target scalar is∇θf​\(θT\(M\)\)⊤​ΔT\(M\)\.\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}\.We initialize

UT\(M\)=∇θf​\(θT\(M\)\)\.U\_\{T\}^\{\(M\)\}=\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)\.Suppose that, before processing update\(t,j\)\(t,j\)in the backward pass, the current adjoint vector isUt\(j\)U\_\{t\}^\{\(j\)\}\. Then, using \([7](https://arxiv.org/html/2610.28883#S3.E7)\), we have

\(Ut\(j\)\)⊤​Δt\(j\)\\displaystyle\\left\(U\_\{t\}^\{\(j\)\}\\right\)^\{\\top\}\\Delta\_\{t\}^\{\(j\)\}=\(Ut\(j\)\)⊤​\(At\(j\)​Δt\(j−1\)\+Bt\(j\)​Δt\(0\)\)\\displaystyle=\\left\(U\_\{t\}^\{\(j\)\}\\right\)^\{\\top\}\\left\(A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(0\)\}\\right\)=\(At\(j\)⊤​Ut\(j\)\)⊤​Δt\(j−1\)\+\(Bt\(j\)⊤​Ut\(j\)\)⊤​Δt\(0\)\.\\displaystyle=\\left\(A\_\{t\}^\{\(j\)\\top\}U\_\{t\}^\{\(j\)\}\\right\)^\{\\top\}\\Delta\_\{t\}^\{\(j\-1\)\}\+\\left\(B\_\{t\}^\{\(j\)\\top\}U\_\{t\}^\{\(j\)\}\\right\)^\{\\top\}\\Delta\_\{t\}^\{\(0\)\}\.\(12\)
Therefore, the adjoint contribution to the previous minibatch iterate isUt\(j−1\)=At\(j\)⊤​Ut\(j\)U\_\{t\}^\{\(j\-1\)\}=A\_\{t\}^\{\(j\)\\top\}U\_\{t\}^\{\(j\)\}, which gives the optimization update:

U←At\(j\)⊤​U=\(I−ηtb​Ht\(j\)\)​U,U\\leftarrow A\_\{t\}^\{\(j\)\\top\}U=\\left\(I\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\}\\right\)U,where we use the symmetry ofHt\(j\)H\_\{t\}^\{\(j\)\}\. The second term in \([12](https://arxiv.org/html/2610.28883#A1.E12)\) contributes to the previous\-stage perturbationΔt\(0\)\\Delta\_\{t\}^\{\(0\)\}rather than toΔt\(j−1\)\\Delta\_\{t\}^\{\(j\-1\)\}\. Hence, during the backward sweep over stagett, all such cross\-stage contributions are accumulated as

Uprev←Uprev\+Bt\(j\)⊤​U\.U\_\{\\mathrm\{prev\}\}\\leftarrow U\_\{\\mathrm\{prev\}\}\+B\_\{t\}^\{\(j\)\\top\}U\.After all updates in stagetthave been processed, the adjoint propagated through the optimization dynamics and the accumulated sampling contribution both act on the same previous\-stage perturbationΔt\(0\)\\Delta\_\{t\}^\{\(0\)\}\. Thus they are combined by

U←U\+Uprev\.U\\leftarrow U\+U\_\{\\mathrm\{prev\}\}\.By repeated application of \([12](https://arxiv.org/html/2610.28883#A1.E12)\), the algorithm maintains the invariant that the current adjointUUsatisfies

∇θf​\(θT\(M\)\)⊤​ΔT\(M\)=U⊤​Δt\(j\)\+Uprev⊤​Δt\(0\)\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}=U^\{\\top\}\\Delta\_\{t\}^\{\(j\)\}\+U\_\{\\mathrm\{prev\}\}^\{\\top\}\\Delta\_\{t\}^\{\(0\)\}whenever the backward pass is positioned at iterateθt\(j\)\\theta\_\{t\}^\{\(j\)\}\. In particular, when the backward pass reaches the injection point\(k,l\)\(k,l\),Δk\(0\)=0\\Delta\_\{k\}^\{\(0\)\}=0, so we have∇θf​\(θT\(M\)\)⊤​ΔT\(M\)=U⊤​Δk\(l\)\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}=U^\{\\top\}\\Delta\_\{k\}^\{\(l\)\}\. The injected perturbation isΔk\(l\)=ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\Delta\_\{k\}^\{\(l\)\}=\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.Substituting this expression gives

∇θf​\(θT\(M\)\)⊤​ΔT\(M\)=⟨U,ηkb​∇θℓ​\(θk\(l−1\),z∗\)⟩\.\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\Delta\_\{T\}^\{\(M\)\}=\\left\\langle U,\\;\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\\right\\rangle\.This is exactly the value returned by Algorithm[2](https://arxiv.org/html/2610.28883#alg2)\. Therefore, Algorithm[2](https://arxiv.org/html/2610.28883#alg2)correctly computes the first\-order RL\-IF estimator for the RL\-LOO effect\.

### A\.3Closed\-Form RL\-IF: Proof of Equation \([8](https://arxiv.org/html/2610.28883#S3.E8)\)

In this section, we show that the recursion in \([7](https://arxiv.org/html/2610.28883#S3.E7)\) directly leads to a compact closed form\. For each staget\>kt\>k, define

At:=∏j=M1At\(j\),Ct:=At\+∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\)\.A\_\{t\}:=\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\},\\qquad C\_\{t\}:=A\_\{t\}\+\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\.\(13\)Here, products are ordered from right to left along the forward trajectory; in particular,

∏j=M1At\(j\)=At\(M\)At\(M−1\)⋯At\(1\)\.\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\}=A\_\{t\}^\{\(M\)\}A\_\{t\}^\{\(M\-1\)\}\\cdots A\_\{t\}^\{\(1\)\}\.Then

RL​\-​IF\(k,l\)​\(z∗,f\)=∇θf​\(θT\(M\)\)⊤​\(∏t=Tk\+1Ct\)​\(∏j=Ml\+1Ak\(j\)\)​ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\mathrm\{RL\}\\text\{\-\}\\mathrm\{IF\}^\{\(k,l\)\}\(z^\{\*\};f\)=\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\left\(\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\\right\)\\left\(\\prod\_\{j=M\}^\{l\+1\}A\_\{k\}^\{\(j\)\}\\right\)\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.\(14\)
##### Proof\.

We prove the closed form in \([14](https://arxiv.org/html/2610.28883#A1.E14)\) by induction\.

First, consider a future staget\>kt\>k\. RecallΔt\(j\)=At\(j\)​Δt\(j−1\)\+Bt\(j\)​Δt\(0\)\.\\Delta\_\{t\}^\{\(j\)\}=A\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}\\Delta\_\{t\}^\{\(0\)\}\.We claim that, for everyr=1,…,Mr=1,\\dots,M,

Δt\(r\)=\(∏j=r1At\(j\)\)​Δt\(0\)\+∑q=1r\(∏j=rq\+1At\(j\)\)​Bt\(q\)​Δt\(0\),\\Delta\_\{t\}^\{\(r\)\}=\\left\(\\prod\_\{j=r\}^\{1\}A\_\{t\}^\{\(j\)\}\\right\)\\Delta\_\{t\}^\{\(0\)\}\+\\sum\_\{q=1\}^\{r\}\\left\(\\prod\_\{j=r\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\Delta\_\{t\}^\{\(0\)\},\(15\)where an empty product is the identity\. Forr=1r=1, \([15](https://arxiv.org/html/2610.28883#A1.E15)\) gives

Δt\(1\)=At\(1\)​Δt\(0\)\+Bt\(1\)​Δt\(0\),\\Delta\_\{t\}^\{\(1\)\}=A\_\{t\}^\{\(1\)\}\\Delta\_\{t\}^\{\(0\)\}\+B\_\{t\}^\{\(1\)\}\\Delta\_\{t\}^\{\(0\)\},which is exactly \([7](https://arxiv.org/html/2610.28883#S3.E7)\)\. Assume \([15](https://arxiv.org/html/2610.28883#A1.E15)\) holds for somer−1r\-1\. Then

Δt\(r\)\\displaystyle\\Delta\_\{t\}^\{\(r\)\}=At\(r\)​Δt\(r−1\)\+Bt\(r\)​Δt\(0\)\\displaystyle=A\_\{t\}^\{\(r\)\}\\Delta\_\{t\}^\{\(r\-1\)\}\+B\_\{t\}^\{\(r\)\}\\Delta\_\{t\}^\{\(0\)\}=At\(r\)​\[\(∏j=r−11At\(j\)\)​Δt\(0\)\+∑q=1r−1\(∏j=r−1q\+1At\(j\)\)​Bt\(q\)​Δt\(0\)\]\+Bt\(r\)​Δt\(0\)\\displaystyle=A\_\{t\}^\{\(r\)\}\\left\[\\left\(\\prod\_\{j=r\-1\}^\{1\}A\_\{t\}^\{\(j\)\}\\right\)\\Delta\_\{t\}^\{\(0\)\}\+\\sum\_\{q=1\}^\{r\-1\}\\left\(\\prod\_\{j=r\-1\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\Delta\_\{t\}^\{\(0\)\}\\right\]\+B\_\{t\}^\{\(r\)\}\\Delta\_\{t\}^\{\(0\)\}=\(∏j=r1At\(j\)\)​Δt\(0\)\+∑q=1r−1\(∏j=rq\+1At\(j\)\)​Bt\(q\)​Δt\(0\)\+Bt\(r\)​Δt\(0\)\\displaystyle=\\left\(\\prod\_\{j=r\}^\{1\}A\_\{t\}^\{\(j\)\}\\right\)\\Delta\_\{t\}^\{\(0\)\}\+\\sum\_\{q=1\}^\{r\-1\}\\left\(\\prod\_\{j=r\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\Delta\_\{t\}^\{\(0\)\}\+B\_\{t\}^\{\(r\)\}\\Delta\_\{t\}^\{\(0\)\}=\(∏j=r1At\(j\)\)​Δt\(0\)\+∑q=1r\(∏j=rq\+1At\(j\)\)​Bt\(q\)​Δt\(0\)\.\\displaystyle=\\left\(\\prod\_\{j=r\}^\{1\}A\_\{t\}^\{\(j\)\}\\right\)\\Delta\_\{t\}^\{\(0\)\}\+\\sum\_\{q=1\}^\{r\}\\left\(\\prod\_\{j=r\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\Delta\_\{t\}^\{\(0\)\}\.Thus the claim holds forrr, and therefore for allr=1,…,Mr=1,\\dots,M\. Settingr=Mr=Mgives

Δt\(M\)=\(∏j=M1At\(j\)\)​Δt\(0\)\+∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\)​Δt\(0\)\.\\Delta\_\{t\}^\{\(M\)\}=\\left\(\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\}\\right\)\\Delta\_\{t\}^\{\(0\)\}\+\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\Delta\_\{t\}^\{\(0\)\}\.Factoring outΔt\(0\)\\Delta\_\{t\}^\{\(0\)\}, we obtain

Δt\(M\)=\[∏j=M1At\(j\)\+∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\)\]​Δt\(0\)=Ct​Δt\(0\)\.\\Delta\_\{t\}^\{\(M\)\}=\\left\[\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\}\+\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\\right\]\\Delta\_\{t\}^\{\(0\)\}=C\_\{t\}\\Delta\_\{t\}^\{\(0\)\}\.\(16\)Next, for the injection stagekk, the perturbation after\(k,l\)\(k,l\)isΔk\(l\)=ηkb​∇θℓ​\(θk\(l−1\),z∗\)\\Delta\_\{k\}^\{\(l\)\}=\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\. Forj=l\+1,…,Mj=l\+1,\\dots,M, the minibatches are fixed, soΔk\(j\)=Ak\(j\)​Δk\(j−1\)\\Delta\_\{k\}^\{\(j\)\}=A\_\{k\}^\{\(j\)\}\\Delta\_\{k\}^\{\(j\-1\)\}\. By another direct induction overjj, this gives

Δk\(M\)=\(∏j=Ml\+1Ak\(j\)\)​Δk\(l\)\.\\Delta\_\{k\}^\{\(M\)\}=\\left\(\\prod\_\{j=M\}^\{l\+1\}A\_\{k\}^\{\(j\)\}\\right\)\\Delta\_\{k\}^\{\(l\)\}\.\(17\)
Finally, applying \([16](https://arxiv.org/html/2610.28883#A1.E16)\) inductively over stagest=k\+1,…,Tt=k\+1,\\dots,Tyields

ΔT\(M\)=\(∏t=Tk\+1Ct\)​Δk\(M\)\.\\Delta\_\{T\}^\{\(M\)\}=\\left\(\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\\right\)\\Delta\_\{k\}^\{\(M\)\}\.Substituting \([17](https://arxiv.org/html/2610.28883#A1.E17)\) and the expression forΔk\(l\)\\Delta\_\{k\}^\{\(l\)\}gives

ΔT\(M\)=\(∏t=Tk\+1Ct\)​\(∏j=Ml\+1Ak\(j\)\)​ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\Delta\_\{T\}^\{\(M\)\}=\\left\(\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\\right\)\\left\(\\prod\_\{j=M\}^\{l\+1\}A\_\{k\}^\{\(j\)\}\\right\)\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.Multiplying both sides by∇θf​\(θT\(M\)\)⊤\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}proves

RL​\-​IF\(k,l\)​\(z∗,f\)=∇θf​\(θT\(M\)\)⊤​\(∏t=Tk\+1Ct\)​\(∏j=Ml\+1Ak\(j\)\)​ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\mathrm\{RL\}\\text\{\-\}\\mathrm\{IF\}^\{\(k,l\)\}\(z^\{\*\};f\)=\\nabla\_\{\\theta\}f\(\\theta\_\{T\}^\{\(M\)\}\)^\{\\top\}\\left\(\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\\right\)\\left\(\\prod\_\{j=M\}^\{l\+1\}A\_\{k\}^\{\(j\)\}\\right\)\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.This completes the proof\.

### A\.4Construction and Justification of the Reference Target

For an attribution window starting at roundkk, we use its terminal checkpointθ¯:=θk\+H\\bar\{\\theta\}:=\\theta\_\{k\+H\}as the reference policy\. We collect an on\-policy rolloutDk\+HD\_\{k\+H\}fromπθ¯\\pi\_\{\\bar\{\\theta\}\}and compute the reference advantages using the critic associated with this checkpoint and the same generalized advantage estimation \(GAE\) procedure and hyperparameters as PPO\. In particular, writingVrefV^\{\\mathrm\{ref\}\}for the reference critic, the estimates take the form

δtref\\displaystyle\\delta\_\{t\}^\{\\mathrm\{ref\}\}=rt\+γ​Vref​\(st\+1\)−Vref​\(st\),\\displaystyle=r\_\{t\}\+\\gamma V^\{\\mathrm\{ref\}\}\(s\_\{t\+1\}\)\-V^\{\\mathrm\{ref\}\}\(s\_\{t\}\),\(18\)A^tref\\displaystyle\\hat\{A\}\_\{t\}^\{\\mathrm\{ref\}\}=∑j=0Lt−1\(γ​λ\)j​δt\+jref,\\displaystyle=\\sum\_\{j=0\}^\{L\_\{t\}\-1\}\(\\gamma\\lambda\)^\{j\}\\delta\_\{t\+j\}^\{\\mathrm\{ref\}\},\(19\)whereLtL\_\{t\}is the number of available steps used in the estimate, with the same termination masking and truncation bootstrapping as in PPO\. The reference rollout, critic, and advantage estimates are held fixed during attribution\. This construction does not involve tuning over candidate reference advantages\.

The terminal checkpoint is used because RL\-LOO evaluates the effect of removing a training occurrence on the policy obtained at the end of the window\. The following result explains the connection between the reference target and the expected return of that policy\.

##### Ideal reference target\.

For the formal statement, let0≤γ<10\\leq\\gamma<1and define the normalized discounted return and discounted state occupancy by

J⁡\(θ\)\\displaystyle J\(\\theta\):=\(1−γ\)​𝔼τ∼πθ​\[∑t=0∞γt​rt\],\\displaystyle:=\(1\-\\gamma\)\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\theta\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\_\{t\}\\right\],\(20\)dπθ​\(s\)\\displaystyle d^\{\\pi\_\{\\theta\}\}\(s\):=\(1−γ\)​∑t=0∞γt​Prπθ⁡\(st=s\)\.\\displaystyle:=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Pr\_\{\\pi\_\{\\theta\}\}\(s\_\{t\}=s\)\.\(21\)The initial\-state distribution is fixed\. LetAπθ¯A^\{\\pi\_\{\\bar\{\\theta\}\}\}denote the exact advantage for the usual discounted return\. Following the negative\-target convention in the main text, define

fref​\(ϕ\):=−𝔼s∼dπθ¯a∼πθ¯\(⋅∣s\)​\[log⁡πϕ​\(a∣s\)​Aπθ¯​\(s,a\)\]\.f\_\{\\mathrm\{ref\}\}\(\\phi\):=\-\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\\sim d^\{\\pi\_\{\\bar\{\\theta\}\}\}\\\\ a\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid s\)\\end\{subarray\}\}\\left\[\\log\\pi\_\{\\phi\}\(a\\mid s\)A^\{\\pi\_\{\\bar\{\\theta\}\}\}\(s,a\)\\right\]\.\(22\)The reference distribution and advantages in this expression are fixed asϕ\\phivaries\. This normalization makes the return\-gradient identity explicit; the corresponding statement for unnormalized discounted return differs by the constant factor1−γ1\-\\gamma\.

Proposition A\.1:Reference target and terminal returnAssume the policy\-gradient identity holds and thatfreff\_\{\\mathrm\{ref\}\}andJJare twice continuously differentiable in a neighborhood ofθ¯\\bar\{\\theta\}\. Letθ¯−z\\bar\{\\theta\}\_\{\-z\}be the terminal parameter after deleting occurrencezz, and setΔz:=θ¯−z−θ¯\\Delta\_\{z\}:=\\bar\{\\theta\}\_\{\-z\}\-\\bar\{\\theta\}\. For sufficiently smallΔz\\Delta\_\{z\},RL​\-​LOO​\(z,fref\)=J⁡\(θ¯\)−J⁡\(θ¯−z\)\+O⁡\(‖Δz‖22\)\.\\mathrm\{RL\\text\{\-\}LOO\}\(z;f\_\{\\mathrm\{ref\}\}\)=J\(\\bar\{\\theta\}\)\-J\(\\bar\{\\theta\}\_\{\-z\}\)\+O\(\\\|\\Delta\_\{z\}\\\|\_\{2\}^\{2\}\)\.\(23\)

##### Proof\.

Since the reference distribution and advantages are fixed, the policy\-gradient theorem gives

∇ϕfref​\(ϕ\)\|ϕ=θ¯\\displaystyle\\left\.\\nabla\_\{\\phi\}f\_\{\\mathrm\{ref\}\}\(\\phi\)\\right\|\_\{\\phi=\\bar\{\\theta\}\}=−𝔼s∼dπθ¯a∼πθ¯\(⋅∣s\)​\[∇θ​log​πθ¯​\(a∣s\)​Aπθ¯​\(s,a\)\]\\displaystyle=\-\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\\sim d^\{\\pi\_\{\\bar\{\\theta\}\}\}\\\\ a\\sim\\pi\_\{\\bar\{\\theta\}\}\(\\cdot\\mid s\)\\end\{subarray\}\}\\left\[\\nabla\_\{\\theta\}\\log\\pi\_\{\\bar\{\\theta\}\}\(a\\mid s\)A^\{\\pi\_\{\\bar\{\\theta\}\}\}\(s,a\)\\right\]\(24\)=−∇J​\(θ¯\)\.\\displaystyle=\-\\nabla J\(\\bar\{\\theta\}\)\.\(25\)Taylor expansion atθ¯\\bar\{\\theta\}therefore yields

fref​\(θ¯−z\)−fref​\(θ¯\)\\displaystyle f\_\{\\mathrm\{ref\}\}\(\\bar\{\\theta\}\_\{\-z\}\)\-f\_\{\\mathrm\{ref\}\}\(\\bar\{\\theta\}\)=−∇J\(θ¯\)⊤Δz\+O\(∥Δz∥22\),\\displaystyle=\-\\nabla J\(\\bar\{\\theta\}\)^\{\\top\}\\Delta\_\{z\}\+O\(\\\|\\Delta\_\{z\}\\\|\_\{2\}^\{2\}\),\(26\)J⁡\(θ¯−z\)−J⁡\(θ¯\)\\displaystyle J\(\\bar\{\\theta\}\_\{\-z\}\)\-J\(\\bar\{\\theta\}\)=∇J\(θ¯\)⊤Δz\+O\(∥Δz∥22\)\.\\displaystyle=\\nabla J\(\\bar\{\\theta\}\)^\{\\top\}\\Delta\_\{z\}\+O\(\\\|\\Delta\_\{z\}\\\|\_\{2\}^\{2\}\)\.\(27\)Combining the two expansions proves \([23](https://arxiv.org/html/2610.28883#A1.E23)\)\.

Thus, with this negative target, a negative RL\-LOO value locally corresponds to an increase in terminal return after deletion\. RL\-IF estimates this counterfactual through the first\-order parameter perturbation\. In practice, we replace the population target and exact advantages with the empirical reference rollout and GAE estimates, introducing sampling and advantage\-estimation error in addition to the first\-order approximation\. Sensitivity to the reference construction is evaluated in Appendix[A\.4\.1](https://arxiv.org/html/2610.28883#A1.SS4.SSS1)\.

#### A\.4\.1Choice of Reference Policy and Rollout

We examine sensitivity to the reference construction by comparing the default terminal checkpoint and its own rollout,\(θk\+H,Dk\+H\)\(\\theta\_\{k\+H\},D\_\{k\+H\}\), with two alternatives\. The first uses the initial checkpoint of the attribution window and its corresponding rollout,\(θk,Dk\)\(\\theta\_\{k\},D\_\{k\}\), following the IIF\-style reference construction\. The second retains the terminal checkpoint but uses the union of rollout data collected within the window\. Although the latter uses more reference data, these rollouts were generated by earlier policies rather than solely by the terminal policy\.

\(θk\+H,Dk\+H\)\(\\theta\_\{k\+H\},\\,D\_\{k\+H\}\)\(default\)\(θk,Dk\)\(\\theta\_\{k\},\\,D\_\{k\}\)\(θk\+H,⋃D\)\(\\theta\_\{k\+H\},\\,\\bigcup D\)

Figure 5:Reference\-construction ablation on Ant and HalfCheetah\. Curves show mean return at the reported training rounds; shaded regions indicate one standard error\.Figure[5](https://arxiv.org/html/2610.28883#A1.F5)shows that the default reference achieves the highest final mean return on both environments\. These results support pairing the terminal reference policy with its own on\-policy rollout\.

## Appendix BDetailed Theorem Derivations

In this section, we provide detailed proofs for the theorems\.

### B\.1Uniform Approximation Error of First\-Order RL\-IF

Theorem B\.1:Uniform approximation error of first\-order RL\-IFAssume thatℓ⁡\(θ,x\)\\ell\(\\theta;x\)is twice differentiable, that its gradient norm is bounded byGG, that its Hessian norm is bounded byΛ\\Lambda, and that its Hessian isLHL\_\{H\}\-Lipschitz continuous inθ\\theta\. Assume also that the rollout distributionPt​\(x∣θ\)P\_\{t\}\(x\\mid\\theta\)has score norm bounded bySSand score Hessian norm bounded byLPL\_\{P\}\. Furthermore, assume that the learning rate satisfiesM​Λ¯​ηt≤1T−kM\\overline\{\\Lambda\}\\eta\_\{t\}\\leq\\frac\{1\}\{T\-k\}for everyk≤t≤Tk\\leq t\\leq T\. Then for the first\-order RL\-IF estimatorΔT\(M\)\\Delta\_\{T\}^\{\(M\)\}, the approximation satisfies‖θ~T\(M\)−θT\(M\)−ΔT\(M\)‖2≤CsmoothM2​b2,\\bigl\\\|\\widetilde\{\\theta\}\_\{T\}^\{\(M\)\}\-\\theta\_\{T\}^\{\(M\)\}\-\\Delta\_\{T\}^\{\(M\)\}\\bigr\\\|\_\{2\}\\leq\\frac\{C\_\{\\text\{smooth\}\}\}\{M^\{2\}b^\{2\}\},whereCsmooth=c​G2​\(LH\+Λ¯​S\+G​LP\)Λ¯3C\_\{\\text\{smooth\}\}=\\frac\{cG^\{2\}\(L\_\{H\}\+\\overline\{\\Lambda\}S\+GL\_\{P\}\)\}\{\\overline\{\\Lambda\}^\{3\}\}andΛ¯=Λ\+G​S\\overline\{\\Lambda\}=\\Lambda\+GSare smoothness constants\.

###### Proof\.

We analyze the population update map and we do not consider minibatch noise in this theorem\. We define the true counterfactual perturbation as

δt\(j\):=θ~t\(j\)−θt\(j\)\.\\delta\_\{t\}^\{\(j\)\}:=\\widetilde\{\\theta\}\_\{t\}^\{\(j\)\}\-\\theta\_\{t\}^\{\(j\)\}\.The first\-order RL\-IF perturbationΔt\(j\)\\Delta\_\{t\}^\{\(j\)\}is defined by \([5](https://arxiv.org/html/2610.28883#S3.E5)\) and satisfies the recursion in Proposition[3\.2](https://arxiv.org/html/2610.28883#S3.SS2)\. At the injection point\(k,l\)\(k,l\), the two coincide exactly:

δk\(l\)=Δk\(l\)=ηkb​∇θℓ​\(θk\(l−1\),z∗\)\.\\delta\_\{k\}^\{\(l\)\}=\\Delta\_\{k\}^\{\(l\)\}=\\frac\{\\eta\_\{k\}\}\{b\}\\nabla\_\{\\theta\}\\ell\(\\theta\_\{k\}^\{\(l\-1\)\},z^\{\*\}\)\.Our goal is to bound the discrepancy

et\(j\):=δt\(j\)−Δt\(j\)e\_\{t\}^\{\(j\)\}:=\\delta\_\{t\}^\{\(j\)\}\-\\Delta\_\{t\}^\{\(j\)\}at the terminal point\(T,M\)\(T,M\)\. Recall the stage update mapFt\(j\)​\(ϑ,β\)F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)defined in \([6](https://arxiv.org/html/2610.28883#S3.E6)\)\. By construction, the post\-intervention trajectory is generated by

θt\(j\)​\(ε\)=Ft\(j\)​\(θt\(j−1\)​\(ε\),θt\(0\)​\(ε\)\)\.\\theta\_\{t\}^\{\(j\)\}\(\\varepsilon\)=F\_\{t\}^\{\(j\)\}\\\!\\left\(\\theta\_\{t\}^\{\(j\-1\)\}\(\\varepsilon\),\\theta\_\{t\}^\{\(0\)\}\(\\varepsilon\)\\right\)\.Sinceℓ⁡\(θ,x\)\\ell\(\\theta;x\)has anLHL\_\{H\}\-Lipschitz Hessian, and the rollout distributionPt​\(x∣θ\)P\_\{t\}\(x\\mid\\theta\)has score norm bounded bySSand score Hessian norm bounded byLPL\_\{P\}, the mapFt\(j\)​\(ϑ,β\)F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)is jointly smooth in\(ϑ,β\)\(\\vartheta,\\beta\)\. We next prove a bound for the remainder:

∥\\displaystyle\\Bigl\\\|Ft\(j\)\(ϑ\+u,β\+v\)−Ft\(j\)\(ϑ,β\)−∂ϑFt\(j\)\(ϑ,β\)u−∂βFt\(j\)\(ϑ,β\)v∥2≤ηt​K2\(∥u∥2\+∥v∥2\)2,\\displaystyle F\_\{t\}^\{\(j\)\}\(\\vartheta\+u,\\beta\+v\)\-F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)\-\\partial\_\{\\vartheta\}F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)u\-\\partial\_\{\\beta\}F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)v\\Bigr\\\|\_\{2\}\\leq\\frac\{\\eta\_\{t\}K\}\{2\}\(\\\|u\\\|\_\{2\}\+\\\|v\\\|\_\{2\}\)^\{2\},\(28\)whereK:=LH\+Λ¯​S\+G​LP,Λ¯:=Λ\+G​S\.K:=L\_\{H\}\+\\overline\{\\Lambda\}S\+GL\_\{P\},\\overline\{\\Lambda\}:=\\Lambda\+GS\.We denotemt\(j\)\(ϑ,β\)=𝔼x∼Pt\(⋅∣β\)\[∇θℓ\(ϑ,x\)\]m\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)=\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\)\}\[\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\)\], for a future roundt\>kt\>k, writem=mt\(j\)m=m\_\{t\}^\{\(j\)\}and define

Rm​\(u,v\):=m⁡\(ϑ\+u,β\+v\)−m⁡\(ϑ,β\)−∂ϑm⁡\(ϑ,β\)​u−∂βm⁡\(ϑ,β\)​v\.R\_\{m\}\(u,v\):=m\(\\vartheta\+u,\\beta\+v\)\-m\(\\vartheta,\\beta\)\-\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\)u\-\\partial\_\{\\beta\}m\(\\vartheta,\\beta\)v\.We decompose this remainder as

Rm​\(u,v\)\\displaystyle R\_\{m\}\(u,v\)=m⁡\(ϑ\+u,β\+v\)−m⁡\(ϑ,β\+v\)−∂ϑm⁡\(ϑ,β\+v\)​u⏟\(I\)\\displaystyle=\\underbrace\{m\(\\vartheta\+u,\\beta\+v\)\-m\(\\vartheta,\\beta\+v\)\-\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\+v\)u\}\_\{\(\\mathrm\{I\}\)\}\+\(∂ϑm⁡\(ϑ,β\+v\)−∂ϑm⁡\(ϑ,β\)\)​u⏟\(II\)\\displaystyle\+\\underbrace\{\\bigl\(\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\+v\)\-\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\)\\bigr\)u\}\_\{\(\\mathrm\{II\}\)\}\+m⁡\(ϑ,β\+v\)−m⁡\(ϑ,β\)−∂βm⁡\(ϑ,β\)​v⏟\(III\)\.\\displaystyle\+\\underbrace\{m\(\\vartheta,\\beta\+v\)\-m\(\\vartheta,\\beta\)\-\\partial\_\{\\beta\}m\(\\vartheta,\\beta\)v\}\_\{\(\\mathrm\{III\}\)\}\.

##### Bounding \(I\)\.

For fixedβ\+v\\beta\+v,

∂ϑm\(ϑ,β\+v\)=𝔼x∼Pt\(⋅∣β\+v\)\[∇θ2ℓ\(ϑ,x\)\]\.\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\+v\)=\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\+v\)\}\\bigl\[\\nabla\_\{\\theta\}^\{2\}\\ell\(\\vartheta,x\)\\bigr\]\.Since the loss Hessian isLHL\_\{H\}\-Lipschitz, we have

‖\(I\)‖2≤LH2​‖u‖22\.\\\|\(\\mathrm\{I\}\)\\\|\_\{2\}\\leq\\frac\{L\_\{H\}\}\{2\}\\\|u\\\|\_\{2\}^\{2\}\.

##### Bounding \(II\)\.

Using the score\-function identity, the Hessian boundΛ\\Lambda, and the score boundSS, we obtain

‖∂ϑm⁡\(ϑ,β\+v\)−∂ϑm⁡\(ϑ,β\)‖2≤Λ​S​‖v‖2\.\\bigl\\\|\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\+v\)\-\\partial\_\{\\vartheta\}m\(\\vartheta,\\beta\)\\bigr\\\|\_\{2\}\\leq\\Lambda S\\\|v\\\|\_\{2\}\.Consequently,

‖\(II\)‖2≤Λ​S​‖u‖2​‖v‖2\.\\\|\(\\mathrm\{II\}\)\\\|\_\{2\}\\leq\\Lambda S\\\|u\\\|\_\{2\}\\\|v\\\|\_\{2\}\.

##### Bounding \(III\)\.

The sampling derivative is

∂βm\(ϑ,β\)=𝔼x∼Pt\(⋅∣β\)\[∇θℓ\(ϑ,x\)∇βlogPt\(x∣β\)⊤\]\.\\partial\_\{\\beta\}m\(\\vartheta,\\beta\)=\\mathbb\{E\}\_\{x\\sim P\_\{t\}\(\\cdot\\mid\\beta\)\}\\left\[\\nabla\_\{\\theta\}\\ell\(\\vartheta,x\)\\nabla\_\{\\beta\}\\log P\_\{t\}\(x\\mid\\beta\)^\{\\top\}\\right\]\.Differentiating once more inβ\\betaproduces a score\-Hessian term and a score outer\-product term\. The gradient boundGG, score boundSS, and score\-Hessian boundLPL\_\{P\}therefore imply

‖\(III\)‖2≤G⁡\(LP\+S2\)2​‖v‖22\.\\\|\(\\mathrm\{III\}\)\\\|\_\{2\}\\leq\\frac\{G\(L\_\{P\}\+S^\{2\}\)\}\{2\}\\\|v\\\|\_\{2\}^\{2\}\.Combining the three bounds yields

‖Rm​\(u,v\)‖2\\displaystyle\\\|R\_\{m\}\(u,v\)\\\|\_\{2\}≤LH2​‖u‖22\+Λ​S​‖u‖2​‖v‖2\+G⁡\(LP\+S2\)2​‖v‖22\\displaystyle\\leq\\frac\{L\_\{H\}\}\{2\}\\\|u\\\|\_\{2\}^\{2\}\+\\Lambda S\\\|u\\\|\_\{2\}\\\|v\\\|\_\{2\}\+\\frac\{G\(L\_\{P\}\+S^\{2\}\)\}\{2\}\\\|v\\\|\_\{2\}^\{2\}≤K2​\(‖u‖2\+‖v‖2\)2,\\displaystyle\\leq\\frac\{K\}\{2\}\(\\\|u\\\|\_\{2\}\+\\\|v\\\|\_\{2\}\)^\{2\},whereK=LH\+Λ​S\+G⁡\(LP\+S2\)\.K=L\_\{H\}\+\\Lambda S\+G\(L\_\{P\}\+S^\{2\}\)\.SinceFt\(j\)​\(ϑ,β\)=ϑ−ηt​mt\(j\)​\(ϑ,β\)F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)=\\vartheta\-\\eta\_\{t\}m\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\), this proves \([28](https://arxiv.org/html/2610.28883#A2.E28)\) fort\>kt\>k\. For the remaining updates of roundkk, the map is independent ofβ\\beta, and the fixed\-minibatch average has anLHL\_\{H\}\-Lipschitz Jacobian\. Its remainder is bounded byηk​LH​‖u‖22/2\\eta\_\{k\}L\_\{H\}\\\|u\\\|\_\{2\}^\{2\}/2, so \([28](https://arxiv.org/html/2610.28883#A2.E28)\) holds there as well\. Applying \([28](https://arxiv.org/html/2610.28883#A2.E28)\) withu=δt\(j−1\)u=\\delta\_\{t\}^\{\(j\-1\)\}andv=δt\(0\)v=\\delta\_\{t\}^\{\(0\)\}gives

δt\(j\)=At\(j\)​δt\(j−1\)\+Bt\(j\)​δt\(0\)\+rt\(j\),\\delta\_\{t\}^\{\(j\)\}=A\_\{t\}^\{\(j\)\}\\delta\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}\\delta\_\{t\}^\{\(0\)\}\+r\_\{t\}^\{\(j\)\},where

‖rt\(j\)‖2≤ηt​K2​\(‖δt\(j−1\)‖2\+‖δt\(0\)‖2\)2\.\\\|r\_\{t\}^\{\(j\)\}\\\|\_\{2\}\\leq\\frac\{\\eta\_\{t\}K\}\{2\}\\left\(\\\|\\delta\_\{t\}^\{\(j\-1\)\}\\\|\_\{2\}\+\\\|\\delta\_\{t\}^\{\(0\)\}\\\|\_\{2\}\\right\)^\{2\}\.\(29\)Subtracting the first\-order recursion yields

et\(j\)=At\(j\)​et\(j−1\)\+Bt\(j\)​et\(0\)\+rt\(j\)\.e\_\{t\}^\{\(j\)\}=A\_\{t\}^\{\(j\)\}e\_\{t\}^\{\(j\-1\)\}\+B\_\{t\}^\{\(j\)\}e\_\{t\}^\{\(0\)\}\+r\_\{t\}^\{\(j\)\}\.\(30\)This also applies to the remaining updates of roundkk, withBk\(j\)=0B\_\{k\}^\{\(j\)\}=0\. We next control the perturbation size\. The uniform Jacobian bounds are

‖∂ϑFt\(j\)​\(ϑ,β\)‖2≤1\+ηt​Λ,‖∂βFt\(j\)​\(ϑ,β\)‖2≤ηt​G​S\.\\\|\\partial\_\{\\vartheta\}F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)\\\|\_\{2\}\\leq 1\+\\eta\_\{t\}\\Lambda,\\qquad\\\|\\partial\_\{\\beta\}F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)\\\|\_\{2\}\\leq\\eta\_\{t\}GS\.Integrating these bounds along the line segment between the two inputs gives

‖Ft\(j\)​\(ϑ\+u,β\+v\)−Ft\(j\)​\(ϑ,β\)‖2≤\(1\+ηt​Λ\)​‖u‖2\+ηt​G​S​‖v‖2\.\\displaystyle\\\|F\_\{t\}^\{\(j\)\}\(\\vartheta\+u,\\beta\+v\)\-F\_\{t\}^\{\(j\)\}\(\\vartheta,\\beta\)\\\|\_\{2\}\\leq\(1\+\\eta\_\{t\}\\Lambda\)\\\|u\\\|\_\{2\}\+\\eta\_\{t\}GS\\\|v\\\|\_\{2\}\.For roundkk, the second term can be omitted\. Thus both the true perturbation and its first\-order approximation grow by at most a factorqt:=1\+ηt​Λ¯q\_\{t\}:=1\+\\eta\_\{t\}\\overline\{\\Lambda\}per update, when measured relative to the maximum perturbation encountered so far\. The learning\-rate assumption implies

∑t=kTM​ηt​Λ¯≤T−k\+1T−k≤2\.\\sum\_\{t=k\}^\{T\}M\\eta\_\{t\}\\overline\{\\Lambda\}\\leq\\frac\{T\-k\+1\}\{T\-k\}\\leq 2\.Starting from the injection bound‖δk\(l\)‖2=‖Δk\(l\)‖2≤ηk​G/b\\\|\\delta\_\{k\}^\{\(l\)\}\\\|\_\{2\}=\\\|\\Delta\_\{k\}^\{\(l\)\}\\\|\_\{2\}\\leq\\eta\_\{k\}G/b, we therefore obtain

D\\displaystyle D:=max\(t,j\)​at or after​\(k,l\)⁡\{‖δt\(j\)‖2,‖Δt\(j\)‖2\}≤ηk​Gb​∏t=kTqtM≤e2​ηk​Gb≤e2​GM​Λ¯​b\.\\displaystyle:=\\max\_\{\(t,j\)\\text\{ at or after \}\(k,l\)\}\\left\\\{\\\|\\delta\_\{t\}^\{\(j\)\}\\\|\_\{2\},\\\|\\Delta\_\{t\}^\{\(j\)\}\\\|\_\{2\}\\right\\\}\\leq\\frac\{\\eta\_\{k\}G\}\{b\}\\prod\_\{t=k\}^\{T\}q\_\{t\}^\{M\}\\leq\\frac\{e^\{2\}\\eta\_\{k\}G\}\{b\}\\leq\\frac\{e^\{2\}G\}\{M\\overline\{\\Lambda\}b\}\.\(31\)Here we usedT−k≥1T\-k\\geq 1in the last inequality\. The sampling perturbation at the start of roundkkis zero\. Consequently, \([29](https://arxiv.org/html/2610.28883#A2.E29)\) gives, for every update after the injection,

‖rt\(j\)‖2≤Rt,Rt:=2​ηt​K​D2\.\\\|r\_\{t\}^\{\(j\)\}\\\|\_\{2\}\\leq R\_\{t\},\\qquad R\_\{t\}:=2\\eta\_\{t\}KD^\{2\}\.
Define

Ek:=maxl≤j≤M∥ek\(j\)∥2,Et:=max0≤j≤M∥et\(j\)∥2\(t\>k\),E\_\{k\}:=\\max\_\{l\\leq j\\leq M\}\\\|e\_\{k\}^\{\(j\)\}\\\|\_\{2\},\\qquad E\_\{t\}:=\\max\_\{0\\leq j\\leq M\}\\\|e\_\{t\}^\{\(j\)\}\\\|\_\{2\}\\quad\(t\>k\),and letat:=M​ηt​Λ¯a\_\{t\}:=M\\eta\_\{t\}\\overline\{\\Lambda\}\. Taking norms in \([30](https://arxiv.org/html/2610.28883#A2.E30)\) and applying the triangle inequality yields

‖et\(j\)‖2≤qt​max⁡\{‖et\(j−1\)‖2,‖et\(0\)‖2\}\+Rt\.\\\|e\_\{t\}^\{\(j\)\}\\\|\_\{2\}\\leq q\_\{t\}\\max\\left\\\{\\\|e\_\{t\}^\{\(j\-1\)\}\\\|\_\{2\},\\\|e\_\{t\}^\{\(0\)\}\\\|\_\{2\}\\right\\\}\+R\_\{t\}\.Fort\>kt\>k, usinget\(0\)=et−1\(M\)e\_\{t\}^\{\(0\)\}=e\_\{t\-1\}^\{\(M\)\}and iterating within the round gives

Et\\displaystyle E\_\{t\}≤qtM​Et−1\+Rt​∑i=0M−1qti\\displaystyle\\leq q\_\{t\}^\{M\}E\_\{t\-1\}\+R\_\{t\}\\sum\_\{i=0\}^\{M\-1\}q\_\{t\}^\{i\}≤eat​\(Et−1\+M​Rt\)\.\\displaystyle\\leq e^\{a\_\{t\}\}\\bigl\(E\_\{t\-1\}\+MR\_\{t\}\\bigr\)\.In roundkk, we instead start fromek\(l\)=0e\_\{k\}^\{\(l\)\}=0\. The remainingM−lM\-lupdates satisfy

Ek≤Rk​∑i=0M−l−1qki≤M​eak​Rk,E\_\{k\}\\leq R\_\{k\}\\sum\_\{i=0\}^\{M\-l\-1\}q\_\{k\}^\{i\}\\leq Me^\{a\_\{k\}\}R\_\{k\},where the sum is empty whenl=Ml=M\. In particular, the error accumulated during roundkkis included\.

Unrolling the round\-wise bounds therefore gives

ET\\displaystyle E\_\{T\}≤M​∑t=kTRt​exp⁡\(∑s=tTas\)\\displaystyle\\leq M\\sum\_\{t=k\}^\{T\}R\_\{t\}\\exp\\left\(\\sum\_\{s=t\}^\{T\}a\_\{s\}\\right\)≤2​e2​M​K​D2​∑t=kTηt\.\\displaystyle\\leq 2e^\{2\}MKD^\{2\}\\sum\_\{t=k\}^\{T\}\\eta\_\{t\}\.Finally,

∑t=kTηt≤T−k\+1\(T−k\)​M​Λ¯≤2M​Λ¯\.\\sum\_\{t=k\}^\{T\}\\eta\_\{t\}\\leq\\frac\{T\-k\+1\}\{\(T\-k\)M\\overline\{\\Lambda\}\}\\leq\\frac\{2\}\{M\\overline\{\\Lambda\}\}\.Combining this with \([31](https://arxiv.org/html/2610.28883#A2.E31)\), we conclude

‖θ~T\(M\)−θT\(M\)−ΔT\(M\)‖2\\displaystyle\\bigl\\\|\\widetilde\{\\theta\}\_\{T\}^\{\(M\)\}\-\\theta\_\{T\}^\{\(M\)\}\-\\Delta\_\{T\}^\{\(M\)\}\\bigr\\\|\_\{2\}=‖eT\(M\)‖2\\displaystyle=\\\|e\_\{T\}^\{\(M\)\}\\\|\_\{2\}≤ET\\displaystyle\\leq E\_\{T\}≤4​e2​K​D2Λ¯\\displaystyle\\leq\\frac\{4e^\{2\}KD^\{2\}\}\{\\overline\{\\Lambda\}\}≤4​e6​G2​\(LH\+Λ¯​S\+G​LP\)M2​b2​Λ¯3\.\\displaystyle\\leq\\frac\{4e^\{6\}G^\{2\}\\bigl\(L\_\{H\}\+\\overline\{\\Lambda\}S\+GL\_\{P\}\\bigr\)\}\{M^\{2\}b^\{2\}\\overline\{\\Lambda\}^\{\\,3\}\}\.This proves the result with the universal constantc=4​e6c=4e^\{6\}\. ∎

### B\.2Relative Bound between Stage\-Wise Operators

Theorem B\.2:Nonconvex relative gap between sampling\-aware and optimization\-only operatorsUnder the smoothness and bounded\-gradient assumptions in Theorem[B\.1](https://arxiv.org/html/2610.28883#A2.SS1), assume in addition that, for each future staget\>kt\>k, the sampling familyPt\(⋅∣θ\)P\_\{t\}\(\\cdot\\mid\\theta\)is KL\-smooth with score second moment bounded byLtL\_\{t\}, and thatηt​Λ<1\\eta\_\{t\}\\Lambda<1\.Then, for every future staget\>kt\>k,‖Ct−At‖2≤ρt​‖At‖2,ρt:=G​Lt​\(1\+ηt​Λ\)M−1Λ\(1−ηt​Λ\)M\.\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq\\rho\_\{t\}\\\|A\_\{t\}\\\|\_\{2\},\\qquad\\rho\_\{t\}:=\\frac\{G\\sqrt\{L\_\{t\}\}\\,\\frac\{\(1\+\\eta\_\{t\}\\Lambda\)^\{M\}\-1\}\{\\Lambda\}\}\{\(1\-\\eta\_\{t\}\\Lambda\)^\{M\}\}\.\(32\)

###### Proof\.

Fix a future staget\>kt\>k\. Recall that

Ct=At\+∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\),At=∏j=M1At\(j\)\.C\_\{t\}=A\_\{t\}\+\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\},\\qquad A\_\{t\}=\\prod\_\{j=M\}^\{1\}A\_\{t\}^\{\(j\)\}\.Therefore,

Ct−At=∑q=1M\(∏j=Mq\+1At\(j\)\)​Bt\(q\)\.C\_\{t\}\-A\_\{t\}=\\sum\_\{q=1\}^\{M\}\\left\(\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\)B\_\{t\}^\{\(q\)\}\.Taking operator norms and using submultiplicativity,

‖Ct−At‖2≤∑q=1M‖∏j=Mq\+1At\(j\)‖2​‖Bt\(q\)‖2\.\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq\\sum\_\{q=1\}^\{M\}\\left\\\|\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\\\|\_\{2\}\\\|B\_\{t\}^\{\(q\)\}\\\|\_\{2\}\.
By smoothness,

‖At\(j\)‖2=‖I−ηtb​Ht\(j\)‖2≤1\+ηt​Λ\.\\\|A\_\{t\}^\{\(j\)\}\\\|\_\{2\}=\\left\\\|I\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\}\\right\\\|\_\{2\}\\leq 1\+\\eta\_\{t\}\\Lambda\.Thus,

‖∏j=Mq\+1At\(j\)‖2≤\(1\+ηt​Λ\)M−q\.\\left\\\|\\prod\_\{j=M\}^\{q\+1\}A\_\{t\}^\{\(j\)\}\\right\\\|\_\{2\}\\leq\(1\+\\eta\_\{t\}\\Lambda\)^\{M\-q\}\.By the bounded\-gradient assumption and the KL\-smoothness assumption,

‖Bt\(q\)‖2≤ηt​G​Lt\.\\\|B\_\{t\}^\{\(q\)\}\\\|\_\{2\}\\leq\\eta\_\{t\}G\\sqrt\{L\_\{t\}\}\.Therefore,

‖Ct−At‖2≤ηt​G​Lt​∑q=1M\(1\+ηt​Λ\)M−q\.\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq\\eta\_\{t\}G\\sqrt\{L\_\{t\}\}\\sum\_\{q=1\}^\{M\}\(1\+\\eta\_\{t\}\\Lambda\)^\{M\-q\}\.Using the geometric sum,

∑q=1M\(1\+ηt​Λ\)M−q=\(1\+ηt​Λ\)M−1ηt​Λ,\\sum\_\{q=1\}^\{M\}\(1\+\\eta\_\{t\}\\Lambda\)^\{M\-q\}=\\frac\{\(1\+\\eta\_\{t\}\\Lambda\)^\{M\}\-1\}\{\\eta\_\{t\}\\Lambda\},we obtain

‖Ct−At‖2≤G​Lt​\(1\+ηt​Λ\)M−1Λ\.\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq G\\sqrt\{L\_\{t\}\}\\frac\{\(1\+\\eta\_\{t\}\\Lambda\)^\{M\}\-1\}\{\\Lambda\}\.
It remains to relate this bound to‖At‖2\\\|A\_\{t\}\\\|\_\{2\}\. Sinceηt​Λ<1\\eta\_\{t\}\\Lambda<1, for every vectorvv,

‖At\(j\)​v‖2=‖v−ηtb​Ht\(j\)​v‖2≥‖v‖2−ηtb​‖Ht\(j\)​v‖2≥\(1−ηt​Λ\)​‖v‖2\.\\\|A\_\{t\}^\{\(j\)\}v\\\|\_\{2\}=\\left\\\|v\-\\frac\{\\eta\_\{t\}\}\{b\}H\_\{t\}^\{\(j\)\}v\\right\\\|\_\{2\}\\geq\\\|v\\\|\_\{2\}\-\\frac\{\\eta\_\{t\}\}\{b\}\\\|H\_\{t\}^\{\(j\)\}v\\\|\_\{2\}\\geq\(1\-\\eta\_\{t\}\\Lambda\)\\\|v\\\|\_\{2\}\.Applying this bound through the product gives

‖At​v‖2≥\(1−ηt​Λ\)M​‖v‖2\.\\\|A\_\{t\}v\\\|\_\{2\}\\geq\(1\-\\eta\_\{t\}\\Lambda\)^\{M\}\\\|v\\\|\_\{2\}\.Hence,

‖At‖2≥\(1−ηt​Λ\)M\.\\\|A\_\{t\}\\\|\_\{2\}\\geq\(1\-\\eta\_\{t\}\\Lambda\)^\{M\}\.Combining this with the absolute bound yields

‖Ct−At‖2≤G​Lt​\(1\+ηt​Λ\)M−1Λ\(1−ηt​Λ\)M​‖At‖2\.\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq\\frac\{G\\sqrt\{L\_\{t\}\}\\frac\{\(1\+\\eta\_\{t\}\\Lambda\)^\{M\}\-1\}\{\\Lambda\}\}\{\(1\-\\eta\_\{t\}\\Lambda\)^\{M\}\}\\\|A\_\{t\}\\\|\_\{2\}\.This proves \([32](https://arxiv.org/html/2610.28883#A2.E32)\)\. ∎

### B\.3Accumulated Relative Gap

Theorem B\.3:Accumulated nonconvex relative gapUnder the assumptions of Theorem[B\.2](https://arxiv.org/html/2610.28883#A2.SS2), for anyT\>kT\>k,‖∏t=Tk\+1Ct−∏t=Tk\+1At‖2≤\[∏t=k\+1T\(1\+ρt\)−1\]​∏t=k\+1T‖At‖2\.\\left\\\|\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\-\\prod\_\{t=T\}^\{k\+1\}A\_\{t\}\\right\\\|\_\{2\}\\leq\\left\[\\prod\_\{t=k\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\-1\\right\]\\prod\_\{t=k\+1\}^\{T\}\\\|A\_\{t\}\\\|\_\{2\}\.\(33\)Furthermore, if the learning rate takes the harmonic formηt=η0M⁡\(t−k\),\\eta\_\{t\}=\\frac\{\\eta\_\{0\}\}\{M\(t\-k\)\},withη0​Λ<1\\eta\_\{0\}\\Lambda<1, and ifLmax:=supk<t≤TLtL\_\{\\max\}:=\\sup\_\{k<t\\leq T\}L\_\{t\}, then, withH:=T−kH:=T\-k,‖∏t=Tk\+1Ct−∏t=Tk\+1At‖2≤\(eα​Hα−1\)​∏t=k\+1T‖At‖2,\\left\\\|\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\-\\prod\_\{t=T\}^\{k\+1\}A\_\{t\}\\right\\\|\_\{2\}\\leq\\left\(e^\{\\alpha\}H^\{\\alpha\}\-1\\right\)\\prod\_\{t=k\+1\}^\{T\}\\\|A\_\{t\}\\\|\_\{2\},\(34\)whereα:=G​Lmax​η0\(1−η0​Λ\)2\.\\alpha:=\\frac\{G\\sqrt\{L\_\{\\max\}\}\\eta\_\{0\}\}\{\(1\-\\eta\_\{0\}\\Lambda\)^\{2\}\}\.

###### Proof\.

We first prove the general accumulated product bound\. Define

PC​\(k,T\)=∏t=Tk\+1Ct,PA​\(k,T\)=∏t=Tk\+1At\.P\_\{C\}\(k,T\)=\\prod\_\{t=T\}^\{k\+1\}C\_\{t\},\\qquad P\_\{A\}\(k,T\)=\\prod\_\{t=T\}^\{k\+1\}A\_\{t\}\.Using the telescoping expansion for products,

PC​\(k,T\)−PA​\(k,T\)=∑s=k\+1T\(∏t=Ts\+1Ct\)​\(Cs−As\)​\(∏t=s−1k\+1At\)\.P\_\{C\}\(k,T\)\-P\_\{A\}\(k,T\)=\\sum\_\{s=k\+1\}^\{T\}\\left\(\\prod\_\{t=T\}^\{s\+1\}C\_\{t\}\\right\)\(C\_\{s\}\-A\_\{s\}\)\\left\(\\prod\_\{t=s\-1\}^\{k\+1\}A\_\{t\}\\right\)\.Taking norms gives

‖PC​\(k,T\)−PA​\(k,T\)‖2≤∑s=k\+1T\(∏t=s\+1T‖Ct‖2\)​‖Cs−As‖2​\(∏t=k\+1s−1‖At‖2\)\.\\\|P\_\{C\}\(k,T\)\-P\_\{A\}\(k,T\)\\\|\_\{2\}\\leq\\sum\_\{s=k\+1\}^\{T\}\\left\(\\prod\_\{t=s\+1\}^\{T\}\\\|C\_\{t\}\\\|\_\{2\}\\right\)\\\|C\_\{s\}\-A\_\{s\}\\\|\_\{2\}\\left\(\\prod\_\{t=k\+1\}^\{s\-1\}\\\|A\_\{t\}\\\|\_\{2\}\\right\)\.By Theorem[B\.2](https://arxiv.org/html/2610.28883#A2.SS2),

‖Cs−As‖2≤ρs​‖As‖2\.\\\|C\_\{s\}\-A\_\{s\}\\\|\_\{2\}\\leq\\rho\_\{s\}\\\|A\_\{s\}\\\|\_\{2\}\.Moreover,

‖Ct‖2≤‖At‖2\+‖Ct−At‖2≤\(1\+ρt\)​‖At‖2\.\\\|C\_\{t\}\\\|\_\{2\}\\leq\\\|A\_\{t\}\\\|\_\{2\}\+\\\|C\_\{t\}\-A\_\{t\}\\\|\_\{2\}\\leq\(1\+\\rho\_\{t\}\)\\\|A\_\{t\}\\\|\_\{2\}\.Substituting these inequalities gives

‖PC​\(k,T\)−PA​\(k,T\)‖2≤∑s=k\+1T\(∏t=s\+1T\(1\+ρt\)​‖At‖2\)​ρs​‖As‖2​\(∏t=k\+1s−1‖At‖2\)\.\\\|P\_\{C\}\(k,T\)\-P\_\{A\}\(k,T\)\\\|\_\{2\}\\leq\\sum\_\{s=k\+1\}^\{T\}\\left\(\\prod\_\{t=s\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\\\|A\_\{t\}\\\|\_\{2\}\\right\)\\rho\_\{s\}\\\|A\_\{s\}\\\|\_\{2\}\\left\(\\prod\_\{t=k\+1\}^\{s\-1\}\\\|A\_\{t\}\\\|\_\{2\}\\right\)\.Factoring out the full product of‖At‖2\\\|A\_\{t\}\\\|\_\{2\}gives

‖PC​\(k,T\)−PA​\(k,T\)‖2≤\[∑s=k\+1Tρs​∏t=s\+1T\(1\+ρt\)\]​∏t=k\+1T‖At‖2\.\\\|P\_\{C\}\(k,T\)\-P\_\{A\}\(k,T\)\\\|\_\{2\}\\leq\\left\[\\sum\_\{s=k\+1\}^\{T\}\\rho\_\{s\}\\prod\_\{t=s\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\\right\]\\prod\_\{t=k\+1\}^\{T\}\\\|A\_\{t\}\\\|\_\{2\}\.Finally, using the identity

∑s=k\+1Tρs​∏t=s\+1T\(1\+ρt\)=∏t=k\+1T\(1\+ρt\)−1,\\sum\_\{s=k\+1\}^\{T\}\\rho\_\{s\}\\prod\_\{t=s\+1\}^\{T\}\(1\+\\rho\_\{t\}\)=\\prod\_\{t=k\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\-1,we obtain

‖∏t=Tk\+1Ct−∏t=Tk\+1At‖2≤\[∏t=k\+1T\(1\+ρt\)−1\]​∏t=k\+1T‖At‖2\.\\left\\\|\\prod\_\{t=T\}^\{k\+1\}C\_\{t\}\-\\prod\_\{t=T\}^\{k\+1\}A\_\{t\}\\right\\\|\_\{2\}\\leq\\left\[\\prod\_\{t=k\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\-1\\right\]\\prod\_\{t=k\+1\}^\{T\}\\\|A\_\{t\}\\\|\_\{2\}\.
We next prove the harmonic\-learning\-rate refinement\. From Theorem[B\.2](https://arxiv.org/html/2610.28883#A2.SS2),

ρt=G​Lt​\(1\+ηt​Λ\)M−1Λ​\(1−ηt​Λ\)M\.\\rho\_\{t\}=G\\sqrt\{L\_\{t\}\}\\frac\{\(1\+\\eta\_\{t\}\\Lambda\)^\{M\}\-1\}\{\\Lambda\(1\-\\eta\_\{t\}\\Lambda\)^\{M\}\}\.Letxt=ηt​Λx\_\{t\}=\\eta\_\{t\}\\Lambda\. Sinceηt=η0/\(M⁡\(t−k\)\)\\eta\_\{t\}=\\eta\_\{0\}/\(M\(t\-k\)\), we have

M​xt=η0​Λt−k≤η0​Λ<1\.Mx\_\{t\}=\\frac\{\\eta\_\{0\}\\Lambda\}\{t\-k\}\\leq\\eta\_\{0\}\\Lambda<1\.Using

\(1\+xt\)M−1≤M​xt1−M​xt,\(1−xt\)−M≤11−M​xt,\(1\+x\_\{t\}\)^\{M\}\-1\\leq\\frac\{Mx\_\{t\}\}\{1\-Mx\_\{t\}\},\\qquad\(1\-x\_\{t\}\)^\{\-M\}\\leq\\frac\{1\}\{1\-Mx\_\{t\}\},we get

ρt≤G​Lt​M​ηt\(1−M​ηt​Λ\)2\.\\rho\_\{t\}\\leq G\\sqrt\{L\_\{t\}\}\\frac\{M\\eta\_\{t\}\}\{\(1\-M\\eta\_\{t\}\\Lambda\)^\{2\}\}\.Since

1−M​ηt​Λ=1−η0​Λt−k≥1−η0​Λ,1\-M\\eta\_\{t\}\\Lambda=1\-\\frac\{\\eta\_\{0\}\\Lambda\}\{t\-k\}\\geq 1\-\\eta\_\{0\}\\Lambda,we have

ρt≤G​Lt​M​ηt\(1−η0​Λ\)2=G​Lt​η0\(1−η0​Λ\)2​1t−k\.\\rho\_\{t\}\\leq\\frac\{G\\sqrt\{L\_\{t\}\}M\\eta\_\{t\}\}\{\(1\-\\eta\_\{0\}\\Lambda\)^\{2\}\}=\\frac\{G\\sqrt\{L\_\{t\}\}\\eta\_\{0\}\}\{\(1\-\\eta\_\{0\}\\Lambda\)^\{2\}\}\\frac\{1\}\{t\-k\}\.UsingLt≤LmaxL\_\{t\}\\leq L\_\{\\max\}gives

ρt≤G​Lmax​η0\(1−η0​Λ\)2​1t−k=αt−k\.\\rho\_\{t\}\\leq\\frac\{G\\sqrt\{L\_\{\\max\}\}\\eta\_\{0\}\}\{\(1\-\\eta\_\{0\}\\Lambda\)^\{2\}\}\\frac\{1\}\{t\-k\}=\\frac\{\\alpha\}\{t\-k\}\.Therefore,

∑t=k\+1Tρt≤α​∑h=1H1h≤α⁡\(1\+log⁡H\)\.\\sum\_\{t=k\+1\}^\{T\}\\rho\_\{t\}\\leq\\alpha\\sum\_\{h=1\}^\{H\}\\frac\{1\}\{h\}\\leq\\alpha\(1\+\\log H\)\.Finally,

∏t=k\+1T\(1\+ρt\)−1≤exp⁡\(∑t=k\+1Tρt\)−1≤exp⁡\(α⁡\(1\+log⁡H\)\)−1\.\\prod\_\{t=k\+1\}^\{T\}\(1\+\\rho\_\{t\}\)\-1\\leq\\exp\\left\(\\sum\_\{t=k\+1\}^\{T\}\\rho\_\{t\}\\right\)\-1\\leq\\exp\\\!\\big\(\\alpha\(1\+\\log H\)\\big\)\-1\.Substituting this into \([33](https://arxiv.org/html/2610.28883#A2.E33)\) proves \([34](https://arxiv.org/html/2610.28883#A2.E34)\)\. ∎

## Appendix CDetailed Experimental Setup

In this section, we provide details of the experimental setup, hyperparameter selection, and implementation details\. All experiments are conducted using Amazon Web Services \(AWS\) c7a high\-performance CPUs and 4 NVIDIA A40 GPUs, each with 46GB of memory\.

### C\.1Experimental Setup for Traditional RL

##### Environment description\.

Table[2](https://arxiv.org/html/2610.28883#A3.T2)summarizes the RL environments used in our experiments\. The MuJoCo environments used in this work are distributed under the Apache 2\.0 License, while the Gymnasium environments and wrappers, including Box2D and FrozenLake environments, are distributed under the MIT License\.

Table 2:Summary of RL environments used in experiments\. MuJoCo environments are from[Todorov et al\. \[2012\]](https://arxiv.org/html/2610.28883#bib.bib12), while Box2D and FrozenLake environments are from Gym/Gymnasium\[[Brockman et al\., 2016](https://arxiv.org/html/2610.28883#bib.bib13),[Towers et al\., 2025](https://arxiv.org/html/2610.28883#bib.bib14)\]\.
##### Training setup\.

We useStable\-Baselines3\[[Raffin et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib15)\]\(MIT license\) as our training framework for all the traditional RL tasks\. We implement a KL\-regularized PPO\[[Schulman et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib5)\], using KL estimated via thek3k\_\{3\}approximation\[[Schulman, 2020](https://arxiv.org/html/2610.28883#bib.bib16)\]

KL^\(πref∥πθ\)=\(r−1\)−logr,r=πθ\(a∣s\)/πref\(a∣s\)\\widehat\{\\mathrm\{KL\}\}\(\\pi\_\{\\mathrm\{ref\}\}\\,\\\|\\,\\pi\_\{\\theta\}\)=\(r\-1\)\-\\log r,\\quad r=\\pi\_\{\\theta\}\(a\\mid s\)/\\pi\_\{\\mathrm\{ref\}\}\(a\\mid s\)
We summarize the detailed hyperparameter selection in table[3](https://arxiv.org/html/2610.28883#A3.T3)\. For base hyperparameters selection, we refer to RL Baselines3 Zoo222[https://github\.com/DLR\-RM/rl\-baselines3\-zoo](https://github.com/DLR-RM/rl-baselines3-zoo), with some minor tuning\. We perform a grid search over KL coefficientsβ\\betaranging from 0\.0 to 5\.0, and select the value that yields the best standard training performance\. Similarly, we search for theρ\\rhothat produces the highest and most stable performance for the baseline method IIF\. For network architecture, we use the defaultMlpPolicyprovided byStable\-Baselines3\.

Table 3:PPO/training hyperparameters per environment\.
##### Evaluation setup\.

During evaluation, we evaluate the stochastic performance of each policy averaged over multiple evaluation episodes\. For most continuous control environments \(Ant, HalfCheetah, Hopper\), we average over 32 episodes by default; for BipedalWalker, we use 100 episodes to reduce variance due to its longer horizon; and for light\-weight environments, LunarLander and FrozenLake, we use 1000 episodes\. Each setup is run with three independent seeds; we report mean±\\pmstandard error\.

### C\.2Experimental Setup for RLHF

##### Base model\.

We use GPT\-Neo 2\.7B\[[Black et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib23)\]\(EleutherAI/gpt\-neo\-2\.7B, MIT license\) as the base language model\. We apply LoRA\[[Hu et al\., 2022](https://arxiv.org/html/2610.28883#bib.bib24)\]for parameter\-efficient fine\-tuning, with rankr=16r=16, scaling factorα=32\\alpha=32, and dropout rate0\.050\.05\.

##### Training data\.

We train on the Real Toxicity Prompts dataset\[[Gehman et al\., 2020](https://arxiv.org/html/2610.28883#bib.bib25)\]\(allenai/real\-toxicity\-prompts, Apache\-2\.0 license\)\. We retain examples with a prompt toxicity score above0\.30\.3, use the 80% training split, and shuffle with a fixed seed\. For each example, we concatenate the prompt and continuation fields, tokenize, and truncate to a length sampled uniformly in\[10,15\]\[10,15\]tokens to form the PPO prompt\. The policy then generates a 30\-token continuation with temperature1\.01\.0and top\-p=1\.0p=1\.0\.

##### Reward model\.

The reward signal is provided by the RoBERTa\-based hate\-speech classifierfacebook/roberta\-hate\-speech\-dynabench\-r4\-target\[[Vidgen et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib26)\]\. Following the reference implementation, we use the raw logit of thenothateclass \(index00\) as the scalar reward: a higher logit indicates less toxic output\.

##### Training setup\.

We optimize a PPO objective with a fixed KL penalty and no probability\-ratio clipping, with a learning rate of5×10−65\\times 10^\{\-6\}\. The per\-token KL divergence from the reference policy is penalized with coefficientβ=0\.4\\beta=0\.4, estimated via thek1k\_\{1\}approximation\[[Schulman, 2020](https://arxiv.org/html/2610.28883#bib.bib16)\]\. Each rollout collects a batch of6464samples; PPO updates use a mini\-batch size of88over44epochs, yielding3232gradient steps per rollout\. We use the Adam optimizer with gradient norm clipping at1\.01\.0and no weight decay\. For each run, we train for300300rollout steps\. All runs use bfloat16 precision with Flash Attention 2\[[Dao, 2024](https://arxiv.org/html/2610.28883#bib.bib17)\]\.

For data attribution setups, we useH=2H=2for RINSE, andρ=10%\\rho=10\\%for both IIF and RINSE for fairness\.

##### Evaluation setup\.

We evaluate on the Wiki\-Toxic dataset\[[Wulczyn et al\., 2017](https://arxiv.org/html/2610.28883#bib.bib27)\]\(OxAISH\-AL\-LLM/wiki\_toxic, test split\), drawn from a different distribution than the training data\. We select the first200200toxic examples \(label=1=1\) and use up to the first2,0002\{,\}000characters of each comment as the evaluation prompt—substantially longer than training prompts and thus more likely to elicit toxic continuations\. Continuations of3030tokens are generated under the same sampling parameters as training\. To guard against reward hacking specific to the training signal, we score continuations with a*separate*classifier,DaNLP/da\-electra\-hatespeech\-detection\[[Pauli et al\., 2021](https://arxiv.org/html/2610.28883#bib.bib28)\], and report the mean toxicity probability across the200200samples\. Each method is run with three independent seeds; we report mean±\\pmstandard error\.

## Appendix DAdditional Experimental Results

### D\.1Further Details on Linear Data Modeling Scores and Evaluation

Directly validating attribution methods using single\-sample leave\-one\-out \(LOO\) correlation can be challenging in online RL due to the extremely low signal\-to\-noise ratio of individual sample effects\. In particular, removing a single rollout record often induces only a negligible change in the final policy behavior, while stochasticity from environment interaction introduces substantial variance across retraining runs\. To obtain a more stable evaluation protocol, we additionally adopt the linear data modeling score \(LDS\), which evaluates whether attribution scores can linearly predict the effect of removing a subset of training data\. Concretely, LDS samples random subsets of rollout records, sums the attribution scores within each subset, and compares the aggregated score against the actual performance change after retraining without that subset\. This evaluation implicitly assumes approximate linear additivity of data influence across samples, and therefore measures how well attribution scores behave as a linear proxy for collective data influence\.

##### Experimental setup\.

For the LDS evaluation, we consider model training with KL\-regularized PPO on two consecutive rounds\. We construct 100 random subsets of training data, each containing 4096 samples\. For each subset, we remove it and retrain the model with future rollout data resampled using the updated policy for 32 independent runs, and compute the average change in validation loss\. All results are further averaged over 3 random seeds\. We conduct the experiments on two MuJoCo benchmark environments\[[Todorov et al\., 2012](https://arxiv.org/html/2610.28883#bib.bib12)\],Ant\-v4andHalfCheetah\-v4\.

Table 4:Linear data modeling scores of attribution methods onAnt\-v4andHalfCheetah\-v4\.Table[4](https://arxiv.org/html/2610.28883#A4.T4)reports LDS results onAnt\-v4andHalfCheetah\-v4\. We additionally report Pearson correlations for completeness, which exhibit the same overall trend as the Spearman correlations in the main text\. Both RL\-IF\-opt and RL\-IF\-full substantially outperform TracIn and random baselines across environments and correlation metrics, indicating that the proposed non\-local attribution framework better captures the long\-term effect of rollout data on downstream policy behavior\.

#### D\.1\.1Direct Leave\-One\-Out Evaluation

To complement the subset\-removal LDS evaluation, we directly evaluate LOO attribution on FrozenLake with attribution horizonH=2H=2\. Unlike the fixed\-rollout experiment reported separately, this evaluation allows future rollout data to change with the perturbed policy\. We use 256 retraining runs to estimate the deletion effects and compare them with the attribution scores of TracIn, RL\-IF\-opt, and RL\-IF\-full\.

Table 5:Direct LOO attribution evaluation on FrozenLake withH=2H=2\. Correlations are reported as per\-seed means with standard errors\.RL\-IF\-full achieves the highest mean correlation in this experiment, outperforming both RL\-IF\-opt and TracIn\. This provides a setting in which explicitly accounting for policy\-induced sampling changes improves agreement with the estimated LOO effects\.

#### D\.1\.2Empirical Characterization of the Sampling Effect

To examine why the sampling contribution differs across environments, we measure the top Hessian eigenvalue of the KL\-regularized loss, denoted byΛ\\Lambda, the rollout\-score second moment, summarized byLmax\\sqrt\{L\_\{\\max\}\}, and the learning rateη\\etawithin the attribution window\. Table[6](https://arxiv.org/html/2610.28883#A4.T6)reports these measurements and the diagnostic ratioη​Lmax/Λ\\eta\\sqrt\{L\_\{\\max\}\}/\\Lambda\.

Table 6:Empirical diagnostics of the sampling contribution\.FrozenLake has a much smaller measured curvature scale and a larger learning rate than Ant and HalfCheetah\. Its reported diagnostic ratio is approximately four orders of magnitude larger than Ant’s\. Together with the direct LOO comparison in Table[5](https://arxiv.org/html/2610.28883#A4.T5), these measurements identify FrozenLake as an empirical setting in which retaining the sampling correction is useful\. In contrast, the full and optimization\-only estimators have nearly identical LDS on Ant and HalfCheetah\.

### D\.2Additional Results for LOO

![Refer to caption](https://arxiv.org/html/2610.28883v1/figures/loo/row_tracin.png)

![Refer to caption](https://arxiv.org/html/2610.28883v1/figures/loo/row_rlinf.png)

Figure 6:Comparison between TracIn and RL\-IF\. The top row shows the comparison between the TracIn method and the ground\-truth LOO values, while the bottom row shows the comparison between RL\-IF and the ground\-truth LOO values under the same held\-out settings\. From left to right, the panels show the attribution horizonHHfrom 1 to 5\. The colorbar \(point index\) is shared across each row and shown once on the right\. Each panel keeps its own axes because the attribution and LOO magnitudes differ across horizons\.Additional LOO results are presented in Figure[6](https://arxiv.org/html/2610.28883#A4.F6)\. We compare our RL\-IF score with TracIn and ground truth \(LOO Retrain\), under the fixed rollout setting, where we do not consider the sampling effect\. RL\-IF shows high correlation with the ground truth LOO retrain, with correlation decaying slowly with respect to the attribution horizon\. TracIn shows high correlation with ground truth when attribution horizon equals to 1, but fails to capture the long horizon effect\.

### D\.3Ablation on Algorithmic Design

In this section, we conduct ablation studies to justify our design choices of the horizon selectionHHin RINSE\.

#### D\.3\.1Ablation on Adaptive Horizon

We ablate the effect of the adaptive horizon by adopting a fixed horizon instead, onAnt\-v4,HalfCheetah\-v4,BipedalWalker\-v3, andLunarLander\-v2\. As shown in Figure[7](https://arxiv.org/html/2610.28883#A4.F7), the adaptive horizon matches or exceeds the fixed variant in every case, and the gap is largest on the harder\-exploration tasks: onAnt\-v4andBipedalWalker\-v3the fixed\-horizon runs progress slowly and erratically, while onLunarLander\-v2the adaptive horizon reaches a higher return with lower variance\. This shows that the adaptive horizon is necessary: it stabilizes training and yields consistently higher\-quality data selection without per\-environment tuning\.

RINSE \+ FixedHHRINSE \+ DynamicHH

Figure 7:Dynamic\-horizon ablation\.Fixed horizon vs\. the adaptive \(dynamic\) horizon\. The dynamic horizon matches or outperforms the fixed variant on all four environments, with the largest gains on the harder\-exploration tasks\.
#### D\.3\.2Effect of Horizon in RINSE

We sweep the horizon overH∈\{1,2,4,5,8,10\}H\\in\\\{1,2,4,5,8,10\\\}onAnt\-v4andHalfCheetah\-v4, holding all other hyperparameters fixed\. Figure[8](https://arxiv.org/html/2610.28883#A4.F8)shows a consistent trend that emerges across both environments\. As the horizonHHgrows, the quality of the selected data improves gradually\. This improvement, however, saturates at a moderate horizon of aroundH=5H=5\. Beyond this point, additional horizon brings no further benefit, and for the largest values it can even slightly degrade selection quality\. This motivates some medium horizons, such asH=5H=5, as a robust default\.

H=1H\{=\}1H=2H\{=\}2H=4H\{=\}4H=5H\{=\}5H=8H\{=\}8H=10H\{=\}10

Figure 8:Horizon ablation\.Effect of the different horizonsH∈\{1,2,4,5,8,10\}H\\in\\\{1,2,4,5,8,10\\\}\. Longer horizons speed up early learning, but the benefit saturates aroundH≈4H\\\!\\approx\\\!4–55\.
#### D\.3\.3Effect of the Discard Ratio

We ablate the discard ratioρ\\rhoonAnt\-v4andHalfCheetah\-v4, sweepingρ∈\{0%,5%,10%,20%,50%,100%\}\\rho\\in\\\{0\\%,5\\%,10\\%,20\\%,50\\%,100\\%\\\}, whereρ=0%\\rho=0\\%recovers standard PPO \(no filtering\) andρ=10%\\rho=10\\%is our default\. Figure[9](https://arxiv.org/html/2610.28883#A4.F9)shows that filtering improves performance across the tested ratios onAnt\-v4, whileHalfCheetah\-v4is more sensitive to aggressive filtering\. OnAnt\-v4, any non\-zero discard ratio substantially outperforms the no\-selection baseline, confirming that removing low\-influence data is beneficial\. OnHalfCheetah\-v4,ρ=50%\\rho=50\\%andρ=100%\\rho=100\\%yield lower final mean returns than the no\-selection baseline\. These results support using a moderate discard ratio\.

ρ=0%\\rho\{=\}0\\%\(baseline\)ρ=5%\\rho\{=\}5\\%ρ=10%\\rho\{=\}10\\%\(default\)ρ=20%\\rho\{=\}20\\%ρ=50%\\rho\{=\}50\\%ρ=100%\\rho\{=\}100\\%

Figure 9:Discard\-ratio ablation\.Effect of the discard ratioρ\\rhoonAnt\-v4andHalfCheetah\-v4\. Moderate ratios \(ρ≈10%\\rho\{\\approx\}10\\%–20%20\\%\) are best, while very aggressive filtering can degrade performance\.

### D\.4Further Details on Time and Sample Efficiency

We provide additional details on the computation of time efficiency \(TE\) and sample efficiency \(SE\), along with extended results\. As described in the main text, both metrics are defined relative to the standard PPO baseline, which is normalized to100%100\\%\. Lower values therefore indicate better efficiency\.

##### Evaluation protocol\.

For each method, we track the training trajectory and record the first time step at which a set of target performance levels is reached\. In the main results, we use three thresholds corresponding to90%90\\%,95%95\\%, and100%100\\%of the peak performance achieved by the standard PPO baseline\. TE and SE are then computed by averaging the relative runtime and data usage required to reach these thresholds\.

In addition to this multi\-level evaluation, we also report a stricter variant based on a single threshold at100%100\\%performance\. This provides a complementary view that focuses on the final stage of training\.

##### Results\.

Table[7](https://arxiv.org/html/2610.28883#A4.T7)summarizes both versions of the efficiency metrics\. The upper block \(“100% only”\) corresponds to the single\-threshold evaluation, while the lower block \(“90/95/100%”\) matches the results reported in the main text\.

Table 7:Relative time efficiency \(TE\) and sample efficiency \(SE\) across environments\. All values are normalized to the PPO baseline \(100%100\\%\); lower is better\. Best results per column are in bold\.Across both evaluation protocols, RINSE improves mean sample efficiency over IIF on all six environments and mean time efficiency on five of the six environments\. The gains are particularly pronounced in continuous control tasks such as Ant and HalfCheetah, where RINSE reduces sample usage by a substantial margin\.

Comparing the two settings, the multi\-threshold evaluation \(90/95/100%\) yields slightly more stable estimates, as it averages performance across different stages of training\. In contrast, the100%100\\%\-only metric is more sensitive to late\-stage behavior but leads to similar overall conclusions\.

Overall, these results suggest that the efficiency improvements of RINSE are not limited to a specific performance regime, but persist throughout the training process\.

#### D\.4\.1Wall\-Clock Runtime Breakdown

Table[8](https://arxiv.org/html/2610.28883#A4.T8)reports the wall\-clock runtime per round for each control environment, separating rollout collection, the initial training pass, attribution, and filtered replay\. The baseline cost is the sum of collection and training, while the additional cost is the sum of attribution and replay\. The final column reports this additional cost relative to the baseline\.

Table 8:Wall\-clock runtime in seconds per round on the traditional RL environments\.Attribution and replay add between62%62\\%and128%128\\%of the baseline per\-round cost in these measurements\. This overhead differs from TE, which measures total time required to reach specified performance thresholds\. Faster progress in return can offset the additional per\-round computation, but the resulting wall\-clock benefit depends on the environment\.

### D\.5Additional Results for RLHF

We provide additional details on RLHF training dynamics and efficiency\.

Figure 10:Training curves of Standard, IIF, and RINSE on RLHF toxicity reduction\.##### Training dynamics\.

Figure[10](https://arxiv.org/html/2610.28883#A4.F10)shows the training curves of Standard, IIF, and RINSE on training toxicity and reward\. Standard and IIF exhibit similar learning dynamics, with IIF slightly faster than Standard, while RINSE achieves more significant toxicity reduction together with higher reward under the same number of training steps\.

Figure 11:Average wall time of Standard, IIF, and RINSE on RLHF\.Figure 12:Per\-step wall\-clock time breakdown for Standard, IIF, and RINSE on RLHF\.

##### Time efficiency and runtime breakdown\.

Figure[12](https://arxiv.org/html/2610.28883#A4.F12)reports the wall\-clock time required for each method to first reach the best test toxicity achieved by Standard on each seed\. RINSE reaches the target substantially faster than Standard, demonstrating that non\-local data filtering accelerates convergence\.

Figure[12](https://arxiv.org/html/2610.28883#A4.F12)further breaks down the per\-step wall\-clock time by training phase\. With the Generalized Gauss–Newton matrix[Martens \[2020\]](https://arxiv.org/html/2610.28883#bib.bib20)approximation, RINSE is only ~1\.5×1\.5\\timesslower than standard training per step\. Although RINSE introduces this additional computation per training step, the overhead is compensated by the reduction in required training steps, leading to improved overall wall\-clock efficiency\.

#### D\.5\.1Generation Quality at Matched Toxicity

We evaluate whether the reduction in toxicity is accompanied by a degradation in generation quality on held\-out Wiki\-Toxic prompts\. We compare RINSE with the unperturbed baseline at a matched toxicity level, so that differences in quality are not explained solely by different degrees of toxicity reduction\.

We assess fluency using the perplexity of the generated continuation under a separate language model, GPT\-2 Large\[[Radford et al\., 2019](https://arxiv.org/html/2610.28883#bib.bib45)\], conditioned on the prompt\. This evaluator is independent of both the policy and the training reward model\. We assess diversity using corpus\-level Distinct\-2 and Distinct\-3, and repetition using the within\-sample repeated\-4\-gram rate, Seq\-rep\-4\.

Table 9:Generation quality on held\-out Wiki\-Toxic prompts at matched toxicity\.At matched toxicity, RINSE achieves lower median perplexity, higher Distinct\-2 and Distinct\-3, and a lower repeated\-4\-gram rate than the unperturbed baseline\. These results provide evidence that the toxicity improvement does not come at the cost of the evaluated fluency and diversity measures\.

## NeurIPS Paper Checklist

1. 1\.Claims
2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
3. Answer:\[Yes\]
4. Justification: We tried our best to ensure that the abstract and introduction accurately reflect the paper’s contributions and scope\.
5. Guidelines: - •The answer\[N/A\]means that the abstract and introduction do not include the claims made in the paper\. - •The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations\. A\[No\]or\[N/A\]answer to this question will not be perceived well by the reviewers\. - •The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings\. - •It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper\.
6. 2\.Limitations
7. Question: Does the paper discuss the limitations of the work performed by the authors?
8. Answer:\[Yes\]
9. Justification: We discuss the limitations in Section[6](https://arxiv.org/html/2610.28883#S6)\.
10. Guidelines: - •The answer\[N/A\]means that the paper has no limitation while the answer\[No\]means that the paper has limitations, but those are not discussed in the paper\. - •The authors are encouraged to create a separate “Limitations” section in their paper\. - •The paper should point out any strong assumptions and how robust the results are to violations of these assumptions \(e\.g\., independence assumptions, noiseless settings, model well\-specification, asymptotic approximations only holding locally\)\. The authors should reflect on how these assumptions might be violated in practice and what the implications would be\. - •The authors should reflect on the scope of the claims made, e\.g\., if the approach was only tested on a few datasets or with a few runs\. In general, empirical results often depend on implicit assumptions, which should be articulated\. - •The authors should reflect on the factors that influence the performance of the approach\. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting\. Or a speech\-to\-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon\. - •The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size\. - •If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness\. - •While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper\. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community\. Reviewers will be specifically instructed to not penalize honesty concerning limitations\.
11. 3\.Theory assumptions and proofs
12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete \(and correct\) proof?
13. Answer:\[Yes\]
14. Justification: We provide the theorems along with the assumptions and proof in Appendix[B](https://arxiv.org/html/2610.28883#A2)
15. Guidelines: - •The answer\[N/A\]means that the paper does not include theoretical results\. - •All the theorems, formulas, and proofs in the paper should be numbered and cross\-referenced\. - •All assumptions should be clearly stated or referenced in the statement of any theorems\. - •The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition\. - •Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material\. - •Theorems and Lemmas that the proof relies upon should be properly referenced\.
16. 4\.Experimental result reproducibility
17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper \(regardless of whether the code and data are provided or not\)?
18. Answer:\[Yes\]
19. Justification: We provide detailed information on the experimental setups in Appendix[C](https://arxiv.org/html/2610.28883#A3), and release our code at[https://github\.com/sx\-liu/RL\-IF](https://github.com/sx-liu/RL-IF)\.
20. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •If the paper includes experiments, a\[No\]answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not\. - •If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable\. - •Depending on the contribution, reproducibility can be accomplished in various ways\. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model\. In general\. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model \(e\.g\., in the case of a large language model\), releasing of a model checkpoint, or other means that are appropriate to the research performed\. - •While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution\. For example 1. \(a\)If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm\. 2. \(b\)If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully\. 3. \(c\)If the contribution is a new model \(e\.g\., a large language model\), then there should either be a way to access this model for reproducing the results or a way to reproduce the model \(e\.g\., with an open\-source dataset or instructions for how to construct the dataset\)\. 4. \(d\)We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility\. In the case of closed\-source models, it may be that access to the model is limited in some way \(e\.g\., to registered users\), but it should be possible for other researchers to have some path to reproducing or verifying the results\.
21. 5\.Open access to data and code
22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
23. Answer:\[Yes\]
25. Guidelines: - •The answer\[N/A\]means that paper does not include experiments requiring code\. - • - •While we encourage the release of code and data, we understand that this might not be possible, so\[No\]is an acceptable answer\. Papers cannot be rejected simply for not including code, unless this is central to the contribution \(e\.g\., for a new open\-source benchmark\)\. - •The instructions should contain the exact command and environment needed to run to reproduce the results\. See the NeurIPS code and data submission guidelines \([https://neurips\.cc/public/guides/CodeSubmissionPolicy](https://neurips.cc/public/guides/CodeSubmissionPolicy)\) for more details\. - •The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc\. - •The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines\. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why\. - •At submission time, to preserve anonymity, the authors should release anonymized versions \(if applicable\)\. - •Providing as much information as possible in supplemental material \(appended to the paper\) is recommended, but including URLs to data and code is permitted\.
26. 6\.Experimental setting/details
27. Question: Does the paper specify all the training and test details \(e\.g\., data splits, hyperparameters, how they were chosen, type of optimizer\) necessary to understand the results?
28. Answer:\[Yes\]
29. Justification: We discuss the results in Section[4](https://arxiv.org/html/2610.28883#S4)and provide the detailed setups in Appendix[C](https://arxiv.org/html/2610.28883#A3)
30. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them\. - •The full details can be provided either with the code, in appendix, or as supplemental material\.
31. 7\.Experiment statistical significance
32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
33. Answer:\[Yes\]
34. Justification: We use at least 3 random seeds in all of our experiments\. We report the standard errors in tables and include error bars in figures\.
35. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The authors should answer\[Yes\]if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper\. - •The factors of variability that the error bars are capturing should be clearly stated \(for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions\)\. - •The method for calculating the error bars should be explained \(closed form formula, call to a library function, bootstrap, etc\.\) - •The assumptions made should be given \(e\.g\., Normally distributed errors\)\. - •It should be clear whether the error bar is the standard deviation or the standard error of the mean\. - •It is OK to report 1\-sigma error bars, but one should state it\. The authors should preferably report a 2\-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified\. - •For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range \(e\.g\., negative error rates\)\. - •If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text\.
36. 8\.Experiments compute resources
37. Question: For each experiment, does the paper provide sufficient information on the computer resources \(type of compute workers, memory, time of execution\) needed to reproduce the experiments?
38. Answer:\[Yes\]
39. Justification: We discuss the compute in Appendix[C](https://arxiv.org/html/2610.28883#A3), and also provide detailed runtime analysis in Section[4](https://arxiv.org/html/2610.28883#S4), Appendix[D\.4](https://arxiv.org/html/2610.28883#A4.SS4)and Appendix[D\.5](https://arxiv.org/html/2610.28883#A4.SS5)
40. Guidelines: - •The answer\[N/A\]means that the paper does not include experiments\. - •The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage\. - •The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute\. - •The paper should disclose whether the full research project required more compute than the experiments reported in the paper \(e\.g\., preliminary or failed experiments that didn’t make it into the paper\)\.
41. 9\.Code of ethics
43. Answer:\[Yes\]
44. Justification: To the best of our knowledge, the research presented in this paper conforms with the NeurIPS Code of Ethics\.
45. Guidelines: - •The answer\[N/A\]means that the authors have not reviewed the NeurIPS Code of Ethics\. - •If the authors answer\[No\], they should explain the special circumstances that require a deviation from the Code of Ethics\. - •The authors should make sure to preserve anonymity \(e\.g\., if there is a special consideration due to laws or regulations in their jurisdiction\)\.
46. 10\.Broader impacts
47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
48. Answer:\[N/A\]
49. Justification: This paper conducts fundamental research in theory, and there is no societal impact of the work performed\.
50. Guidelines: - •The answer\[N/A\]means that there is no societal impact of the work performed\. - •If the authors answer\[N/A\]or\[No\], they should explain why their work has no societal impact or why the paper does not address societal impact\. - •Examples of negative societal impacts include potential malicious or unintended uses \(e\.g\., disinformation, generating fake profiles, surveillance\), fairness considerations \(e\.g\., deployment of technologies that could make decisions that unfairly impact specific groups\), privacy considerations, and security considerations\. - •The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments\. However, if there is a direct path to any negative applications, the authors should point it out\. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation\. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster\. - •The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from \(intentional or unintentional\) misuse of the technology\. - •If there are negative societal impacts, the authors could also discuss possible mitigation strategies \(e\.g\., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML\)\.
51. 11\.Safeguards
52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse \(e\.g\., pre\-trained language models, image generators, or scraped datasets\)?
53. Answer:\[N/A\]
54. Justification: This paper does not release high\-risk models, datasets, or generative systems\.
55. Guidelines: - •The answer\[N/A\]means that the paper poses no such risks\. - •Released models that have a high risk for misuse or dual\-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters\. - •Datasets that have been scraped from the Internet could pose safety risks\. The authors should describe how they avoided releasing unsafe images\. - •We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort\.
56. 12\.Licenses for existing assets
57. Question: Are the creators or original owners of assets \(e\.g\., code, data, models\), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
58. Answer:\[Yes\]
59. Justification: We have cited the datasets, models, code frameworks and included their licenses in Appendix[C](https://arxiv.org/html/2610.28883#A3)\.
60. Guidelines: - •The answer\[N/A\]means that the paper does not use existing assets\. - •The authors should cite the original paper that produced the code package or dataset\. - •The authors should state which version of the asset is used and, if possible, include a URL\. - •The name of the license \(e\.g\., CC\-BY 4\.0\) should be included for each asset\. - •For scraped data from a particular source \(e\.g\., website\), the copyright and terms of service of that source should be provided\. - •If assets are released, the license, copyright information, and terms of use in the package should be provided\. For popular datasets,[paperswithcode\.com/datasets](https://paperswithcode.com/datasets)has curated licenses for some datasets\. Their licensing guide can help determine the license of a dataset\. - •For existing datasets that are re\-packaged, both the original license and the license of the derived asset \(if it has changed\) should be provided\. - •If this information is not available online, the authors are encouraged to reach out to the asset’s creators\.
61. 13\.New assets
62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
63. Answer:\[N/A\]
64. Justification: This paper does not release new assets\.
65. Guidelines: - •The answer\[N/A\]means that the paper does not release new assets\. - •Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates\. This includes details about training, license, limitations, etc\. - •The paper should discuss whether and how consent was obtained from people whose asset is used\. - •At submission time, remember to anonymize your assets \(if applicable\)\. You can either create an anonymized URL or include an anonymized zip file\.
66. 14\.Crowdsourcing and research with human subjects
67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation \(if any\)?
68. Answer:\[N/A\]
69. Justification: This paper does not include crowdsourcing experiments or research with human subjects\.
70. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper\. - •According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector\.
71. 15\.Institutional review board \(IRB\) approvals or equivalent for research with human subjects
72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board \(IRB\) approvals \(or an equivalent approval/review based on the requirements of your country or institution\) were obtained?
73. Answer:\[N/A\]
74. Justification: This paper does not include research with human subjects\.
75. Guidelines: - •The answer\[N/A\]means that the paper does not involve crowdsourcing nor research with human subjects\. - •Depending on the country in which research is conducted, IRB approval \(or equivalent\) may be required for any human subjects research\. If you obtained IRB approval, you should clearly state this in the paper\. - •We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution\. - •For initial submissions, do not include any information that would break anonymity \(if applicable\), such as the institution conducting the review\.
76. 16\.Declaration of LLM usage
77. Question: Does the paper describe the usage of LLMs if it is an important, original, or non\-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does*not*impact the core methodology, scientific rigor, or originality of the research, declaration is not required\.
78. Answer:\[N/A\]
79. Justification: This paper does not involve LLMs as any important, original, or non\-standard components\.
80. Guidelines: - •The answer\[N/A\]means that the core method development in this research does not involve LLMs as any important, original, or non\-standard components\. - •Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described\.