CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning

# CODS: Iterative Bellman-Residual Data Selection for Reusable Offline Reinforcement Learning
Source: [https://arxiv.org/html/2608.07719](https://arxiv.org/html/2608.07719)
Ibne Farabi Shihab1Sanjeda Akter1Abu Sa\-Adat Mohamed Moon\-Im Al Ahsan2 Md Najmus Swaqeeb2Anuj Sharma3 1Department of Computer Science, Iowa State University 2Department of Computer Science & Engineering, BRAC University 3Department of Civil, Construction & Environmental Engineering, Iowa State University ishihab@iastate\.edu,sanjeda@iastate\.edu abu\.sa\.adat\.mohamed\.moon\.im\.al\.ahsan@g\.bracu\.ac\.bd, md\.najmus\.swaqeeb@g\.bracu\.ac\.bd

###### Abstract

Offline reinforcement learning repeatedly trains policies from a fixed transition pool, making redundant data costly across seeds and hyperparameters, while naive subsampling can remove rare transitions needed for long\-horizon credit assignment\. We introduce CODS, a critic\-guided selector that alternates between fitting an algorithm\-matched critic and acquiring high\-residual transitions before freezing a reusable subset\. Unlike prioritized replay, CODS produces a static artifact; unlike one\-shot residual selection, it refreshes scores as the critic changes\. At a 10% budget, CODS retains 96\.6% of eligible\-pool performance across 20 valid D4RL task–algorithm cells\. It exceeds ReDOR and OPER on 19/20 cells and every other subset baseline on 20/20; all six subset advantages remain significant under predeclared hierarchical inference with Holm correction\. Holding total selector updates fixed, five acquisition rounds improve four representative cells by 11\.23 points over one round and saturate thereafter\. Equal\-pass and equal\-hour evaluations clarify that reuse, rather than a single\-run speedup, creates the compute advantage\. Mechanism and corruption interventions expose both useful sparse\-reward enrichment and sensitivity to outliers\. Finally, a whole\-trace extension retains 95\.4% of pooled ALFWorld success and 96\.5% of pooled GSM8K exact match\. CODS is therefore a reusable selection procedure, not a formal coreset guarantee\.

CODS: Iterative Bellman\-Residual Data Selection for Reusable Offline Reinforcement Learning

Ibne Farabi Shihab1††thanks:Corresponding author:ishihab@iastate\.eduSanjeda Akter1Abu Sa\-Adat Mohamed Moon\-Im Al Ahsan2Md Najmus Swaqeeb2Anuj Sharma31Department of Computer Science, Iowa State University2Department of Computer Science & Engineering, BRAC University3Department of Civil, Construction & Environmental Engineering, Iowa State Universityishihab@iastate\.edu,sanjeda@iastate\.eduabu\.sa\.adat\.mohamed\.moon\.im\.al\.ahsan@g\.bracu\.ac\.bd,md\.najmus\.swaqeeb@g\.bracu\.ac\.bd

## 1Introduction

Offline reinforcement learning learns a policy from a fixed collection of transitions rather than through further environment interaction\(Levineet al\.,[2020](https://arxiv.org/html/2608.07719#bib.bib22)\)\. Benchmark datasets such as D4RL and RL Unplugged have made this setting reproducible, but they also expose a practical cost that receives less attention: the same pool is often processed again for every seed, architecture, and regularization choice\(Fuet al\.,[2020](https://arxiv.org/html/2608.07719#bib.bib4); Gulcehreet al\.,[2021](https://arxiv.org/html/2608.07719#bib.bib23)\)\. Uniformly shrinking that pool is risky\. Dense regions may contain many similar transitions, whereas a small number of reward\-bearing or connective transitions can determine whether temporal\-difference learning propagates value across a long horizon\.

This tension makes offline data selection different from assigning each transition a permanent notion of importance\. A transition with a small error under an initial critic may become informative after nearby value estimates change, while an initially surprising transition may cease to matter once its local error has been fitted\. One\-shot selection ignores this movement\. Dynamic replay priorities track it, but they remain coupled to one training run and therefore cannot be frozen and reused across a sweep\.

CODS addresses this gap with an iterative but ultimately static selector\. It starts from a small random burn\-in set, fits the critic used by the downstream offline RL algorithm, adds a batch of high\-residual transitions, and refits before the next acquisition\. Once the budget is reached, CODS freezes the selected indices and initializes each downstream run from scratch\. The resulting object is a reusable data subset, not a weighted replay distribution and not a coreset in the formal approximation sense\.

The empirical study asks whether a frozen 10% subset retains performance, whether iterative rescoring improves over the identical one\-shot rule, how its gains vary with budget and selection seed, and when its construction cost can be amortized\. Across a provenance\-screened matrix of 20 D4RL task–algorithm cells, CODS has the strongest reported subset mean in 19 cells relative to ReDOR and OPER and in all 20 relative to the remaining subset baselines\. Fixed\-selector\-budget ablations isolate rescoring across four configurations, while matched\-update, matched\-pass, and matched\-hour evaluations separate subset quality from compute savings\.

The contribution is consequently both methodological and empirical\. CODS turns a moving Bellman\-error signal into a fixed artifact, evaluates a common subset budget against geometric, return\-based, one\-shot residual, resampling, and gradient\-matching alternatives, and isolates acquisition rounds from final subset size and selector updates\. The analysis is completed by multi\-task budget curves, seed\-variance estimates, sparse\-reward removal interventions, corruption and duplication stress tests, cross\-algorithm reuse, and full OPER results\. Appendix[C](https://arxiv.org/html/2608.07719#A3)develops the design rationale and reuse interface in full\.

Two boundaries keep these contributions precise\. First, the theoretical calculation only relates residual magnitude to a local information\-gain proxy under strong conditioning assumptions; its empirical worst\-case factor is too loose to certify the observed ranking\. Second, transition\-wise acquisition is appropriate for the Markov control experiments but not automatically for text\. Our language experiments therefore adapt CODS to select complete, prefix\-closed traces with a language\-model critic rather than treating tokens as independent D4RL transitions\. This extension connects the controlled RL study to offline value learning for reasoning and language agents\(Wanget al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib32)\)without claiming that the two data modalities are identical\.

## 2Related Work

CODS is a data selector rather than a new policy\-learning objective\. We therefore evaluate it with three representative offline RL algorithms\. TD3\+BC regularizes policy improvement toward behavior cloning\(Fujimoto and Gu,[2021](https://arxiv.org/html/2608.07719#bib.bib31)\); CQL penalizes high values for actions outside the data distribution\(Kumaret al\.,[2020](https://arxiv.org/html/2608.07719#bib.bib11)\); and IQL uses expectile value learning without explicitly estimating the behavior policy\(Kostrikovet al\.,[2021](https://arxiv.org/html/2608.07719#bib.bib10)\)\. Matching the acquisition residual to each backup lets the selection rule follow the error optimized by the corresponding learner\.

Experience replay and offline data reweighting offer the closest comparisons\. Prioritized experience replay updates sampling probabilities during an active training run\(Schaulet al\.,[2015](https://arxiv.org/html/2608.07719#bib.bib16)\)\. ReD rebalances offline data using trajectory return\(Yueet al\.,[2022](https://arxiv.org/html/2608.07719#bib.bib25)\), and OPER uses fixed advantage\- or return\-derived priorities for offline resampling\(Yueet al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib28)\)\. These approaches change how often examples are visited but do not construct an iteratively refined, frozen subset\. Trajectory\-level replay further shows that transition\-wise sampling can disrupt temporal structure, a concern that becomes more important for sparse rewards and language trajectories\(Liuet al\.,[2024](https://arxiv.org/html/2608.07719#bib.bib27)\)\.

ReDOR directly targets offline dataset reduction by matching the full\-data actor–critic gradient with a selected subset\(Yanget al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib21)\)\. It supplies a stronger optimization objective than a scalar Bellman residual, but requires more expensive gradient representations and an OMP\-style selection procedure\. CODS instead asks whether repeatedly updating a cheap, algorithm\-specific residual is sufficient\. The distinction is empirical rather than categorical: both methods depend on the critic used during selection, and a faithful comparison must include their complete selection costs\.

In supervised learning, subset selection has been driven by geometric coverage, influence, example difficulty, and gradient matching\(Sener and Savarese,[2017](https://arxiv.org/html/2608.07719#bib.bib26); Koh and Liang,[2017](https://arxiv.org/html/2608.07719#bib.bib8); Paulet al\.,[2021](https://arxiv.org/html/2608.07719#bib.bib24); Mirzasoleimanet al\.,[2020](https://arxiv.org/html/2608.07719#bib.bib13)\)\. CODS borrows the iterative rhythm of pool\-based active learning, but no new labels are queried\. Bellman backups provide a model\-dependent signal over an already observed pool\. This makes the selector simple, while also exposing it to critic misspecification and high\-residual outliers\.

The baselines test complementary notions of importance rather than a single hierarchy\. Appendix[D](https://arxiv.org/html/2608.07719#A4)compares their operational assumptions and records the implementation checks needed for a fair selector audit\.

Offline RL is increasingly connected to NLP through multi\-step reasoning and the training of language agents\. OREO, for example, learns token\- or step\-level values from fixed reasoning traces and uses a soft Bellman equation for credit assignment\(Wanget al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib32)\)\. The common structure is learning credit from a fixed collection of sequential decisions, which makes reusable data selection relevant in both domains\. Our extension preserves this structure by scoring complete traces with a language\-model value critic and retaining every prefix needed to reproduce a chosen action\. The ALFWorld and GSM8K results in Appendix[O\.11](https://arxiv.org/html/2608.07719#A15.SS11)are therefore direct language\-domain evidence for the adapted selector, while the transition\-level claims remain grounded in D4RL\.

## 3Critic\-Guided Offline Data Selection

Let𝒟=\{\(si,ai,ri,si′,di\)\}i=1N\\mathcal\{D\}=\\\{\(s\_\{i\},a\_\{i\},r\_\{i\},s^\{\\prime\}\_\{i\},d\_\{i\}\)\\\}\_\{i=1\}^\{N\}be an offline transition pool, with terminal indicatordid\_\{i\}\. Given budgetBB, CODS returns indices𝒮K⊂\{1,…,N\}\\mathcal\{S\}\_\{K\}\\subset\\\{1,\\ldots,N\\\}that are subsequently reused by the downstream learner\. The selector begins with a uniformly sampled burn\-in set𝒮0\\mathcal\{S\}\_\{0\}of sizeb0b\_\{0\}\. At roundkk, it evaluates the current critic on every unselected transition, adds thebkb\_\{k\}largest absolute residuals, and updates the critic on the enlarged working set\. The budgets satisfyb0\+∑k=1Kbk=Bb\_\{0\}\+\\sum\_\{k=1\}^\{K\}b\_\{k\}=B\.

The selection critic and final learner have separate roles\. Critic fitting during acquisition determines only the selected indices\. After roundKK, every downstream configuration is initialized independently and trained from scratch on the frozen subset; neither the acquisition critic nor its optimizer state is reused\. This separation prevents selection\-stage optimization from becoming unreported pretraining and is what permits one subset to be reused across compatible runs\.

Algorithm 1:CODS

Figure[1](https://arxiv.org/html/2608.07719#S3.F1)makes the separation explicit\. Only the acquisition loop changes the subset\. Once frozen, its output may feed multiple downstream runs, while any optional monitoring statistic remains outside the selection path\.

Unselected poolScore with currentBellman residualAdd top\-bkb\_\{k\}transitionsUpdate selectioncriticFreeze𝒮K\\mathcal\{S\}\_\{K\}and train afreshKKroundsFigure 1:CODS separates iterative acquisition from downstream training\. Residuals are recomputed after each critic update; the final indices are then frozen and reused\. Optional diagnostics do not enter this path\.### 3\.1Algorithm\-matched residuals

The acquisition score must be interpreted relative to a backup operator\. CODS therefore follows the downstream target convention while ranking by the absolute, unpenalized Bellman residual, so algorithm\-specific regularization magnitudes do not directly determine the ordering\.

The residual is matched to the base algorithm\. For TD3\+BC, CODS uses

δTD3\+BC​\(τ\)=\\displaystyle\\delta^\{\\mathrm\{TD3\+BC\}\}\(\\tau\)=\{\}r\+γ​\(1−d\)​minj∈\{1,2\}⁡Qθj′​\(s′,a′\)\\displaystyle r\+\\gamma\(1\-d\)\\min\_\{j\\in\\\{1,2\\\}\}Q\_\{\\theta^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\\prime\}\)\(1\)−Qθ​\(s,a\)\.\\displaystyle\\quad\-Q\_\{\\theta\}\(s,a\)\.wherea′=πϕ′​\(s′\)\+ϵa^\{\\prime\}=\\pi\_\{\\phi^\{\\prime\}\}\(s^\{\\prime\}\)\+\\epsilonincludes the clipped target\-policy smoothing noise of the downstream implementation\. For CQL, the critic is trained with its conservative regularizer but the acquisition score uses the unpenalized TD residual,

δCQL​\(τ\)\\displaystyle\\delta^\{\\mathrm\{CQL\}\}\(\\tau\)=r\+γ​\(1−d\)​minj=1,2⁡Qθj′​\(s′,a′\)\\displaystyle=r\+\\gamma\(1\-d\)\\min\_\{j=1,2\}Q\_\{\\theta^\{\\prime\}\_\{j\}\}\(s^\{\\prime\},a^\{\\prime\}\)\(2\)−Qθ\(s,a\),a′∼πϕ\(⋅∣s′\)\.\\displaystyle\\quad\-Q\_\{\\theta\}\(s,a\),\\qquad a^\{\\prime\}\\sim\\pi\_\{\\phi\}\(\\cdot\\mid s^\{\\prime\}\)\.This choice prevents the magnitude of the conservative penalty from directly determining which transitions are retained\. IQL instead uses its learned expectile value,

δIQL​\(τ\)=r\+γ​\(1−d\)​Vψ​\(s′\)−Qθ​\(s,a\)\.\\delta^\{\\mathrm\{IQL\}\}\(\\tau\)=r\+\\gamma\(1\-d\)V\_\{\\psi\}\(s^\{\\prime\}\)\-Q\_\{\\theta\}\(s,a\)\.\(3\)In all three cases, CODS ranks candidates by\|δ​\(τ\)\|\|\\delta\(\\tau\)\|without reward, trajectory, or density bonuses\.

The default experiment usesB=0\.10​NB=0\.10N,b0=0\.02​Nb\_\{0\}=0\.02N,K=5K=5, andbk=0\.016​Nb\_\{k\}=0\.016N\. The rule is deliberately minimal: it imposes neither a trajectory quota nor a geometric coverage constraint\. Consequently, it can select corrupted outliers or many nearby transitions when the critic assigns them large errors\. The corruption and duplicate\-density interventions in Appendix[O\.7](https://arxiv.org/html/2608.07719#A15.SS7)measure both failures directly and evaluate clipped\-ensemble and residual–coverage variants as targeted ablations\.

### 3\.2Why the ranking is recomputed

After an acquisition, the critic can reduce old errors and expose new inconsistencies; recomputing the pool ranking is the mechanism that distinguishes CODS from a fixed top\-BBresidual list\. The burn\-in remains a source of bias rather than a trusted core, and the frozen artifact is reused only after selection ends\. Appendix[C](https://arxiv.org/html/2608.07719#A3)gives the full rationale, while Appendix[E](https://arxiv.org/html/2608.07719#A5)specifies batch tradeoffs, invariants, and complexity\.

## 4What Residual Scoring Can Justify

Bellman residual magnitude is often interpreted as a measure of how much a critic can still learn from a transition\. The following local calculation gives that intuition a precise but narrow form\. Let

Iloc​\(τ\)=12​δθ​\(τ\)2​gτ⊤​ℐθ−1​gτ,gτ=∇θQθ​\(s,a\),I\_\{\\mathrm\{loc\}\}\(\\tau\)=\\frac\{1\}\{2\}\\delta\_\{\\theta\}\(\\tau\)^\{2\}g\_\{\\tau\}^\{\\top\}\\mathcal\{I\}\_\{\\theta\}^\{\-1\}g\_\{\\tau\},\\qquad g\_\{\\tau\}=\\nabla\_\{\\theta\}Q\_\{\\theta\}\(s,a\),\(4\)whereℐθ\\mathcal\{I\}\_\{\\theta\}is positive definite on the span of the candidate gradients\.

The proof is in Appendix[A](https://arxiv.org/html/2608.07719#A1)\. This comparison does not imply pairwise rank preservation: the omitted gradient–curvature term can differ across transitions\. In the empirical audit, the worst\-case factor is approximately2×1032\\times 10^\{3\}, and only 0\.02% of sampled pairs meet the sufficient separation condition\. We therefore use the result to delimit the heuristic, not to explain the empirical gains\. In particular, CODS has no established policy\-recovery, regret, monotone\-submodular, or formal coreset guarantee\.

The gap between this score comparison and a policy guarantee is substantial because acquisition changes the critic, actor, and future targets\. Appendix[G](https://arxiv.org/html/2608.07719#A7)records the omitted gradient–curvature quantities and the required empirical audit\.

## 5Experiments

The experiments proceed from quality to explanation and then to cost\. The 20\-cell fixed\-update matrix tests whether a 10% subset preserves performance\. Fixed\-selector\-budget round ablations isolate rescoring, and multi\-task budget curves locate the observed saturation region\. Nested selection and downstream seeds quantify both uncertainty sources\. Sparse\-reward removal, corruption, duplication, and transfer studies then test when the residual signal helps or fails\. Finally, equal\-update, equal\-pass, and equal\-hour protocols determine which compute claims follow from reusable subsets\. A whole\-trace language experiment asks whether the iterative principle survives the change from Markov transitions to textual action sequences\.

### 5\.1Tasks, baselines, and protocol

We use D4RL v2 locomotion datasets for HalfCheetah, Walker2d, and Hopper at medium and medium\-expert quality, together with AntMaze\-medium\-play\(Fuet al\.,[2020](https://arxiv.org/html/2608.07719#bib.bib4)\)\. Each selector is paired with TD3\+BC, CQL, or IQL\. Random selection establishes an unstructured lower baseline\. k\-center uses Euclidean distance in coordinate\-wise z\-scored state–action space\. ReD uses trajectory return, Static\-PER uses the burn\-in critic once and is therefore identical to CODS withK=1K=1, OPER performs its published offline resampling procedure, and ReDOR approximates the full\-pool gradient direction with an OMP\-style objective\. Appendix[B](https://arxiv.org/html/2608.07719#A2)gives the exact selector schedule, while Appendix[O\.1](https://arxiv.org/html/2608.07719#A15.SS1)audits the two strongest external implementations\.

Ten percent of the original trajectories are held out from both selection and training for an optional diagnostic described in Appendix[L](https://arxiv.org/html/2608.07719#A12)\. The eligible\-pool baseline therefore uses the remaining 90% of the original dataset\. Every subset contains transitions equal to 10% of the original dataset, or 11\.1% of the eligible pool\. Calling the eligible\-pool condition “100%” would obscure this distinction, so Table[1](https://arxiv.org/html/2608.07719#S5.T1)labels it as Pool\.

All primary quality comparisons train the downstream learner for exactly 1,000,000 gradient updates with batch size 256\. Evaluation occurs every 5,000 updates over ten deterministic episodes, and the reported score averages the final ten checkpoints\. This equal\-update design controls optimizer exposure but causes each selected transition to be revisited more often\. The separate equal\-pass and equal\-amortized\-hour measurements in Appendix[O\.9](https://arxiv.org/html/2608.07719#A15.SS9)expose that distinction rather than attributing all savings to data quality\.

The primary matrix excludes one configuration before aggregation\. TD3\+BC is at the AntMaze task floor and lacks informative subset\-baseline comparisons\. The IQL AntMaze row is retained after reconciling every method to the manifest\-verified expectile and temperature\(τ,β\)=\(0\.9,10\)\(\\tau,\\beta\)=\(0\.9,10\), fixed code revision, selected indices, and checkpoint rule\. Appendix[J](https://arxiv.org/html/2608.07719#A10)documents both decisions and explains why an unreconciled historical IQL value is not reused\.

All selectors receive the same eligible pool and cardinality; Appendix[D](https://arxiv.org/html/2608.07719#A4)gives the fairness audit\. Each subset method uses five independently selected artifacts, and each artifact is evaluated with three downstream seeds, giving 15 nested evaluations per populated cell\. Pool uses ten downstream seeds\. The CQL Hopper\-medium reversal remains in the matrix, and the nested design preserves selection\-seed variation instead of treating downstream initialization as the only source of uncertainty\.

Table 1:D4RL normalized scores under an equal downstream budget of 1,000,000 updates\. Subsets contain transitions equal to 10% of the original dataset; Pool uses the 90% eligible training pool after the diagnostic holdout\. Subset entries are means±\\pmstandard deviations over five selection seeds and three downstream seeds; Pool uses ten downstream seeds\. Bold marks the highest reported subset mean\. The 20\-cell primary comparison excludes only†\\daggerthe floor\-level TD3\+BC AntMaze cell\. The IQL AntMaze row is the manifest\-verified\(0\.9,10\)\(0\.9,10\)reconciliation\. Appendix Table[13](https://arxiv.org/html/2608.07719#A15.T13)reports all 20 OPER cells\.
### 5\.2Subset quality

Across the 20 valid cells, CODS retains 96\.6% of the aggregate eligible\-pool score\. We predeclare Pool as a retention comparison, not as one of the subset hypotheses\. The confirmatory family therefore contains exactly six comparisons: ReDOR, ReD, Static\-PER, OPER, k\-center, and Random\. Confidence intervals come from a dataset\-blocked hierarchical bootstrap that resamples datasets, then selection artifacts, then downstream seeds\. One\-sidedpp\-values come from a paired mixed\-effects model with fixed learner effects and random intercepts for dataset, dataset–learner cell, and selection artifact within cell, calibrated by 100,000 null parametric\-bootstrap replicates and Holm\-adjusted over the six\-method family\. This model\-based test, detailed in Appendix[K](https://arxiv.org/html/2608.07719#A11), does not treat seven dataset blocks or dependent seeds as independent sign flips\.

Table[2](https://arxiv.org/html/2608.07719#S5.T2)shows a2\.772\.77\-point mean gap to Pool and positive CODS differences against all six subset baselines, from\+4\.72\+4\.72over ReDOR to\+27\.12\+27\.12over Random\. Each subset interval excludes zero at Holm\-adjustedp<0\.001p<0\.001\. The claim is therefore preservation of most pooled performance with a much smaller artifact, not statistical equivalence to Pool\.

Table 2:Primary comparison over 20 valid cells\. Pool is the separately predeclared retention comparison and is not in the Holm family; the six subset comparisons use hierarchical bootstrap CIs and mixed\-model parametric\-bootstrappp\-values\.The aggregate is not driven by one learner \(Table[3](https://arxiv.org/html/2608.07719#S5.T3)\): sum\-based retention is 96\.6% for the six eligible TD3\+BC cells, 97\.0% for seven CQL cells, and 96\.1% for seven IQL cells, 96\.6% overall\. Appendix[H](https://arxiv.org/html/2608.07719#A8)gives these per\-algorithm summaries and discusses the CQL Hopper\-medium reversal without relabeling it as a win\.

Table 3:Retention by downstream learner over the 20 valid cells \(sum of CODS scores over sum of Pool scores within each row\)\.
### 5\.3Does rescoring matter?

Static\-PER and CODS withK=1K=1are the same one\-shot rule when burn\-in, residual, and budget are matched\. Table[4](https://arxiv.org/html/2608.07719#S5.T4)therefore merges them rather than presenting duplicate labels as separate methods\. On Walker2d\-medium with TD3\+BC, the mean rises from 65\.4 for one\-shot selection to 74\.7 with three rounds and 78\.5 with five rounds\. Ten rounds produce no further gain\. A fixed\-critic\-budget replication on four representative cells \(Walker2d/TD3\+BC, HalfCheetah/CQL, Hopper/IQL, AntMaze/CQL\), holding total selection\-critic updates constant across round counts, confirms this is a cross\-task effect rather than a Walker\-only observation \(Appendix[O\.3](https://arxiv.org/html/2608.07719#A15.SS3)\): the four\-cell mean rises from 49\.95 atK=1K\{=\}1to 61\.18 atK=5K\{=\}5\(\+11\.23\+11\.23, 95% CI\[\+8\.86,\+13\.74\]\[\+8\.86,\+13\.74\],p<0\.001p<0\.001\) andK=10K\{=\}10is−0\.40\-0\.40relative toK=5K\{=\}5\(CI\[−1\.18,\+0\.37\]\[\-1\.18,\+0\.37\]\), supporting saturation rather than monotonic improvement\. Five rounds are not claimed to be universally optimal\.

Table 4:Acquisition\-round ablation at a 10% budget on Walker2d\-medium with TD3\+BC\. Entries average five selection seeds and three downstream seeds\.Because the final cardinality and the 100,000\-update selector budget are fixed, the gain fromK=1K=1toK=5K=5cannot be attributed to more selected transitions or more critic updates\. Appendix[O\.3](https://arxiv.org/html/2608.07719#A15.SS3)repeats this design with five selection seeds on four configurations, so the observed gain includes selection\-seed uncertainty rather than a single burn\-in artifact\.

### 5\.4How does performance change with budget?

The Walker2d sweep in Figure[2](https://arxiv.org/html/2608.07719#S5.F2)complements the round ablation: CODS rises sharply between 2% and 5%, reaches 94% of Pool at 10%, and changes little beyond 15%\. Appendix[O\.4](https://arxiv.org/html/2608.07719#A15.SS4)repeats the same budgets on four representative configurations\. Their four\-cell mean gains per budget point are 5\.49 from 2% to 5%, 0\.87 from 5% to 10%, and 0\.23 from 10% to 20%\. Thus diminishing marginal gains are cross\-task in this sample, although 10% remains a common evaluation budget rather than a universal optimum\.

0224466881010121214141616181820202020404060608080Data budget \(%\)Normalized scoreRandomReDORCODSPoolFigure 2:Budget sweep for Walker2d\-medium with TD3\+BC\. Points average five selection seeds and three downstream seeds; bands show marginal standard deviations\.The multi\-task table provides the broader evidence\. Figure[2](https://arxiv.org/html/2608.07719#S5.F2)remains useful because it shows the full Walker2d curve and uncertainty bands\. We report finite\-difference slopes directly from the observed means rather than fitting an unspecified saturation model\. Appendix[H](https://arxiv.org/html/2608.07719#A8)gives the corresponding interpretation\.

## 6Reuse Cost Model

Equal\-update scores do not demonstrate lower training time, because minibatch updates cost roughly the same regardless of the pool from which they are sampled\. The timing ledger therefore separates one\-time selection from three downstream protocols on Walker2d\-medium\. A Pool run over the matched number of dataset passes costs 1\.8 hours, subset training costs 0\.18 hours per configuration, and CODS selection costs 3\.0 hours once\. For a sweep ofSSconfigurations, the epoch\-scaled costs are

Cpool​\(S\)=1\.8​S,CCODS​\(S\)=3\.0\+0\.18​S\.C\_\{\\mathrm\{pool\}\}\(S\)=1\.8S,\\qquad C\_\{\\mathrm\{CODS\}\}\(S\)=3\.0\+0\.18S\.
The break\-even count is⌈3\.0/\(1\.8−0\.18\)⌉=2\\lceil 3\.0/\(1\.8\-0\.18\)\\rceil=2\. AtS=20S=20, the ledger gives 36\.0 hours for the pool and 6\.6 hours for CODS, a5\.45×5\.45\\timesarithmetic reduction\. Table[5](https://arxiv.org/html/2608.07719#S6.T5)includes ReDOR under the same accounting\.

Table 5:Reported wall\-clock cost model for 20 epoch\-scaled configurations on Walker2d\-medium with TD3\+BC\. The table establishes amortization arithmetic, not matched\-performance speedup\.A matched\-pass run scores 75\.6 rather than the equal\-update 78\.5, retaining 90\.3% rather than 93\.8% of the 83\.7 Pool score\. At 20 reuses, that condition realizes the ledger’s5\.45×5\.45\\timestotal\-time reduction; at one reuse, CODS still costs 3\.18 hours versus 1\.8 for Pool\. Equal updates yield only a1\.04×1\.04\\timestotal\-time reduction at 20 reuses\. The equal\-amortized\-hour condition retains 95\.1% of Pool because its shared selection cost is spread across the sweep\. Appendix[O\.9](https://arxiv.org/html/2608.07719#A15.SS9)reports the three protocols together\.

Reuse is therefore the source of the arithmetic advantage, not faster single\-run convergence\. Appendix[I](https://arxiv.org/html/2608.07719#A9)derives the general break\-even rule, tabulates the one\- and two\-run cases, and gives the complete job\-accounting template\.

## 7Discussion and Conclusion

CODS is built around one observation: transition importance changes as the critic changes\. Recomputing an algorithm\-specific residual before each acquisition batch outperforms freezing the initial ranking on Walker2d and on the four\-cell fixed\-budget replication\. The 20\-cell matrix then shows that the same procedure remains effective across three offline RL objectives\. Freezing the indices creates a further operational advantage over dynamic prioritization when compatible runs reuse one artifact\. The completion experiments in Appendix[O](https://arxiv.org/html/2608.07719#A15)connect that headline result to baseline fidelity, cross\-task budgets, selection stability, learner\-specific sparse\-reward mechanisms, robustness failures, transfer, matched compute, and language traces\.

The primary comparison gives this observation a broad, if still benchmark\-limited, empirical base\. CODS retains 96\.6% of the aggregate eligible\-pool score at a budget equal to 10% of the original dataset\. Its advantage over return, geometry, and one\-shot residual baselines appears in every valid cell, while its comparison with gradient matching is closer and includes one 0\.4\-point reversal\. OPER is likewise competitive in one cell, but its complete 20\-cell column leaves CODS ahead by 6\.84 points on average\. This pattern is more informative than claiming uniform dominance: inexpensive residual scoring is competitive when refreshed, whereas richer gradient and resampling signals remain serious alternatives\.

The AntMaze interventions sharpen the mechanism claim without extrapolating across learners\. CQL\- and IQL\-specific selectors both enrich reward\-bearing, bottleneck, and goal\-proximal transitions, and category\-excluding replacements lower the corresponding learner’s return\. The CQL intervention is used only for CQL and the IQL intervention only for IQL\. By contrast, the failed TD3\+BC learner demonstrates the opposite boundary: iteration can refresh a ranking but cannot manufacture a meaningful backup from a critic at the task floor\. Its composition audit stays descriptive and is not used to explain successful navigation\.

The evidence still does not support a formal coreset story\. The local proxy bound is too loose to certify the ranking, and the equal\-update protocol alone does not prove time efficiency\. The matched\-pass result quantifies the quality lost when repeated exposure is removed, while the matched\-hour result locates a measured point on the reuse frontier\. In language, the whole\-trace adaptation outperforms random, diversity, one\-shot, and ReDOR selectors on ALFWorld and GSM8K, and fragmentation hurts both tasks\. These results broaden the empirical domain while also showing that the selection unit must follow the task’s sequential structure\.

The reuse analysis remains conditional on workload\. The ledger breaks even after two epoch\-scaled configurations, and the measured 20\-run equal\-pass condition reduces total time by5\.45×5\.45\\timeswhile retaining 90\.3% of Pool on Walker2d\-medium\. A single run is slower after selection is charged, and equal updates offer almost no time reduction\. CODS is therefore most useful for repeated compatible training, not as a universal accelerator\.

Within this scope, the conclusion is concrete: CODS is a simple iterative selector whose 10% subsets preserve most eligible\-pool performance across 20 state\-based cells, improve over one\-shot residual selection under a fixed selector budget, and transfer to whole language traces when prefix structure is preserved\. Its output is a reusable index artifact rather than a training\-time priority distribution\. The released arrays, manifests, schedules, and selected indices make those claims auditable while leaving the theoretical guarantee deliberately modest\.

## Limitations

CODS can mistake noise for information because large Bellman residuals also arise from corrupted rewards, stochastic targets, and critic misspecification\. The corruption study confirms this failure: at 10% reward corruption, raw CODS falls from 58\.0 to 27\.8 on CQL AntMaze\. Clipped\-ensemble scoring recovers to 45\.4 but does not remove the failure\. Duplicate\-density and burn\-in studies likewise show that pure residual selection is neither density\-invariant nor insensitive to initialization\. These variants are diagnostic ablations, not a replacement for the minimal method\.

Transition\-wise selection can fragment trajectories\. For the Markov benchmarks, each tuple carries its next state and terminal flag, so the critic can form a one\-step target even when adjacent transitions are absent, although sparse\-reward learning may still benefit from connected segments\. The language adaptation enforces whole\-trace, prefix\-closed selection, and its fragmentation ablation confirms that this constraint matters\. Segment\-level acquisition for long continuous\-control trajectories remains untested\.

The experimental evidence is limited to seven state\-based D4RL datasets and three continuous\-control algorithms, plus a sequence\-level language\-agent extension on ALFWorld and GSM8K\. The TD3\+BC AntMaze cell is excluded as a documented learner failure; the IQL AntMaze cell is reconciled to its manifest\-verified\(0\.9,10\)\(0\.9,10\)setting\. Confirmatory inference uses a dataset\-blocked hierarchical bootstrap over datasets, selection seeds, and downstream seeds; the four\-cell nested study attributes 71% of variance to dataset/configuration, 17% to selection seed, and 12% to downstream seed, so selection\-seed replication is necessary and downstream initialization is not the only uncertainty source\. The optional held\-out diagnostic assumes trajectory exchangeability but has no role in selection and provides no guarantee for the learned policy\.

The equal\-update protocol isolates optimization exposure rather than data\-processing cost, and subset transitions are revisited more frequently\. Equal\-pass and equal\-hour results reduce this ambiguity but cover only Walker2d\-medium with TD3\+BC and one 20\-configuration reuse pattern\. The selector remains slower for a single configuration\. The language evaluation covers one embodied benchmark and one mathematical\-reasoning benchmark with one 7B model; it does not establish transfer to dialogue, open\-ended generation, other model scales, or online interaction\.

## Ethical Considerations

The study uses existing simulated\-control and public language benchmarks and involves no human participants or newly collected personal data\. Its principal responsible\-reporting risk is empirical overstatement\. We exclude the floor\-level TD3\+BC AntMaze configuration, retain the reconciled IQL configuration, distinguish measured scores from amortization arithmetic, and report negative robustness results\. The anonymous artifact includes raw evaluation arrays, selected indices, run manifests, and scripts that regenerate every table and figure\.

## References

- A\. N\. Angelopoulos, S\. Bates, E\. J\. Candès, M\. I\. Jordan, and L\. Lei \(2025\)Learn then test: calibrating predictive algorithms to achieve risk control\.The Annals of Applied Statistics19\(2\),pp\. 1641–1662\.Cited by:[Appendix L](https://arxiv.org/html/2608.07719#A12.p1.1)\.
- A\. N\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. Schuster \(2022\)Conformal risk control\.arXiv preprint arXiv:2208\.02814\.Cited by:[Appendix L](https://arxiv.org/html/2608.07719#A12.p1.1)\.
- N\. Angelopoulos Anastasios and B\. Stephen \(2021\)A gentle introduction to conformal prediction and distribution\-free uncertainty quantification\.arXiv preprint arXiv:2107\.07511\.Cited by:[Appendix L](https://arxiv.org/html/2608.07719#A12.p1.1)\.
- K\. Cobbe, V\. Kosaraju, M\. Bavarian, M\. Chen, H\. Jun, L\. Kaiser, M\. Plappert, J\. Tworek, J\. Hilton, R\. Nakano, C\. Hesse, and J\. Schulman \(2021\)Training verifiers to solve math word problems\.arXiv preprint arXiv:2110\.14168\.External Links:[Link](https://arxiv.org/abs/2110.14168)Cited by:[§O\.11](https://arxiv.org/html/2608.07719#A15.SS11.p1.1)\.
- J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. Levine \(2020\)D4rl: datasets for deep data\-driven reinforcement learning\.arXiv preprint arXiv:2004\.07219\.Cited by:[§1](https://arxiv.org/html/2608.07719#S1.p1.1),[§5\.1](https://arxiv.org/html/2608.07719#S5.SS1.p1.1)\.
- S\. Fujimoto and S\. S\. Gu \(2021\)A minimalist approach to offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.34\.External Links:[Link](https://arxiv.org/abs/2106.06860)Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p1.1)\.
- C\. Gulcehre, Z\. Wang, A\. Novikov, T\. L\. Paine, S\. G\. Colmenarejo, K\. Zolna, R\. Agarwal, J\. Merel, D\. Mankowitz, C\. Paduraru, G\. Dulac\-Arnold, J\. Li, M\. Norouzi, M\. Hoffman, O\. Nachum, G\. Tucker, N\. Heess, and N\. de Freitas \(2021\)RL unplugged: a suite of benchmarks for offline reinforcement learning\.External Links:2006\.13888,[Link](https://arxiv.org/abs/2006.13888)Cited by:[§1](https://arxiv.org/html/2608.07719#S1.p1.1)\.
- P\. W\. Koh and P\. Liang \(2017\)Understanding black\-box predictions via influence functions\.InInternational conference on machine learning,pp\. 1885–1894\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p4.1)\.
- I\. Kostrikov, A\. Nair, and S\. Levine \(2021\)Offline reinforcement learning with implicit q\-learning\.arXiv preprint arXiv:2110\.06169\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p1.1)\.
- A\. Kumar, A\. Zhou, G\. Tucker, and S\. Levine \(2020\)Conservative q\-learning for offline reinforcement learning\.Advances in neural information processing systems33,pp\. 1179–1191\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p1.1)\.
- S\. Levine, A\. Kumar, G\. Tucker, and J\. Fu \(2020\)Offline reinforcement learning: tutorial, review, and perspectives on open problems\.arXiv preprint arXiv:2005\.01643\.Cited by:[§1](https://arxiv.org/html/2608.07719#S1.p1.1)\.
- J\. Liu, Y\. Ma, J\. Hao, Y\. Hu, Y\. Zheng, T\. Lv, and C\. Fan \(2024\)A trajectory perspective on the role of data sampling techniques in offline reinforcement learning\.InProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems,pp\. 1229–1237\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p2.1)\.
- B\. Mirzasoleiman, J\. Bilmes, and J\. Leskovec \(2020\)Coresets for data\-efficient training of machine learning models\.InInternational Conference on Machine Learning,pp\. 6950–6960\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p4.1)\.
- M\. Paul, S\. Ganguli, and G\. K\. Dziugaite \(2021\)Deep learning on a data diet: finding important examples early in training\.Advances in neural information processing systems34,pp\. 20596–20607\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p4.1)\.
- T\. Schaul, J\. Quan, I\. Antonoglou, and D\. Silver \(2015\)Prioritized experience replay\.arXiv preprint arXiv:1511\.05952\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p2.1)\.
- O\. Sener and S\. Savarese \(2017\)Active learning for convolutional neural networks: a core\-set approach\.arXiv preprint arXiv:1708\.00489\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p4.1)\.
- M\. Shridhar, X\. Yuan, M\. Côté, Y\. Bisk, A\. Trischler, and M\. Hausknecht \(2021\)ALFWorld: aligning text and embodied environments for interactive learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=0IOX0YcCdTn)Cited by:[§O\.11](https://arxiv.org/html/2608.07719#A15.SS11.p1.1)\.
- M\. F\. Taufiq, J\. Ton, R\. Cornish, Y\. W\. Teh, and A\. Doucet \(2022\)Conformal off\-policy prediction in contextual bandits\.Advances in Neural Information Processing Systems35,pp\. 31512–31524\.Cited by:[Appendix L](https://arxiv.org/html/2608.07719#A12.p1.1)\.
- V\. Vovk, A\. Gammerman, and G\. Shafer \(2005\)Algorithmic learning in a random world\.Springer\.Cited by:[Appendix L](https://arxiv.org/html/2608.07719#A12.p1.1)\.
- H\. Wang, S\. Hao, H\. Dong, S\. Zhang, Y\. Bao, Z\. Yang, and Y\. Wu \(2025\)Offline reinforcement learning for LLM multi\-step reasoning\.InFindings of the Association for Computational Linguistics: ACL 2025,pp\. 8881–8893\.External Links:[Link](https://aclanthology.org/2025.findings-acl.464/)Cited by:[§O\.11](https://arxiv.org/html/2608.07719#A15.SS11.p3.6),[§1](https://arxiv.org/html/2608.07719#S1.p6.1),[§2](https://arxiv.org/html/2608.07719#S2.p6.1)\.
- Q\. A\. Yang, B\. Yang, B\. Zhang, B\. Hui, B\. Zheng, B\. Yu, C\. Li, D\. Liu, F\. Huang, G\. Dong, H\. Wei, H\. Lin, J\. Yang, J\. Tu, J\. Zhang, J\. Yang, J\. Yang, J\. Zhou, J\. Lin, K\. Dang, K\. Lu, K\. Bao, K\. Yang, L\. Yu, M\. Li, M\. Xue, P\. Zhang, Q\. Zhu, R\. Men, R\. Lin, T\. Li, T\. Xia, X\. Ren, X\. Ren, Y\. Fan, Y\. Su, Y\. Zhang, Y\. Wan, Y\. Liu, Z\. Cui, Z\. Zhang, Z\. Qiu, S\. Quan, and Z\. Wang \(2024\)Qwen2\.5 technical report\.ArXivabs/2412\.15115\.External Links:[Link](https://api.semanticscholar.org/CorpusID:274859421)Cited by:[§O\.11](https://arxiv.org/html/2608.07719#A15.SS11.p1.1)\.
- Y\. Yang, Q\. Wang, C\. Li, H\. Hu, C\. Wu, Y\. Jiang, D\. Zhong, Z\. Zhang, Q\. Zhao, C\. Zhang, and X\. Bo \(2025\)Fewer may be better: enhancing offline reinforcement learning with reduced dataset\.External Links:2502\.18955,[Link](https://arxiv.org/abs/2502.18955)Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p3.1)\.
- Y\. Yue, B\. Kang, X\. Ma, Z\. Xu, G\. Huang, and S\. Yan \(2022\)Boosting offline reinforcement learning via data rebalancing\.arXiv preprint arXiv:2210\.09241\.Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p2.1)\.
- Y\. Yue, B\. Kang, X\. Ma, Q\. Yang, G\. Huang, S\. Song, and S\. Yan \(2025\)Decoupled prioritized resampling for offline rl\.External Links:2306\.05412,[Link](https://arxiv.org/abs/2306.05412)Cited by:[§2](https://arxiv.org/html/2608.07719#S2.p2.1)\.

## Appendix AProof of the Local Score Comparison

###### Proof of Remark[1](https://arxiv.org/html/2608.07719#Thmtheorem1)\.

Becauseℐθ\\mathcal\{I\}\_\{\\theta\}is positive definite on the span of the candidate gradients, its eigenvalues on that span obey0<λmin​\(ℐθ\)≤λmax​\(ℐθ\)<∞0<\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\\leq\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)<\\infty\. For everyxxin the span, the Rayleigh quotient for the inverse gives

∥x∥22λmax​\(ℐθ\)≤x⊤​ℐθ−1​x≤∥x∥22λmin​\(ℐθ\)\.\\frac\{\\lVert x\\rVert\_\{2\}^\{2\}\}\{\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\\leq x^\{\\top\}\\mathcal\{I\}\_\{\\theta\}^\{\-1\}x\\leq\\frac\{\\lVert x\\rVert\_\{2\}^\{2\}\}\{\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\.\(5\)Applying Equation[5](https://arxiv.org/html/2608.07719#A1.E5)tox=gτx=g\_\{\\tau\}and usingmg≤∥gτ∥2≤Mgm\_\{g\}\\leq\\lVert g\_\{\\tau\}\\rVert\_\{2\}\\leq M\_\{g\}yields

mg2λmax​\(ℐθ\)≤gτ⊤​ℐθ−1​gτ≤Mg2λmin​\(ℐθ\)\.\\frac\{m\_\{g\}^\{2\}\}\{\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\\leq g\_\{\\tau\}^\{\\top\}\\mathcal\{I\}\_\{\\theta\}^\{\-1\}g\_\{\\tau\}\\leq\\frac\{M\_\{g\}^\{2\}\}\{\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\.\(6\)Multiplying byδθ​\(τ\)2/2\\delta\_\{\\theta\}\(\\tau\)^\{2\}/2gives

mg2​δθ​\(τ\)22​λmax​\(ℐθ\)≤Iloc​\(τ\)≤Mg2​δθ​\(τ\)22​λmin​\(ℐθ\)\.\\frac\{m\_\{g\}^\{2\}\\delta\_\{\\theta\}\(\\tau\)^\{2\}\}\{2\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\\leq I\_\{\\mathrm\{loc\}\}\(\\tau\)\\leq\\frac\{M\_\{g\}^\{2\}\\delta\_\{\\theta\}\(\\tau\)^\{2\}\}\{2\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\.\(7\)Taking square roots,

mg2​λmax​\(ℐθ\)​\|δθ​\(τ\)\|≤Iloc​\(τ\)≤Mg2​λmin​\(ℐθ\)​\|δθ​\(τ\)\|\.\\frac\{m\_\{g\}\}\{\\sqrt\{2\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\}\|\\delta\_\{\\theta\}\(\\tau\)\|\\leq\\sqrt\{I\_\{\\mathrm\{loc\}\}\(\\tau\)\}\\\\\[\-2\.0pt\] \\leq\\frac\{M\_\{g\}\}\{\\sqrt\{2\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\}\|\\delta\_\{\\theta\}\(\\tau\)\|\.\(8\)The ratio between the upper and lower constants is

Mgmg​λmax​\(ℐθ\)λmin​\(ℐθ\)=Mgmg​κ,\\frac\{M\_\{g\}\}\{m\_\{g\}\}\\sqrt\{\\frac\{\\lambda\_\{\\max\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\{\\lambda\_\{\\min\}\(\\mathcal\{I\}\_\{\\theta\}\)\}\}=\\frac\{M\_\{g\}\}\{m\_\{g\}\}\\sqrt\{\\kappa\},which proves the stated comparison\. Since the quadratic form may vary among candidates, the inequalities do not imply pairwise rank preservation\. ∎

## Appendix BImplementation Details

All control experiments use D4RL v2 with Gym 0\.23\.1 and MuJoCo 2\.1\.0\. Actors and critics are two\-layer multilayer perceptrons with 256 hidden units and ReLU activations\. Adam uses learning rate3×10−43\\times 10^\{\-4\}, batch size 256, discountγ=0\.99\\gamma=0\.99, and target update rate 0\.005\. TD3\+BC uses behavior\-cloning coefficientα=2\.5\\alpha=2\.5\. CQL uses conservative coefficientα=5\.0\\alpha=5\.0for Walker2d and Hopper andα=10\.0\\alpha=10\.0for HalfCheetah; its AntMaze rewards are shifted by−1\-1and scaled by 10\. IQL uses expectile 0\.7 and inverse temperature 3 on locomotion, and the manifest\-verified\(τ,β\)=\(0\.9,10\)\(\\tau,\\beta\)=\(0\.9,10\)setting on AntMaze\. States are normalized for TD3\+BC\.

CODS uses five acquisition rounds, a 2% burn\-in set, and batches of 1\.6% of the original dataset\. The selector receives exactly 100,000 critic updates:U0=25,000U\_\{0\}=25\{,\}000burn\-in updates andUk=15,000U\_\{k\}=15\{,\}000after each of the five acquisitions\. The fixed\-budget round ablation keepsU0=25,000U\_\{0\}=25\{,\}000and divides the remaining 75,000 updates evenly, givingUk=75,000U\_\{k\}=75\{,\}000, 25,000, 15,000, and 7,500 forK=1,3,5,10K=1,3,5,10, respectively\. Pool scoring uses minibatches of 4,096 candidates; the target network receives its soft update after every critic step, optimizer state continues across acquisition rounds, and deterministic index order breaks score ties\. Separate seeds govern burn\-in selection, stochastic residual targets, and downstream initialization\. The selected\-index filename is keyed by dataset, base algorithm, preprocessing hash, and selection seed, independently of downstream runs\.

The main comparison gives every final learner 1,000,000 updates\. Evaluations occur every 5,000 updates with ten deterministic episodes, and each reported score averages the final ten checkpoints\. The 10% trajectory holdout is disjoint from selection and training\. Subset budgets are measured against the original dataset, whereas Pool refers to the remaining 90% eligible training pool\.

For k\-center, state and action coordinates are z\-scored before Euclidean distance is computed\. ReDOR uses the last hidden layer of the burn\-in critic and candidate minibatches of 2,048 transitions in its OMP\-style objective\. The local implementation was checked against the official code, representation, stopping rule, regularization, and search space; Appendix[O\.1](https://arxiv.org/html/2608.07719#A15.SS1)reports the resulting fidelity gaps\. Static\-PER and CODS with one acquisition batch share one implementation, selected indices, and seed convention\.

## Appendix CExtended Design Rationale

Offline RL development normally compares several policy objectives, tunes conservative penalties or advantage temperatures, and repeats promising configurations across random seeds\. If every run scans the same million\-transition pool for the same number of dataset passes, total data movement and gradient computation scale with the entire sweep\. A useful selector should therefore do more than identify a small training set for one model\. It should produce a stable artifact whose one\-time construction cost can be shared across compatible runs, while retaining the transitions on which later value estimates depend\.

Three requirements follow from that use case\. The score should be tied to the backup of the learner that will consume the subset, because TD3\+BC, CQL, and IQL do not form identical value targets\. It should be recomputed after the critic changes, because the relevant errors are non\-stationary\. Selection must also end before downstream training begins, so that all reported learners start afresh from a common subset and the acquisition critic is not hidden pretraining\. CODS is a minimal procedure satisfying these requirements\.

The term “selection” is important\. CODS neither changes the reward nor assigns persistent sampling weights to retained transitions\. Once the indices are frozen, the downstream learner samples uniformly from that subset using its ordinary training code\. This interface makes the method auditable: a selected\-index file, dataset version, and selection manifest determine the data seen by every subsequent run\. It also makes failures visible\. If the burn\-in critic is uninformative or noisy transitions dominate its residuals, iterative acquisition can preserve the wrong examples; the method contains no theorem that automatically repairs such a critic\.

The contribution consequently differs from both replay and policy learning\. Methodologically, CODS turns a moving Bellman\-error signal into a fixed training artifact and specifies algorithm\-matched residuals for three offline learners\. Empirically, it evaluates one subset budget across a task–algorithm matrix and isolates acquisition rounds\. Computationally, it separates subset quality at equal update counts from amortization under an epoch\-scaled ledger\. Conflating those protocols would turn a measured data\-quality result into an unmeasured single\-run speedup claim\.

The ranking is recomputed because value errors move\. Suppose a one\-shot selector ranks two transitions before either has been used\. The first may have a large residual because its predecessor values have not propagated\. After it is added and the critic is updated, its error can contract while the second becomes newly inconsistent with the revised value surface\. A fixed top\-BBranking cannot react\. CODS commits only the nextbkb\_\{k\}transitions and then scores the remaining pool again\. It is greedy with respect to a moving model, not a fixed set function\.

This view clarifies the burn\-in set\. Burn\-in supplies enough data to define an initial critic, but it is not assumed to be representative and is never treated as a trusted core\. Its indices count against the total budget, and later rounds inherit any blind spots it creates\. Iteration gives newly acquired transitions an opportunity to change the ranking; it does not guarantee recovery from an initially biased critic\.

CODS is complete when it writes the selected indices\. A subset selected with a given base algorithm can be reused across that algorithm’s seeds and hyperparameters without rerunning acquisition\. Appendix[O\.8](https://arxiv.org/html/2608.07719#A15.SS8)also measures cross\-algorithm reuse: matched selectors are best for all three downstream learners, while mismatched artifacts lose 1\.4–3\.9 retention points\. The primary matrix therefore uses algorithm\-specific artifacts, and the transfer result quantifies rather than assumes their portability\.

## Appendix DBaseline Implementations and Fairness

The baselines represent distinct hypotheses about useful offline data\. k\-center preserves geometric coverage after z\-scoring states and actions\. ReD uses trajectory return and therefore favors high\-return behavior\. Static\-PER ranks by the absolute residual of the burn\-in critic and performs no rescoring\. ReDOR approximates the full\-pool actor–critic gradient with an OMP\-style objective\. Random selection gives the expected performance of an unstructured subset\. Table[6](https://arxiv.org/html/2608.07719#A4.T6)makes their operational differences explicit\.

Table 6:Operational comparison of data\-use strategies\. “Updated” means that the score changes after critic fitting; “reusable” means that the output can be fixed before independent downstream runs\.The distinction between static and dynamic methods changes the control\. Dynamic replay can adapt to each downstream seed, but its adaptation cost is paid within every run and its sampled sequence cannot be reused directly\. A static selector pays before downstream optimization and must work without seeing the final learner\. Static\-PER is consequently the closest ablation: with the same burn\-in, residual, tie rule, and budget, it is exactly CODS with one acquisition batch\. A separate name must not be used to imply an additional method difference\.

ReDOR supplies a stronger representation than a scalar residual but requires candidate gradients and OMP\-style optimization\. CODS asks whether repeatedly refreshing a cheap score is sufficient\. k\-center and ReD span complementary failure modes\. Geometry can remain useful when a critic is poor but ignores value propagation; return can retain successful behavior but omit low\-return connectors\. CODS follows current Bellman inconsistency and has no independent diversity guarantee\. None of these axes creates a universal ranking of selectors\.

Every subset method must draw from the same eligible pool and receive the same cardinality\. Random selection is without replacement\. k\-center normalization statistics are fit only on the eligible pool\. ReD, Static\-PER, and CODS share the trajectory split\. ReDOR may use the common burn\-in critic representation, but its complete selection cost is charged to ReDOR\. Any hyperparameter search unique to one selector must be reported and budgeted rather than hidden in preprocessing\.

ReDOR uses gradients of the last critic hidden layer and candidate minibatches of 2,048\. The fidelity audit in Table[12](https://arxiv.org/html/2608.07719#A15.T12)matches the official representation, OMP stopping rule, regularization, and search space; local scores differ by at most 0\.4 points on the four audited cells\. OPER is implemented as its published offline resampling procedure rather than relabeled Static\-PER\. Table[13](https://arxiv.org/html/2608.07719#A15.T13)reports every OPER cell, and the corresponding manifests charge all baseline\-specific preprocessing and selection time\.

## Appendix ESelection Schedule, Invariants, and Complexity

Acquisition batches trade adaptivity for cost\. Very small batches approximate sequential rescoring but require many pool\-wide forward passes and critic updates\. One large batch is inexpensive but collapses to one\-shot selection\. The default schedule assigns 2% of the original dataset to burn\-in and 1\.6% to each of five later rounds, placing the experiment between those extremes\. TheKKablation holds the final 10% cardinality fixed, but its interpretation also depends on how selection\-critic updates are allocated\.

EveryKKcondition uses the 100,000\-update schedule reported in Appendix[B](https://arxiv.org/html/2608.07719#A2): 25,000 burn\-in updates followed by an equal division of 75,000 post\-acquisition updates\. Thus changingKKchanges only how often the ranking is refreshed and how the fixed post\-burn\-in budget is partitioned\. The manifest records the 4,096\-transition scoring batch, per\-step target updates, continued optimizer state, deterministic tie rule, and all selection seeds\. These fields make the fixed\-critic\-budget attribution directly auditable\.

Several invariants make the selected artifact auditable\. Candidate and diagnostic trajectories are disjoint before scoring\. Selected indices are unique, acquisition batches do not overlap, and burn\-in plus later batches equals the declared budget\. Ties are broken deterministically\. Downstream initialization does not alter the index file\. Target\-policy noise and candidate sampling during acquisition belong to a recorded selection seed rather than to a downstream seed\. Dataset order and preprocessing hashes are stored with the indices so that an integer index cannot silently refer to another dataset revision\.

At roundkk, a direct implementation evaluatesN−\|𝒮k−1\|N\-\|\\mathcal\{S\}\_\{k\-1\}\|candidates\. If one scoring forward pass costscfc\_\{f\}and the critic receivesUkU\_\{k\}updates of costcuc\_\{u\}, acquisition work is approximately

Cselect=cf​∑k=1K\(N−\|𝒮k−1\|\)\+cu​∑k=0KUk\.C\_\{\\mathrm\{select\}\}=c\_\{f\}\\sum\_\{k=1\}^\{K\}\\bigl\(N\-\|\\mathcal\{S\}\_\{k\-1\}\|\\bigr\)\+c\_\{u\}\\sum\_\{k=0\}^\{K\}U\_\{k\}\.\(9\)Streaming top\-bkb\_\{k\}selection avoids storing all scores, keeping additional memory proportional to the candidate minibatch and a size\-bkb\_\{k\}heap\. A full sort is simpler but costsO​\(N​log⁡N\)O\(N\\log N\)per round; streaming selection isO​\(N​log⁡bk\)O\(N\\log b\_\{k\}\)after scoring\. Both produce the same indices under the same deterministic tie rule\.

## Appendix FRobustness and Additional Evaluation Protocols

The primary matrix establishes mean subset quality, while the completion experiments test the main robustness threats under shared design rules\. Table[7](https://arxiv.org/html/2608.07719#A6.T7)summarizes the controls; their measured outcomes appear in Appendix[O](https://arxiv.org/html/2608.07719#A15)\. Each uses genuine selected\-index artifacts and raw downstream arrays, and every level was fixed before inspecting the corresponding outcome\.

Table 7:Design of the completed robustness controls\. Primary outcomes are normalized return, selected\-index overlap, and complete selection plus training time\.Selection stability should be reported at two levels\. Pairwise Jaccard overlap measures whether two seeds retain the same indices, while downstream return measures whether different but functionally equivalent subsets exist\. Low overlap does not imply failure if returns are stable, and high overlap is not sufficient if every seed locks onto the same corrupted outliers\. Rank correlation between residual lists across rounds can additionally show how much rescoring changes the candidate order\.

The budget study uses independently generated artifacts at 2, 5, 10, 15, and 20% on the four representative configurations\. Independent selection gives every budget its best unconstrained artifact; it does not imply that a 5% subset is nested inside a 10% subset\. The same convention is used for every selector, while the fixed\-KKexperiment alone holds the final cardinality fixed to isolate rescoring\.

The targeted variants do not change the paper’s central method\. Residual clipping replaces\|δ\|\|\\delta\|bymin⁡\(\|δ\|,c\)\\min\(\|\\delta\|,c\), the ensemble score takes the median across independently initialized critics, and the residual–coverage hybrid reserves 20% of each acquisition batch for farthest\-first points\. Appendix[O\.7](https://arxiv.org/html/2608.07719#A15.SS7)first measures raw CODS and then reports each variant only on the failure it is designed to address\.

## Appendix GLocal Information\-Proxy Audit

The local score comparison is useful only if its omitted gradient–curvature factor is empirically characterized\. For a fixed critic checkpoint, a reproducible audit samples candidate transitions, computes‖gτ‖2\\\|g\_\{\\tau\}\\\|\_\{2\}, estimates extremal eigenvalues of the damped information matrix on the sampled gradient span, and compares residual order with the full local proxy\. Damping, gradient layer, sample size, and numerical solver tolerance must be reported because each can materially change the condition number\.

The current worst\-case factor of approximately2×1032\\times 10^\{3\}combines the observed gradient\-norm ratio with the square root of the estimated condition number\. At that factor, the sufficient separation needed to certify a pair order is so large that the observed certified rate rounds to zero\. This is a negative diagnostic result: the bound does not explain why CODS works\. It should be released alongside, rather than replaced by, less conservative rank statistics\.

Table 8:Released quantities for the local information\-proxy audit\. Values aggregate the four representative critic checkpoints\.
## Appendix HExtended Interpretation of the Reported Results

Table[9](https://arxiv.org/html/2608.07719#A8.T9)groups the 20 valid cells by downstream learner\. CODS retains 96\.6% of the Pool sum over the six eligible TD3\+BC cells, 97\.0% over seven CQL cells, and 96\.1% over seven IQL cells \(the seventh IQL cell is the reconciled AntMaze row\)\. These ratios show that the overall 96\.6% figure is not produced solely by one learner, but they should not be treated as a new metric across unrelated benchmark scales\.

Table 9:Descriptive summary by downstream algorithm over the retained cells\. Retention is the sum of CODS scores divided by the sum of Pool scores within each row\.On medium\-expert locomotion, CODS trails Pool by 1\.6–4\.7 points while maintaining a consistent advantage over the populated subset baselines\. On medium locomotion, the Pool gap is often smaller, although Walker2d with TD3\+BC remains a visible exception at 5\.2 points\. The two sparse\-reward AntMaze cells are the CQL cell \(CODS 58\.0 vs Pool 61\.2, ReDOR 45\.0\) and the reconciled IQL cell \(CODS 63\.4 vs Pool 72\.6, ReDOR 56\.8\); both benefit from the enriched connective transitions characterized in Appendix[O\.6](https://arxiv.org/html/2608.07719#A15.SS6)\.

The one reversal is CQL Hopper\-medium, where ReDOR scores 57\.2 and CODS 56\.8\. Both reported standard deviations exceed the 0\.4\-point difference\. The row is retained and is not relabeled as a CODS win\. Across the matrix, the highest sample mean is descriptive rather than a per\-cell significance decision; paired arrays and multiplicity control are needed for inferential labels\.

Relative to Static\-PER, CODS has a higher reported mean in all 20 valid cells\. The controlledKKablation uses the same burn\-in, tie rule, final cardinality, and 100,000 selector updates on four representative cells \(Appendix[O\.3](https://arxiv.org/html/2608.07719#A15.SS3)\)\. Its mean improves by 8\.05 points from one to three rounds and 3\.18 points from three to five rounds, while ten rounds changes the mean by−0\.40\-0\.40\. Because the final cardinality and selector\-update total are fixed, this experiment isolates the value of refreshing the ranking, subject to the usual interaction between the chosen acquisition partition and critic optimization\.

The budget curve supplies a complementary view\. CODS rises quickly from 2 to 5%, changes modestly by 10%, and nearly reaches Pool at 15%\. At 20%, its plotted mean slightly exceeds Pool\. The fixed\-update protocol means a larger subset receives fewer expected revisits per transition, so the curve should not be read as a pure information\-per\-example law\. The 10% budget was the common primary condition rather than a fitted optimum\.

## Appendix IComplete Compute Accounting

Letcpc\_\{p\}be the cost of one epoch\-scaled Pool run,csc\_\{s\}the cost of one subset run, andcac\_\{a\}acquisition cost\. ForSScompatible downstream configurations,

Cp​\(S\)=S​cp,Cs​\(S\)=ca\+S​cs\.C\_\{p\}\(S\)=Sc\_\{p\},\\qquad C\_\{s\}\(S\)=c\_\{a\}\+Sc\_\{s\}\.\(10\)Ifcp\>csc\_\{p\}\>c\_\{s\}, the selected approach becomes strictly cheaper whenS\>ca/\(cp−cs\)S\>c\_\{a\}/\(c\_\{p\}\-c\_\{s\}\)\. With\(cp,cs,ca\)=\(1\.8,0\.18,3\.0\)\(c\_\{p\},c\_\{s\},c\_\{a\}\)=\(1\.8,0\.18,3\.0\)hours, one configuration costs 1\.8 hours for Pool and 3\.18 for CODS, two cost 3\.6 and 3\.36 hours, and twenty cost 36\.0 and 6\.6 hours\. Reuse, not faster single\-run convergence, creates the reported ratio\.

The same calculation for ReDOR uses the supplied 4\.5\-hour acquisition cost and the same 0\.18\-hour subset training cost, giving 8\.1 hours over twenty runs\. These figures remain a ledger until linked to job identifiers and score logs\. A complete accounting includes dataset loading, feature construction, acquisition\-critic fitting, pool scoring, selector optimization, final training, evaluation, and failed or restarted jobs\. It should report hardware model, software stack, mixed\-precision setting, and whether acquisition and training share cached data\.

Matched\-performance efficiency requires a frontier rather than one total\. Table[22](https://arxiv.org/html/2608.07719#A15.T22)therefore reports equal optimizer updates, equal dataset passes, and equal amortized GPU hours, with acquisition charged at 20 reuses\. The artifact additionally records final return, area under the learning curve, time to the predeclared target, acquisition cost, and reuse count\. These views distinguish subset quality, amortized processing savings, and end\-to\-end acceleration; broader hardware and task coverage remains future work\.

## Appendix JExcluded AntMaze Configurations

The TD3\+BC AntMaze\-medium\-play learner reports8\.4±3\.68\.4\\pm 3\.6for Pool and7\.2±3\.27\.2\\pm 3\.2for CODS, placing both conditions at the task floor\. Historical values of 71\.2 and 66\.1 cannot be tied to a configuration, checkpoint, and raw episode\-return log in the supplied source\. The clean floor\-level row is retained in Table[1](https://arxiv.org/html/2608.07719#S5.T1), but the configuration is excluded from subset comparisons and mechanism claims\.

The IQL AntMaze cell has now been reconciled\. The row uses the standard AntMaze IQL setting\(τ,β\)=\(0\.9,10\)\(\\tau,\\beta\)=\(0\.9,10\), a fixed code revision, identical selected indices across the three nested downstream seeds, and final\-checkpoint averaging identical to the locomotion rows; under that manifest the run reports Pool 72\.6, CODS 63\.4, ReDOR 56\.8, OPER 55\.6, ReD 52\.9, Static\-PER 52\.1, k\-center 47\.7, and Random 16\.7\. This resolves the earlier conflict, in which a sensitivity record reported 39\.5 for\(0\.7,3\)\(0\.7,3\)and 60\.1 for\(0\.9,10\)\(0\.9,10\)while an aggregate table attributed 68\.1 to\(0\.7,3\)\(0\.7,3\): the historical 68\.1 value is*not*reused, and only the manifest\-verified\(0\.9,10\)\(0\.9,10\)row enters the 20\-cell primary matrix\. Every score in the reconciled row maps to the exact hyperparameters, code revision, selected indices, checkpoint rule, and raw seed arrays recorded in the artifact\.

## Appendix KStatistical Inference and Secondary Cross\-Cell Test

The raw arrays preserve dataset, learner, selection\-seed, and downstream\-seed identifiers\. For each comparison, the 95% interval is obtained from 100,000 hierarchical bootstrap replicates\. A replicate samples the seven datasets with replacement; within each sampled dataset it resamples paired selection\-seed labels and then paired downstream\-seed labels\. Repeated occurrences of a dataset carry all of its learner cells together, preserving dataset\-level dependence\. Pool has no selection stage, so its downstream seeds are resampled within dataset–learner cell\. Percentile intervals are computed from the resulting mean paired differences\.

The six confirmatory subset tests use the paired differences

dj​a​s​t=μ\+λa\+uj\+wj​a\+qj​a​s\+ϵj​a​s​t,d\_\{jast\}=\\mu\+\\lambda\_\{a\}\+u\_\{j\}\+w\_\{ja\}\+q\_\{jas\}\+\\epsilon\_\{jast\},\(11\)wherejjindexes dataset,aalearner,ssselection seed, andttdownstream seed\. Learner effectsλa\\lambda\_\{a\}are fixed;uju\_\{j\},wj​aw\_\{ja\}, andqj​a​sq\_\{jas\}are independent Gaussian random intercepts for dataset, dataset–learner cell, and selection artifact within cell\. The residual term retains the paired downstream\-seed variation\. For each baseline, a one\-sidedpp\-value is the proportion, with the standard plus\-one correction, of 100,000 null parametric\-bootstrap refits whose estimatedμ\\muis at least the observed estimate\. Holm correction is applied once across ReDOR, ReD, Static\-PER, OPER, k\-center, and Random\. Pool was predeclared as a separate retention comparison and receives an interval but no entry in this six\-hypothesis family\. Thusp<0\.001p<0\.001is model\-based hierarchical inference, not an impossible exact sign\-flip result from only seven dataset blocks\.

For continuity with the cell\-level analysis, Table[10](https://arxiv.org/html/2608.07719#A11.T10)also reports exact one\-sided Wilcoxon signed\-rank calculations on the 20 valid cell means\. The maximum positive\-rank sum is210210\. ReDOR and OPER each have one negative difference of the smallest absolute rank, givingW\+=209W^\{\+\}=209andp=2/220=1\.91×10−6p=2/2^\{20\}=1\.91\\times 10^\{\-6\}\. The other four baselines are positive in all 20 cells, givingW\+=210W^\{\+\}=210andp=2−20=9\.54×10−7p=2^\{\-20\}=9\.54\\times 10^\{\-7\}\. These secondary values agree in direction with the hierarchical analysis but are not confirmatory because cells sharing a dataset are dependent and cell means discard seed uncertainty\.

Table 10:Secondary exact signed\-rank calculations over 20 valid cell means\. Confirmatory inference instead uses the nested arrays\.
## Appendix LOptional Held\-Out Value Diagnostic

The original study reserved 10% of trajectories for an optional monitoring statistic inspired by conformal prediction\(Vovket al\.,[2005](https://arxiv.org/html/2608.07719#bib.bib29); Angelopoulos Anastasios and Stephen,[2021](https://arxiv.org/html/2608.07719#bib.bib30); Angelopouloset al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib1),[2022](https://arxiv.org/html/2608.07719#bib.bib2); Taufiqet al\.,[2022](https://arxiv.org/html/2608.07719#bib.bib18)\)\. For calibration valuesZ1,…,ZmZ\_\{1\},\\ldots,Z\_\{m\}, the reported lower empirical quantile is

q^α=Z\(⌈\(m\+1\)​α⌉\)\.\\widehat\{q\}\_\{\\alpha\}=Z\_\{\(\\lceil\(m\+1\)\\alpha\\rceil\)\}\.This quantity does not score transitions, alter the selected indices, or stop the fixed\-KKacquisition schedule\. Removing it while fixing the candidate pool and random seed leaves CODS unchanged\. Because an adaptively selected policy is not exchangeable with a reused calibration sample without further conditions, we make no coverage claim for policy performance\. The diagnostic is retained only to document why the training pool contains 90% rather than 100% of the original trajectories\.

## Appendix MDescriptive AntMaze Composition Audit

Table[11](https://arxiv.org/html/2608.07719#A13.T11)preserves the supplied composition audit for the floor\-level TD3\+BC AntMaze critic\. CODS retains these transition categories at approximately 1\.6–2\.0 times the 10% random rate\. Since the critic does not solve the task, the audit cannot explain successful navigation and is not referenced by the main empirical argument\. The former qualitative heatmap and residual\-evolution plot are omitted because they were generated from the same failed critic and would add visual confidence without valid mechanistic evidence\.

Table 11:Descriptive TD3\+BC AntMaze subset composition at a 10% budget\. The floor\-level critic makes these enrichments non\-explanatory\.
## Appendix NArtifact Requirements

The anonymous artifact contains the genuine evaluation arrays underlying every aggregate: five selection seeds by three downstream seeds for each subset method and cell, ten downstream seeds for Pool, and the corresponding per\-episode returns and checkpoint summaries\. No seed value is reconstructed from a mean or standard deviation\. The hierarchy encoded in those arrays is the one used by the confidence intervals, variance components, paired tests, and Holm\-adjusted comparisons in this paper\.

Each array is linked by a machine\-checkable manifest to its configuration, code revision, selection and downstream seeds, selected\-transition indices, checkpoint identities, environment versions, and preprocessing hashes\. The same manifest recordsU0:KU\_\{0:K\}, candidate\-scoring batch size, optimizer continuation, and target\-update frequency; it reconciles IQL AntMaze and marks TD3\+BC AntMaze as excluded\. Regeneration scripts rebuild every table and figure from these arrays and fail if an expected seed, cell, or manifest field is absent\.

## Appendix OExperimental Completion and Robustness Results

This appendix collects the completion experiments referenced in the main text\. Unless a table states otherwise, all selectors use the same eligible 90% pool and 10% budget\. Every subset method uses five selection seeds, each evaluated with three downstream seeds \(15 nested observations per cell\); Pool uses ten downstream seeds\. The four representative cells are Walker2d\-medium/TD3\+BC, HalfCheetah\-medium/CQL, Hopper\-medium/IQL, and AntMaze\-medium\-play/CQL\. Confirmatory intervals use the dataset\-blocked hierarchical bootstrap, while the six subset hypotheses use the paired mixed\-effects parametric\-bootstrap tests defined in Appendix[K](https://arxiv.org/html/2608.07719#A11)\. Pool remains a separate retention comparison\.

### O\.1Baseline fidelity

The primary comparison uses official or author\-validated implementations\. The local ReDOR reproduction is within 0\.4 normalized points of the official implementation on the four representative cells and its selection cost differs by less than 6%; OPER is implemented as its actual offline resampling procedure rather than using Static\-PER as a proxy\.

Table 12:Implementation\-fidelity audit on the four representative cells\. Gap is the maximum absolute local\-minus\-official difference\.
### O\.2Complete OPER comparison

OPER is a fully populated baseline rather than a proxy or a four\-cell audit\. Table[13](https://arxiv.org/html/2608.07719#A15.T13)gives all 20 valid cells\. CODS is higher in 19 cells; the only reversal is CQL Hopper\-medium, where OPER is higher by 0\.2 points\. The mean paired difference is 6\.84 points, matching Table[2](https://arxiv.org/html/2608.07719#S5.T2)exactly\.

Table 13:Complete equal\-update OPER comparison\. Means±\\pmSD use five selection seeds and three downstream seeds\.Δ\\Deltais CODS minus OPER\.
### O\.3Iterative rescoring across tasks

Holding total selection\-critic updates constant across round counts, Table[14](https://arxiv.org/html/2608.07719#A15.T14)spans the four representative cells\. The hierarchical estimate forK=5K\{=\}5vsK=1K\{=\}1is\+11\.23\+11\.23points \(95% CI\[\+8\.86,\+13\.74\]\[\+8\.86,\+13\.74\],p<0\.001p<0\.001\);K=10K\{=\}10vsK=5K\{=\}5is−0\.40\-0\.40\(CI\[−1\.18,\+0\.37\]\[\-1\.18,\+0\.37\]\), supporting saturation rather than monotonic improvement\.

Table 14:Fixed\-critic\-budget acquisition\-round ablation across four representative cells \(mean±\\pmSD\)\.
### O\.4Multi\-task budget curves

The four\-cell mean is 40\.35 at 2%, 56\.83 at 5%, 61\.18 at 10%, and 63\.45 at 20%\. Direct finite differences are therefore 5\.49 points per budget percentage point from 2% to 5%, 0\.87 from 5% to 10%, and 0\.23 from 10% to 20%\. These are descriptive slopes computed from Table[15](https://arxiv.org/html/2608.07719#A15.T15), not coefficients from an unreleased mixed model\. Marginal gains diminish after 10% on these cells, but 10% is not claimed universally optimal\.

Table 15:Budget sweep across four representative cells\. Entries average five selection seeds and three downstream seeds\.
### O\.5Selection stability

Five selection seeds separate burn\-in sensitivity from downstream\-seed noise\. Exact index overlap is moderate while downstream performance is far more stable; successive candidate\-ranking correlations rise \(≈0\.42→0\.56→0\.68→0\.76\\approx 0\.42\\to 0\.56\\to 0\.68\\to 0\.76\), so early rounds change the ordering most\.

Table 16:Selection\-seed stability\. CV is the coefficient of variation of performance across selection seeds\.
### O\.6Learner\-specific sparse\-reward mechanism tests

The CQL and IQL analyses use their own successful AntMaze critics, manifests, and selected indices; neither learner’s composition is used to explain the other\. Categories are fixed from the environment before inspecting scores\. A transition is reward\-bearing when its raw environment reward is positive before CQL reward shifting\. A maze cell is a bottleneck when removing it disconnects the start and goal regions in the four\-neighbor graph of the published layout, and a transition is goal\-proximal when its raw\(x,y\)\(x,y\)coordinate is within 2\.0 environment units of the goal\. Categories may overlap, but each intervention is performed separately\.

For an intervention, every selected transition in the target category is replaced by an eligible\-pool transition outside that category, matched on normalized trajectory\-position decile and 20\-nearest\-neighbor state\-density decile in z\-scored state–action space\. Bottleneck and goal\-proximal replacements also match the raw reward indicator\. Reward\-bearing replacements cannot preserve reward sign while excluding the category; consequently, that intervention partly tests direct reward availability as well as which rewarded transitions CODS retained\. Equal\-random controls replace the same number of transitions under the position and density matches without targeting a category\.

Table 17:Learner\-specific AntMaze enrichment and category\-excluding removal interventions\. CIs are paired hierarchical\-bootstrap intervals\.
### O\.7Robustness to critic and data failures

Raw residual selection is sensitive to reward outliers; a clipped\-ensemble variant recovers part but not all of the loss and is treated as an ablation\. A 2% burn\-in is the smallest tested budget near the plateau, and a pure residual score is not density\-invariant under duplication\.

Table 18:Reward corruption \(CQL AntMaze\): replace a fraction of eligible rewards with 99th\-percentile magnitude and random sign\.Table 19:Burn\-in sensitivity over the 20\-cell matrix\.Table 20:Duplicate\-density stress on CQL AntMaze\.
### O\.8Cross\-algorithm reuse

The transfer matrix reports percentages of each learner’s Pool score over the common locomotion cells\. Matched selection is best in every column, so a single TD3\+BC subset cannot be reused for CQL and IQL without cost\.

Table 21:Cross\-algorithm selector reuse \(% of each learner’s Pool score\)\. Rows are the selector critic; columns are the downstream learner\.
### O\.9Matched compute and reuse

The5\.45×5\.45\\timesfigure is arithmetic under an epoch\-scaled ledger; both the equal\-update and equal\-pass conditions are reported\. At 20 reuses on Walker2d/TD3\+BC, CODS reduces epoch\-scaled total time from 36\.0 to 6\.6 hours under equal dataset passes while retaining 90\.3% of Pool\. Under equal updates, it retains 93\.8% but gives no meaningful training\-time reduction\.

Across the 20\-configuration sweep, Pool and CODS configuration rankings have Spearman correlation 0\.89 with 4/5 top\-five overlap, supporting subset reuse for tuning without claiming identical rankings\.

Table 22:Compute protocols on Walker2d\-medium with TD3\+BC\. Retention and total\-time reduction are reported for 20 reuses\.
### O\.10Variance components and theory audit

The four\-cell nested study attributes 71% of variance to dataset/configuration, 17% to selection seed, and 12% to downstream seed\. The theory remains modest even where empirical rank association is positive: the worst\-case score factor\(Mg/mg\)​κ\(M\_\{g\}/m\_\{g\}\)\\sqrt\{\\kappa\}is2\.0×1032\.0\\times 10^\{3\}, residual\-vs\-local\-proxy Spearman is0\.63±0\.050\.63\{\\pm\}0\.05, top\-decile overlap is0\.58±0\.040\.58\{\\pm\}0\.04, and the certified pair\-order rate is0\.02%0\.02\\%\. Residuals are a useful surrogate, but the worst\-case bound certifies essentially none of the ranking; these do not support a coreset, policy\-recovery, regret, or submodular\-greedy guarantee\.

### O\.11Sequence\-level language extension

We evaluate the sequence adaptation on ALFWorld embodied interaction\(Shridharet al\.,[2021](https://arxiv.org/html/2608.07719#bib.bib33)\)and GSM8K mathematical reasoning\(Cobbeet al\.,[2021](https://arxiv.org/html/2608.07719#bib.bib34)\)\. A frozen Qwen2\.5\-7B\-Instruct generator\(Yanget al\.,[2024](https://arxiv.org/html/2608.07719#bib.bib35)\)constructs each offline pool before any selector is trained\. For ALFWorld, we generate 16 trajectories for every task in the standard training split using temperature 0\.8, nucleus probability 0\.95, and at most 50 environment actions; reporting uses the standard unseen evaluation split\. For GSM8K, we generate eight newline\-delimited reasoning traces for each of the 7,473 training problems under the same sampling parameters, giving 59,784 traces; reporting uses the 1,319\-problem test split\. Pools, prompts, and sampled traces are fixed and shared by all selectors\.

In ALFWorld,sts\_\{t\}contains the task goal, current observation, and complete action–observation prefix, whileata\_\{t\}is the next textual environment action\. In GSM8K,sts\_\{t\}contains the problem and complete reasoning prefix, whileata\_\{t\}is the next newline\-delimited reasoning step\. The terminal reward is one for environment success or exact final numeric answer and zero otherwise; intermediate rewards are zero andγ=1\\gamma=1for both episodic tasks\. Invalid ALFWorld actions remain in the trace with the environment’s resulting observation, so the offline construction does not silently filter failures\.

The critic follows the IQL form used in OREO\-style offline reasoning\(Wanget al\.,[2025](https://arxiv.org/html/2608.07719#bib.bib32)\)\. Qwen2\.5 final\-token states feed scalarQθ​\(st,at\)Q\_\{\\theta\}\(s\_\{t\},a\_\{t\}\)andVψ​\(st\)V\_\{\\psi\}\(s\_\{t\}\)heads\. TheQQhead minimizes squared error tort\+\(1−dt\)​Vψ​\(st\+1\)r\_\{t\}\+\(1\-d\_\{t\}\)V\_\{\\psi\}\(s\_\{t\+1\}\); the value head minimizes the expectile loss withτ=0\.9\\tau=0\.9on the stop\-gradient differenceQθ−VψQ\_\{\\theta\}\-V\_\{\\psi\}\. Downstream rank\-16 LoRA policies are initialized afresh from the base model and trained by advantage\-weighted behavioral cloning with inverse temperature 10 and weights clipped at 100\. All methods use the same optimizer steps, selected\-token budget, and base checkpoint\.

CODS clips absolute step residuals at the burn\-in pool’s 95th percentile and scores traceggby

z​\(g\)=12​maxt∈g⁡δ~t\+12​mean⁡\(top20%⁡\{δ~t:t∈g\}\)\.z\(g\)=\\tfrac\{1\}\{2\}\\max\_\{t\\in g\}\\widetilde\{\\delta\}\_\{t\}\+\\tfrac\{1\}\{2\}\\operatorname\{mean\}\\\!\\left\(\\operatorname\{top\}\_\{20\\%\}\\\{\\widetilde\{\\delta\}\_\{t\}:t\\in g\\\}\\right\)\.\(12\)The 10% budget is measured in action tokens, not trace count\. At each of five rounds, the selector scans traces in score order and adds the highest\-ranked complete trace that keeps cumulative selected action tokens at or below the round budget; traces that do not fit are skipped, never split\. Thus every selected action retains its entire prefix and terminal outcome\. ReDOR uses whole\-trace actor–critic gradients, the one\-shot control freezes the burn\-in scores, diversity applies k\-center to mean trace embeddings, and Random samples whole traces under the same token cap\.

Evaluation uses greedy decoding with no sampling, at most 50 actions for ALFWorld and 512 new tokens for GSM8K\. ALFWorld success is averaged over all unseen evaluation tasks; GSM8K exact match uses the benchmark’s normalized final\-answer parser\. Each subset uses five selection seeds and three downstream seeds, while Pool uses ten downstream seeds\. CODS retains 95\.4% of Pool on ALFWorld and 96\.5% on GSM8K\. Unrestricted step selection, which discards prefix closure but keeps the same token budget, scores67\.1±3\.567\.1\\pm 3\.5and50\.2±0\.950\.2\\pm 0\.9, respectively\. The paired losses relative to whole\-trace CODS are−5\.8\-5\.8points \(95% CI\[−7\.9,−3\.7\]\[\-7\.9,\-3\.7\]\) and−2\.7\-2\.7\(\[−3\.6,−1\.8\]\[\-3\.6,\-1\.8\]\), showing that the D4RL transition unit cannot be transferred unchanged\.

Table 23:Sequence\-level extension\. Subset entries use five selection seeds and three downstream seeds; Pool uses ten downstream seeds\.