Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

# Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning
Source: [https://arxiv.org/html/2609.12014](https://arxiv.org/html/2609.12014)
Cody FlemingAffiliation:Iowa State UniversityAffiliation:Ames, IA, USAEmail:[\{aharoon,flemingc\}@iastate\.edu](mailto:)

###### Abstract

Safe offline reinforcement learning assumes a cost function on every transition\. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget\. Certified safety curation answers with a filter\-then\-clone pipeline: a state\-only value trained from segment comparisons scores whole trajectories, Learn\-then\-Test calibration certifies a selection threshold under a distribution\-free\(α,δ\)\(\\alpha,\\delta\)bound on the unsafe fraction of the selection, and behavior cloning follows\. We are not aware of prior work certifying the composition of a training set for offline RL or imitation\. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories\. The policies satisfy the cost budget on eleven of fifteen DSRL tasks, one short of cloning the ground\-truth safe subset, which needs a label on every trajectory; the uncertified variant reaches twelve\. Retrained on the certified selection, the strongest full\-label method becomes safe where no setting of its own cost target rescues it\. Refusal is predictable: the certificate’s probability has a closed form in the purity the pool attains, which the calibration sample estimates and the scorer enters only through\.

## 1Introduction

Offline reinforcement learning promises policy improvement from logged data alone, and its safety\-constrained variant additionally requires the learned policy to keep expected cost below a budget\([Liu et al\., 2023a](https://arxiv.org/html/2609.12014#bib.bib4)\)\. Existing safe offline RL methods require a cost function evaluated on every transition of the dataset\([Liu et al\., 2023b](https://arxiv.org/html/2609.12014#bib.bib20);[Zheng et al\., 2024](https://arxiv.org/html/2609.12014#bib.bib19);[Gong et al\., 2025](https://arxiv.org/html/2609.12014#bib.bib5)\)\. On benchmarks it is given; the difficulty is upstream, in writing that function down\. Specifying a cost that faithfully captures what one means by unsafe is the problem reward engineering has proven to be, and it is why preference\-based learning exists\([Christiano et al\., 2017](https://arxiv.org/html/2609.12014#bib.bib24)\)\. Where no trustworthy cost function can be specified, a practitioner may still be able to say which of two short clips looks safer, and to judge occasionally whether an episode went over budget\. This paper asks what safe offline RL can still deliver from those two weaker signals alone\. Recent work has begun importing preferences into safe offline RL\([Gong et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib2);[Burnwal et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib15)\)\. None of it, however, carries a statistical guarantee relating supervision to the safety of what is learned\.

Figure 1:Certified safety curation\. A state\-only value trained from segment comparisons, which carry an ordering and no magnitudes, scores whole trajectories\. A threshold cuts the score axis and200200episodic budget labels audit what survives, certifying the selection at\(α,δ\)\(\\alpha,\\delta\)or refusing and returning a conservative selection reported as uncertified; either way the selection is cloned, and a certified selection may first be curated for reward \(Section[4\.4](https://arxiv.org/html/2609.12014#S4.SS4)\)\. Paths and densities are illustrative, not measured\.The question we study is whether a*state\-only*safety value, one that never sees actions and is trained only from comparisons, can identify safe training data well enough to be certified\. Our answer is certified safety curation \(Figure[1](https://arxiv.org/html/2609.12014#S1.F1)\), a filter\-then\-clone method whose central deliverable is a statistical guarantee on the data it trains on: the value scores whole trajectories, a threshold selects, and the threshold is chosen by Learn\-then\-Test calibration\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.12014#bib.bib11)\)against a small labeled sample, yielding a distribution\-free guarantee\. With probability at least1−δ1\-\\deltaover the calibration draw, the unsafe\-trajectory fraction of the selected training set is at mostα\\alpha\. The guarantee consumes 200 budget\-exceedance judgments, one bit per sampled trajectory, where competing methods require a cost value on every transition of every trajectory\.

An empirical analysis of independent interest grounds the design: the natural alternative, advantage\-weighted regression\([Peng et al\., 2019](https://arxiv.org/html/2609.12014#bib.bib6)\), fails for two compounding reasons, and both point the same way, to using the value where preference training identifies it and where selection keeps behaviors intact, the scale of whole trajectories\.

Our contributions are a granularity analysis with oracle controls establishing that the readout scale, not the supervision, is the binding choice, including that the intermediate segment scale fails with full labels \(BC\-Safe\-Seg is safe on two of fifteen tasks against twelve at trajectory scale, Section[5](https://arxiv.org/html/2609.12014#S5)\); certified curation, Learn\-then\-Test calibration of the selection threshold against 200 budget\-exceedance labels, with an\(α,δ\)\(\\alpha,\\delta\)guarantee on training\-set composition that refuses exactly where the score cannot concentrate safety belowα\\alpha\(Section[4\.3](https://arxiv.org/html/2609.12014#S4.SS3)\); and a measured account of what the supervision buys: CDT violates on eight Goal and Button tasks at every cost target down to a fifth of the budget yet is safe on the certified selection, while a classifier trained on the calibration labels alone matches the calibrated filter’s safe\-task count and replacing its bits with numeric costs does not improve it, so what separates methods is certification rather than scorer accuracy or supervision richness \(Section[6](https://arxiv.org/html/2609.12014#S6)\)\.

## 2Related Work

##### Filtered and weighted behavior cloning\.

Selecting or reweighting data before cloning is well established for reward: percentile behavior cloning ranks trajectories by return\([Chen et al\., 2021](https://arxiv.org/html/2609.12014#bib.bib7);[Emmons et al\., 2021](https://arxiv.org/html/2609.12014#bib.bib8)\), BAIL selects transitions near a return envelope\([Chen et al\., 2020](https://arxiv.org/html/2609.12014#bib.bib9)\), CRR and AWR weight transitions by learned advantages\([Wang et al\., 2020](https://arxiv.org/html/2609.12014#bib.bib10);[Peng et al\., 2019](https://arxiv.org/html/2609.12014#bib.bib6)\), and concurrent work scores demonstrations with learned feedback models before cloning\([Liang et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib17);[Beck, 2025](https://arxiv.org/html/2609.12014#bib.bib16)\)\. This line targets task success; none addresses a safety constraint, and every threshold in it is hand\-tuned\.

##### Safe offline RL\.

Methods with full cost labels include constrained sequence modeling\([Liu et al\., 2023b](https://arxiv.org/html/2609.12014#bib.bib20)\), feasibility\-guided diffusion\([Zheng et al\., 2024](https://arxiv.org/html/2609.12014#bib.bib19)\), and trajectory classification\([Gong et al\., 2025](https://arxiv.org/html/2609.12014#bib.bib5)\)\. The DSRL benchmark’s BC\-Safe baseline clones trajectories under the cost budget, a label on every trajectory\([Liu et al\., 2023a](https://arxiv.org/html/2609.12014#bib.bib4)\)\. Preference and weak\-label variants are emerging: PReSa consumes binary segment safety labels inside a Lagrangian objective\([Gong et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib2)\); OSIL learns a per\-step cost model from dataset\-level preferences and applies it as a cloning penalty\([Burnwal et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib15)\); PREFINE fine\-tunes implicit reward and cost models from preferences for safety alignment\([Verma et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib3)\)\. None selects training data with a preference\-learned value, and none carries a statistical guarantee\.

##### Distribution\-free risk control\.

Learn\-then\-Test and risk\-controlling prediction sets calibrate model configurations with finite\-sample guarantees\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.12014#bib.bib11);[Bates et al\., 2021](https://arxiv.org/html/2609.12014#bib.bib12);[Angelopoulos et al\., 2024](https://arxiv.org/html/2609.12014#bib.bib13)\)\. In RL and control, published applications are at deployment time: conformal off\-policy intervals\([Taufiq et al\., 2022](https://arxiv.org/html/2609.12014#bib.bib21)\), runtime monitors\([Feldman et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib18)\), and high\-confidence policy improvement\([Thomas et al\., 2015](https://arxiv.org/html/2609.12014#bib.bib22);[Laroche et al\., 2019](https://arxiv.org/html/2609.12014#bib.bib23)\)\. The nearest statistical relative is conformal selection\([Jin and Candès, 2023](https://arxiv.org/html/2609.12014#bib.bib14)\), which screens candidate units under false discovery rate control; FDR is an expectation bound, weaker than the\(α,δ\)\(\\alpha,\\delta\)high\-probability bound we calibrate\. Our calibration is standard Learn\-then\-Test with an exact hypergeometric substitution for the finite pool; the contribution is the object it certifies, the composition of a training set, and the empirical study of what that certificate delivers\.

## 3Preliminaries

A constrained MDP\(𝒮,𝒜,P,r,c,b\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,c,b\)maximizes return subject toJc​\(π\)≤bJ\_\{c\}\(\\pi\)\\leq b; the offline setting provides a static dataset𝒟=\{τi\}i=1N\\mathcal\{D\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}\. We call a trajectory unsafe when its episodic cost exceeds the budget and writeu⁡\(τ\)∈\{0,1\}u\(\\tau\)\\in\\\{0,1\\\}for this indicator\. A constrained MDP suppliesccby definition and benchmarks apply it to every transition; our method never reads it, using two weaker signals instead: a preference dataset𝒟pref\\mathcal\{D\}\_\{\\mathrm\{pref\}\}of pairs of length\-HHsegments\(σ\+,σ−\)\(\\sigma^\{\+\},\\sigma^\{\-\}\), each labeled only with which is safer, an ordering rather than a magnitude; and, used only for calibration, a*cost label*, the binary indicatoru⁡\(τ\)u\(\\tau\), on each ofnntrajectories drawn uniformly without replacement from𝒟\\mathcal\{D\}, which makes the finite\-population test of Section[4\.3](https://arxiv.org/html/2609.12014#S4.SS3)exact\. We call whoever supplies these signals the*labeler*; in our experiments it is the benchmark’s own cost function, which lets us study the algorithm without confounding it with labeler error\. We report raw reward and episodic cost against each task’s budget; a policy is safe when its expected cost is within budget \(Appendix[G](https://arxiv.org/html/2609.12014#A7)\)\.

## 4Method: Certified Safety Curation

The method has four stages: value learning, trajectory scoring, calibrated selection, and cloning\. Figure placement and pseudocode are deferred to Appendix[A](https://arxiv.org/html/2609.12014#A1)\.

### 4\.1Safety value from segment preferences

We learn a state\-only valueVθ:𝒮→ℝV\_\{\\theta\}:\\mathcal\{S\}\\to\\mathbb\{R\}by maximum likelihood under a Bradley\-Terry model\([Bradley and Terry, 1952](https://arxiv.org/html/2609.12014#bib.bib25)\)over segment sums,

ℒ⁡\(θ\)=−𝔼\(σ\+,σ−\)∼𝒟pref​\[log⁡logistic⁡\(∑t=1HVθ​\(st\+\)−∑t=1HVθ​\(st−\)\)\],\\mathcal\{L\}\(\\theta\)=\-\\mathbb\{E\}\_\{\(\\sigma^\{\+\},\\sigma^\{\-\}\)\\sim\\mathcal\{D\}\_\{\\mathrm\{pref\}\}\}\\Big\[\\log\\mathrm\{logistic\}\\Big\(\\textstyle\\sum\_\{t=1\}^\{H\}V\_\{\\theta\}\(s\_\{t\}^\{\+\}\)\-\\sum\_\{t=1\}^\{H\}V\_\{\\theta\}\(s\_\{t\}^\{\-\}\)\\Big\)\\Big\],\(1\)wherelogistic⁡\(z\)=1/\(1\+e−z\)\\mathrm\{logistic\}\(z\)=1/\(1\+e^\{\-z\}\)\. We train an ensemble ofK=3K=3independently initialized networks and writeV¯\\bar\{V\}for their mean\. The ensemble is not what makes selection work, and single members match its filter precision to within0\.0060\.006\(Appendix[M](https://arxiv.org/html/2609.12014#A13)\); we keep it because member disagreement is the diagnostic Section[5](https://arxiv.org/html/2609.12014#S5)uses to locate where the value is and is not identified\. Equation[1](https://arxiv.org/html/2609.12014#S4.E1)supervises only segment sums: any perturbation whose sums agree across every preference pair is invisible to the loss, so per\-state values are underdetermined and aggregation is the value’s reliable regime, which Section[5](https://arxiv.org/html/2609.12014#S5)quantifies\.

### 4\.2Trajectory scoring and selection

Each trajectory receives the scoreg⁡\(τ\)=1\|τ\|​∑st∈τV¯​\(st\)g\(\\tau\)=\\frac\{1\}\{\|\\tau\|\}\\sum\_\{s\_\{t\}\\in\\tau\}\\bar\{V\}\(s\_\{t\}\), and the selection at thresholdλ∗\\lambda^\{\*\}isS⁡\(λ∗\)=\{τi:g⁡\(τi\)≥λ∗\}S\(\\lambda^\{\*\}\)=\\\{\\tau\_\{i\}:g\(\\tau\_\{i\}\)\\geq\\lambda^\{\*\}\\\}\. The policy is obtained by behavior cloning onS⁡\(λ∗\)S\(\\lambda^\{\*\}\)\. Averaging over hundreds of states cancels the per\-state noise that Equation[1](https://arxiv.org/html/2609.12014#S4.E1)leaves unconstrained, and selection at the trajectory level preserves whole multi\-step avoidance behaviors that per\-transition reweighting fragments\.

### 4\.3Calibrated threshold with a composition guarantee

The threshold is the method’s only free parameter, and we select it by Learn\-then\-Test \(LTT\)\([Angelopoulos et al\., 2025](https://arxiv.org/html/2609.12014#bib.bib11)\)\. Let𝒞\\mathcal\{C\}be a calibration sample ofnntrajectories drawn uniformly without replacement from𝒟\\mathcal\{D\}, each carrying its binary labelu⁡\(τ\)u\(\\tau\)\. For a decreasing grid of candidate thresholdsλ1\>λ2\>⋯\\lambda\_\{1\}\>\\lambda\_\{2\}\>\\cdots\(score quantiles from0\.850\.85down to0\.300\.30\), letNj=\|S⁡\(λj\)\|N\_\{j\}=\|S\(\\lambda\_\{j\}\)\|be the selection size, and letmjm\_\{j\}andkjk\_\{j\}be the number of calibration trajectories aboveλj\\lambda\_\{j\}and the number of those that are unsafe\. Because the pool is finite and the selection sizes are known, the null hypothesis that the selection contains more thanα​Nj\\alpha N\_\{j\}unsafe trajectories admits an exact test: under the least\-favorable null the unsafe calibration count is hypergeometric, andpj=Fhyp​\(kj,Nj,⌊α​Nj⌋\+1,mj\)p\_\{j\}=F\_\{\\mathrm\{hyp\}\}\\big\(k\_\{j\};\\,N\_\{j\},\\,\\lfloor\\alpha N\_\{j\}\\rfloor\+1,\\,m\_\{j\}\\big\)is a super\-uniform p\-value, whereFhypF\_\{\\mathrm\{hyp\}\}is the hypergeometric CDF\. Strict fixed\-sequence testing tests the hypotheses in order, each at levelδ\\delta, stops at the first failure to reject, and returns the last rejected threshold; if the first test fails, no threshold is certified\.

###### Proposition 1\.

The returned thresholdλ^\\hat\{\\lambda\}satisfies

Pr\[a threshold is certified∧unsafe^\(S\(λ^\)\)\>α\]≤δ,\\Pr\\Big\[\\,\\textnormal\{a threshold is certified\}\\ \\wedge\\ \\widehat\{\\mathrm\{unsafe\}\}\\big\(S\(\\hat\{\\lambda\}\)\\big\)\>\\alpha\\,\\Big\]\\;\\leq\\;\\delta,\(2\)whereunsafe^​\(S\)=1\|S\|​∑τ∈Su⁡\(τ\)\\widehat\{\\mathrm\{unsafe\}\}\(S\)=\\frac\{1\}\{\|S\|\}\\sum\_\{\\tau\\in S\}u\(\\tau\)is the unsafe fraction for the dataset𝒟\\mathcal\{D\}itself and the probability is over the uniform calibration draw\. A proof is given in Appendix[B](https://arxiv.org/html/2609.12014#A2)\.

Validity needs no assumption that the unsafe fraction decreases as the threshold rises: testing in a fixed order and stopping at the first failure means a true null is never tested after a rejection, and monotonicity affects only*power*\. Both the grid and the score are fixed before the calibration sample is drawn\. The guarantee is transductive over the finite pool, holds per calibration run with no joint claim across tasks or seeds, and the score’s dependence on that same pool is treated in Appendix[B](https://arxiv.org/html/2609.12014#A2)\. It is also unconditional, bounding the joint probability of certifying and violating rather than the violation probability given a certificate, which can exceedδ\\deltawhere certification is rare \(Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)reports both\)\. The grid starts at the0\.850\.85quantile because deeper thresholds are statistically uncertifiable at practicalnnand sit in the score tail the Bradley\-Terry null space leaves unidentified \(Section[5](https://arxiv.org/html/2609.12014#S5)\); when no threshold certifies, the procedure returns an explicitly uncertified conservative selection, keeping the top fraction of trajectories given by a one\-sided Clopper\-Pearson lower bound on the pool safe mass computed from the calibration labels already spent\.

### 4\.4Certification\-gated reward\-aware curation

A safety\-only filter discards the dataset’s reward information, so within a selection we optionally retain the top half by episodic return before cloning\. Because return and cost correlate on contaminated selections, this is safe exactly when the selection is clean, so we gate it on the certificate, which serves double duty as a guarantee on composition and a decision signal for how aggressively to curate\. The hard rule is the extreme point of a family we also instantiate softly:*certified return\-weighted cloning*resamples the certified selection with probability proportional toexp⁡\(clip⁡\(z⁡\(Ri\),±κ\)\)\\exp\(\\mathrm\{clip\}\(z\(R\_\{i\}\),\\pm\\kappa\)\), wherez⁡\(Ri\)z\(R\_\{i\}\)standardizes episodic return within the selection and the clipκ\\kappasets the return\-seeking intensity\. This is the operator the granularity analysis prescribes: the exponentiated weighting of advantage\-weighted regression, moved to the trajectory scale where selection composes and run only behind the certificate\. Its safety on certified selections is robust toκ\\kappa, and beyond the certified composition it is not \(Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2), Appendix[J](https://arxiv.org/html/2609.12014#A10)\)\.

## 5Why Trajectory Scale: an Oracle\-Controlled Analysis

The algorithm of Appendix[A](https://arxiv.org/html/2609.12014#A1)clones a*selection*: the value picks trajectories and behavior cloning does the rest\. The standard alternative reweights individual transitions instead, and everything analyzed in this section is that alternative, not a component of our method\. It is advantage\-weighted regression\([Peng et al\., 2019](https://arxiv.org/html/2609.12014#bib.bib6)\): form a per\-transition advantage from the value, weight each transition byexp⁡\(A/β\)\\exp\(A/\\beta\), and fit a policy to the reweighted data\. The value is a scoring signal in both cases; only the granularity of its use differs\. The analysis uses nine DSRL tasks\([Liu et al\., 2023a](https://arxiv.org/html/2609.12014#bib.bib4)\), the five velocity\-constrained locomotion tasks and the four Goal\-family navigation tasks, called the*analysis tasks*throughout; Section[6](https://arxiv.org/html/2609.12014#S6)evaluates the full fifteen\-task suite\. Labels are exact, isolating algorithmic questions from labeler noise, and segment\-level labeling makes the value’s supervision genuinely sub\-trajectory \(Appendix[M](https://arxiv.org/html/2609.12014#A13)\)\.

The analysis separates two obstacles\. The first concerns how the value is used and holds even when the value is exact; the second concerns what preference training leaves undetermined in the value itself\. Throughout,*per\-transition*use means weighting each state\-action pair by its own advantage, while*trajectory\-scale*use means aggregating the value over a whole trajectory and selecting on that aggregate\.

##### Obstacle 1: reweighting transitions does not produce trajectory\-level safety, even with exact values\.

We substitute two ground\-truth values into the identical advantage\-weighted extraction on all nine tasks: the per\-step oracleV=−cV=\-c, the exact generator of the preferences, and the discounted cost\-to\-go oracle\. Neither improves on the learned value in any cell but one \(Table[6](https://arxiv.org/html/2609.12014#A4.T6)\), and on all five velocity tasks the learned value is safe while at least one oracle violates, reaching cost235235and169169against budget2020on HalfCheetah and Ant\. The learned value’s generalization, spreading sparse cost events onto neighboring states, is therefore essential signal rather than approximation error; widening the advantage window degrades safety monotonically, and the same reweighting destroys BC\-Safe’s safety \(Appendix[D](https://arxiv.org/html/2609.12014#A4)\)\. The mechanism is compositional: exponentiated per\-step weights reweight actions the dataset already contains at each state, while safety is carried by whole multi\-step avoidance behaviors that reweighting fragments\. Value quality is not the limit, the granularity of its use is, and Appendix[B](https://arxiv.org/html/2609.12014#A2)proves a case where the failure is unavoidable \(Proposition[3](https://arxiv.org/html/2609.12014#Thmproposition3)\)\. The obstacle is scoped rather than absolute: on PointGoal1 five of the eleven per\-transition configurations we tested reach the budget, at reward above our clone’s \(13\.713\.7against7\.57\.5\) but with far less margin \(22\.122\.1against8\.88\.8, budget2525\), while on CarGoal2 and PointGoal2 none of nine does, the closest at cost26\.926\.9and the rest between2929and7272\(Appendices[D](https://arxiv.org/html/2609.12014#A4)and[E](https://arxiv.org/html/2609.12014#A5)\)\.

##### Obstacle 2: preference training additionally underdetermines the learned value at per\-transition scale\.

The learned value carries a second burden the oracles do not, which Table[5](https://arxiv.org/html/2609.12014#A4.T5)quantifies at both scales\. Trajectory\-level ranking is reliable everywhere, per\-transition agreement is not: cross\-seed advantage rank correlation and top\-one\-percent overlap vary widely across tasks, so the tail exponential weighting concentrates on is seed\-specific, consistent with the Bradley\-Terry null space\. A matched control with dense per\-step supervision disagrees comparably, so the per\-transition scale is unreliable under either supervision, and neither held\-out accuracy nor per\-transition reliability predicts the policy; the two can even invert \(Appendix[D](https://arxiv.org/html/2609.12014#A4)\)\. What limits the per\-transition route is the operator, not the value it is given\.

###### Observation 1\(Granularity principle\)\.

Per\-transition extraction fails twice over\. Reweighting transitions does not produce trajectory\-level safety even under exact values \(Obstacle 1\), and preference\-trained values are unidentified at per\-transition scale while reliable at trajectory scale \(Obstacle 2\)\. Trajectory\-scale selection sidesteps both\. It uses the value where preference training identifies it, and it keeps whole behaviors intact rather than reweighting their parts\.

The positive half of the principle has a direct witness\. Behavior cloning on the ground\-truth safe subset \(BC\-Safe\) satisfies the budget on eight of nine tasks, including PointGoal1, which no per\-transition operator moved\. Our method replaces BC\-Safe’s full labeling with the preference\-learned score and adds the certificate\.

##### What the value encodes\.

Inspection corroborates the principle: saliency recovers the cost\-defining observation dimension on four of five velocity tasks,V¯\\bar\{V\}declines as violations approach on every task, explaining its advantage over the exact but sparse per\-step oracle, and on navigationV¯\\bar\{V\}binned over the workspace recovers a hazard field it was never given; the diagnostics flatten precisely where the method trails the strongest baseline \(Figure[4](https://arxiv.org/html/2609.12014#A6.F4), Appendix[F](https://arxiv.org/html/2609.12014#A6)\)\.

## 6Experiments

### 6\.1Setup

We evaluate on fifteen DSRL tasks\([Liu et al\., 2023a](https://arxiv.org/html/2609.12014#bib.bib4)\): five velocity\-constrained locomotion tasks and ten SafetyGymnasium navigation tasks spanning the Goal, Button, and Circle families for the Point and Car robots, the union of our comparators’ task sets at their budgets \(20 velocity, 25 navigation\)\. Datasets are the standard DSRL releases; all methods share architectures, the DSRL evaluation protocol, and identical preference data \(Appendix[M](https://arxiv.org/html/2609.12014#A13)\)\. We compare against BC\-All, the full\-label filters BC\-Safe and BC\-Safe\-Seg\([Liu et al\., 2023a](https://arxiv.org/html/2609.12014#bib.bib4);[Gong et al\., 2026](https://arxiv.org/html/2609.12014#bib.bib2)\), random\-, return\-, and bottom\-return\-selection controls at matched selection size \(the pool’s ground\-truth safe count, so selection signal is compared at equal budget\), the full\-label methods CDT\([Liu et al\., 2023b](https://arxiv.org/html/2609.12014#bib.bib20)\), CPQ\([Xu et al\., 2022](https://arxiv.org/html/2609.12014#bib.bib26)\)and COptiDICE\([Lee et al\., 2022](https://arxiv.org/html/2609.12014#bib.bib27)\)trained in\-codebase at matched cost limits \(Appendix[G](https://arxiv.org/html/2609.12014#A7)\), and CPL\([Hejna et al\., 2024](https://arxiv.org/html/2609.12014#bib.bib1)\)consuming exactly our preference data\. Our own configurations are the V\-filter at the calibration\-estimated fraction \(ground\-truth\-matched and fixed top\-quartile fractions retained as diagnostics\), the calibrated filter atα=0\.25\\alpha=0\.25,δ=0\.1\\delta=0\.1,n=200n=200, its certification\-gated reward\-aware variant, and certified return\-weighted cloning\. Headline configurations and controls report mean and 95 percent bootstrap CIs over five seeds; analysis sweeps use three\. A five\-task BulletSafetyGym suite, held out as a transfer test with every hyperparameter frozen, is reported in Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)and Appendix[L](https://arxiv.org/html/2609.12014#A12)\.

### 6\.2Main results

Table 1:Raw reward and cost on fifteen DSRL tasks \(five seeds, mean±\\pm95% CI; budgets 20/25; bold cost within budget; BC\-All is a single training with 100\-episode evaluation; V\-filter is the uncalibrated filter at the calibration\-estimated fraction; Calibrated is the full procedure of Section[4\.3](https://arxiv.org/html/2609.12014#S4.SS3), certifying where it can and otherwise returning the conservative selection with an explicit refusal\)\.†: on the three certified tasks the calibrated column applies the certification\-gated sub\-selection; elsewhere the two coincide\.We judge methods by the constrained objective itself\. A method earns credit first for satisfying the cost budget, and only then for the reward it achieves while doing so\. Bold cells in every table mark budget satisfaction\. Table[1](https://arxiv.org/html/2609.12014#S6.T1)presents the main comparison\. Thresholds and budgets are stated in raw cost throughout, so raw units are primary here; Table[10](https://arxiv.org/html/2609.12014#A7.T10)restates cost across every method in normalized units for comparison with the literature\.

##### The calibrated filter, the self\-contained deliverable\.

One fixed procedure across tasks, consuming preferences plus 200 calibration labels, is safe on eleven of fifteen, a count owned by the full procedure including its uncertified fallback\. The certificate itself fires on three tasks atα=0\.25\\alpha=0\.25and on eight of the nine analysis tasks atα=0\.40\\alpha=0\.40\(Section[6\.4](https://arxiv.org/html/2609.12014#S6.SS4)\), and on uncertified tasks the deliverable is the conservative selection with an explicit no\-certificate report\. Its failures are Hopper \(marginal\); Swimmer, where only the estimated\-fraction V\-filter is marginally within budget \(20 against BC\-Safe’s 72\) while the calibrated fallback lands at 45; and PointButton2 and PointCircle2, which have no selectable safe subset, where every method fails and the calibration refuses to certify\.

##### The value against ground truth\.

The V\-filter at the calibration\-estimated fraction, which reads no ground\-truth quantity, is safe on twelve of fifteen tasks; BC\-Safe, which needs a budget label on every trajectory, is also safe on twelve\. Their failure sets differ rather than nest: ours violates on Hopper, which BC\-Safe handles; BC\-Safe violates on Swimmer at more than three times the budget, which ours handles; both violate on PointButton2 and PointCircle2\. The two also sit differently inside the budget: on the eleven where both are safe ours runs at lower mean cost on eight, mean normalized cost0\.480\.48against BC\-Safe’s0\.540\.54\(Appendix[G](https://arxiv.org/html/2609.12014#A7)\)\. Cells near the boundary are counted descriptively, without a test\. The mechanism is graded versus binary signal: the value ranks trajectories by distance from the budget, while the label admits everything up to it, and the clone inherits that boundary\.

##### The certificate as a gate\.

The gated reward\-aware variant is certification\-safe\. On the three certified tasks it keeps cost within budget in every cell with reward held within confidence intervals, and uncertified tasks keep their conservative selections; ungated, the same sub\-selection on Hopper’s contaminated selection multiplies mean cost to 148, and the gate refuses there \(per\-cell detail in Appendix[H](https://arxiv.org/html/2609.12014#A8)\)\. Since sub\-selection discards trajectories but never adds them, reward rises only where the certified selection is reward\-rich; the return\-weighted operator below is the stronger form of the same idea\. The contribution here is the certificate and the weaker supervision it needs, not raw policy performance\.

##### Transfer\.

The unchanged pipeline transfers to five BulletSafetyGym tasks with new embodiments, episode lengths of 100 to 300, and a harder budget, satisfying it on all five with the guarantee holding in every validation cell \(Appendix[L](https://arxiv.org/html/2609.12014#A12)\)\. Across both suites the certificate fires on 4 of 20 tasks atα=0\.25\\alpha=0\.25; yield is governed by the purity margin of Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3), not raw safe mass: Swimmer has less safe mass than Ant yet certifies far more often because its score separates the safe trajectories better\.

##### External baselines\.

Against full\-label offline safe RL and direct preference\-based policy learning \(Table[11](https://arxiv.org/html/2609.12014#A7.T11), Figure[6](https://arxiv.org/html/2609.12014#A7.F6), Appendix[G](https://arxiv.org/html/2609.12014#A7)\), CDT is safe on all five velocity tasks and both Circle tasks with reward above ours wherever it is safe, but violates at*every*cost target on the eight Goal and Button tasks, its realized cost barely responding to the knob, so the lever is inoperative where selectivity is needed\. Supervision does not substitute for readout scale: these three read a cost on every transition and keep seven, four and two of fifteen within budget, while a filter consuming200200bits keeps eleven \(Appendix[K](https://arxiv.org/html/2609.12014#A11)\)\. CPQ and COptiDICE are safe on four and two of fifteen, several cells degenerately, and CPL, consuming exactly our preference data, on four: the same supervision that curates safely fails as a direct policy\-optimization signal\. Every filter\-then\-clone method in our study is safer than every optimization\-based method, the granularity principle at benchmark scale\.

##### Handing the selection to another learner\.

Because the certificate describes the data rather than the policy, the certified selection can be handed to any offline learner\. Retraining CDT on the certified selections of the three tasks that certify atα=0\.25\\alpha=0\.25makes every run safe, 27 of 27 across three seeds and three distinct selections per task, three of them out of specification \(Tables[18](https://arxiv.org/html/2609.12014#A10.T18)and[19](https://arxiv.org/html/2609.12014#A10.T19)\): on both navigation tasks, where full\-data CDT violates by nearly 50 percent, it runs at cost 5 to 11, and on HalfCheetah it retains CDT’s full reward \(2694 against our clone’s 2102\) at cost 2\. On navigation CDT trails our own clone, so the certified data costs it reward there\. Only three tasks appear because only three certify, which is refusal rather than a choice of favorable cases\. Attribution is controlled twice: full\-data CDT without augmentation remains severely unsafe on navigation \(costs 52 and 47\), and no cost target down to a fifth of the budget rescues it \(Appendix[K](https://arxiv.org/html/2609.12014#A11)\), so only curation moves it inside budget\. The same holds for CPL, a weaker learner: it is within budget on all 27 runs, and on PointGoal1, where full\-data CPL violates on every seed at cost 40\.7, the certified selection brings it to 9\.8, so curation is not tuned to one consumer\. At the looserα=0\.40\\alpha=0\.40the three diverge sharply: our clone stays safe on 22 of 24 task\-selections, CPL on 16, and CDT violates on every navigation task on at least one \(Appendix[J](https://arxiv.org/html/2609.12014#A10)\)\. Averaging learners outvote the unsafe tail while return\-conditioned learners seek exactly it, so composing with them requires the tighter certificate, givingα=0\.25\\alpha=0\.25an empirical justification beyond conservatism\. Certified curation is thus a supervision\-agnostic front end for stronger learners wherever their violations are data\-driven; where the operator itself violates on a benign pool it cannot help, a boundary one transfer task exposes \(Appendix[L](https://arxiv.org/html/2609.12014#A12)\)\.

##### A certificate\-native return\-seeking consumer\.

The certificate’s value depends on its consumer, so we derive one from the pipeline itself: certified return\-weighted cloning \(Section[4\.4](https://arxiv.org/html/2609.12014#S4.SS4)\), run only on certified selections\. Across the three certifying tasks’ nine certified selections, the CarRun echo and four intensities, 27 of the 28 within\-specification cells are safe, and the exception, CarRun’s hard top\-half rule on one seed, shows that resampling rather than return\-seeking is what carries safety\. The operator converts certification into reward: on HalfCheetah it lifts the clone’s 2102 to 2361 on the deployed selection and up to 2516 across certified selections at cost below 1, recovering up to two thirds of the gap to CDT retrained on the same data \(2694\), and on CarRun it is safe where full\-data CDT violates on all seeds\. Beyond the certified composition the same knob is not safe, the worst tail selection surviving only its gentlest intensity, so the certificate is what decouples the operator’s safety from its aggressiveness \(Table[23](https://arxiv.org/html/2609.12014#A10.T23); detail in Appendices[J](https://arxiv.org/html/2609.12014#A10)and[L](https://arxiv.org/html/2609.12014#A12)\)\.

### 6\.3Guarantee validation

If safety can be decoded from state, when can the decoding be trusted? We validate Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)on the nine analysis tasks across 2000 fresh calibration draws per task, seed, and sizen∈\{50,100,200,400\}n\\in\\\{50,100,200,400\\\}\(Figure[7](https://arxiv.org/html/2609.12014#A8.F7), Appendix[H](https://arxiv.org/html/2609.12014#A8)\)\. The unconditional false\-certification rate, the probability of certifying a threshold whose selection is out of specification, stays belowδ=0\.1\\delta=0\.1in every cell, the worst being0\.0700\.070\(PointGoal2, seed 1,n=400n=400; Clopper\-Pearson0\.0600\.060to0\.0830\.083\), and a maximum over cells is itself upward biased\. Certification rates increase withnnand with achievable purity: PointGoal1 certifies in about two thirds of draws atn=200n=200and on more than nine tenths byn=400n=400, while Hopper, Ant and PointGoal2 certify never or almost never, correctly, since their top\-quantile selections genuinely exceedα\\alpha\. Refusal is the guarantee working, and it is predictable: define a pool’s*purity margin*asα\\alphaminus the smallest unsafe fraction any threshold on the grid can achieve, that is, how far the purest attainable selection sits below the risk target\. Certification rate tracks this margin almost exactly across the fourteen validated tasks \(Spearman0\.990\.99\), and again when segment length rather than task is varied \(Spearman0\.9880\.988over 27 cells, Appendix[H](https://arxiv.org/html/2609.12014#A8)\)\. The relationship is a threshold rather than a trend: every task with a negative margin certifies essentially never, while positive margins yield rates rising with it, and varying the margin continuously on constructed pools traces the same threshold at zero \(Figure[3](https://arxiv.org/html/2609.12014#A2.F3)\)\.

###### Proposition 2\(Certification dichotomy\)\.

Writeuj=Uj/Nju\_\{j\}=U\_\{j\}/N\_\{j\}for the unsafe fraction of the selection at grid thresholdλj\\lambda\_\{j\}, and letumin=minj⁡uju\_\{\\min\}=\\min\_\{j\}u\_\{j\}\.

1. 1\.Ifumin\>αu\_\{\\min\}\>\\alpha, thenPr⁡\[certify\]≤δ\\Pr\[\\textnormal\{certify\}\]\\leq\\deltafor every calibration sizenn\.
2. 2\.Ifu1<αu\_\{1\}<\\alpha, thenPr⁡\[certify\]→1\\Pr\[\\textnormal\{certify\}\]\\to 1asn→Nn\\to N\.

Proof in Appendix[B](https://arxiv.org/html/2609.12014#A2)\.

Part \(i\) is the formal content of refusal: on such a pool no threshold can purify belowα\\alpha, and no labeling budget changes that\. Part \(ii\) makes the label cost forecastable, which Proposition[4](https://arxiv.org/html/2609.12014#Thmproposition4)turns into an exact budget, matching the resampled rates within0\.0160\.016on average\. The same resample yields the deployer\-facing conditional ratePr⁡\[violate∣certified\]\\Pr\[\\text\{violate\}\\mid\\text\{certified\}\], small where certification is common and large where it is rare \(Ant1\.001\.00at rate0\.010\.01\), so the certification rate is the trust signal to read alongside the certificate \(Table[14](https://arxiv.org/html/2609.12014#A8.T14)\)\.

Two boundaries are worth stating plainly\. Certification is stricter than policy safety requires: the binary indicator counts cost 26 against budget 25 as fully unsafe, so selections can exceedα\\alphawhile their clones stay within budget; the gap between eleven safe tasks and three certifying is this conservatism, priced and reported\. Composition is also not sufficient: BC\-Safe’s training set has unsafe fraction exactly zero, an implicit certificate atα=0\\alpha=0, yet its clone violates on three of twenty tasks, from the label’s boundary near budget and from cloning error in the large ones \(Swimmer at 72\), separated in Appendix[H](https://arxiv.org/html/2609.12014#A8)\. The certificate audits the data; the favorable mapping to policy behavior is carried by margin\-ranked selection, and we report the two separately\. Because that mapping is empirical, we also certify the deployed policy from the same one bit per episode: Clopper\-Pearson putsPr\[cost\>budget\]\\Pr\[\\text\{cost\}\>\\text\{budget\}\]at0\.2560\.256for the calibrated selection against0\.6840\.684for cloning everything and0\.2460\.246for the full\-label oracle, and an empirical\-Bernstein bound on𝔼⁡\[cost\]\\mathbb\{E\}\[\\text\{cost\}\]certifies nine of fifteen tasks on every seed, the shortfall against eleven safe tasks being the price of a finite\-sample bound\. Both need rollouts, so they complement the offline certificate rather than extend it \(Appendix[C](https://arxiv.org/html/2609.12014#A3)\)\.

### 6\.4Ablations

##### Selection signal\.

Matched\-size random\- and return\-selection controls isolate the safety value’s contribution \(Table[13](https://arxiv.org/html/2609.12014#A8.T13)\)\. Return selection, the percentile\-BC analogue, is unsafe on all fifteen tasks and on several actively concentrates cost; random selection is unsafe on fourteen\. Bottom\-return selection, the strongest form of the concern that the score merely anti\-ranks return, is safe on eight of fifteen yet catastrophically unsafe on four of five velocity tasks \(costs 48 to 189 against budget 20\) even though its selected trajectories are low\-cost\. Score correlations complete the account: the score tracks cost on every task \(Spearman−0\.55\-0\.55to−0\.93\-0\.93\), is at most weakly aligned with return on velocity, and anti\-correlates with return only on navigation, where return and cost are themselves positively correlated, as any safety\-tracking score there must \(Table[12](https://arxiv.org/html/2609.12014#A8.T12)\)\.

##### Calibration size and guarantee level\.

Certification rate falls gracefully withnn: atn=50n=50the procedure certifies rarely and the guarantee still holds, so a small budget costs conservatism, never silent failure\. Sweepingα\\alphamaps the operating curve:0\.050\.05certifies nowhere,0\.100\.10only on PointGoal1,0\.250\.25is the practical operating point, and0\.400\.40certifies broadly except on Hopper, which never certifies at any level, correctly\. Certification grows withnnthroughout, so tighter guarantees are a matter of labels and tighterα\\alphayields safer, lower\-reward policies \(Figures[7](https://arxiv.org/html/2609.12014#A8.F7),[8](https://arxiv.org/html/2609.12014#A8.F8)and[10](https://arxiv.org/html/2609.12014#A11.F10); Appendices[H](https://arxiv.org/html/2609.12014#A8)and[K](https://arxiv.org/html/2609.12014#A11)\)\.

##### Labels\-only filtering\.

The calibration sample could train a filter instead of auditing one\. A supervised classifier on only thennlabels, with our architecture, scoring rule, and fractions, is safe on 11 to 13 of fifteen across budgetsn∈\{50,100,200,400\}n\\in\\\{50,100,200,400\\\}, and splitting the budget between training and calibration certifies at a rate comparable to ours\. At matched budget, preferences buy neither additional safety nor certification\. Both routes rest on the same binary judgment, whether an episode exceeded its budget; labels\-only spends those judgments on training, while the preference route scores from ordinal comparisons and holds all 200 back to audit\. What preferences buy is the audit, not the accuracy\. Safety here is decodable from state alone, so the scorer is largely interchangeable and the curation carries the result, as Proposition[4](https://arxiv.org/html/2609.12014#Thmproposition4)explains: the scorer enters the certification rate only through the purity the pool attains \(Appendices[H](https://arxiv.org/html/2609.12014#A8)and[I](https://arxiv.org/html/2609.12014#A9)\)\.

##### Further controls\.

Five controls in Appendix[H](https://arxiv.org/html/2609.12014#A8)bound what the calibration sample and the score contribute: held\-out preferences in place of labeled calibration are safe on four of fifteen; label noise, a Boltzmann labeler and a halved pair budget degrade precision mildly and never certify falsely; numeric costs in place of the200200bits buy nothing; the budget\-native risk is strictly more conservative; andQ⁡\(s,a\)Q\(s,a\)forfeits policy\-agnostic scoring\.

## 7Limitations

The certificate covers training\-set composition, not the resulting policy’s cost; the mapping between them is empirically favorable here, but a starved selection can underfit into an unsafe policy, mitigated but not eliminated by the fallback’s minimum size and addressed from the other side by the deployment\-time certificates of Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)\. The guarantee consumes 200 labeled trajectories, which the preference\-only variant removes at the cost of the certificate\. All preferences are synthesized from ground\-truth episodic costs, so the weak\-oracle regime is modeled, under label noise and a Boltzmann labeler, rather than entered with human labelers\. The extreme tail of the trajectory score is unreliable, which the capped grid works around; tail\-robust scoring remains open\. The return\-weighted operator must not be run on uncertified selections: on the worst observed out\-of\-specification selection its safety survives only at the gentlest intensity\. Three scoping notes close: a labels\-only classifier matches the comparison route’s safe\-task count, so that route’s advantage is the held\-out audit rather than coverage; the action\-conditioned ablation covers five tasks; and composability spans three tasks atα=0\.25\\alpha=0\.25and eight atα=0\.40\\alpha=0\.40, three selections each \(Appendix[J](https://arxiv.org/html/2609.12014#A10)\)\.

## 8Conclusion

We presented certified safety curation: a state\-only safety value learned from segment comparisons selects whole trajectories, Learn\-then\-Test calibrates the threshold, and the certified selection is cloned under a distribution\-free guarantee on its composition\. Safety in these domains is decodable from state alone at trajectory scale, from supervision far weaker than a cost function, and a labels\-only classifier matches that coverage; what separates methods is the certificate and its refusals\. The selection also transfers: the strongest full\-label baseline retrained on it becomes safe on navigation tasks it otherwise violates\. Offline the data bounds what any method can achieve, and certifying it is a guarantee available before a policy is trained or run\.

#### AI use statement

We used generative AI tools as a coding and editing assistant: for writing and refactoring experiment\-orchestration and analysis scripts, for regenerating tables and figures from archived records, and for copy\-editing prose\. We did not use generative AI to originate the research question, the method, the theoretical results, or their proofs, and no experimental result was produced or altered by an AI tool outside the logged pipeline described in the paper\. All AI\-assisted code was reviewed by the authors and its outputs cross\-checked against the archived evaluation records; every numerical claim in the text is verified against those records by the auditing scripts accompanying the paper\. We take responsibility for the final content of this work, including all text, claims, and artifacts\.

#### Ethics statement

This work studies safety constraint satisfaction in offline reinforcement learning using public simulated benchmarks; it involves no human subjects, no personal data, and no new data collection\. The preference labels are synthesized from benchmark cost functions rather than elicited from people, which we state as a limitation rather than a feature\. One dual\-use consideration is specific to this paper: the certificate bounds the composition of a training set, not the behavior of the resulting policy\. We report certification and policy outcomes separately throughout, and Section[7](https://arxiv.org/html/2609.12014#S7)states that boundary explicitly\.

#### Reproducibility statement

All experiments use public DSRL datasets\. Every table regenerates from archived evaluation records via the scripts accompanying the paper, and the guarantee figure regenerates from archived calibration statistics\. Hyperparameters are listed in Appendix[M](https://arxiv.org/html/2609.12014#A13); no per\-task tuning is used for the calibrated method\.

## References

- Angelopouloset al\.\(2024\)A\. Angelopoulos, S\. Bates, A\. Fisch, L\. Lei, and T\. SchusterConformal risk control\.InInternational conference on learning representations,Vol\.2024,pp\. 55198–55218\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Angelopouloset al\.\(2025\)A\. N\. Angelopoulos, S\. Bates, E\. J\. Candès, M\. I\. Jordan, and L\. LeiLearn then test: calibrating predictive algorithms to achieve risk control\.The Annals of Applied Statistics19,pp\. 1641–1662\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p2.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1),[§4\.3](https://arxiv.org/html/2609.12014#S4.SS3.p1.1)\.
- Bateset al\.\(2021\)S\. Bates, A\. Angelopoulos, L\. Lei, J\. Malik, and M\. JordanDistribution\-free, risk\-controlling prediction sets\.Journal of the ACM \(JACM\)68,pp\. 1–34\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Beck \(2025\)J\. BeckOffline rlaif: piloting vlm feedback for rl via sfo\.arXiv preprint arXiv:2503\.01062\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Bradley and Terry \(1952\)R\. A\. Bradley and M\. E\. TerryRank analysis of incomplete block designs: i\. the method of paired comparisons\.Biometrika39,pp\. 324–345\.Cited by:[§4\.1](https://arxiv.org/html/2609.12014#S4.SS1.p1.1)\.
- Burnwalet al\.\(2026\)R\. Burnwal, N\. P\. Bhatt, and B\. RavindranOSIL: learning offline safe imitation policies with safety inferred from non\-preferred trajectories\.arXiv preprint arXiv:2602\.11018\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1)\.
- Chenet al\.\(2021\)L\. Chen, K\. Lu, A\. Rajeswaran, K\. Lee, A\. Grover, M\. Laskin, P\. Abbeel, A\. Srinivas, and I\. MordatchDecision transformer: reinforcement learning via sequence modeling\.Advances in neural information processing systems34,pp\. 15084–15097\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Chenet al\.\(2020\)X\. Chen, Z\. Zhou, Z\. Wang, C\. Wang, Y\. Wu, and K\. RossBail: best\-action imitation learning for batch deep reinforcement learning\.Advances in Neural Information Processing Systems33,pp\. 18353–18363\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Christianoet al\.\(2017\)P\. F\. Christiano, J\. Leike, T\. Brown, M\. Martic, S\. Legg, and D\. AmodeiDeep reinforcement learning from human preferences\.Advances in neural information processing systems30\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1)\.
- Emmonset al\.\(2021\)S\. Emmons, B\. Eysenbach, I\. Kostrikov, and S\. LevineRvs: what is essential for offline rl via supervised learning?\.arXiv preprint arXiv:2112\.10751\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Feldmanet al\.\(2026\)A\. O\. Feldman, J\. A\. Vincent, M\. Adang, J\. Low, and M\. SchwagerLearning robot safety from sparse human feedback using conformal prediction\.IEEE Transactions on Robotics\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Gonget al\.\(2025\)Z\. Gong, A\. Kumar, and P\. VarakanthamOffline safe reinforcement learning using trajectory classification\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.39,pp\. 16880–16887\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1)\.
- Gonget al\.\(2026\)Z\. Gong, P\. Varakantham, and A\. KumarOffline safe policy optimization from heterogeneous feedback\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems,AAMAS ’26,pp\. 3573–3575\.Note:Extended abstractExternal Links:[Document](https://dx.doi.org/10.65109/XGRR7655)Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Hejnaet al\.\(2024\)J\. Hejna, R\. Rafailov, H\. Sikchi, C\. Finn, S\. Niekum, W\. B\. Knox, and D\. SadighContrastive preference learning: learning from human feedback without reinforcement learning\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 18770–18798\.Cited by:[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Jin and Candès \(2023\)Y\. Jin and E\. J\. CandèsSelection by prediction with conformal p\-values\.Journal of Machine Learning Research24,pp\. 1–41\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Larocheet al\.\(2019\)R\. Laroche, P\. Trichelair, and R\. T\. Des CombesSafe policy improvement with baseline bootstrapping\.InInternational conference on machine learning,pp\. 3652–3661\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Leeet al\.\(2022\)J\. Lee, C\. Paduraru, D\. J\. Mankowitz, N\. Heess, D\. Precup, K\. Kim, and A\. GuezCoptidice: offline constrained reinforcement learning via stationary distribution correction estimation\.arXiv preprint arXiv:2204\.08957\.Cited by:[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Lianget al\.\(2026\)A\. Liang, Y\. Korkmaz, J\. Zhang, M\. Hwang, A\. Anwar, S\. Kaushik, A\. Shah, A\. S\. Huang, L\. Zettlemoyer, D\. Fox,et al\.Robometer: scaling general\-purpose robotic reward models via trajectory comparisons\.arXiv preprint arXiv:2603\.02115\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Liuet al\.\(2023a\)Z\. Liu, Z\. Guo, H\. Lin, Y\. Yao, J\. Zhu, Z\. Cen, H\. Hu, W\. Yu, T\. Zhang, J\. Tan,et al\.Datasets and benchmarks for offline safe reinforcement learning\.arXiv preprint arXiv:2306\.09303\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1),[§5](https://arxiv.org/html/2609.12014#S5.p1.1),[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Liuet al\.\(2023b\)Z\. Liu, Z\. Guo, Y\. Yao, Z\. Cen, W\. Yu, T\. Zhang, and D\. ZhaoConstrained decision transformer for offline safe reinforcement learning\.InInternational conference on machine learning,pp\. 21611–21630\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1),[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Penget al\.\(2019\)X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. LevineAdvantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.arXiv preprint arXiv:1910\.00177\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p3.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2609.12014#S5.p1.1)\.
- Rashidinejadet al\.\(2021\)P\. Rashidinejad, B\. Zhu, C\. Ma, J\. Jiao, and S\. RussellBridging offline reinforcement learning and imitation learning: a tale of pessimism\.Advances in Neural Information Processing Systems34,pp\. 11702–11716\.Cited by:[Appendix C](https://arxiv.org/html/2609.12014#A3.p1.1)\.
- Taufiqet al\.\(2022\)M\. F\. Taufiq, J\. Ton, R\. Cornish, Y\. W\. Teh, and A\. DoucetConformal off\-policy prediction in contextual bandits\.Advances in Neural Information Processing Systems35,pp\. 31512–31524\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Thomaset al\.\(2015\)P\. Thomas, G\. Theocharous, and M\. GhavamzadehHigh confidence policy improvement\.InInternational Conference on Machine Learning,pp\. 2380–2388\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px3.p1.1)\.
- Vermaet al\.\(2026\)R\. Verma, B\. Kulur, S\. Chawla, and B\. RavindranPREFINE: preference\-based implicit reward and cost fine\-tuning for safety alignment\.InProceedings of the 25th International Conference on Autonomous Agents and Multiagent Systems,AAMAS ’26,pp\. 1004–1012\.External Links:[Document](https://dx.doi.org/10.65109/SDRB4374)Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1)\.
- Wanget al\.\(2020\)Z\. Wang, A\. Novikov, K\. Zolna, J\. S\. Merel, J\. T\. Springenberg, S\. E\. Reed, B\. Shahriari, N\. Siegel, C\. Gulcehre, N\. Heess,et al\.Critic regularized regression\.Advances in Neural Information Processing Systems33,pp\. 7768–7778\.Cited by:[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px1.p1.1)\.
- Xieet al\.\(2021\)T\. Xie, N\. Jiang, H\. Wang, C\. Xiong, and Y\. BaiPolicy finetuning: bridging sample\-efficient offline and online reinforcement learning\.Advances in neural information processing systems34,pp\. 27395–27407\.Cited by:[Appendix C](https://arxiv.org/html/2609.12014#A3.p1.1)\.
- Xuet al\.\(2022\)H\. Xu, X\. Zhan, and X\. ZhuConstraints penalized q\-learning for safe offline reinforcement learning\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.36,pp\. 8753–8760\.Cited by:[§6\.1](https://arxiv.org/html/2609.12014#S6.SS1.p1.1)\.
- Zhanet al\.\(2022\)W\. Zhan, B\. Huang, A\. Huang, N\. Jiang, and J\. LeeOffline reinforcement learning with realizability and single\-policy concentrability\.InConference on Learning Theory,pp\. 2730–2775\.Cited by:[Appendix C](https://arxiv.org/html/2609.12014#A3.p1.1)\.
- Zhenget al\.\(2024\)Y\. Zheng, J\. Li, D\. Yu, Y\. Yang, S\. Li, X\. Zhan, and J\. LiuSafe offline reinforcement learning with feasibility\-guided diffusion model\.InInternational Conference on Learning Representations,Vol\.2024,pp\. 3838–3867\.Cited by:[§1](https://arxiv.org/html/2609.12014#S1.p1.1),[§2](https://arxiv.org/html/2609.12014#S2.SS0.SSS0.Px2.p1.1)\.

## Appendix AAlgorithm

Certified safety curation\.1\.TrainKKvalue networks on𝒟pref\\mathcal\{D\}\_\{\\mathrm\{pref\}\}by Equation[1](https://arxiv.org/html/2609.12014#S4.E1); setV¯\\bar\{V\}to the ensemble mean\.2\.Score every trajectory:g⁡\(τ\)=1\|τ\|​∑st∈τV¯​\(st\)g\(\\tau\)=\\frac\{1\}\{\|\\tau\|\}\\sum\_\{s\_\{t\}\\in\\tau\}\\bar\{V\}\(s\_\{t\}\)\.3\.Drawnncalibration trajectories uniformly; label episodic costs\.4\.Fixed\-sequence LTT over score quantiles0\.850\.85down to0\.300\.30with exact hypergeometric tests at levelδ\\deltaagainst risk targetα\\alpha; return the last certified threshold, or an uncertified selection at the calibration\-estimated safe\-mass fraction with an explicit no\-certificate report\.5\.If certified, optionally curate for reward, retaining the top half by return or resampling trajectories with probability proportional toexp⁡\(clip⁡\(z⁡\(R\),±κ\)\)\\exp\(\\mathrm\{clip\}\(z\(R\),\\pm\\kappa\)\)\.6\.Behavior\-clone the selection\.

## Appendix BProofs

Propositions[1](https://arxiv.org/html/2609.12014#Thmproposition1)and[2](https://arxiv.org/html/2609.12014#Thmproposition2)are proved in turn, followed by two results stated only here: a construction separating the operator classes of Section[5](https://arxiv.org/html/2609.12014#S5), and a closed form for the certification probability\.

Fix the threshold gridλ1\>λ2\>⋯\>λJ\\lambda\_\{1\}\>\\lambda\_\{2\}\>\\cdots\>\\lambda\_\{J\}in advance\. For eachjjwriteSj=S⁡\(λj\)⊆𝒟S\_\{j\}=S\(\\lambda\_\{j\}\)\\subseteq\\mathcal\{D\},Nj=\|Sj\|N\_\{j\}=\|S\_\{j\}\|, and letUjU\_\{j\}denote the number of unsafe trajectories inSjS\_\{j\}\. The null hypothesis at stepjjisℋj:Uj\>α​Nj\\mathcal\{H\}\_\{j\}:U\_\{j\}\>\\alpha N\_\{j\}, equivalentlyUj≥Uj∗U\_\{j\}\\geq U\_\{j\}^\{\*\}withUj∗=⌊α​Nj⌋\+1U\_\{j\}^\{\*\}=\\lfloor\\alpha N\_\{j\}\\rfloor\+1\. The composition guarantee of Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)rests on two lemmas\.

###### Lemma 1\(Conditional distribution\)\.

Let𝒞\\mathcal\{C\}be drawn uniformly without replacement from𝒟\\mathcal\{D\}, and condition onmj=\|𝒞∩Sj\|m\_\{j\}=\|\\mathcal\{C\}\\cap S\_\{j\}\|\. Then𝒞∩Sj\\mathcal\{C\}\\cap S\_\{j\}is a uniform without\-replacement sample of sizemjm\_\{j\}fromSjS\_\{j\}, sokj\|mj∼Hyp⁡\(Nj,Uj,mj\)k\_\{j\}\\mid m\_\{j\}\\sim\\mathrm\{Hyp\}\(N\_\{j\},U\_\{j\},m\_\{j\}\)\.

###### Proof\.

LetA⊆SjA\\subseteq S\_\{j\}with\|A\|=mj\|A\|=m\_\{j\}andB⊆𝒟∖SjB\\subseteq\\mathcal\{D\}\\setminus S\_\{j\}with\|B\|=n−mj\|B\|=n\-m\_\{j\}\. Since𝒞\\mathcal\{C\}is a uniformly randomnn\-subset of𝒟\\mathcal\{D\},Pr\[𝒞=A∪B\]=\(Nn\)−1\\Pr\[\\mathcal\{C\}=A\\cup B\]=\\binom\{N\}\{n\}^\{\-1\}, which does not depend onAAorBB\. Summing over the admissibleBBgivesPr⁡\[𝒞∩Sj=A∣mj\]=\(N−Njn−mj\)/\(\(Njmj\)​\(N−Njn−mj\)\)=\(Njmj\)−1\\Pr\[\\mathcal\{C\}\\cap S\_\{j\}=A\\mid m\_\{j\}\]=\\binom\{N\-N\_\{j\}\}\{n\-m\_\{j\}\}/\\big\(\\binom\{N\_\{j\}\}\{m\_\{j\}\}\\binom\{N\-N\_\{j\}\}\{n\-m\_\{j\}\}\\big\)=\\binom\{N\_\{j\}\}\{m\_\{j\}\}^\{\-1\}, the same value for every suchAA\. Hence𝒞∩Sj\\mathcal\{C\}\\cap S\_\{j\}is a uniformmjm\_\{j\}\-subset ofSjS\_\{j\}, and counting its unsafe elements giveskj\|mj∼Hyp⁡\(Nj,Uj,mj\)k\_\{j\}\\mid m\_\{j\}\\sim\\mathrm\{Hyp\}\(N\_\{j\},U\_\{j\},m\_\{j\}\)\. ∎

###### Lemma 2\(Super\-uniformity\)\.

Definepj=Fhyp​\(kj,Nj,Uj∗,mj\)p\_\{j\}=F\_\{\\mathrm\{hyp\}\}\(k\_\{j\};N\_\{j\},U\_\{j\}^\{\*\},m\_\{j\}\)\. Ifℋj\\mathcal\{H\}\_\{j\}holds, thenPr⁡\[pj≤δ∣mj\]≤δ\\Pr\[p\_\{j\}\\leq\\delta\\mid m\_\{j\}\]\\leq\\deltafor everymjm\_\{j\}, and hence marginally\.

###### Proof\.

The argument has three steps\.

*Stochastic monotonicity\.*IfU≤U′U\\leq U^\{\\prime\}thenHyp\(N,U,m\)⪯stHyp\(N,U′,m\)\\mathrm\{Hyp\}\(N,U,m\)\\preceq\_\{\\mathrm\{st\}\}\\mathrm\{Hyp\}\(N,U^\{\\prime\},m\)\. Couple the two by drawing a single uniformmm\-subsetMMof\{1,…,N\}\\\{1,\\dots,N\\\}and declaring the firstUUindices successes in one marginal and the firstU′U^\{\\prime\}in the other\. Both marginals are correct, and\|M∩\{1,…,U\}\|≤\|M∩\{1,…,U′\}\|\|M\\cap\\\{1,\\dots,U\\\}\|\\leq\|M\\cap\\\{1,\\dots,U^\{\\prime\}\\\}\|pointwise, which is stochastic dominance\.

*Dominance of thepp\-value\.*Underℋj\\mathcal\{H\}\_\{j\}we haveUj≥Uj∗U\_\{j\}\\geq U\_\{j\}^\{\*\}, so by Lemma[1](https://arxiv.org/html/2609.12014#Thmlemma1)and the previous stepkj⪰stXk\_\{j\}\\succeq\_\{\\mathrm\{st\}\}XforX∼Hyp⁡\(Nj,Uj∗,mj\)X\\sim\\mathrm\{Hyp\}\(N\_\{j\},U\_\{j\}^\{\*\},m\_\{j\}\)\. The mapk↦Fhyp​\(k,Nj,Uj∗,mj\)k\\mapsto F\_\{\\mathrm\{hyp\}\}\(k;N\_\{j\},U\_\{j\}^\{\*\},m\_\{j\}\)is non\-decreasing, and a non\-decreasing map preserves stochastic order, sopj⪰stFhyp\(X;Nj,Uj∗,mj\)p\_\{j\}\\succeq\_\{\\mathrm\{st\}\}F\_\{\\mathrm\{hyp\}\}\(X;N\_\{j\},U\_\{j\}^\{\*\},m\_\{j\}\)\.

*Super\-uniformity\.*For any random variableXXwith CDFFFand anyδ∈\(0,1\)\\delta\\in\(0,1\),Pr\[F\(X\)≤δ\]≤δ\\Pr\[F\(X\)\\leq\\delta\]\\leq\\delta: if no point of the support satisfiesF⁡\(x\)≤δF\(x\)\\leq\\deltathe probability is zero, and otherwise, writingx∗x^\{\*\}for the largest such point,FFnon\-decreasing gives\{F\(X\)≤δ\}=\{X≤x∗\}\\\{F\(X\)\\leq\\delta\\\}=\\\{X\\leq x^\{\*\}\\\}, whose probability isF⁡\(x∗\)≤δF\(x^\{\*\}\)\\leq\\delta\. Applying this toXXabove and combining with the previous step,Pr\[pj≤δ∣mj\]≤Pr\[Fhyp\(X\)≤δ\]≤δ\\Pr\[p\_\{j\}\\leq\\delta\\mid m\_\{j\}\]\\leq\\Pr\[F\_\{\\mathrm\{hyp\}\}\(X\)\\leq\\delta\]\\leq\\deltafor everymjm\_\{j\}, and averaging overmjm\_\{j\}gives the marginal statement\. ∎

###### Proof of Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)\.

Letttbe the index of the first true null in the fixed order, if any exists; if none exists, no certification can be false and the claim is vacuous\. The strict stopping rule testsℋ1,ℋ2,…\\mathcal\{H\}\_\{1\},\\mathcal\{H\}\_\{2\},\\ldotsin order and stops at the first failure to reject\. Reachingℋt\\mathcal\{H\}\_\{t\}requires rejectingℋ1,…,ℋt−1\\mathcal\{H\}\_\{1\},\\ldots,\\mathcal\{H\}\_\{t\-1\}, all of which are false nulls, so no error is committed before steptt\. Suppose the procedure certifies and the returned threshold isλ^=λȷ^\\hat\{\\lambda\}=\\lambda\_\{\\hat\{\\jmath\}\}withunsafe^​\(S⁡\(λȷ^\)\)\>α\\widehat\{\\mathrm\{unsafe\}\}\(S\(\\lambda\_\{\\hat\{\\jmath\}\}\)\)\>\\alpha, the violation event of Equation[2](https://arxiv.org/html/2609.12014#S4.E2); that is,ℋȷ^\\mathcal\{H\}\_\{\\hat\{\\jmath\}\}is true\. Sincettis the least index with a true null,ȷ^≥t\\hat\{\\jmath\}\\geq t\. The strict stopping rule returnsλȷ^\\lambda\_\{\\hat\{\\jmath\}\}only if it rejectedℋ1,…,ℋȷ^\\mathcal\{H\}\_\{1\},\\ldots,\\mathcal\{H\}\_\{\\hat\{\\jmath\}\}in order, so in particular it tested and rejectedℋt\\mathcal\{H\}\_\{t\}\. The false\-certification event is therefore contained in\{ℋttested and rejected\}⊆\{pt≤δ\}\\\{\\mathcal\{H\}\_\{t\}\\text\{ tested and rejected\}\\\}\\subseteq\\\{p\_\{t\}\\leq\\delta\\\}, whose probability is at mostδ\\deltaby Lemma[2](https://arxiv.org/html/2609.12014#Thmlemma2)\. The bound holds without any monotonicity assumption onj↦Uj/Njj\\mapsto U\_\{j\}/N\_\{j\}; monotonicity only affects which hypotheses are reachable, that is, power\. ∎

One dependence deserves an explicit statement\. The scoreggis trained on preferences synthesized from the episodic costs of the same pool𝒟\\mathcal\{D\}, sogg, and hence everySjS\_\{j\}, is a function of the pool’s labels\. Validity is unaffected:ggand the setsSjS\_\{j\}are fixed before𝒞\\mathcal\{C\}is drawn and do not depend on which trajectories land in𝒞\\mathcal\{C\}, so the exchangeability argument of Lemma[1](https://arxiv.org/html/2609.12014#Thmlemma1)applies verbatim\. What the construction does forfeit is independent test data for the score itself, which is why the guarantee is on the selection’s composition and not on any property ofgg\. As a further remark, calibration trajectories whose labels reveal them unsafe could be excluded from the selection, which can only improve its composition; the reported results do not apply this and are conservative in that respect\.

A corollary transfers the composition bound to the return\-weighted operator of Section[4\.4](https://arxiv.org/html/2609.12014#S4.SS4)\. Resampling the certified selection with weightswi∈\[e−κ,eκ\]w\_\{i\}\\in\[e^\{\-\\kappa\},e^\{\\kappa\}\]multiplies its unsafe\-to\-safe odds by at moste2​κe^\{2\\kappa\}, so the certified boundα\\alphaon the unsafe fraction becomes a bound ofe2​κ​α/\(e2​κ​α\+1−α\)e^\{2\\kappa\}\\alpha/\(e^\{2\\kappa\}\\alpha\+1\-\\alpha\)on the unsafe mass of the resampling distribution\. The bound is informative only at smallκ\\kappa\(atα=0\.25\\alpha=0\.25it already exceeds one half byκ=1\\kappa=1\), so the intensity\-robustness observed in Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)is an empirical property of the score\-return geometry of certified selections rather than a consequence of the guarantee; the guarantee’s role is to pin the composition the operator starts from\. Certifying the operator’s weighted risk directly, rather than through this corollary, does not help at our label budget\. The resampling weights are a function of returns and of the selection, so the normalizer is known exactly and only the weighted unsafe mass needs estimating, but that quantity ranges over\[e−κ,eκ\]\[e^\{\-\\kappa\},e^\{\\kappa\}\], and atκ=2\\kappa=2a distribution\-free bound needs roughly10310^\{3\}labeled trajectories*inside*the selection, about67006700calibration labels at a 15 percent selection, against the 200 we spend\. The composition risk is certifiable at this budget precisely because it is a bounded count rather than a weighted average\.

###### Proof of Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)\.

\(i\) Ifumin\>αu\_\{\\min\}\>\\alphathenuj\>αu\_\{j\}\>\\alphafor everyjj, so any returned threshold hasunsafe^​\(S​\(λ^\)\)\>α\\widehat\{\\mathrm\{unsafe\}\}\(S\(\\hat\{\\lambda\}\)\)\>\\alpha\. Certification therefore implies the violation event, andPr\[certify\]=Pr\[certify∧unsafe^\>α\]≤δ\\Pr\[\\textnormal\{certify\}\]=\\Pr\[\\textnormal\{certify\}\\wedge\\widehat\{\\mathrm\{unsafe\}\}\>\\alpha\]\\leq\\deltaby Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1), for anynn\. \(ii\) Atn=Nn=Nthe calibration sample is the whole pool, som1=N1m\_\{1\}=N\_\{1\}andk1=U1k\_\{1\}=U\_\{1\}deterministically, andp1=Fhyp​\(U1,N1,U1∗,N1\)p\_\{1\}=F\_\{\\mathrm\{hyp\}\}\(U\_\{1\};N\_\{1\},U\_\{1\}^\{\*\},N\_\{1\}\)is00whenU1<U1∗=⌊α​N1⌋\+1U\_\{1\}<U\_\{1\}^\{\*\}=\\lfloor\\alpha N\_\{1\}\\rfloor\+1, which is implied byu1<αu\_\{1\}<\\alpha\. The first hypothesis is then rejected with certainty and a threshold is returned\. Forn<Nn<Nthe same event gives a lower bound: whenever the calibration draw happens to contain all ofS⁡\(λ1\)S\(\\lambda\_\{1\}\)we havem1=N1m\_\{1\}=N\_\{1\}and the argument above applies verbatim, soPr\[certify\]≥Pr\[m1=N1\]=\(N−N1n−N1\)/\(Nn\)\\Pr\[\\textnormal\{certify\}\]\\geq\\Pr\[m\_\{1\}=N\_\{1\}\]=\\binom\{N\-N\_\{1\}\}\{n\-N\_\{1\}\}/\\binom\{N\}\{n\}, which tends to one asn→Nn\\to N\. ∎

##### Discussion of Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)\.

The two halves are not symmetric: part \(i\) is a corollary of Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)rather than new machinery, while part \(ii\) concerns the first tested threshold, since the fixed\-sequence walk stops at its first failure; when the score ranks well the two coincide, becauseuju\_\{j\}is smallest at the top quantile\.

Refusal is therefore a correctness property, not a conservative heuristic\. The second half is a question of labels, and its cost is forecastable: the budget at which a task first certifies in half of draws falls off as a power of the margin, shallower than a fixed\-threshold test would give because the walk reaches deeper thresholds asnngrows \(Appendix[H](https://arxiv.org/html/2609.12014#A8)\)\. Since the margin is estimable from the calibration sample, a practitioner can forecast before spending labels whether a certificate is attainable and what it would cost\. A rare certificate on a mostly\-refusing task is weak evidence; the conditional rates the same resample yields are reported in Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)and Table[14](https://arxiv.org/html/2609.12014#A8.T14)\. The validation resamples calibration draws with the score fixed, the theorem’s own setting, so it checks the implementation and finite\-sample constants rather than testing an assumption\.

##### A construction separating the operator classes\.

The following minimal CMDP makes the operator half of Obstacle 1 provable rather than only measured\. Consider a deterministicTT\-step corridor whose state is the time index, so the state records no action history; at every step two actions are available, one safe \(per\-step cost 0\) and one risky \(cost 1\), the budget is 0, and the behavior policy takes the safe action with probabilityps∈\(0,1\)p\_\{\\mathrm\{s\}\}\\in\(0,1\)independently at each step\. We state the population version; empirical action frequencies concentrate aroundpsp\_\{\\mathrm\{s\}\}at each step by Hoeffding’s inequality\.

###### Proposition 3\.

In the corridor CMDP, per\-transition weighted cloning with weightsw⁡\(s,a\)w\(s,a\)and safe\-to\-risky weight ratioWWat every state yields the product policy with per\-step safe probabilityps′=ps​W/\(ps​W\+1−ps\)p^\{\\prime\}\_\{\\mathrm\{s\}\}=p\_\{\\mathrm\{s\}\}W/\(p\_\{\\mathrm\{s\}\}W\+1\-p\_\{\\mathrm\{s\}\}\)\. Consequently: \(i\) state\-only weights giveW=1W=1, so the cloned policy equals the behavior policy, with all\-safe probabilitypsTp\_\{\\mathrm\{s\}\}^\{T\}; \(ii\) reaching all\-safe probability1−ϵ1\-\\epsilonrequiresW≥1−psps\(\(1−ϵ\)−1/T−1\)−1W\\geq\\frac\{1\-p\_\{\\mathrm\{s\}\}\}\{p\_\{\\mathrm\{s\}\}\}\\,\\big\(\(1\-\\epsilon\)^\{\-1/T\}\-1\\big\)^\{\-1\}, which approaches1−psps⋅T/ln⁡11−ϵ\\frac\{1\-p\_\{\\mathrm\{s\}\}\}\{p\_\{\\mathrm\{s\}\}\}\\cdot T/\\ln\\frac\{1\}\{1\-\\epsilon\}, linear in the horizon, so weights bounded by any fixed clip fail beyond a horizon threshold; \(iii\) trajectory\-level selection of the zero\-cost trajectories followed by unweighted cloning yields the exactly safe policy whenever the dataset contains at least one such trajectory, an event of expected countN​psTNp\_\{\\mathrm\{s\}\}^\{T\}\.

###### Proof\.

Weighted cloning at statessmaximizes∑aca​log⁡π⁡\(a∣s\)\\sum\_\{a\}c\_\{a\}\\log\\pi\(a\\mid s\)over the simplex, whereca=μ⁡\(a\)​w​\(s,a\)\>0c\_\{a\}=\\mu\(a\)w\(s,a\)\>0is the population weight of actionaa\. The objective is strictly concave on the relative interior and tends to−∞\-\\inftyat the boundary, so it has a unique maximizer there; the stationarity condition of∑aca​log⁡πa−η⁡\(∑aπa−1\)\\sum\_\{a\}c\_\{a\}\\log\\pi\_\{a\}\-\\eta\(\\sum\_\{a\}\\pi\_\{a\}\-1\)isca/πa=ηc\_\{a\}/\\pi\_\{a\}=\\etafor everyaa, and normalizing givesπa=ca/∑bcb\\pi\_\{a\}=c\_\{a\}/\\sum\_\{b\}c\_\{b\}, that isπ⁡\(a∣s\)∝μ⁡\(a\)​w​\(s,a\)\\pi\(a\\mid s\)\\propto\\mu\(a\)w\(s,a\)\. With two actions of weights in ratioWWthis isps′p^\{\\prime\}\_\{\\mathrm\{s\}\}as stated; rollout steps are independent because the state carries no action history\. \(i\) is immediate\. For \(ii\),\(ps′\)T≥1−ϵ\(p^\{\\prime\}\_\{\\mathrm\{s\}\}\)^\{T\}\\geq 1\-\\epsilonforcesps′≥qp^\{\\prime\}\_\{\\mathrm\{s\}\}\\geq qwithq=\(1−ϵ\)1/Tq=\(1\-\\epsilon\)^\{1/T\}, andps​W/\(ps​W\+1−ps\)≥qp\_\{\\mathrm\{s\}\}W/\(p\_\{\\mathrm\{s\}\}W\+1\-p\_\{\\mathrm\{s\}\}\)\\geq qrearranges toW≥1−psps⋅q1−q=1−psps\(\(1−ϵ\)−1/T−1\)−1→1−psps⋅Tln⁡11−ϵW\\geq\\frac\{1\-p\_\{\\mathrm\{s\}\}\}\{p\_\{\\mathrm\{s\}\}\}\\cdot\\frac\{q\}\{1\-q\}=\\frac\{1\-p\_\{\\mathrm\{s\}\}\}\{p\_\{\\mathrm\{s\}\}\}\\big\(\(1\-\\epsilon\)^\{\-1/T\}\-1\\big\)^\{\-1\}\\to\\frac\{1\-p\_\{\\mathrm\{s\}\}\}\{p\_\{\\mathrm\{s\}\}\}\\cdot\\frac\{T\}\{\\ln\\frac\{1\}\{1\-\\epsilon\}\}\. For \(iii\), every transition in a zero\-cost trajectory takes the safe action, so cloning the selection reproduces it deterministically\. ∎

The construction is faithful to the paper’s setting twice over\. Segment\-sum preference training receives zero signal here, since every trajectory visits the identical state sequence, so a state\-only value necessarily lands in case \(i\); and advantage weights built from observed per\-step costs have the fixed ratioW=e1/βW=e^\{1/\\beta\}, which case \(ii\) defeats at long horizons, the deployed clip of 20 included\. The two operator classes thus pay different prices\. Reweighting needs a weight range growing with the horizon, while selection needs a safe trajectory to exist, at expected countN​psTNp\_\{\\mathrm\{s\}\}^\{T\}under per\-step\-independent behavior\. The benchmarks supply safe trajectories at 9 to 46 percent of each pool precisely because real behavior is correlated across steps rather than independent, and that correlation is what whole\-trajectory selection preserves and per\-transition reweighting fragments\. The corridor isolates the operator’s structural limitation; the benchmark\-scale claim rests on the empirical sweep \(Appendix[E](https://arxiv.org/html/2609.12014#A5)\), since real states partially record consequences, which is why one navigation task escapes at a sharp enough temperature\. Figure[2](https://arxiv.org/html/2609.12014#A2.F2)simulates the construction with a benchmark\-like behavior mixture \(20 percent all\-safe expert trajectories, per\-step\-independent otherwise\): unweighted cloning collapses immediately, the deployed clip of 20 collapses by the benchmarks’ horizon of 1000, a clip of 400 is already bending, and selection holds at one throughout because the expert subset always exists, with theory \(dashed\) matching simulation \(markers\)\.

Figure 2:The corridor construction, simulated \(ps=0\.7p\_\{\\mathrm\{s\}\}=0\.7, expert mass0\.20\.2,N=104N=10^\{4\}, 50 replicates\): all\-safe probability of the cloned policy against horizon\. Open markers are simulation; dashed lines are the closed forms of Proposition[3](https://arxiv.org/html/2609.12014#Thmproposition3)for the three per\-transition weight clips, which the simulation tracks closely\. Trajectory selection \(solid, filled\) has no closed form here\.
##### Certification probability in closed form\.

Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)separates the two regimes but leaves the rate between them asymptotic\. Because the walk is a strict fixed sequence, stopping at the first failure to reject, a certificate is returned if and only if the first and most selective thresholdλ1\\lambda\_\{1\}is rejected\. That collapses the whole procedure to one hypergeometric event and makes the rate exact\.

###### Proposition 4\(Certification rate\)\.

LetN1=\|S⁡\(λ1\)\|N\_\{1\}=\|S\(\\lambda\_\{1\}\)\|, letU1U\_\{1\}be the number of unsafe trajectories it contains, and letk∗=⌊α​N1⌋\+1k^\{\*\}=\\lfloor\\alpha N\_\{1\}\\rfloor\+1\. Withmmthe number of calibration trajectories falling inS⁡\(λ1\)S\(\\lambda\_\{1\}\)andkkthe unsafe count among them,

Pr\[certify\]=∑mPr\[m\]∑k:F⁡\(k,N1,k∗,m\)≤δPr\[k∣m\],\\Pr\[\\textnormal\{certify\}\]=\\sum\_\{m\}\\Pr\[m\]\\sum\_\{k\\,:\\,F\(k;N\_\{1\},k^\{\*\},m\)\\leq\\delta\}\\Pr\[k\\mid m\],\(3\)wherem∼Hypergeom⁡\(N,N1,n\)m\\sim\\mathrm\{Hypergeom\}\(N,N\_\{1\},n\),k\|m∼Hypergeom⁡\(N1,U1,m\)k\\mid m\\sim\\mathrm\{Hypergeom\}\(N\_\{1\},U\_\{1\},m\), andFFis the hypergeometric CDF\.

###### Proof\.

The walk testsH1,H2,…H\_\{1\},H\_\{2\},\\dotsin order and halts at the first hypothesis it fails to reject, so it returns a certificate if and only if it rejectsH1H\_\{1\}; every deeper threshold is reached only afterH1H\_\{1\}has already been rejected\. Rejection atλ1\\lambda\_\{1\}occurs exactly when the exact test’spp\-valueF⁡\(k,N1,k∗,m\)F\(k;N\_\{1\},k^\{\*\},m\)is at mostδ\\delta\. The calibration sample is a uniform draw ofnnfrom theNNtrajectories of𝒟\\mathcal\{D\}, of whichN1N\_\{1\}lie inS⁡\(λ1\)S\(\\lambda\_\{1\}\), so the numbermmfalling inside the selection isHypergeom⁡\(N,N1,n\)\\mathrm\{Hypergeom\}\(N,N\_\{1\},n\)\. Conditioned onmm, thosemmtrajectories are a uniform draw fromS⁡\(λ1\)S\(\\lambda\_\{1\}\), which containsU1U\_\{1\}unsafe ones, sok∼Hypergeom⁡\(N1,U1,m\)k\\sim\\mathrm\{Hypergeom\}\(N\_\{1\},U\_\{1\},m\)\. Summing the rejection indicator against these two distributions gives Equation[3](https://arxiv.org/html/2609.12014#A2.E3)\. Both halves of Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)follow: ifu1\>αu\_\{1\}\>\\alphathenU1≥k∗U\_\{1\}\\geq k^\{\*\}and the inner sum is the level of the test, at mostδ\\delta; and ifu1<αu\_\{1\}<\\alphathe rejection event atλ1\\lambda\_\{1\}has probability tending to one, as shown in the proof of Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)\. The first half in fact needs onlyu1\>αu\_\{1\}\>\\alpharather thanumin\>αu\_\{\\min\}\>\\alpha, since certification turns onℋ1\\mathcal\{H\}\_\{1\}alone\. ∎

Three things follow\. Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)is recovered as a corollary: whenu1\>αu\_\{1\}\>\\alphathe inner sum is bounded byδ\\delta, and whenu1<αu\_\{1\}<\\alphait tends to one asn→Nn\\to N\. The scorer enters Equation[3](https://arxiv.org/html/2609.12014#A2.E3)only throughU1U\_\{1\}, so the certification rate is a property of the purity the pool makes attainable rather than of how the ranking was produced, which is why interchanging the scorer leaves it essentially unchanged\. And the expression inverts: given a target rate, it returns the calibration budget that achieves it, so a practitioner can price the labels before spending them\. On PointGoal1 it asks forn≥360n\\geq 360to reach a rate of0\.90\.9, where the deployedn=200n=200realizes0\.730\.73\.

Table[2](https://arxiv.org/html/2609.12014#A2.T2)compares Equation[3](https://arxiv.org/html/2609.12014#A2.E3)against the resampled rates\. Across all108108cells the mean absolute error is0\.0160\.016and the correlation is0\.9950\.995, with the residuals consistent with the Monte\-Carlo error of a200200\-draw estimate\. Equation[3](https://arxiv.org/html/2609.12014#A2.E3)is the operative form; a readable relaxation of it is not\. Writingϵ=α−u1\\epsilon=\\alpha\-u\_\{1\}for the purity margin, three Hoeffding steps for sampling without replacement givePr\[certify∣m\]≥1−exp\(−mϵ2/2\)\\Pr\[\\textnormal\{certify\}\\mid m\]\\geq 1\-\\exp\(\-m\\epsilon^\{2\}/2\)wheneverm≥2​ln⁡\(1/δ\)/ϵ2m\\geq 2\\ln\(1/\\delta\)/\\epsilon^\{2\}\. That bound is vacuous on every cell here, because the first grid threshold keeps only1515percent of the pool and so receives about3030of the200200calibration trajectories, far short of what the condition asks\. It is reported because it names the governing quantity,m​ϵ2m\\epsilon^\{2\}: a wide margin on a thin selection certifies rarely, which is why deepening the grid start does not help\.

Table 2:Certification rate, Equation[3](https://arxiv.org/html/2609.12014#A2.E3)against the200200\-draw resample \(predicted/observed, averaged over three pipeline seeds\)\. Across all108108cells the mean absolute error is0\.0160\.016and the Pearson correlation0\.9950\.995\.
##### Controlled contamination\.

The nine pools offer one purity margin each, and none sits near zero, so they cannot show whether the dichotomy is a threshold or merely a trend\. Figure[3](https://arxiv.org/html/2609.12014#A2.F3)varies the margin continuously by subsampling each pool to eight target unsafe fractions, dropping unsafe trajectories to purify it or safe ones to dirty it, recomputing the grid quantiles on the constructed pool as the procedure would on a dataset of that shape, and running the identical calibration walk on it with scores fixed\. Across198198constructed pools spanning margins from−0\.35\-0\.35to\+0\.25\+0\.25, the certification rate is essentially zero at every non\-positive margin, with a maximum of0\.0750\.075againstδ=0\.1\\delta=0\.1, and rises through a sigmoid to one by a margin of about0\.20\.2\. Equation[3](https://arxiv.org/html/2609.12014#A2.E3)tracks every point, with mean absolute error0\.0110\.011and correlation0\.9990\.999, and residuals consistent with the Monte\-Carlo error of the200200\-draw estimate\. Eight of the nine tasks cross the threshold in both directions; Swimmer’s pool is too clean to be dirtied below zero without falling under the minimum pool size and contributes positive margins only\. Nothing is retrained: the sweep reuses the deployed scorers and changes only what is in the pool\.

Figure 3:Certification against controlled contamination\. Each pool is subsampled to eight target unsafe fractions and the full calibration walk is rerun on the result \(200200draws,n=200n=200, scores fixed\); the horizontal axis is the constructed pool’s purity marginα−u1\\alpha\-u\_\{1\}\. Crosses are Equation[3](https://arxiv.org/html/2609.12014#A2.E3)evaluated on that pool’s own\(N,N1,U1\)\(N,N\_\{1\},U\_\{1\}\)\. The rate is at most0\.0750\.075at every non\-positive margin \(dotted line atδ\\delta\) and rises to one by a margin of about0\.20\.2;198198pools over nine tasks\.

## Appendix CCertificates on the deployed policy

The certificate of Section[4\.3](https://arxiv.org/html/2609.12014#S4.SS3)bounds the composition of the training set, not the cost of the policy learned from it, and Section[7](https://arxiv.org/html/2609.12014#S7)states that boundary\. The gap can be closed from the other side, without any concentrability argument\([Rashidinejad et al\., 2021](https://arxiv.org/html/2609.12014#bib.bib28);[Xie et al\., 2021](https://arxiv.org/html/2609.12014#bib.bib29);[Zhan et al\., 2022](https://arxiv.org/html/2609.12014#bib.bib30)\)and without strengthening the supervisor\. Once training ends the clone is a fixed function, so its evaluation episodes are independent draws from the task’s initial\-state distribution and the number exceeding the budget is binomial\. Both certificates below therefore consume exactly the supervision the pipeline already assumes: one bit per episode, whether it went over budget\. Each bound holds per cell, with no joint claim across the seventy\-five; counts of certifying tasks aggregate separate bounds rather than asserting a simultaneous guarantee\. They differ from the composition certificate in one respect that matters\. That certificate is computed from the dataset alone and is available before the policy is trained or run; these are computed from executed episodes, so they are deployment\-time guarantees and require the interaction an offline setting may not permit\. We report them as a complement to the offline certificate, not a replacement for it\.

##### Violation probability\.

Counting the episodes over budget among the100100evaluation episodes already reported per cell and inverting the exact binomial gives a Clopper\-Pearson upper limit onPr\[cost\>budget\]\\Pr\[\\text\{cost\}\>\\text\{budget\}\]at levelδ\\delta, reported in Table[3](https://arxiv.org/html/2609.12014#A3.T3), whose pooled values Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)quotes: curation cuts the deployment violation probability by roughly two thirds and lands within a point of the full\-label oracle, on a criterion stricter than the one the benchmark scores\. The same table records a property of that criterion rather than of any method here: a quarter of episodes exceed budget even for the oracle, because mean\-cost safety constrains an average and leaves the tail free\.

##### Expected cost\.

The tail bound is not the paper’s safety criterion, which is𝔼⁡\[cost\]≤\\mathbb\{E\}\[\\text\{cost\}\]\\leqbudget, and bits carry no magnitudes\. Bounding the mean needs the cost values on the evaluation episodes, and it needs more of them: with episodic cost in\[0,T\]\[0,T\]the range term of an empirical\-Bernstein bound dominates atn=100n=100however favorable the observed costs are\. We therefore rolled out20002000fresh episodes per cell on the same reported policies, with no retraining, and applied Maurer\-Pontil with the range fixed a priori atT=1000T=1000rather than taken from the observed maximum\. Table[4](https://arxiv.org/html/2609.12014#A3.T4)reports the result:𝔼⁡\[cost\]\\mathbb\{E\}\[\\text\{cost\}\]is certified within budget on nine of the fifteen tasks, on every seed, atδ=0\.1\\delta=0\.1\. Two tasks are safe by point estimate but do not certify, and both are single\-seed effects rather than marginal tasks: CarButton2 misses on one seed by1\.11\.1, and PointCircle1 carries one broken seed at mean cost59\.159\.1against4\.74\.7to7\.37\.3for the others\. Such seeds are not visible before deployment\. Across the7575calibrated cells, no selection statistic separates a failing seed from its siblings: task\-demeaned, the Spearman correlation of the20002000\-episode cost with selection size, kept fraction and realized contamination is0\.270\.27,0\.250\.25and0\.230\.23, and in each task with a seed\-level failure one failing seed has the task’s smallest selection and another its largest\. Most such failures are broad rather than rare: on five of the seven the median episode itself exceeds the budget, so the clone has learned an unsafe behavior rather than an occasionally unlucky one\. The other two are tail\-driven, PointCircle1 seed44and Swimmer seed44carrying medians of1414and1212against budgets of2525and2020, so both failure modes occur\. That is the case for a certificate on the deployed policy: the composition certificate audits the data, and the seed that misuses it can only be caught by running it\.

The larger evaluation also re\-tests the headline\. Comparing the20002000\-episode estimates against the100100\-episode ones, all fifteen safe\-or\-unsafe verdicts are unchanged and the safe\-task count is eleven under both, so Table[1](https://arxiv.org/html/2609.12014#S6.T1)’s result replicates at twenty times the evaluation budget\.

Table 3:Deployment violation probability: exact Clopper\-Pearson upper limit onPr\[cost\>budget\]\\Pr\[\\text\{cost\}\>\\text\{budget\}\]atδ=0\.1\\delta=0\.1, from the100100\-episode evaluations of Table[1](https://arxiv.org/html/2609.12014#S6.T1)\(five seeds pooled per task; BC\-All is a single training\)\. Pooling makes the bound a statement about a policy drawn uniformly from the five training seeds, not about any one deployed clone; per\-seed bounds are wider\.Table 4:Expected\-cost certificate from20002000fresh episodes per seed on the deployed policies: mean cost, the Maurer\-Pontil upper bound atδ=0\.1\\delta=0\.1taken over the worst seed, and the budget\. Bold marks a bound within budget; the final column requires every seed to certify\.

## Appendix DFull extraction sweep

This appendix reports the sweeps behind Section[5](https://arxiv.org/html/2609.12014#S5)in full: the value’s agreement with held\-out preferences at two scales, the oracle\-value substitutions, and the per\-transition configurations tested\.

Table 5:The value at two scales on the nine analysis tasks, three seeds each\. Accuracy ranges over tasks and is the fraction of held\-out pairs the value orders the same way as their ground\-truth summed costs, on pairs drawn from the bottom and top cost quartiles\. The per\-transition rows range over every seed pair of every task, on 5000 matched transitions\.Table 6:Oracle\-value ablation on all nine analysis tasks with the BC\-All reference \(oracles: three seeds; learned value: five seeds; bold marks mean cost within budget\)\.The full oracle\-value ablation is in Table[6](https://arxiv.org/html/2609.12014#A4.T6)\. One cell has an oracle improving on the learned value: on Swimmer the per\-step oracle extracts a cost\-safe policy \(19 against the learned value’s 61\)\. The improvement requires the dense ground\-truth per\-step cost, which is unavailable under preference supervision, and Swimmer is the task no configuration in the study solves from preferences, so the exception sharpens the supervision boundary without touching the operator\-class reading on the six diagnostic tasks\. Table[7](https://arxiv.org/html/2609.12014#A4.T7)reports the per\-transition operator sweep on the nine analysis tasks \(three seeds; the exp/h=1h\{=\}1reference appears in Table[6](https://arxiv.org/html/2609.12014#A4.T6)\)\. Columns: rank and binary weight transforms; counterfactual scoring with the learned value \(per\-state\-normalized softmax\); and the action\-conditioned oracle, a value distilled from full ground\-truth cost\-to\-go and scored counterfactually through the dynamics ensemble\. On the failing navigation tasks \(CarGoal2, PointGoal1, PointGoal2\), no weight transform escapes the cost plateau; only the action\-conditioned oracle, which consumes dense cost labels unavailable under preference supervision, reaches safety there, at severe reward cost\. Varying the extraction temperature rather than the transform does reach the budget, on PointGoal1 and on neither of the other two \(Appendix[E](https://arxiv.org/html/2609.12014#A5)\)\. On the velocity tasks the transforms trade reward for cost without dominating the exponential reference, and on CarGoal1, the one navigation task where per\-transition extraction is cost\-safe throughout, the weighted variants attain somewhat higher reward at two to three times the selection route’s cost\.

##### Matched\-supervision disagreement control\.

To separate the Bradley\-Terry null space from ordinary seed variance, we compute the same cross\-seed advantage statistics for cost\-to\-go ensembles trained with dense per\-step supervision \(three seeds, identical architecture, 5000 fixed transitions per task\)\. Disagreement is not specific to preference training\. On seven of the nine analysis tasks the supervised ensembles agree less by advantage rank correlation than the preference ensembles on the same samples, and top\-one\-percent overlaps are similar in range\. The null\-space argument still explains why preference training cannot identify per\-transition values; the control shows that dense supervision does not identify them either at these sample sizes, so the granular scale is unreliable under both supervisions, which is the fact the design responds to\. The identification deficit is quantifiable\. The 1000 training pairs constrain 1\.4 to 4\.2 percent of each pool’s states, with segment overlap below half a percent, so roughly 97 percent of visited states receive no direct constraint and, within the covered states, each appears in about one segment sum; the score’s behavior everywhere else is generalization, which is exactly what the oracle comparison of Section[5](https://arxiv.org/html/2609.12014#S5)finds to be the essential signal\.

Table 7:Extraction operator sweep on the nine analysis tasks \(three seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\. The oracle substitutions and the exponential reference appear in Table[6](https://arxiv.org/html/2609.12014#A4.T6)\.
##### Probes behind Obstacle 1\.

Two probes support the granularity reading in Section[5](https://arxiv.org/html/2609.12014#S5)\. Multi\-step advantages, formed over anhh\-step window, degrade safety monotonically inhh: ath=5h=5cost worsens on seven of the nine analysis tasks relative toh=1h=1, HalfCheetah moving from4\.84\.8to121121\. Applying the same exponentiated reweighting on top of the ground\-truth\-filtered BC\-Safe policy destroys its safety on five of the seven tasks where that policy satisfies the budget, so the failure is not a deficiency of the learned value: it survives substituting a policy trained on exactly the safe data\. On the failing navigation tasks every value source, learned or oracle, sits in the same cost band well above budget\.

##### The value at two scales\.

Table[5](https://arxiv.org/html/2609.12014#A4.T5)gives the ranges behind Obstacle 2\. Trajectory\-level held\-out accuracy runs from0\.7980\.798to1\.0001\.000, while cross\-seed advantage rank correlation spans0\.360\.36to0\.970\.97and top\-one\-percent overlap0\.150\.15to0\.890\.89\. The diagnostics do not predict the policy\. Hopper \(0\.9800\.980accuracy\) and Swimmer \(0\.9880\.988\) carry two of the best values into the two worst policies, and Hopper also has among the strongest cross\-seed agreement \(ρ=0\.91\\rho=0\.91\) yet violates on every seed at cost43\.543\.5against budget2020, while CarGoal1 has the weakest agreement of the nine \(ρ=0\.49\\rho=0\.49\) and is safe on every seed at16\.216\.2against2525\. Across the nine tasks the route is safe on two of the five tasks withρ≥0\.85\\rho\\geq 0\.85and on two of the three below it\.

## Appendix EExtraction temperature, weight clip, and score aggregation

Two sweeps were registered with committed predictions before they were run\. Both are reported here, whether or not the prediction survived\.

##### Extraction temperature and weight clip\.

Table[8](https://arxiv.org/html/2609.12014#A5.T8)varies the advantage\-weighted extraction’s temperatureβ\\betaover\{0\.1,0\.3,3\}\\\{0\.1,0\.3,3\\\}at clip2020, and the weight clip over\{5,100\}\\\{5,100\\\}atβ=1\\beta=1, on the nine analysis tasks at three seeds\. All six cells share one trainer and one seed protocol, so the columns are directly comparable; the deployed arm elsewhere in the paper uses an ensemble value and five seeds and is not a controlled comparison against them\. The registered prediction was that no setting reaches the budget on the three failing navigation tasks, since the weight range the corridor argument requires grows with the horizon and the sweep’s largest ratio is far below it\. Sharpening the weights trades cost against reward monotonically, and the prediction holds on two of the three\. On CarGoal2 no setting falls below29\.029\.0and on PointGoal2 none below41\.841\.8, against a budget of2525\. On PointGoal1 it fails\. Cost falls monotonically as the weights sharpen, from30\.53±1\.6730\.53\\pm 1\.67atβ=3\\beta=3to22\.05±1\.3322\.05\\pm 1\.33atβ=0\.3\\beta=0\.3and16\.40±0\.7116\.40\\pm 0\.71atβ=0\.1\\beta=0\.1, the last two inside the budget\. The safety is paid for in reward, which falls from17\.5317\.53to10\.8010\.80across that range\. We therefore scope Obstacle 1 rather than state it without qualification\. Per\-transition reweighting reaches the budget at no swept setting on two of the three failing navigation tasks, and on the third only by sharpening far enough to give up a third of the reward, where it still runs at nearly twice the selection route’s cost \(16\.4016\.40against8\.808\.80\) though at higher reward \(10\.8010\.80against7\.477\.47\)\. This is the behavior the corridor construction anticipates once real states partially record consequences, which is where the benchmark departs from the idealized corridor\.

Table 8:Extraction temperature and weight clip on the nine analysis tasks \(three seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\. These are the five registered cells, all from one trainer at one seed protocol\.
##### Score aggregation\.

Table[9](https://arxiv.org/html/2609.12014#A5.T9)replaces the deployed mean aggregator with the minimum and the tenth percentile of the per\-state value along a trajectory, filtering at the calibration\-estimated fraction\. Both registered predictions hold\. The minimum inherits the score’s unreliable extreme tail and is safe on four of the nine tasks against the mean’s eight\. The tenth percentile tracks the mean closely everywhere, and is safe on all nine\. It rescues Hopper, at cost88against the mean’s2626on a budget of2020, a task the registration named in advance as one where the mean\-aggregated filter fails\. That is a pre\-registered improvement on a single task rather than a sweep\-wide gain, and it softens rather than removes the open problem of tail\-robust scoring\.

Table 9:Score aggregators on the nine analysis tasks, V\-filter at the calibration\-estimated fraction \(three seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\. The mean column is the deployed aggregator\.

## Appendix FValue diagnostics: anticipation, traces, and the safety landscape

Figure[4](https://arxiv.org/html/2609.12014#A6.F4)reports the two temporal diagnostics behind the inspection paragraph of Section[5](https://arxiv.org/html/2609.12014#S5)\. Binned by steps until the next violation, the ensemble mean falls by two to four within\-task standard deviations between the most distant bin and imminent violation on all nine tasks; individual bins reverse by up to 0\.9 standard deviations, so the signal is a graded early warning rather than a strictly ordered countdown\. In the single\-trajectory trace, per\-state values swing across several units while their rolling mean moves stably through the episode’s cost\-event bands, which is Observation[1](https://arxiv.org/html/2609.12014#Thmobservation1)at the resolution of one episode\. The same network is noisy per state and reliable in aggregate\.

Figure 4:Left:V¯\\bar\{V\}declines as violations approach \(per\-task z\-scores\)\. Right: per\-state values along one unsafe trajectory \(episodic cost 40, budget 20\) are noisy while their rolling mean is stable\.Figure[5](https://arxiv.org/html/2609.12014#A6.F5)renders the learned value over the workspace on three independently sampled PointGoal1 layouts with the same value network \(100 rollouts per layout; BC\-All and BC\-Safe policies with action\-noise episodes for coverage\)\. The low\-value basin tracks the hazard field in each layout, with state\-level Spearman correlations of0\.430\.43,0\.440\.44, and0\.200\.20betweenV¯\\bar\{V\}and distance to the nearest hazard, the values labeled in the figure\. The alignment largely collapses when the same states are frozen at zero velocity \(ρ\\rhoof0\.090\.09,0\.270\.27, and0\.150\.15on layouts 7, 11 and 23\), so the value encodes proximity and motion jointly on the visited distribution rather than position alone\. Correlations computed on cell means rather than states are weaker \(0\.170\.17,0\.290\.29,0\.220\.22\): binning discards the visit density that concentrates states in the region where the value varies most, so each cell counts equally regardless of how well the behavior data supports it\. Two secondary structures are visible\. Hazard clusters produce deeper basins than isolated hazards, which the behavior data rarely visits\. And in the second layout the goal itself lies inside the hazard field, since reaching it requires transiting low\-value states, the geometric signature of tasks where any safety filter must either sacrifice reward or refuse\.

![Refer to caption](https://arxiv.org/html/2609.12014v1/landscape_pointgoal1_layouts.png)Figure 5:Safety landscape on three independently sampled PointGoal1 layouts \(reset seeds 7, 11, 23\), same value network\. Mean ensembleV¯\\bar\{V\}over the states visited by 100 rollouts per layout, alternating the BC\-All and BC\-Safe behavior policies with action noise so that coverage spans both the hazard field and the safe region; cells with fewer than 25 visits are left gray\. Dashed circles are ground\-truth hazards, which the model never sees, and G marks the goal\. Right:V¯\\bar\{V\}in distance bins, labeled with the state\-level Spearman correlation per layout\.
## Appendix GExternal baselines and labeling cost

This appendix reports the external baselines per task, in normalized and in raw units, and places every method by the supervision it consumes\.

Table 10:Normalized cost \(Cn=C/budgetC\_\{n\}=C/\\text\{budget\}; safe whenCn≤1C\_\{n\}\\leq 1, bold\) across all methods in the study, for at\-a\-glance comparison in the benchmark’s standard units; rewards and intervals appear in Table[1](https://arxiv.org/html/2609.12014#S6.T1)and Appendix[G](https://arxiv.org/html/2609.12014#A7)\.Table[11](https://arxiv.org/html/2609.12014#A7.T11)reports the full\-label baselines and CPL per task; Figure[6](https://arxiv.org/html/2609.12014#A7.F6)summarizes the study by the kind of supervisor each method assumes against tasks kept within budget\. Two details support the main\-text reading\. CDT’s navigation costs run from roughly 1\.5 times budget on the Goal tasks to nearly 5 times on the Button tasks, and Appendix[K](https://arxiv.org/html/2609.12014#A11)shows they stay there at every conditioning target\. CPQ’s safety is largely degenerate, pairing near\-zero cost with collapsed reward on three of its four safe cells \(Walker2d, Ant, Swimmer\); PointGoal1 is its one safe cell with intact reward\. COptiDICE’s two safe cells, HalfCheetah and CarGoal1, give up 23 to 36 percent of BC\-All’s reward\. In the figure, full\-label methods are placed at their best configuration, CDT at the most conservative target of its sweep; no external baseline exceeds the calibrated filter’s eleven safe tasks, CDT reaching seven of fifteen, while BC\-Safe reaches twelve on ground\-truth labels\. BC\-Safe\-Seg, which clones the ground\-truth\-safe*segments*rather than trajectories, is safe on two of fifteen despite full labels\. Selection at segment scale fragments the multi\-step avoidance behaviors that carry safety, a selection\-level counterpart of the granularity principle at the scale between per\-transition reweighting and whole\-trajectory selection\. Table[10](https://arxiv.org/html/2609.12014#A7.T10)gives the normalized\-cost comparison across all methods; normalized rewardRnR\_\{n\}uses dataset return extrema and is recoverable from the raw values reported here\.

Table 11:External baselines: full\-label offline safe RL trained in\-codebase \(three seeds; CDT’s full cost\-target sweep, including its safest target per task, is in Appendix[K](https://arxiv.org/html/2609.12014#A11)\), CPL on our exact preference data, and BC\-Safe\-Seg, the segment\-level full\-label clone \(five seeds\)\. Bold cost cells are within budget\.Figure 6:Supervision against safety coverage\. The horizontal axis is the strongest judgment a method’s supervisor must be able to make, given by the tick labels, rather than how much of it is consumed; methods sit on their tier and spread sideways only to separate ties\. The vertical axis counts the fifteen tasks kept within budget\.*Calibrated*is the procedure this paper proposes, V\-filter its uncalibrated ablation, and labels\-only and labels\-split the controls that spend the 200 labels on training a classifier rather than on auditing one\. The two ringed methods are the only ones shown that issue a certificate or an explicit refusal\. CDT, CPQ and COptiDICE are external baselines at their best configuration, CDT at the safest target of its sweep \(Appendix[K](https://arxiv.org/html/2609.12014#A11)\); rewards appear in Tables[1](https://arxiv.org/html/2609.12014#S6.T1)and[11](https://arxiv.org/html/2609.12014#A7.T11)\.
## Appendix HExtended ablation discussions

### H\.1Controls on the score and the selection

##### Calibration\-source and score controls\.

Replacing the labeled calibration entirely with held\-out preferences, thresholding at the0\.90\.9quantile of dispreferred\-parent scores, is safe on four of fifteen tasks and unsafe on the rest, including every low\-safe\-rate velocity task, so labeled calibration earns its keep exactly where safety is scarce\. Flipping up to 30 percent of labels, a Boltzmann labeler that errs on borderline pairs, or a budget of 100 pairs degrades filter precision only mildly and never produces a false certification: validity flows from the clean calibration sample, not from score quality\. The same machinery certifies the budget\-native risk, the selection’s mean episodic cost, in a strictly more conservative variant that certifies only on PointGoal1 with zero false certifications\. Finally,Q⁡\(s,a\)Q\(s,a\)under the identical Bradley\-Terry loss leaves four of five representative tasks unchanged and is suggestively worse on Hopper while forfeiting the state\-only value’s policy\-agnostic scoring, and oracle reward\-to\-go through the identical extraction raises reward while cost explodes, so the pipeline transmits whatever signal the value carries\.

Figure 7:Guarantee validation over 2000 calibration draws per cell\. Left: unconditional false\-certification rate against the levelδ=0\.1\\delta=0\.1\(dashed\); the guarantee holds in every task, seed, and calibration size\. Right: certification rate as a function of calibration sizenn; tasks whose selections cannot meetα\\alphacertify rarely and at rates Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)bounds belowδ\\delta\(Ant0\.0060\.006, PointGoal20\.0470\.047, Hopper0\.0000\.000\)\.This appendix carries the selection\-signal control table, the full versions of the ablation discussions summarized in Section[6\.4](https://arxiv.org/html/2609.12014#S6.SS4), and per\-cell detail for the gated variant of Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)\.

##### Selection\-signal controls\.

Table[13](https://arxiv.org/html/2609.12014#A8.T13)reports both matched\-size controls in full\. Return selection buys reward on all fifteen tasks and pays for it with higher cost than random selection on thirteen of the fifteen, so ranking by return is not merely uninformative about safety but anti\-aligned with it in these datasets; random selection’s cost, by contrast, tracks each pool’s base contamination and reads as a no\-information reference for the selection operator\. Table[12](https://arxiv.org/html/2609.12014#A8.T12)reports the score’s Spearman correlations with episodic cost and return per task, and the cost\-return correlation that makes some anti\-alignment with return unavoidable on the navigation tasks\. The V\-filter’s second fraction completes a small sweep over selection size: at a fixed top 25 percent it is safe on ten of fifteen, close to the estimated fraction everywhere except Hopper, whose ground\-truth safe pool is 11 percent of its dataset, so a 25 percent selection necessarily dilutes it \(cost 66 against 18 at the matched size\)\. Safety tracks whether the fraction stays within the task’s safe mass, which is exactly the quantity the calibrated threshold adapts to per task\. The bottom\-return column is the sharpest of the three controls\. Its selected data is genuinely low\-cost \(on HalfCheetah, kept\-set mean trajectory cost 39\.9 against the pool’s 115\.4\), yet its clone runs at cost 109: low\-return trajectories are behaviorally incoherent, and cloning them reproduces neither their returns nor their costs, a second demonstration that training\-set composition alone does not determine policy safety\. Where bottom\-return is safe, it is safe by collapse \(Swimmer, reward 4 at cost 1\) or with thinner margins than the V\-filter at the same selection size \(PointGoal1 cost 20\.1 against 10\.7\), and it breaks on PointGoal2 \(43\.7\) where the V\-filter is safe at 19\.8\. On PointCircle1, where cost and return are nearly collinear \(\+0\.89\+0\.89\), the score itself approaches an anti\-ranking of return \(−0\.95\-0\.95\) and anti\-return selection is accordingly safe there; the velocity tasks are where the two signals separate\.

Table 12:Spearman correlations of the trajectory scoreggwith episodic cost and episodic return \(mean over V\-ensemble seeds\), and between cost and return, per task\.Table 13:Selection\-signal controls at matched selection sizes \(five seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\. Ranking by return, the percentile\-BC analogue, is unsafe on every task; random selection is unsafe on fourteen of fifteen; bottom\-return selection is safe on eight of fifteen and catastrophically unsafe on the velocity tasks\. The safety value is doing the work\.
##### Gated reward\-aware variant: per\-cell detail\.

On the three certified tasks the gated sub\-selection keeps cost within budget in every cell, HalfCheetah 3\.2 to 4\.1, CarGoal1 5\.7 to 5\.6, and PointGoal1 6\.6 to 6\.3, all three within confidence intervals, with reward held within confidence intervals throughout\. Ungated, the identical top\-half sub\-selection applied to Hopper’s contaminated selection multiplies mean cost to 148, reaching 487 on one seed; the certificate’s refusal is what prevents this, the decision\-signal role of Section[4\.4](https://arxiv.org/html/2609.12014#S4.SS4)in action\.

##### Pipeline sanity and the action\-conditioned oracle\.

Two further controls close the analysis\. Feeding oracle reward\-to\-go through the identical extraction machinery, a single\-task control on HalfCheetah over three seeds, raises reward above behavior cloning on every seed while cost explodes, confirming the pipeline transmits whatever signal the value carries; its gentleness, a few percent of movement, is the compositional mechanism of Obstacle 1\. And granting the per\-transition route everything it lacks at once, an action\-conditioned, long\-horizon, generalizing value distilled from full ground\-truth cost labels, scored counterfactually through the dynamics ensemble, still does not reach safety at its documented configuration \(PointGoal1 cost31\.531\.5, CarGoal239\.539\.5, PointGoal249\.649\.6, against budget 25\), and it buys reward rather than safety \(on PointGoal2, reward12\.212\.2against the selection route’s6\.26\.2\)\. Even the maximally endowed per\-transition operator does not produce trajectory\-level safety on these tasks, which is Obstacle 1 in its strongest form\.

##### Action\-conditioned value\.

ReplacingV⁡\(s\)V\(s\)withQ⁡\(s,a\)Q\(s,a\)trained by the identical Bradley\-Terry segment\-sum loss, changing only the input to the concatenation of state and action, leaves four of five representative tasks unchanged \(HalfCheetah, PointGoal1, PointGoal2, CarButton1: safe, comparable reward\)\. On Hopper, the task with the scarcest safe fraction,QQexceeds the budget on two of three seeds \(costs 35\.8, 37\.2, 14\.2; mean 29\.1\) whereVVis safe at18±718\\pm 7over five seeds; given Hopper’s seed variance this is suggestive rather than conclusive, but the direction is consistent with the mechanism we would expect, since conditioning on actions ties the score to the behavior policy’s action choices at exactly the states where selection is hardest\. Action\-conditioning also costs the properties that the state\-only value provides elsewhere in the paper\. The safety landscape and constraint\-saliency analyses of Section[5](https://arxiv.org/html/2609.12014#S5)readVVdirectly as a function of the state, whileQQadmits no such reading without first marginalizing over actions under some policy\. And althoughVVis shaped by the behavior distribution it was trained on, it does not condition on actions, so the same scorer applies to any trajectory regardless of which policy produced it, whereasQQ’s scores are meaningful only on the support of the behavior policy’s actions\.

##### Segment length\.

Rerunning the entire pipeline atH∈\{10,50\}H\\in\\\{10,50\\\}on the nine analysis tasks \(re\-segmentation, preference regeneration, value ensembles, calibrated filter, three seeds each\) leaves the deployed filter’s safe count unchanged at 7 of 9 for everyHH, with the identical failures \(Hopper, Swimmer\)\. Certification is the more sensitive quantity, but three seeds estimate it far too coarsely to compare arms, so we resample20002000calibration draws per task and seed and report rates\. Mean certification rate declines monotonically as segments shorten,0\.2040\.204atH=50H=50,0\.1570\.157atH=30H=30, and0\.1020\.102atH=10H=10, tracking the mean purity margin over the same arms \(−0\.002\-0\.002,−0\.015\-0\.015,−0\.072\-0\.072\)\. This is a decline rather than a collapse, and it is worth stating that the realized three\-seed counts suggest otherwise: PointGoal1 happened to certify on none of its threeH=10H=10seeds despite a true rate of0\.690\.69, which is the kind of artifact a seed count produces and a resampled rate does not\. Across all2727task\-length cells the margin predicts the rate almost exactly \(Spearman0\.9880\.988,p<10−20p<10^\{\-20\}\), an independent replication of the law of Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)under a design change rather than across tasks: cells with negative margin certify at mean rate0\.0150\.015and never above0\.0570\.057, while positive\-margin cells average0\.2840\.284\. The guarantee holds in every arm, with worst unconditional false\-certification0\.0440\.044,0\.0570\.057, and0\.0540\.054againstδ=0\.1\\delta=0\.1\. Segment length is therefore not a load\-bearing choice for whether the resulting policy is safe, and it moves certification only through the purity it leaves attainable\.

Figure 8:The certification operating curve on both suites\. Certification rate against the guarantee levelα\\alpha\(n=200n=200; resampled calibration draws; mean over V\-ensemble seeds\)\. Rates rise withα\\alphain the order of each pool’s safe mass; where a pool cannot meet a level, the rate there stays at or near zero and within theδ\\deltathe guarantee prices\.

### H\.2Properties of the certificate

##### Conditional false\-certification\.

Table[14](https://arxiv.org/html/2609.12014#A8.T14)reports, per task atn=200n=200andα=0\.25\\alpha=0\.25from the 2000\-draw resample, the certification rate, the unconditional false\-certification rate the theorem bounds, and the conditional violation rate a certificate\-holder faces\. The unconditional rate is belowδ\\deltaeverywhere; the conditional rate is governed by the certification rate and ranges from0\.070\.07\(PointGoal1\) to near one on tasks certifying in under five percent of draws, where the handful of certificates that do fire are disproportionately the optimistic draws\. The practical reading is in Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)\. Certification rate is itself a trust signal\.

Table 14:Certification rate, unconditional false\-certification, and conditional violation rate atn=200n=200,α=0\.25\\alpha=0\.25\(2000 draws per seed, seed means; final column: certified draws pooled over seeds\)\.
##### Multiple\-testing procedures over the grid\.

Fixed\-sequence testing could in principle be beaten by procedures that reach deep thresholds without passing shallow ones\. On the archived scores \(2000 draws per cell\), Bonferroni and Holm at levelδ\\deltaover the twelve\-point grid certify far less everywhere, PointGoal1 at0\.540\.54against fixed\-sequence’s0\.860\.86and near zero elsewhere, with validity intact for all three\. Theδ/J\\delta/Jpenalty costs more than deep\-threshold access returns atn=200n=200\. Certification yield is limited by per\-test power at this calibration size, not by the choice of procedure, and the deployed fixed\-sequence walk is the power\-optimal option among those tested\.

##### Label complexity of certification\.

Table[15](https://arxiv.org/html/2609.12014#A8.T15)reports the calibration budget at which each task first certifies in half of draws, obtained by extending the resampling sweep ton∈\{800,1600,3200\}n\\in\\\{800,1600,3200\\\}on the tasks with positive purity margin\. Regressing the crossing point on the margin giveslog⁡n50=−1\.57​log⁡\(margin\)\+1\.72\\log n\_\{50\}=\-1\.57\\log\(\\text\{margin\}\)\+1\.72withR2=0\.996R^\{2\}=0\.996\. Two tasks with small positive margins \(Swimmer at0\.0310\.031, Walker2d at0\.0140\.014\) do not reach half certification even atn=3200n=3200, consistent with the fit’s prediction of budgets in the thousands; every task with a non\-positive margin certifies at essentially zero for every budget, as Proposition[2](https://arxiv.org/html/2609.12014#Thmproposition2)requires\.

Table 15:Calibration budgetn50n\_\{50\}at which certification reaches half of draws, against the purity margin \(nine analysis tasks and five transfer tasks; resampling atnnup to32003200\)\.
##### A two\-tier fallback, considered and rejected\.

When no threshold certifies, the procedure returns the fraction given by a Clopper\-Pearson lower bound on the pool’s safe mass\. A natural alternative reruns the identical fixed\-sequence test at even\-odds evidence \(δ=0\.5\\delta=0\.5\) and takes the deepest passing threshold\. We reject it on principle\. Relaxing the evidence standard buys a deeper selection by abandoning the level at which the guarantee is stated, and the result is still returned as uncertified, so the procedure acquires the appearance of a test without its protection; theα\\alphalever relaxes the target instead and keeps a certificate\. Its scores do not argue against it\. Across nine tasks it changes no safety verdict except Swimmer, which it converts \(44\.744\.7to9\.39\.3\), though it raises mean cost on five of the nine, including PointCircle2 \(87\.287\.2to112\.5112\.5\) and Hopper \(22\.922\.9to32\.932\.9\)\.

##### Mean\-cost certificate\.

The same machinery certifies a budget\-native risk directly, by replacing the unsafe\-fraction risk with the mean episodic cost of the selection \(Hoeffding\-without\-replacement p\-values, target, namely selection mean cost at or below the budget\) yields a strictly more conservative variant that certifies only on PointGoal1 \(5 to 8 percent of draws atn=200n=200\) and records zero false certifications anywhere\. The unsafe\-fraction form remains the practical operating certificate; the mean\-cost form documents that the framework extends to CMDP\-native risks without modification\. It also admits a loose bridge to policy cost: writingπS\\pi\_\{S\}for the behavior policy of the selection andϵ\\epsilonfor the cloning error in average total variation, standard imitation arguments giveJc​\(π\)≤Jc​\(πS\)\+2​T2​cmax​ϵJ\_\{c\}\(\\pi\)\\leq J\_\{c\}\(\\pi\_\{S\}\)\+2T^\{2\}c\_\{\\max\}\\,\\epsilon, andJc​\(πS\)J\_\{c\}\(\\pi\_\{S\}\)concentrates around the selection mean cost that this certificate bounds\. TheT2T^\{2\}compounding makes the bound far too loose to certify in practice, but it separates the remaining gap into exactly two quantities\. The certified selection mean and the cloning error\.

### H\.3What the supervision buys

##### Preference noise and budget\.

Flipping preference labels degrades the pipeline gracefully and never breaks the certificate\. Across noise levels up to 30 percent and budgets down to 100 pairs, the worst false\-certification rate remains atδ\\delta, because validity flows from the clean calibration sample rather than from score quality\. Filter precision is nearly flat under noise on the navigation tasks and degrades mildly on velocity tasks at the highest levels; 100 preference pairs already match the full budget within a few points of precision on every analysis task\. One effect deserves note: on Walker2d, mild label noise*improves*the score \(precision0\.770\.77to0\.860\.86, certification rate0\.010\.01to0\.760\.76at five percent noise\), acting as a regularizer on precisely the score tail whose pathology the threshold grid works around; the guarantee holds throughout\. A Boltzmann labeler, which errs preferentially on borderline pairs \(P⁡\(prefer safer\)=logistic⁡\(cost gap/Tlab\)P\(\\text\{prefer safer\}\)=\\mathrm\{logistic\}\(\\text\{cost gap\}/T\_\{\\mathrm\{lab\}\}\),Tlab=3T\_\{\\mathrm\{lab\}\}=3, realized error rates 5 to 6 percent\), leaves navigation\-task precision unchanged within±0\.01\\pm 0\.01and degrades velocity tasks mildly \(at worst−0\.20\-0\.20on Swimmer\), again without a single false certification: structured, boundary\-concentrated labeler error behaves no worse for the certificate than uniform flips\.

##### Labels\-only filtering\.

This control puts the calibration sample to that alternative use, training a supervised state\-level safety classifier on only thennlabeled trajectories, using the same architecture, trajectory scoring rule, and matched selection fractions as the preference route, and clones its selection\. Across label budgetsn∈\{50,100,200,400\}n\\in\\\{50,100,200,400\\\}it is safe on 11, 12, 11, and 13 of fifteen tasks, competitive on average, and evidence that filter\-then\-clone, not the preference machinery alone, carries much of the safety on this benchmark\. Two properties separate the routes\. First, which tasks come out safe under labels\-only depends on the label draw\. Its safe\-task set overlaps across independent draws at a mean Jaccard of 0\.85, dropping to 0\.61 atn=100n=100, with Swimmer’s cost ranging from 21 to 174 across budgets, Hopper safe only atn=400n=400, and PointButton2 and PointCircle2 alternating without pattern\. The variation is attributable to the draw rather than the learner, retrained with three initializations on one fixed draw, the labels\-only safe\-task set overlaps at 0\.95, above the preference route’s 0\.85 across value\-ensemble seeds, so safe\-set stability is not an advantage of the preference route\. What the draw sensitivity does mark is that the labels\-only deliverable depends on whichnntrajectories were labeled, and a practitioner holding one draw cannot tell which tasks failed; the preference route’s score does not consume the labels at all\. Second, at every budget the labels\-only route consumes its labels in training, so no valid certificate or refusal can be issued from the same sample; matching our deliverable would require a second labeled sample of equal size\. The preference route separates the two supervision roles by construction\. Preferences carry the scoring, and the labels remain held out to certify or refuse\. Splitting the label budget between the two roles completes the comparison, with our exact total budget divided 100 for training and 100 for calibration, the labels\-only route certifies at a rate comparable to ours \(9 against 11 percent of seed\-runs\) and is equally safe, and at double the budget it certifies more often \(24 percent\) and again matches on safety\. The certifications concentrate on disjoint task sets, ours on HalfCheetah, CarGoal1, and PointGoal1 and the split’s on Walker2d and PointCircle1, so the two scorers are complementary rather than redundant\. Each concentrates certifiable purity where the other’s score has a weak tail\. At matched label budget, preferences therefore buy neither additional safety nor additional certification on this benchmark; their measured advantage is the graded selection on Hopper, and their structural advantage is that the budget judgments remain held out, so the sample that scores can also certify\. Three kinds of judgment appear in this study:*ordinal*, which of two clips is safer;*binary*, whether one episode exceeded its budget; and*cardinal*, the numeric cost of a transition\. Labels\-only and our calibration rest on the identical binary judgment, and the routes differ in how each spends it: labels\-only consumes its labels in training, so a certificate needs a second sample or a split of the first, while the preference route carries scoring on ordinal comparisons and leaves every budget judgment held out\. BC\-Safe needs that same binary judgment on every trajectory rather than on 200; the offline safe\-RL baselines go further and presume a cardinal cost on every transition\. Which route is available is a property of the application, not of the budget\. Appendix[I](https://arxiv.org/html/2609.12014#A9)reports the full sweep\.

##### Cardinal supervision at matched budget\.

The three judgments are ordered, and the labels\-only route occupies the middle rung, consuming the same binary over\-budget bit the calibration sample does and leaving the cardinal rung untested\. This control tests it\. Everything is held fixed against the labels\-only arm, the same script, the same200200trajectories, the same per\-task selection fraction, architecture, seeds, clone and evaluation; only the target changes, from the bit to the standardised numeric episodic cost, with a regression loss in place of the classification one\. The registered prediction, that magnitudes buy nothing at matched budget if safety is decodable from the ordering alone, holds\. Table[16](https://arxiv.org/html/2609.12014#A8.T16)reports both arms: the cardinal selection is*less*pure on eleven of the fifteen tasks, by0\.0240\.024on average, and its clone is safe on twelve tasks against the bit route’s eleven, a difference of one task with flips in both directions and well inside the draw\-to\-draw variation of Appendix[I](https://arxiv.org/html/2609.12014#A9)\. Supervision richness is therefore not the binding choice either, which is what makes the readout scale of Section[5](https://arxiv.org/html/2609.12014#S5)the one that is\.

Table 16:Binary against cardinal supervision at matched budget: the same filter trained on the over\-budget bit and on the numeric episodic cost of the same200200trajectories \(three seeds; kept\-unsafe is the true unsafe fraction of the selection, lower better; bold marks clone cost within budget\)\.
##### Why labels\-only works\.

An episodic label is not one bit\. It weakly labels every state of a thousand\-step trajectory, so 200 trajectory labels yield roughly 200,000 labeled states\. A subsampling test shows this multiplication is not the mechanism\. Shrinking the states seen per labeled trajectory a hundredfold, to roughly 2,000 states in total, degrades selection purity only from 0\.29 to 0\.35 kept\-unsafe and leaves the safe\-task count essentially unchanged \(13 and 10 of fifteen at tenfold and hundredfold reduction, against 11 at full\)\. Labels\-only is strong because trajectory\-level safety in these environments is carried by low\-dimensional, strongly separable state features, the same structure the saliency and landscape analyses of Section[5](https://arxiv.org/html/2609.12014#S5)identify\. This answers the paper’s opening question in its sharpest form\. Safety in these tasks is decodable from state alone, easily enough that the discriminating questions are not whether a scorer can be learned but whether its selections can be certified and how little supervision, of what kind, the certification requires\.

## Appendix ILabels\-only control across label budgets

Table[17](https://arxiv.org/html/2609.12014#A9.T17)reports the labels\-only filtering control of Section[6\.4](https://arxiv.org/html/2609.12014#S6.SS4)at every label budget \(three seeds per cell; the classifier, scoring rule, and matched selection fractions are identical to the preference route\)\. Safe\-task counts are stable in aggregate while the membership of the safe set shifts across budgets, which is the instability the main text describes\. In the split variants the certified population is the dataset minus the classifier’s training trajectories, over which the calibration half is exactly a uniform without\-replacement sample, so Proposition[1](https://arxiv.org/html/2609.12014#Thmproposition1)applies unchanged; training trajectories that are individually labeled safe are then added to the cloned selection, which can only lower its unsafe fraction below the certified level\.

Table 17:Labels\-only filtering control across calibration budgetsnn\(three seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\.
## Appendix JComposability and contamination tolerance

On the two navigation tasks CDT on the certified selection trails our own clone \(8\.4 against 9\.0 on CarGoal1; 4\.4 against 7\.5 on PointGoal1\), so the certified data costs it reward there, and atα=0\.40\\alpha=0\.40PointGoal1 moves from cost 5\.4 to 41\.5 as realized contamination rises\. Table[18](https://arxiv.org/html/2609.12014#A10.T18)reports the composability comparison at the deployed levelα=0\.25\\alpha=0\.25that Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)summarizes: full\-data CDT with and without augmentation, CDT retrained on the certified selection, our calibrated clone, certified return\-weighted cloning, and CPL on the same certified selection\.

Table 18:Composability atα=0\.25\\alpha=0\.25: CDT and our certified operator on the deployed certified selection per task, three learner seeds each, with the two further distinct selections per task in Table[19](https://arxiv.org/html/2609.12014#A10.T19)\. The clone column is our deployed pipeline end to end over its five seeds, so it is scored on the selection each seed produces rather than on the single selection the CDT and operator columns share\. CDT’s frontier\-fitting augmentation presumes a full cost spectrum and cannot run on curated data, so certified runs disable it; the no\-augmentation column controls for that on full data\. The operator column is certified return\-weighted cloning atκ=2\\kappa=2\. CPL is given the same certified selection as CDT, with its preference budget redrawn inside it, so both external learners are scored on the same data\. Bold cost is within budget\.Table[20](https://arxiv.org/html/2609.12014#A10.T20)reports CDT, our calibrated clone and CPL atα=0\.40\\alpha=0\.40for the eight tasks that certify at that level \(three seeds each; the pipeline’s deployed certified selection per task, with the full three\-selection grid in Table[21](https://arxiv.org/html/2609.12014#A10.T21); Hopper never certifies\), alongside full\-data CDT with and without augmentation for reference\. Our clone is safe on all eight; CDT is safe on six and violates on PointGoal1 and PointGoal2, the tasks whose selections carry the largest usable unsafe tails\. The no\-augmentation control seals the attribution across the table: full\-data CDT without augmentation is safe on the velocity tasks but severely unsafe on all four navigation tasks \(costs 47 to 109\), so wherever certified\-selection CDT is safe on navigation, the safety is due to the certified data and not to the disabled augmentation\. Against full\-data CDT, theα=0\.40\\alpha=0\.40selection lowers cost on six of the eight tasks and leaves Walker2d unchanged near zero; the exception is PointGoal1, where the loosely curated pool is worse than no curation at all \(41\.5 against 36\.8\), consistent with the selection concentrating high\-return marginal violators that return conditioning then exploits\. Against the tighter level on the three shared tasks,α=0\.25\\alpha=0\.25is safer for CDT on both navigation tasks \(costs 5\.5 and 5\.4 against 15\.2 and 41\.5\) and equivalent on HalfCheetah \(1\.8 against 0\.6, both far within budget\), which is the dose\-response the main text summarizes\.

##### Selection robustness across distinct certified selections\.

Independent calibration draws frequently select the identical threshold, with repeated draws coinciding for five of the eight tasks atα=0\.40\\alpha=0\.40and two of three atα=0\.25\\alpha=0\.25, and a 500\-draw resample shows the certified threshold concentrating, on Swimmer’s top quantile in 85 percent of certified draws\. The concentration is itself a deployment\-stability property, so the grids are built on the three most probable*distinct*certified selections per task, ordered by realized contaminationα^\\hat\{\\alpha\}\(Table[21](https://arxiv.org/html/2609.12014#A10.T21)\); the distribution’s rare tail contains the certificate’s worst outputs, and the grids deliberately include them\. Atα=0\.25\\alpha=0\.25, all nine distinct selections across the three tasks are CDT\-safe, 27 of 27 runs \(Table[19](https://arxiv.org/html/2609.12014#A10.T19)\), including three out\-of\-specification selections \(HalfCheetah at realized0\.2870\.287and0\.3510\.351, costs0\.030\.03and0\.770\.77; CarGoal1 at0\.2660\.266, cost5\.935\.93\), with per\-task mean costs spanning at most5\.25\.2\. Atα=0\.40\\alpha=0\.40every learner degrades with realized contamination, at very different thresholds\. CDT violates on every navigation task on at least one selection, CarGoal1 moving safe, borderline, unsafe as contamination rises \(0\.1870\.187,0\.3530\.353,0\.3750\.375\)\. CPL, safe on all 27 runs atα=0\.25\\alpha=0\.25, is safe on only 16 of 24 here, failing on all three Walker2d selections, both Swimmer tail selections and three navigation selections, so the dose\-response that justifies the tighter level holds for the second external learner as well as the first\. Our clone, an averaging learner, absorbs far more: it is safe on 22 of 24 task\-selections \(Table[22](https://arxiv.org/html/2609.12014#A10.T22)\); its two failures are Swimmer’s tail selections at realized0\.3820\.382and0\.4620\.462, the latter above even the certified level, so the averaging learner’s tolerance has a measured boundary rather than being unbounded, and on the0\.3820\.382selection the roles reverse, CDT safe at18\.318\.3while the clone violates at23\.623\.6\. One exception is honest to note\. On PointGoal2 the ordering is not monotone in contamination \(26\.526\.5,23\.223\.2,28\.528\.5at contamination0\.2690\.269,0\.3350\.335,0\.3820\.382\), so selection size interacts with composition and the contamination reading is a strong tendency, not a law\. The composite picture stands\. At the tight level realized composition is pinned and every distinct selection composes, while at the loose level the certified band is wide enough that consumer safety depends on which selection the certificate outputs\.

Table 19:Distinct\-selection grid atα=0\.25\\alpha=0\.25: the three distinct certified selections per task for the three tasks that certify at this level, with the reward and mean cost of CDT and of CPL on each identical selection \(three seeds each; bold marks mean cost within budget\)\. All 27 CDT runs and all 27 CPL runs are within budget, including the three selections whose realized contaminationα^\\hat\{\\alpha\}exceeds the certified level\.Table 20:Composability atα=0\.40\\alpha=0\.40on the deployed certified selection per task \(three seeds, mean±\\pm95% CI; bold marks mean cost within budget\)\. Column naming follows Table[18](https://arxiv.org/html/2609.12014#A10.T18): the clone column is our deployed pipeline end to end, so it is scored on the selection each seed produces rather than on the single selection the CDT and CPL columns share\. The operator column of Table[18](https://arxiv.org/html/2609.12014#A10.T18)has no counterpart here because certified return\-weighted cloning is characterized atα=0\.25\\alpha=0\.25\(Table[23](https://arxiv.org/html/2609.12014#A10.T23)\)\. Both external learners, CDT and CPL, are scored on the same selection as our own clone, with CPL’s preference budget redrawn inside it\.Table 21:Distinct\-selection grid atα=0\.40\\alpha=0\.40: the three most probable distinct certified selections per task, ordered by increasing realized contaminationα^\\hat\{\\alpha\}, with the reward and mean cost of CDT and of CPL on the identical selection, matching Table[19](https://arxiv.org/html/2609.12014#A10.T19)\(three seeds each; bold marks mean cost within budget\)\. The pipeline’s deployed selection per task, whose position in the ordering varies, is the one reported in Table[20](https://arxiv.org/html/2609.12014#A10.T20)\. PointGoal2’s selections are drawn at pipeline seed 1 because seed 0 does not certify at this level, and a certified selection cannot be reported from a draw that refused; the refused draw’s clone would itself have been safe \(cost21\.021\.0against budget 25\), so the substitution is conservative rather than favorable\.
##### Contamination tolerance of the clone\.

The composability tables \(Tables[18](https://arxiv.org/html/2609.12014#A10.T18),[19](https://arxiv.org/html/2609.12014#A10.T19),[20](https://arxiv.org/html/2609.12014#A10.T20)and[21](https://arxiv.org/html/2609.12014#A10.T21)\) report external learners on certified data\. The clone is our own consumer, not an external one, so its behavior across the same selections is a learner\-sensitivity control rather than a composability result, reported separately in Table[22](https://arxiv.org/html/2609.12014#A10.T22): it measures how much contamination an averaging learner absorbs, which is what sets the level the external learners need\. The comparison is against the tight level, where the clone is within budget on every certifying task \(Table[18](https://arxiv.org/html/2609.12014#A10.T18)\); what Table[22](https://arxiv.org/html/2609.12014#A10.T22)measures is how far that tolerance extends once the certified band is widened\. That table repeats neither CDT nor CPL, which Table[21](https://arxiv.org/html/2609.12014#A10.T21)already gives on the identical selections, and its clone differs from the calibrated column of Table[20](https://arxiv.org/html/2609.12014#A10.T20), which is the pipeline end to end on the selection each seed produces rather than cloning on a fixed given selection\.

Table 22:Contamination tolerance of the clone: behavior cloning on the identicalα=0\.40\\alpha=0\.40selections of Table[21](https://arxiv.org/html/2609.12014#A10.T21), ordered by increasing realized contaminationα^\\hat\{\\alpha\}\(three seeds each; bold marks mean cost within budget\)\.
##### The composition\-to\-cost mapping, measured\.

Figure[9](https://arxiv.org/html/2609.12014#A10.F9)plots clone cost over budget against realized contamination for 235 archived selection\-clone pairs spanning certified selections, ground\-truth and control selections, and both V\-filter fractions across the fifteen main tasks, with a single archived CarRun pair included\. Task\-demeaned, contamination alone explainsR2=0\.38R^\{2\}=0\.38of clone cost, so the mapping the paper calls empirically favorable is a measured, monotone tendency rather than an assertion; the remaining variance is carried partly by the selection’s score margin \(adding margin statistics raisesR2R^\{2\}to0\.500\.50\) and partly by learner effects the composability grids characterize\.

Figure 9:Clone cost against realized selection contaminationα^\\hat\{\\alpha\}, 235 selection\-clone pairs, all but one from the fifteen DSRL tasks \(colors: tasks; dashed line: budget; symlog cost axis\)\.
##### The return\-weighted operator across intensities\.

Table[23](https://arxiv.org/html/2609.12014#A10.T23)reports certified return\-weighted cloning \(Section[4\.4](https://arxiv.org/html/2609.12014#S4.SS4)\) on the same distinctα=0\.25\\alpha=0\.25selections, at clipsκ∈\{1,2,3\}\\kappa\\in\\\{1,2,3\\\}and the hard top\-half rule, plus the CarRun certified selection from the transfer suite\. The grid substantiates the two claims made in Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)\. Every within\-specification cell is safe at every intensity, with reward on HalfCheetah rising in the return\-seeking intensity, and the single selection whose realized contamination substantially exceeds the certified level \(0\.3510\.351\) is safe only atκ=1\\kappa=1, with the stronger settings mining the contaminated return tail exactly as the gating rationale predicts\. The milder out\-of\-specification selections \(0\.2660\.266,0\.2870\.287\) remain safe in every arm, so the failure requires both a contaminated selection and an aggressive intensity; the certificate removes the first factor and thereby makes the second free to tune\.

Table 23:Certified return\-weighted cloning across intensities on the distinctα=0\.25\\alpha=0\.25certified selections and the CarRun echo selection \(three seeds, mean±\\pm95% CI; bold marks mean cost within budget;α^\\hat\{\\alpha\}is the selection’s realized contamination\)\.

## Appendix KCDT cost\-target sweep

CDT conditions on a target cost at deployment, so reporting it at a single target risks understating it\. We retrain CDT on full data at each task’s budget \(three seeds, identical protocol\) and evaluate every checkpoint across cost targets down to a fifth of the budget, 100 episodes per cell \(Tables[24](https://arxiv.org/html/2609.12014#A11.T24)and[25](https://arxiv.org/html/2609.12014#A11.T25)\)\.

The sweep separates two regimes\. Where the pool admits it, conditioning below budget delivers real safety at high reward\. All five velocity tasks, both Circle tasks, and, at target 2, all five BulletSafetyGym tasks, in each case with reward above every filter in the study\. Where safety requires selectivity, the knob does not work at all: on the eight Goal and Button tasks CDT violates at every target, and realized cost is nearly flat in the target, moving from40\.940\.9to37\.837\.8on CarGoal1 and sitting near100100on PointButton1 across the whole range against budget 25\. Figure[10](https://arxiv.org/html/2609.12014#A11.F10)shows the two families side by side\. The entire CDT curve lies right of the budget on the Goal tasks, whichever target is chosen, while ourα\\alphasweep traverses a frontier left of it\. The gap between a conditioning target and realized cost is itself the diagnosis: CDT fits a return\-cost frontier that these datasets do not contain, so asking for a lower cost cannot produce trajectories the data never demonstrated\. This is the same reading the granularity analysis gives from the other direction, and it is why retraining CDT on a certified selection succeeds on exactly the tasks where its own conditioning fails \(Section[6\.2](https://arxiv.org/html/2609.12014#S6.SS2)\)\.

Figure 10:Reward against cost over budget on the nine analysis tasks \(dashed line: the budget\)\. Blue: CDT on full data across its cost\-target sweep, one point per target \(values in Table[24](https://arxiv.org/html/2609.12014#A11.T24)\)\. Orange: our calibrated filter acrossα∈\{0\.10,0\.25,0\.40\}\\alpha\\in\\\{0\.10,0\.25,0\.40\\\}, tighterα\\alphamoving down\-left\. Teal: the V\-filter at the calibration\-estimated fraction\. Green: BC\-Safe with full labels\. On the Goal tasks the entire CDT curve sits right of the budget line, so no target choice reaches the constraint, while theα\\alphasweep traverses a genuine frontier left of it, at or inside BC\-Safe’s cost on every navigation task\.Table 24:CDT on full data across cost targets, fifteen DSRL tasks \(reward/cost, three seeds, 100 episodes per cell; bold marks mean cost within the task budget of 20 or 25\)\. The final column is the budget\-matched target; velocity tasks have no target 15\.Table 25:CDT on full data across cost targets, five BulletSafetyGym tasks \(reward/cost, three seeds; bold marks mean cost within budget 10\)\.
## Appendix LTransfer to BulletSafetyGym

The V\-filter satisfies the budget on four of the five, the calibrated procedure on all five\. Full\-data CDT also meets this budget on all five once its cost target is conditioned to 2, at reward well above every filter, so on this suite the harder budget separates the methods by labeling cost rather than by safety\. Table[26](https://arxiv.org/html/2609.12014#A12.T26)reports the pipeline on five BulletSafetyGym tasks \(BallRun, BallCircle, CarCircle, CarRun, DroneRun; 651 to 1990 trajectories; episode lengths 100 to 300; DroneRun is the one dataset in the study with true terminal states\) at budget 10, the strictest of the DSRL protocol’s three evaluation thresholds \(10, 20, 40\), with every hyperparameter unchanged from the main study and no per\-task tuning\. The slice is hard: ground\-truth safe fractions are 9\.0, 9\.7, 11\.4, 46\.2, and 28\.0 percent, and the 200 calibration labels are 10 to 31 percent of these smaller pools, extending the main study’s 5\-to\-22\-percent range\. Short episodes are a different regime rather than a harder one for segment supervision, since a length\-30 segment covers up to 30 percent of an episode; episode lengths are fixed except on DroneRun, where the score is essentially uncorrelated with length \(ρ=0\.13\\rho=0\.13\)\. The score transfers\. Its Spearman correlation with episodic cost is−0\.81\-0\.81to−0\.94\-0\.94on every task, and it is doing the work\. Random selection at the identical matched fractions is unsafe on four of five tasks \(costs 32 to 79\), safe only on CarRun’s 46\-percent\-safe pool\. The strongest full\-label baseline behaves differently here than on the main suite: full\-data CDT at the budget\-matched target 10 is unsafe on four of five tasks \(costs 11\.1 to 14\.5\), but conditioning its target down to 2 makes it safe on all five \(costs 0\.1 to 9\.4\) at reward far above every filter \(Appendix[K](https://arxiv.org/html/2609.12014#A11)\)\. Unlike the Goal and Button tasks, where the same sweep leaves cost flat and far above budget, these pools are conditioning\-tractable, so the transfer suite tests the pipeline’s portability rather than exhibiting a baseline failure\. Cloning everything is unsafe on four of five tasks \(costs 24 to 65\); the calibrated procedure satisfies the budget on all five, one of them \(DroneRun\) by the reward collapse just described, and the estimated\-fraction V\-filter on all five as well\. Certification concentrates where safe mass supports it\. CarRun certifies on five of five seeds, DroneRun on one, and the remaining three refuse on every seed, with the 2000\-draw validation holding in every cell \(worst unconditional false\-certification rate0\.060\.06, 95 percent Clopper\-Pearson0\.0500\.050to0\.0710\.071\)\. The operating curve transfers as well \(Figure[8](https://arxiv.org/html/2609.12014#A8.F8)\): resampling the calibration at looser levels, certification rates rise in safe\-mass order, DroneRun from0\.200\.20atα=0\.25\\alpha=0\.25to0\.870\.87at0\.400\.40, BallRun from0\.000\.00to0\.060\.06over the same range, while CarRun certifies almost always \(0\.990\.99atα=0\.25\\alpha=0\.25\) and BallCircle and CarCircle never, so theα\\alphalever behaves on the new domain exactly as on the main suite\.

One cell carries the section’s real limitation\. On DroneRun every filter, ours and the full\-label one, drives cost to zero, but reward falls from383383under unfiltered cloning to160160for BC\-Safe and4242for the calibrated filter\. Safety here is bought by collapse rather than by selection, and the pipeline has no mechanism for noticing that its selection has become behaviorally degenerate while still satisfying the budget, a limitation this suite exposes for the first time\. The DroneRun lesson does not: a pre\-registered check ran the certification machinery on the trivial selectionS=𝒟S=\\mathcal\{D\}, in both its composition and mean\-cost forms, and it fires nowhere, because DroneRun’s pool has mean cost 50\.5 against budget 10 and 72 percent unsafe trajectories while its clone is safe by policy averaging\. Neither composition\-level test can detect that filtering is unwarranted; the detection appears to require a policy\-level signal, which is the composition\-to\-policy gap of Section[6\.3](https://arxiv.org/html/2609.12014#S6.SS3)in its converse direction\.

##### Composability is regime\-dependent\.

A pre\-registered rule ran the composability echo on CarRun once full\-data CDT violated there\. The original certified selection for this echo was lost, so it was re\-derived by the same certification procedure from the current value ensemble, which yields a purer and smaller selection than the one first used \(realized contamination0\.0510\.051over 98 trajectories,1515percent of the pool, against0\.1380\.138over roughly half\)\. On it, behavior cloning is safe at cost0\.650\.65across three seeds\. CDT was unsafe on all three seeds of the original selection \(11\.411\.4to19\.619\.6against budget 10\)\. On the regenerated selection it violates on one seed of three \(0\.00\.0,7\.07\.0and11\.111\.1\), so the purer selection helps without taming it \(Table[23](https://arxiv.org/html/2609.12014#A10.T23)\)\. Unlike the main suite, where certified selections tamed CDT in all 27 runs, curation is not sufficient here, and the reason is diagnostic: on CarRun even cloning the whole pool is safe \(BC\-All at cost0\.00\.0\), so CDT’s violation is caused by its operator rather than by the data, and curation cannot remove a violation the data did not cause\. The certified return\-weighted operator completes the diagnosis\. On the same selection the three return\-weighted intensities are safe on every seed \(reward 480 to 513, cost at most3\.73\.7, Table[23](https://arxiv.org/html/2609.12014#A10.T23)\), so return\-seeking as such is not what violates on CarRun; extrapolative return conditioning is\. The hard top\-half rule is the exception, violating on one seed of three \(cost 30\.0\), which locates the boundary at truncation rather than at return\-seeking: resampling preserves the selection’s composition where hard truncation to the 49 highest\-return trajectories does not\. Across the four composability cases the pattern is consistent\. Certified curation cures the full\-label learner exactly where cloning the full pool is itself unsafe \(all three main\-suite tasks, BC\-All costs 34 to 86\), and cannot where the pool is benign and the operator alone violates, the boundary the granularity analysis predicts\.

Table 26:BulletSafetyGym transfer at budget 10 \(mean±\\pm95% CI where multiple seeds; bold marks mean cost within budget; BC\-All deterministic; CDT trained on the full data and evaluated at the budget\-matched target of 10, with its tighter targets in Table[25](https://arxiv.org/html/2609.12014#A11.T25)\)\. CarRun certifies on five of five seeds; every other task refuses on every seed\.

## Appendix MHyperparameters

Value ensembles:K=3K=3, two\-layer MLPs of width 256, 300 epochs, batch 512, Adam3×10−43\\times 10^\{\-4\}, 1000 preference pairs of length 30 per task\. Preference pairs are sampled from parent trajectories in the bottom and top quartiles of episodic cost, and labeled by the two segments’ own summed costs, ties skipped\. Behavior cloning: 100 epochs, batch 512\. CPL baseline and CPL composability arms: behavior\-cloning pretraining for 50 epochs then 300 epochs of CPL, batch 512, Adam3×10−43\\times 10^\{\-4\}, BC regularization weight0\.10\.1, temperature11, conservative bias0\.50\.5, on the same 1000 preference pairs per task the value ensemble consumes\. On a certified selection the same budget of 1000 pairs is redrawn from within the selection, since pairs are sampled to contrast episodic cost and almost none of the pool’s pairs have both endpoints inside a selection\. Calibration:α=0\.25\\alpha=0\.25,δ=0\.1\\delta=0\.1,n=200n=200, threshold grid at score quantiles\{0\.85,0\.80,…,0\.30\}\\\{0\.85,0\.80,\\dots,0\.30\\\}, uncertified fallback fraction from the one\-sided Clopper\-Pearson lower bound \(levelδ\\delta\) on pool safe mass, minimum selection 50 trajectories\. The grid starts at0\.850\.85rather than deeper in the tail because above it too few calibration trajectories clear the threshold for the exact test to reject atδ\\delta\. All200200labels are spent before the threshold is known, and the uncertified fallback uses every one, but the accepted test rests on themmthat land above the chosen threshold:m=28m=28,3131and2828on HalfCheetah, CarGoal1 and PointGoal1, givingp=0\.014p=0\.014,0\.0680\.068and0\.0430\.043againstδ=0\.1\\delta=0\.1\. Reward\-aware sub\-selection: top 50 percent by episodic return, applied only under certification\. Certified return\-weighted cloning, meaning trajectory resampling with probability proportional toexp⁡\(clip⁡\(z⁡\(R\),±κ\)\)\\exp\(\\mathrm\{clip\}\(z\(R\),\\pm\\kappa\)\), return standardized within the selection, selection size preserved,κ∈\{1,2,3\}\\kappa\\in\\\{1,2,3\\\}reported withκ=2\\kappa=2in Table[18](https://arxiv.org/html/2609.12014#A10.T18), applied only under certification\. Advantage\-weighted extraction \(analysis section\): weightsexp⁡\(A/β\)\\exp\(A/\\beta\)withβ=1\\beta=1, weight clip 20, advantage windowh=1h=1unless stated\. Evaluation: 100 episodes, deterministic policy mean\. Sensitivity: averaged over its members, a single network matches theK=3K=3filter precision within0\.0060\.006on all nine analysis tasks, though individual members vary by up to0\.0750\.075, so the ensemble size is not load\-bearing for selection and serves the aggregation diagnostics; a segment\-length sweep appears in Appendix[H](https://arxiv.org/html/2609.12014#A8)\.