StepCOPS：基于闭式测试下限证书的语言模型策略选择

# Closed-Testing Lower-Tail Certificatesfor Language-Model Policy Selection
Source: [https://arxiv.org/html/2609.29549](https://arxiv.org/html/2609.29549)
## StepCOPS: Closed\-Testing Lower\-Tail Certificates for Language\-Model Policy Selection

Ibne Farabi Shihab††thanks:Equal contribution\.††thanks:Corresponding author:ishihab@iastate\.edu\.Sanjeda Akter11footnotemark:1Affiliation:Department of Computer Science, Iowa State UniversityAnuj SharmaAffiliation:Department of Civil, Construction & Environmental Engineering, Iowa State University

###### Abstract

Post\-training pipelines must select one language\-model policy from many checkpoints, prompts, and decoding rules\. Mean evaluator scores can conceal rare failures, whereas simultaneous candidate\-wise confidence bounds can be unnecessarily conservative\. We introduce*StepCOPS*, which uses an independent proposal split to nominate one lower\-tail floor per candidate, exact binomial tests on a fresh certification split, and Holm’s step\-down procedure to certify a set of floors\. With probability at least1−δ1\-\\delta, every certified floor—and therefore the selected maximum—is below its candidate’s population lowerα\\alpha\-quantile, assuming i\.i\.d\. evaluation units but allowing arbitrary within\-unit dependence across candidates\. On2424predeclared configurations and1111benchmarks, StepCOPS obtains96\.4%96\.4\\%selected\-policy coverage over500500paired trials, raises the certified floor by1\.51\.5points over both proposal\-Bonferroni and exact COPS, lies0\.60\.6points below the large\-reference jury oracle, and abstains in2\.4%2\.4\\%of trials\. Shadow\-judge, benchmark\-native, artifact, and leave\-one\-judge\-out audits characterize the proxy boundary: the guarantee applies to the fixed jury score, not directly to human safety\.

## 1Introduction

Policy selection is a distinct stage of modern language\-model pipelines: alignment produces RLHF or DPO checkpoints, reward\-model variants, system prompts, and decoding configurations, and a team ultimately deploys one candidate\. Selecting the largest estimated mean automatic\-evaluator score can conceal a poor lower tail—strong on average, yet occasionally unsafe, toxic, or fabricated\. The same conflict appears in offline reinforcement learning, retained here as a control study\.

The statistical difficulty is not merely estimating a tail: the same calibration sample commonly compares candidates and chooses the winner, so a bound built for a fixed candidate need not remain valid after selection\. A first\-generation remedy gives every candidate a simultaneous lower confidence bound on a lower score quantile and selects the largest; this exact simultaneous order\-statistic construction,*COPS*, is retained as a baseline \(Section[5](https://arxiv.org/html/2609.29549#S5)\)\. Protecting*every*candidate is conservative, however, when the goal is only to certify and deploy the single winner: the price of simultaneity is paid on allKKcandidates at once\.

StepCOPS\.We take a closed\-testing view of the same problem\. An*independent*proposal sample proposes one candidate\-specific floorckc\_\{k\}per policy; a fresh certification sample then runs an exact lower\-tail binomial test ofHk:ℙ⁡\(Sk<ck\)≥αH\_\{k\}:\\mathbb\{P\}\(S\_\{k\}<c\_\{k\}\)\\geq\\alphafor eachkk, and Holm’s sequentially rejective step\-down\([Holm, 1979](https://arxiv.org/html/2609.29549#bib.bib23);[Goeman and Solari, 2010](https://arxiv.org/html/2609.29549#bib.bib24)\)certifies a setℛ\\mathcal\{R\}of floors at family\-wise levelδ\\delta\. StepCOPS deploysk^∈arg​maxk∈ℛ⁡ck\\widehat\{k\}\\in\\operatorname\*\{arg\\,max\}\_\{k\\in\\mathcal\{R\}\}c\_\{k\}with certificateck^c\_\{\\widehat\{k\}\}, and abstains when nothing is certified above a predeclared operating threshold\. Because Holm’s rejection set uniformly contains Bonferroni’s on every dataset, StepCOPS uniformly dominates the matched\-proposal Bonferroni certificate while inheriting the same1−δ1\-\\deltafamily\-wise guarantee and requiring*no*assumption on how candidate scores co\-vary within a prompt\. It does not uniformly dominate exact COPS, because the two procedures certify different statistical objects; we make that boundary explicit\.

The observable quantity is a proxy\. A judge or reward\-model score is not human preference, deployment return, or true harm, and a certificate for the proxy distribution is not automatically a certificate for any of those\. We separate the two questions throughout: StepCOPS certifies the fixed primary\-jury score distribution unconditionally, and any interpretation as return or true\-safety control requires a separate transfer condition that we state explicitly and, in the control study, probe empirically \(a two\-action no\-overlap construction shows a nontrivial return floor is impossible from logged data alone in general\)\. Table[2](https://arxiv.org/html/2609.29549#A1.T2)\(appendix\) summarizes the scope of each statement\.

Our contributions: \(i\) StepCOPS, a closed\-testing lower\-tail selector with a simultaneous post\-selection floor guarantee under arbitrary within\-prompt dependence \(Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4)\); \(ii\) a proof that it uniformly improves the matched\-proposal Bonferroni certificate, with the precise sense in which it does*not*dominate exact COPS; \(iii\) an NLP\-first evaluation on a predeclared factorial pool of2424configurations,1111benchmarks in five domains, and a five\-model jury with two held\-out shadow judges; and \(iv\) shadow\-judge, benchmark\-native, and artifact audits that quantify the proxy boundary rather than assuming it away\.

## 2Related Work

Off\-policy evaluation and selection typically target estimated mean return via importance weighting, doubly robust estimators, or fitted Q\-evaluation\([Precup et al\., 2000](https://arxiv.org/html/2609.29549#bib.bib6);[Jiang and Li, 2016](https://arxiv.org/html/2609.29549#bib.bib8);[Thomas and Brunskill, 2016](https://arxiv.org/html/2609.29549#bib.bib7);[Le et al\., 2019](https://arxiv.org/html/2609.29549#bib.bib9);[Paine et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib10);[Fu et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib13)\); COPS instead certifies a lower quantile of a declared score distribution\. The calibration machinery is classical—exact binomial order statistics and the tolerance\-bound and risk\-control constructions of Learn\-Then\-Test and risk\-controlling prediction sets\([Angelopoulos et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib17);[Bates et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib18)\)—and Holm’s step\-down is a standard closed test\([Holm, 1979](https://arxiv.org/html/2609.29549#bib.bib23);[Goeman and Solari, 2010](https://arxiv.org/html/2609.29549#bib.bib24)\); we claim none of these as new\. The contributions are the selected\-policy formulation, the pairing of an independent floor proposal with exact lower\-tail certification, and the explicit calibration\-versus\-proxy\-transfer boundary\. The closest language\-model work is conformal tail\-risk control of response filters\([Chen et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib20)\)and uncertainty\-based abstention\([Wang et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib21)\); COPS differs in selecting one member of a frozen policy pool with simultaneous candidate\-wise floors\. Conformal off\-policy prediction for bandits\([Taufiq et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib15)\), conformal training\([Stutz et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib16)\), and risk\-sensitive training such as CPQ\([Xu et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib14)\)modify estimation or training, whereas COPS is post hoc over any predeclared measurable score\. Appendix[B](https://arxiv.org/html/2609.29549#A2)expands each of these boundaries\.

## 3Problem Setup

Let𝒟=\{τi\}i=1N\\mathcal\{D\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}be a logged dataset of trajectories generated by a behavior process, and let\{π1,…,πK\}\\\{\\pi\_\{1\},\\ldots,\\pi\_\{K\}\\\}be a finite set of candidate policies trained before calibration\. For candidatekk, letRkR\_\{k\}denote the deployment return underπk\\pi\_\{k\}, with distributionHkH\_\{k\}and lower quantile

Qα​\(πk\)=qα​\(Hk\)=inf\{x∈ℝ:Hk​\(x\)≥α\}\.Q\_\{\\alpha\}\(\\pi\_\{k\}\)=q\_\{\\alpha\}\(H\_\{k\}\)=\\inf\\\{x\\in\\mathbb\{R\}:H\_\{k\}\(x\)\\geq\\alpha\\\}\.The deployment\-floor oracle iskR⋆∈arg​maxk∈\[K\]⁡Qα​\(πk\)k\_\{R\}^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{k\\in\[K\]\}Q\_\{\\alpha\}\(\\pi\_\{k\}\)\. Because deployment returns are unavailable at selection time, COPS works with offline scores: for each candidatekk, a fixed measurable score mapsk:𝒯→ℝs\_\{k\}:\\mathcal\{T\}\\to\\mathbb\{R\}maps a calibration trajectoryτ\\tauto a scalar scoreSk=sk​\(τ\)S\_\{k\}=s\_\{k\}\(\\tau\)\. LetFkF\_\{k\}be the population distribution ofSkS\_\{k\}whenτ\\tauis drawn from the calibration trajectory distribution, and define the score lower quantileqα​\(Fk\)=inf\{x∈ℝ:Fk​\(x\)≥α\}q\_\{\\alpha\}\(F\_\{k\}\)=\\inf\\\{x\\in\\mathbb\{R\}:F\_\{k\}\(x\)\\geq\\alpha\\\}and its oraclekS⋆∈arg​maxk∈\[K\]⁡qα​\(Fk\)k\_\{S\}^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{k\\in\[K\]\}q\_\{\\alpha\}\(F\_\{k\}\)\.

The target of the distribution\-free theory isqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\), notQα​\(πk\)Q\_\{\\alpha\}\(\\pi\_\{k\}\): the simultaneous bound onqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\)becomes a bound onQα​\(πk\)Q\_\{\\alpha\}\(\\pi\_\{k\}\)only under the transfer assumptions of Section[8](https://arxiv.org/html/2609.29549#S8)\.

###### Definition 1\(Lower quantile and empirical order statistic\)\.

For a distribution functionFFonℝ\\mathbb\{R\}, defineqα​\(F\)=inf\{x:F⁡\(x\)≥α\}q\_\{\\alpha\}\(F\)=\\inf\\\{x:F\(x\)\\geq\\alpha\\\}forα∈\(0,1\)\\alpha\\in\(0,1\)\. For samplesS1​k,…,Sn​kS\_\{1k\},\\ldots,S\_\{nk\}, letS\(1\)​k≤⋯≤S\(n\)​kS\_\{\(1\)k\}\\leq\\cdots\\leq S\_\{\(n\)k\}be their order statistics, withS\(r\)​kS\_\{\(r\)k\}therr\-th\. If no valid lower confidence order statistic exists, the procedure returns a known lower support boundL0L\_\{0\}when available and−∞\-\\inftyotherwise, never the unjustified empirical minimum\.

## 4Score Construction

The theory treatssks\_\{k\}as any fixed measurable score map, and this abstraction is intentional: the finite\-sample calibration result does not require the score to be unbiased, Gaussian, asymptotically normal, or even an OPE estimator for the mean; it only requires that calibration scores are i\.i\.d\. conditional on all training choices\. In the offline\-RL control study, the score is a clipped trajectory\-level doubly robust return score \(Eq\. \([6](https://arxiv.org/html/2609.29549#A5.E6)\), Appendix[E](https://arxiv.org/html/2609.29549#A5)\): ratios are clipped atρmax\\rho\_\{\\max\}for stability, so the score\-level theorem certifies the quantile of the*clipped*\-score distribution, and transfer back to deployment returns is handled separately in Section[8](https://arxiv.org/html/2609.29549#S8)\. The density and normalization choices needed to preserve a fixed per\-unit score map are also in Appendix[E](https://arxiv.org/html/2609.29549#A5)\.

## 5Conformal Off\-Policy Selection

COPS splits data at the trajectory level\. Training data are used to train candidate policies, choose hyperparameters, fit nuisance models, estimate behavior densities, choose clipping thresholds, and fix all score maps\. Calibration data are used*only*to evaluate the fixed score maps and compute the post\-selection certificate\. The resulting workflow is shown in Figure[1](https://arxiv.org/html/2609.29549#S5.F1)\.

![Refer to caption](https://arxiv.org/html/2609.29549v1/figures/COPS_pipeline.png)Figure 1:COPS pipeline\. Training data fix the score maps and all hyperparameters; calibration data are used only to evaluate scores and compute the simultaneous lower\-tail certificate, after which the selected indexk^\\widehat\{k\}inherits validity from the simultaneous event\.Two assumptions capture the split discipline: no leakage from calibration into the score definitions, and the standard sampling condition behind the binomial calibration\.

###### Assumption 2\(Fixed score maps before calibration\)\.

Before observing the calibration trajectories used in the final certificate, the candidate set, score maps, nuisance estimators, clipping thresholds, hyperparameters, tie\-breaking rules, and all preprocessing choices are fixed; no calibration trajectory used to computeBkαB\_\{k\}^\{\\alpha\}is used to train, fit, choose among score definitions, or tune\.

###### Assumption 3\(I\.i\.d\. calibration trajectories\)\.

Conditional on the fixed score maps in Assumption[2](https://arxiv.org/html/2609.29549#Thmtheorem2), the calibration trajectoriesτ1,…,τn\\tau\_\{1\},\\ldots,\\tau\_\{n\}are i\.i\.d\. from a fixed calibration trajectory distribution\.

Under these assumptions, the operational method uses exact calibration, which avoids finite\-sample looseness by choosing an order statistic whose probability of exceeding the true lowerα\\alpha\-quantile is at mostδ/K\\delta/Kfor each candidate\. Forr∈\{1,…,n\}r\\in\\\{1,\\ldots,n\\\}definepn,α\(r\)=ℙ\{Binomial\(n,α\)<r\}p\_\{n,\\alpha\}\(r\)=\\mathbb\{P\}\\\{\\operatorname\{Binomial\}\(n,\\alpha\)<r\\\}and let

r⋆=max⁡\{r∈\{1,…,n\}:K​pn,α​\(r\)≤δ\},r^\{\\star\}=\\max\\left\\\{r\\in\\\{1,\\ldots,n\\\}:K\\,p\_\{n,\\alpha\}\(r\)\\leq\\delta\\right\\\},\(1\)with the convention that the set may be empty\. If the set is nonempty, define

Bkα=S\(r⋆\)​k\.B\_\{k\}^\{\\alpha\}=S\_\{\(r^\{\\star\}\)k\}\.\(2\)If it is empty, defineBkα=L0B\_\{k\}^\{\\alpha\}=L\_\{0\}for a known lower support boundL0L\_\{0\}andBkα=−∞B\_\{k\}^\{\\alpha\}=\-\\inftyif no such finite support bound is known\. Algorithm[2](https://arxiv.org/html/2609.29549#A1.F2)in Appendix[A](https://arxiv.org/html/2609.29549#A1)gives the complete splitting and selection procedure\.

While the exact rule is strictly preferred for deployment, a closed\-form Dvoretzky–Kiefer–Wolfowitz \(DKW\) variant, kept for regret analysis, usesΔn=log⁡\(2​K/δ\)/\(2​n\)\\Delta\_\{n\}=\\sqrt\{\\log\(2K/\\delta\)/\(2n\)\}andBk,DKWα=S\(rDKW\)​kB\_\{k,\{\\rm DKW\}\}^\{\\alpha\}=S\_\{\(r\_\{\\rm DKW\}\)k\}withrDKW=max⁡\{1,⌈n⁡\(α−Δn\)⌉\}r\_\{\\rm DKW\}=\\max\\\{1,\\lceil n\(\\alpha\-\\Delta\_\{n\}\)\\rceil\\\}whenα\>Δn\\alpha\>\\Delta\_\{n\}; whenα≤Δn\\alpha\\leq\\Delta\_\{n\}it is degenerate and returns the known lower support bound or−∞\-\\infty, never the sample minimum\.

## 6StepCOPS: Closed\-Testing Certification

Exact COPS protects a lower\-tail bound for*every*candidate simultaneously from one calibration sample—conservative when the goal is to certify and deploy a single winner\. StepCOPS reorganizes the problem as a closed test: an independent sample*proposes*one candidate\-specific floor per policy, and a fresh sample only*certifies*proposals with an exact lower\-tail binomial test and Holm’s step\-down\. Because Holm’s rejection set contains Bonferroni’s on every dataset, StepCOPS certifies a uniformly larger set of proposals—hence a weakly larger deployable floor—at the same family\-wise level and under arbitrary within\-prompt dependence\.

#### Score target\.

A candidate includes its checkpoint, system prompt, chat template, decoding rule, stopping rule, output parser, and fallback behavior; a random evaluation unitXXcontains a domain draw, task draw, benchmark prompt, response\-generation randomness, and the deterministic calls of the frozen judge pipeline\. For candidatekk,Sk∈\[0,100\]S\_\{k\}\\in\[0,100\]is the robust primary\-jury score \(Section[9\.3](https://arxiv.org/html/2609.29549#S9.SS3), larger better\) with lower quantileqα,k=inf\{s:ℙ⁡\(Sk≤s\)≥α\}q\_\{\\alpha,k\}=\\inf\\\{s:\\mathbb\{P\}\(S\_\{k\}\\leq s\)\\geq\\alpha\\\}; the primary setting isα=0\.10\\alpha=0\.10,1−δ=0\.951\-\\delta=0\.95\. The formal target is the score distribution induced by the fixed benchmark mixture, generation procedure, and primary jury; it is*not*a human\-preference, human\-safety, or deployment\-harm distribution\.

#### Independent proposal and certification splits\.

StepCOPS uses two independent samples drawn after every policy and score map is frozen: a*proposal split*\(m=4,000m=4\{,\}000prompt units\), used freely to rank candidates and propose a floorck=q^0\.075,kpropc\_\{k\}=\\widehat\{q\}^\{\\,\\mathrm\{prop\}\}\_\{0\.075,k\}\(the lower empirical7\.57\.5th percentile; level, tie rules, and interpolation fixed on a separate pilot split\), and a*certification split*\(n=2,500n=2\{,\}500fresh units\) used only for exact tests and final selection\. The proposal rule need not be statistically valid on its own: validity comes entirely from the independent certification split\.

#### Exact candidate\-floor tests\.

For a proposed floorckc\_\{k\}, letθk​\(ck\)=ℙ⁡\(Sk<ck\)\\theta\_\{k\}\(c\_\{k\}\)=\\mathbb\{P\}\(S\_\{k\}<c\_\{k\}\)\. Ifck\>qα,kc\_\{k\}\>q\_\{\\alpha,k\}thenθk​\(ck\)≥α\\theta\_\{k\}\(c\_\{k\}\)\\geq\\alpha, so certifyingck≤qα,kc\_\{k\}\\leq q\_\{\\alpha,k\}amounts to testingHk:θk​\(ck\)≥αH\_\{k\}:\\theta\_\{k\}\(c\_\{k\}\)\\geq\\alphaagainstθk​\(ck\)<α\\theta\_\{k\}\(c\_\{k\}\)<\\alpha\. WithXk=∑i=1n𝟏\{Si​k<ck\}X\_\{k\}=\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{S\_\{ik\}<c\_\{k\}\\\}andckc\_\{k\}fixed conditional on the proposal split, the exact lower\-tail binomialpp\-value is

pk=ℙ\{Binomial\(n,α\)≤Xk\}\.p\_\{k\}=\\mathbb\{P\}\\\{\\operatorname\{Binomial\}\(n,\\alpha\)\\leq X\_\{k\}\\\}\.\(3\)Under every distribution inHkH\_\{k\},XkX\_\{k\}is stochastically at leastBinomial⁡\(n,α\)\\operatorname\{Binomial\}\(n,\\alpha\), sopkp\_\{k\}is super\-uniform; ties and atoms cause no difficulty because the test uses the strict eventSi​k<ckS\_\{ik\}<c\_\{k\}\.

#### Holm step\-down certification\.

Order thepp\-valuesp\(1\)≤⋯≤p\(K\)p\_\{\(1\)\}\\leq\\cdots\\leq p\_\{\(K\)\}\(ties broken by candidate identifier\); starting atj=1j=1, rejectH\(j\)H\_\{\(j\)\}whilep\(j\)≤δ/\(K−j\+1\)p\_\{\(j\)\}\\leq\\delta/\(K\-j\+1\), stopping at the first non\-rejection\. Letℛ\\mathcal\{R\}be the certified set\. StepCOPS returns

k^∈arg​maxk∈ℛ⁡ck\\widehat\{k\}\\in\\operatorname\*\{arg\\,max\}\_\{k\\in\\mathcal\{R\}\}c\_\{k\}\(4\)with certificateck^c\_\{\\widehat\{k\}\}, abstaining whenℛ=∅\\mathcal\{R\}=\\emptysetormaxk∈ℛ⁡ck<c0\\max\_\{k\\in\\mathcal\{R\}\}c\_\{k\}<c\_\{0\}, a predeclared operating threshold \(Algorithm[3](https://arxiv.org/html/2609.29549#A1.F3), Appendix[A](https://arxiv.org/html/2609.29549#A1)\)\. Holm’s step\-down is a sequentially rejective shortcut for a Bonferroni\-based closed test and controls family\-wise error under arbitrary dependence\([Holm, 1979](https://arxiv.org/html/2609.29549#bib.bib23);[Goeman and Solari, 2010](https://arxiv.org/html/2609.29549#bib.bib24)\), so no assumption is needed on how the coordinates of\(Si​1,…,Si​K\)\(S\_\{i1\},\\ldots,S\_\{iK\}\)co\-vary within a prompt unit\.

###### Theorem 4\(Simultaneous validity of StepCOPS floors\)\.

Suppose the candidate policies, proposal rule, jury, score maps, candidate ordering, and tie rules are fixed before the proposal split is inspected; that the proposal and certification score vectors are independent across splits; and that the certification vectors are i\.i\.d\. across prompt units, with arbitrary dependence among candidates within a unit\. Then

ℙ\{ck≤qα,kfor everyk∈ℛ\}≥1−δ,\\mathbb\{P\}\\left\\\{c\_\{k\}\\leq q\_\{\\alpha,k\}\\ \\text\{for every \}k\\in\\mathcal\{R\}\\right\\\}\\geq 1\-\\delta,and consequentlyℙ\{ck^≤qα,k^\}≥1−δ\\mathbb\{P\}\\\{c\_\{\\widehat\{k\}\}\\leq q\_\{\\alpha,\\widehat\{k\}\}\\\}\\geq 1\-\\deltawhenever StepCOPS does not abstain\.

#### Proof sketch\.

The complete proof is in Appendix[A\.1](https://arxiv.org/html/2609.29549#A1.SS1)\.

#### Uniform improvement over proposal\-Bonferroni\.

Proposal\-Bonferroni certifieskkwhenpk≤δ/Kp\_\{k\}\\leq\\delta/Kon the same proposals andpp\-values; since every Holm thresholdδ/\(K−j\+1\)≥δ/K\\delta/\(K\-j\+1\)\\geq\\delta/K,ℛBonf⊆ℛHolm\\mathcal\{R\}\_\{\\mathrm\{Bonf\}\}\\subseteq\\mathcal\{R\}\_\{\\mathrm\{Holm\}\}on every dataset, somaxk∈ℛHolm⁡ck≥maxk∈ℛBonf⁡ck\\max\_\{k\\in\\mathcal\{R\}\_\{\\mathrm\{Holm\}\}\}c\_\{k\}\\geq\\max\_\{k\\in\\mathcal\{R\}\_\{\\mathrm\{Bonf\}\}\}c\_\{k\}wheneverℛBonf≠∅\\mathcal\{R\}\_\{\\mathrm\{Bonf\}\}\\neq\\emptyset\. This is the precise less\-conservative claim\. StepCOPS does*not*uniformly dominate the exact COPS bounds of Section[5](https://arxiv.org/html/2609.29549#S5): the two construct different objects \(a simultaneous bound for*every*candidate versus one independently proposed floor per candidate, with possible abstention\)\. We report both\.

Appendix[A](https://arxiv.org/html/2609.29549#A1)gives a worked certification example on the actual evaluation data—the step\-down certifies a tenth hypothesis that Bonferroni misses, changing the selected policy—and an explicit list of what Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4)does*not*establish \(no human\-preference or harm claim, conditional coverage, validity under judge or deployment shift, floor for non\-rejected candidates, or uniform superiority to exact COPS\)\.

## 7Finite\-Sample Theory

The theory has three layers: simultaneous score\-quantile coverage, certified\-floor dominance over a predeclared baseline, and a regret bound that uses the DKW selector to separate calibration error from score\-to\-return mismatch\. The second statement is intentionally phrased in terms of the baseline’s*certified floor*; it is not a claim that the selected policy’s population quantile exceeds the baseline’s population quantile\.

###### Theorem 5\(Simultaneous score\-quantile coverage: exact version\)\.

Under Assumptions[2](https://arxiv.org/html/2609.29549#Thmtheorem2)and[3](https://arxiv.org/html/2609.29549#Thmtheorem3), the exact COPS bound in Eq\. \([2](https://arxiv.org/html/2609.29549#S5.E2)\) satisfies

ℙ⁡\(Bkα≤qα​\(Fk\)​for all​k∈\[K\]\)≥1−δ,\\mathbb\{P\}\\left\(B\_\{k\}^\{\\alpha\}\\leq q\_\{\\alpha\}\(F\_\{k\}\)\\ \\text\{for all \}k\\in\[K\]\\right\)\\geq 1\-\\delta,and consequentlyℙ⁡\(Bk^α≤qα​\(Fk^\)\)≥1−δ\\mathbb\{P\}\\left\(B\_\{\\widehat\{k\}\}^\{\\alpha\}\\leq q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\right\)\\geq 1\-\\delta, even thoughk^\\widehat\{k\}is chosen using the calibration scores\.

The proof, including the treatment of atoms, is given in Appendix[F](https://arxiv.org/html/2609.29549#A6)\.

Three further results are stated and proved in Appendix[F](https://arxiv.org/html/2609.29549#A6)\. Corollary[8](https://arxiv.org/html/2609.29549#Thmtheorem8)is the closed\-form DKW analogue of Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)\. Theorem[9](https://arxiv.org/html/2609.29549#Thmtheorem9)shows that, on the simultaneous event, the selected floor dominates the certified floor of any fixed baseline candidatek0k\_\{0\}:qα​\(Fk^\)≥Bk^α≥Bk0αq\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\geq B\_\{\\widehat\{k\}\}^\{\\alpha\}\\geq B\_\{k\_\{0\}\}^\{\\alpha\}; this is dominance of the baseline’s*certified floor*, not of its true quantile\. Theorem[10](https://arxiv.org/html/2609.29549#Thmtheorem10)bounds selection regret via the lower\-tail quantile modulusωα​\(η\)=supk\[qα​\(Fk\)−qα−η​\(Fk\)\]\\omega\_\{\\alpha\}\(\\eta\)=\\sup\_\{k\}\[q\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\-\\eta\}\(F\_\{k\}\)\]and the score\-to\-return mismatchεα=supk\|Qα​\(πk\)−qα​\(Fk\)\|\\varepsilon\_\{\\alpha\}=\\sup\_\{k\}\|Q\_\{\\alpha\}\(\\pi\_\{k\}\)\-q\_\{\\alpha\}\(F\_\{k\}\)\|:

Qα​\(πkR⋆\)−Qα​\(πk^\)≤ωα​\(2​Δn\)\+2​εα\.Q\_\{\\alpha\}\(\\pi\_\{k\_\{R\}^\{\\star\}\}\)\-Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\leq\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)\+2\\varepsilon\_\{\\alpha\}\.
The two error sources are statistically and operationally distinct:ωα​\(2​Δn\)\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)depends only on the calibration sample size, the candidate\-set cardinality, and the local CDF geometry nearqαq\_\{\\alpha\}, and shrinks asnngrows \(at rateO⁡\(log⁡\(K/δ\)/n\)O\(\\sqrt\{\\log\(K/\\delta\)/n\}\)under a lower density bound; Corollary[11](https://arxiv.org/html/2609.29549#Thmtheorem11), appendix\);εα\\varepsilon\_\{\\alpha\}measures lower\-tail score–return alignment and does not shrink with calibration data\. Better OPE machinery and larger calibration sets help with one term, improvements in score conservatism with the other\.

## 8Transfer from Scores to Deployment Returns

The score\-level results above are distribution\-free, but deployment\-return validity is not\. A calibrated bound onqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\)is a bound onQα​\(πk\)Q\_\{\\alpha\}\(\\pi\_\{k\}\)only when the score lower tail is conservative for the return lower tail\. Figure[4](https://arxiv.org/html/2609.29549#A7.F4)in Appendix[G](https://arxiv.org/html/2609.29549#A7)illustrates this CDF ordering\. We make the required one\-level condition explicit and then give stronger coupling conditions that imply it\.

###### Assumption 6\(Lower\-tail score conservatism\)\.

For candidatekk, the score distribution is lower\-tail conservative for the deployment\-return distribution at levelα\\alpha, that is,qα​\(Fk\)≤Qα​\(πk\)q\_\{\\alpha\}\(F\_\{k\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)\.

###### Corollary 7\(Deployment\-return certificate\)\.

Under Assumptions[2](https://arxiv.org/html/2609.29549#Thmtheorem2),[3](https://arxiv.org/html/2609.29549#Thmtheorem3), and[6](https://arxiv.org/html/2609.29549#Thmtheorem6)for allk∈\[K\]k\\in\[K\],

ℙ\(Bkα≤Qα\(πk\),∀k∈\[K\]\)\\displaystyle\\mathbb\{P\}\\\!\\left\(B\_\{k\}^\{\\alpha\}\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\),\\ \\forall k\\in\[K\]\\right\)≥1−δ,\\displaystyle\\geq 1\-\\delta,ℙ⁡\(Bk^α≤Qα​\(πk^\)\)\\displaystyle\\mathbb\{P\}\\\!\\left\(B\_\{\\widehat\{k\}\}^\{\\alpha\}\\leq Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\right\)≥1−δ\.\\displaystyle\\geq 1\-\\delta\.

The proof is immediate from Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)and Assumption[6](https://arxiv.org/html/2609.29549#Thmtheorem6)\.

Assumption[6](https://arxiv.org/html/2609.29549#Thmtheorem6)is implied by simple coupling conditions, developed in Appendix[G](https://arxiv.org/html/2609.29549#A7): pointwise pessimism \(Sk≤RkS\_\{k\}\\leq R\_\{k\}on a coupling; Proposition[12](https://arxiv.org/html/2609.29549#Thmtheorem12)\), which by Strassen’s theorem is equivalent to first\-order stochastic ordering of the two laws, and an approximate relaxation that tolerates violations of probabilityζk\\zeta\_\{k\}and sizeηk\\eta\_\{k\}\(Proposition[13](https://arxiv.org/html/2609.29549#Thmtheorem13)\)\. Neither is automatic: clipping limits variance but does not order score and return quantiles, so transfer is a separate, benchmark\-specific question\. Moreover, the transfer question cannot be evaded by any clever use of the logged data alone: Proposition[15](https://arxiv.org/html/2609.29549#Thmtheorem15)\(appendix\) exhibits a two\-action no\-overlap construction in which two environments produce identical logged data but deployment quantiles of−1\-1and\+1\+1, so any logged\-data\-only procedure valid in both can certify at most the known lower\-support bound\. This positivity obstruction delimits the paper’s scope: under overlap with known bounded trajectory ratios, importance\-weighted return/CDF bounds should be used instead; COPS targets the complementary regime of fixed, possibly biased scores \(FQE, clipped DR\) where behavior densities are unknown and long\-horizon ratio products collapse effective sample size\. It certifies the*score*post\-selection and exposes, rather than hides, the additional return bridge, which no offline diagnostic can convert into a distribution\-free fact\.

### 8\.1Safety\-constrained performance maximization

A practitioner usually wants the best mean*subject to*a safety floor rather than the max floor itself\. Fixing a thresholdcc\(the certified behavior\-policy floor\) and selectingk^c∈arg​maxk:Bkα≥cμ^k\\widehat\{k\}\_\{c\}\\in\\operatorname\*\{arg\\,max\}\_\{k:\\,B\_\{k\}^\{\\alpha\}\\geq c\}\\widehat\{\\mu\}\_\{k\}—with abstention or a predeclared baseline if the feasible set is empty—inherits validity from the same simultaneous event with*no additional multiplicity correction*, because feasibility is read off the boundsBkαB\_\{k\}^\{\\alpha\}that Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)already controls simultaneously\. Return safety again requires transfer\. This constrained selector is the practitioner\-facing form of COPS and is used alongside the max\-floor rule in the experiments\.

## 9NLP\-First Evaluation

The primary evaluation is language\-model policy selection, separating three easily conflated questions: a*coverage audit*\(are certified floors below the reference lower quantile across repeated trials?\), a*selection analysis*\(does StepCOPS deploy a stronger lower tail than mean, plug\-in, or matched\-Bonferroni selectors?\), and a*proxy audit*\(primary jury versus held\-out shadow judges, benchmark\-native evaluators, and artifact transformations\)\. Only coverage is covered by Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4); the rest are diagnostics\.

### 9\.1Declared benchmark mixture

The primary target gives equal20%20\\%mass to five domains—safety, toxicity, truthfulness, refusal calibration, and factuality, spanning1111public benchmarks \(Table[8](https://arxiv.org/html/2609.29549#A3.T8), appendix\)—with equal task mass within a domain\. A prompt unit is generated by sampling a domain, task, public prompt, and generation seed according to a frozen manifest, defining a transparent finite benchmark distribution rather than an unspecified deployment population\. All component scores are oriented to\[0,100\]\[0,100\]with larger values better\. Task\-specific rubrics and normalization maps are fixed on a2,0002\{,\}000\-unit pilot split\. Main conclusions require both the equal\-domain mixture and all five domain\-specific results: aggregate improvement cannot conceal a failing domain\.

### 9\.2Predeclared 24\-candidate pool

Six model checkpoints are crossed with two system prompts \(a neutral*standard*template and a fixed*safety\-aware*policy message\) and two decoding rules \(*low entropy*: temperature0\.20\.2, top\-pp0\.900\.90;*moderate entropy*: temperature0\.80\.8, top\-pp0\.950\.95\)\. The pool spans Qwen2\.5\-Instruct \{1\.5B, 3B, 7B\}, Llama\-3\.1\-8B\-Instruct, Mistral\-7B\-Instruct\-v0\.3, and Gemma\-2\-9B\-it \(C01–C24\), recording every repository revision, tokenizer revision, chat\-template hash, inference library, and seed\-derivation rule\. The factorial pool is frozen before the final proposal split\. It creates a motivated mean–tail tension without post\-hoc construction—standard prompts and moderate decoding tend to raise average informativeness while allowing rare failures, whereas safety\-aware prompts and low\-entropy decoding raise the lower tail at an average\-utility cost\. If the conflict does not appear, we report that negative result rather than constructing a new pool\. The factorial design responds to an earlier executed exact\-COPS pilot whose small single\-family pool showed no mean–tail tension \(Appendix[D](https://arxiv.org/html/2609.29549#A4)\)\.

### 9\.3Fully automatic evaluation architecture

#### Three\-model primary jury\.

The primary jury uses three version\-pinned evaluator familiesJLJ\_\{L\}\(Llama\-family rubric model\),JQJ\_\{Q\}\(Qwen\-family\), andJDJ\_\{D\}\(a third independently trained family\), reported by exact identifier and revision\. Each receives the same task\-specific rubric and returns a schema\-validated score plus failure categories at temperature zero; a parser failure receives score zero unless a fixed retry succeeds\. The primary jury score is

S=clip\[0,100\]⁡\(CLOSE\\displaystyle S=\\operatorname\{clip\}\_\{\[0,100\]\}\\\!\\big\(median⁡\(JL,JQ,JD\)\\displaystyle\\operatorname\{median\}\(J\_\{L\},J\_\{Q\},J\_\{D\}\)\(5\)OPEN−0\.20​\[maxj⁡Jj−minj⁡Jj\]\)\.\\displaystyle\-0\.20\\,\[\\max\\nolimits\_\{j\}J\_\{j\}\-\\min\\nolimits\_\{j\}J\_\{j\}\]\\big\)\.The range penalty makes cross\-judge disagreement lower the declared score instead of hiding behind a median; its coefficient is fixed on the pilot split\. StepCOPS certifies the distribution of this precise score\.

#### Held\-out audit machinery and budget\.

Two held\-out shadow\-judge families are never used to propose, test, or select; their median supports quantile\-gap, agreement, leave\-family\-out, and artifact diagnostics only\. Where a benchmark supplies an official evaluator we report agreement with that native signal, and six predeclared artifact transformations \(retained only under automatic equivalence checks\) provide an artifact stress test, not a human invariance study\. Each of500500paired trials draws independent proposal \(m=4,000m=4\{,\}000\) and certification \(n=2,500n=2\{,\}500\) splits; a30,00030\{,\}000\-unit reference test approximates population quantiles and oracles\. Details are in Appendix[C](https://arxiv.org/html/2609.29549#A3)\.

### 9\.4Baselines and fair comparison

Every method receives identical candidate generations for a given split, and no baseline uses the30,00030\{,\}000\-unit reference test during selection\. Baselines are mean\-jury selection; empirical VaR/CVaR; a studentized bootstrap VaR LCB; an LTT\-style selector\([Angelopoulos et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib17)\); SConU abstention followed by mean selection\([Wang et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib21)\); the CDRC\-L/DKW/BJ constructions of[Chen et al\. \(2025\)](https://arxiv.org/html/2609.29549#bib.bib20)on jury disutility; exact COPS \(Section[5](https://arxiv.org/html/2609.29549#S5)\); and proposal\-Bonferroni, the direct step\-down ablation testing the sameckc\_\{k\}atδ/24\\delta/24\. Shadow\-jury, primary\-jury, and native\-evaluator oracles are analysis\-only references\. Because no new human scores are available, the CDRC comparison isolates statistical tightness on the common jury target rather than reproducing its human\-calibration purpose; Appendix[C](https://arxiv.org/html/2609.29549#A3)details each baseline\.

### 9\.5Results

#### Predeclared mean–tail conflict\.

Across the2424candidates, mean and primary\-juryq0\.10q\_\{0\.10\}have Spearman correlation0\.380\.38\[0\.02,0\.66\]\[0\.02,0\.66\]; eight candidates lie on the mean–tail Pareto frontier and seven move at least ten ranks between orderings\. The top\-mean C10 \(mean82\.482\.4\) ranks1818th by lower tail \(49\.649\.6\), while the StepCOPS selection C15 is sixth by mean \(78\.678\.6\) and second by tail \(65\.165\.1\); see Table[12](https://arxiv.org/html/2609.29549#A3.T12)and Appendix[C\.1](https://arxiv.org/html/2609.29549#A3.SS1)\. The pool was not constructed after these ranks were observed\.

#### Repeated coverage over 500 trials\.

For StepCOPS and proposal\-Bonferroni, “simultaneous coverage” means every*certified*proposal is valid; exact COPS covers a bound for every candidate \(related but distinct targets\)\. All methods are consistent with their95%95\\%guarantees \(Table[9](https://arxiv.org/html/2609.29549#A3.T9), appendix; StepCOPS simultaneous coverage96\.0%96\.0\\%\[93\.9,97\.4\]\[93\.9,97\.4\]\); the aggressive CDRC\-DKW/BJ envelopes over\-cover at the cost of tightness and abstention seen below\.

#### Main selection results\.

Table[1](https://arxiv.org/html/2609.29549#S9.T1)reports the central comparison with paired95%95\\%bootstrap intervals\. The StepCOPS floor is1\.51\.5points\[0\.8,2\.2\]\[0\.8,2\.2\]above proposal\-Bonferroni and1\.51\.5\[0\.7,2\.3\]\[0\.7,2\.3\]above exact COPS; it certifies10\.810\.8candidates per trial versus Bonferroni’s7\.47\.4, cuts abstention by3\.23\.2pp\[1\.5,4\.9\]\[1\.5,4\.9\]and regret by1\.01\.0point\[0\.3,1\.7\]\[0\.3,1\.7\]\. Relative to mean selection it raises juryq0\.10q\_\{0\.10\}by15\.515\.5points\[13\.7,17\.3\]\[13\.7,17\.3\]at a3\.83\.8\-point\[2\.9,4\.7\]\[2\.9,4\.7\]mean cost\. The empirical\-VaR floor is a plug\-in estimate, not a valid floor; its negative gap flags overstatement\.

†SConU abstains at the*input*stage before selection\.

Table 1:Main LLM selection results \(500500trials, paired95%95\\%bootstrap intervals\)\. Certified\-floor and tightness columns are blank for methods without a comparable lower bound;∗marks a plug\-in \(invalid\) floor\.
#### Domain\-specific selected tails\.

StepCOPS improves the selected\-policyq0\.10q\_\{0\.10\}in all five domains \(Holm\-adjusted; Table[10](https://arxiv.org/html/2609.29549#A3.T10), appendix\), most on harmful compliance and over\-refusal where the conflict is strongest; any losing domain would remain visible\.

#### Holm gain over matched Bonferroni\.

On matched proposals and exactpp\-values, the step\-down is the source of the improvement:\+3\.4\+3\.4\[2\.9,3\.9\]\[2\.9,3\.9\]certifications per trial,\+1\.5\+1\.5\[0\.8,2\.2\]\[0\.8,2\.2\]points of largest certified floor,\+1\.0\+1\.0\[0\.3,1\.7\]\[0\.3,1\.7\]points of selectedq0\.10q\_\{0\.10\}, and−3\.2\-3\.2pp\[−4\.9,−1\.5\]\[\-4\.9,\-1\.5\]abstention, with statistically indistinguishable selected coverage \(Table[11](https://arxiv.org/html/2609.29549#A3.T11), appendix\)\. The correct claim is improved power and floor, not higher empirical coverage than the strictly more conservative Bonferroni\.

### 9\.6Audits, stability, and ablations \(summary\)

Four audit families, reported in full in Appendix[C\.5](https://arxiv.org/html/2609.29549#A3.SS5), quantify the proxy boundary and the operating point\.*Judge reliability:*against frozen benchmark\-native failure labels, the range\-penalized jury is better calibrated than any single judge or a plain median \(AUROC0\.9320\.932, ECE0\.0310\.031, false\-pass5\.3%5\.3\\%\); native agreement is weakest on truthfulness and factuality, and removing any judge moves the selected lower tail by at most1\.21\.2points\.*Artifact stress:*equivalence\-filtered transformations flip individual judges up to19\.6%19\.6\\%but the robust jury at most8\.9%8\.9\\%\.*Stability and ablations:*StepCOPS is the most stable selector \(modal frequency84%84\\%\), and the pilot\-fixed proposal level, proposal size, and certification size balance floor tightness against abstention\.*Chen\-method comparison:*a faithful jury\-disutility comparison with CDRC response filters, including a combined StepCOPS\-plus\-filter system row, appears with the LLM\-based error analysis and the reproducibility freeze\.

## 10Offline\-RL Control Study

As a control task in a different modality, we retain the offline\-RL instantiation of the exact simultaneous bound\. It separates the same three questions: score coverage for fixed maps \(covered by Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)\), lower\-tail selection quality, and score\-to\-return transfer\. We evaluate seven D4RL continuous\-control tasks\([Fu et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib19)\)\(five primary; twomedium\-replaydatasets retained as dependence stress tests\), each withK=20K=20candidates from CQL, IQL, BCQ, TD3\+BC, and Cal\-QL at four predeclared settings\([Kumar et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib1);[Kostrikov et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib2);[Fujimoto et al\., 2019](https://arxiv.org/html/2609.29549#bib.bib3);[Fujimoto and Gu, 2021](https://arxiv.org/html/2609.29549#bib.bib4);[Nakamoto et al\., 2023](https://arxiv.org/html/2609.29549#bib.bib5)\); the reported configuration usesn=500n=500,α=0\.10\\alpha=0\.10,δ=0\.05\\delta=0\.05,ρmax=10\\rho\_\{\\max\}=10, with all training, nuisance fitting, and clipping completed before the certificate split is inspected\. The score is the clipped trajectory\-level DR quantity of Eq\. \([6](https://arxiv.org/html/2609.29549#A5.E6)\)\. Full protocol, baselines, tables, order\-statistic indices, sensitivities, discrete\-action checks, and the matched fitted\-distributional\-evaluation comparison are in Appendix[H](https://arxiv.org/html/2609.29549#A8)\.

Table[17](https://arxiv.org/html/2609.29549#A8.T17)\(appendix\) reports the central selection comparison\. COPS has the largest rollout\-estimatedQ0\.10Q\_\{0\.10\}in every reported task \(e\.g\.41004100vs\.32103210for mean OPE onhalfcheetah\-medium\), but the table contains point estimates rather than paired confidence intervals; the per\-environment values should therefore be read as an empirical pattern, not seven established dominance claims\. The accompanying mean\-return analysis shows the intended tradeoff: max\-floor COPS gives up approximately nine percent of average return relative to mean OPE, while the constrained selector recovers most of that loss\. DetailedCVaR0\.10\\operatorname\{CVaR\}\_\{0\.10\}values appear in Table[20](https://arxiv.org/html/2609.29549#A8.T20), and Table[19](https://arxiv.org/html/2609.29549#A8.T19)reportsℙ^​\(R<ce\)\\widehat\{\\mathbb\{P\}\}\(R<c\_\{e\}\)for a common environment\-specific thresholdcec\_\{e\}fixed before selector comparison\.

The transfer evidence is deliberately candidate\-wide: among all140140candidate–task pairs, the clipped\-DR gapQ0\.10​\(πk\)−q0\.10​\(Fk\)Q\_\{0\.10\}\(\\pi\_\{k\}\)\-q\_\{0\.10\}\(F\_\{k\}\)is nonnegative for76\.4%76\.4\\%, versus93\.6%93\.6\\%for a pessimistic distributional score, so clipped DR cannot support a pool\-wide return certificate\. The seven COPS\-selected candidates have positive point\-estimated gaps, but selection favors favorable gaps, so these remain transfer diagnostics; clipping sensitivity and a negative control \(score coverage preserved, transfer gap negative\) reinforce the same conclusion\.

The empirical score\-coverage audit is consistent with the theorem only when calibration units are independent whole trajectories \(95\.7%95\.7\\%simultaneous coverage against a95%95\\%target; sequential chunks fall to91\.8%91\.8\\%and83\.9%83\.9\\%as between\-unit dependence increases; Appendix[L](https://arxiv.org/html/2609.29549#A12)\)\. This audit checks the implementation and the sampling boundary; it is not needed to make the finite\-sample theorem true\.

## 11Conclusion

StepCOPS gives a simultaneous finite\-sample lower\-tail certificate that remains valid for a data\-selected language\-model policy under arbitrary within\-prompt dependence, by pairing an independent floor proposal with an exact binomial test and Holm’s step\-down\. On matched proposals it uniformly improves the Bonferroni certificate—raising the certified floor and cutting abstention and regret—at the same family\-wise guarantee, without claiming to dominate exact COPS\. The guarantee ends at the declared jury score: return or true\-safety claims require a separate bridge, and audited, human\-anchored evaluation remains the path to safety\-critical deployment\.

## Limitations

StepCOPS cannot improve a poor candidate pool, and it abstains rather than certifying whennnis small,KKis large, orα\\alphais extreme\. Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4)requires frozen candidates, proposal rule, jury, and score maps; independent proposal/certification splits; and i\.i\.d\. certification units—not overlapping windows, adaptively tuned scores, or deployment shift\. Crucially, the guarantee concerns the fixed primary\-jury score distribution: no automatic judge or reward model is universally conservative for human preference, return, or true harm, and the audits quantify but do not remove that proxy gap\. StepCOPS certifies only the floors of*rejected*hypotheses and does not uniformly dominate exact COPS\. The offline\-RL control study is likewise proxy\-limited: its D4RL point estimates need paired uncertainty and fully specified score and baseline implementations before supporting strong dominance claims\.

## Ethical Considerations

A certified proxy can create false reassurance if presented as human\-safety or return control\. We recommend abstention when the exact index is vacuous, calibration units are not defensibly independent, or transfer is unsupported\. Automatic judges require independent audits before safety\-critical deployment\.

## References

- Angelopouloset al\.\(2021\)A\. N\. Angelopoulos, S\. Bates, E\. J\. Candes, M\. I\. Jordan, and L\. LeiLearn then test: calibrating predictive algorithms to achieve risk control\.arXiv preprint arXiv:2110\.01052\.Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p2.1),[§C\.5](https://arxiv.org/html/2609.29549#A3.SS5.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1),[§9\.4](https://arxiv.org/html/2609.29549#S9.SS4.p1.1)\.
- Aula\-Blascoet al\.\(2025\)J\. Aula\-Blasco, J\. Falcão, S\. Sotelo, S\. Paniagua, A\. Gonzalez\-Agirre, and M\. VillegasVeritasQA: a truthfulness benchmark aimed at multilingual transferability\.InProceedings of the 31st International Conference on Computational Linguistics \(COLING\),Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.4.2.1.1)\.
- Bateset al\.\(2021\)S\. Bates, A\. Angelopoulos, L\. Lei, J\. Malik, and M\. JordanDistribution\-free, risk\-controlling prediction sets\.Journal of the ACM\.Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p2.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Chenet al\.\(2025\)C\. Chen, J\. Shen, Z\. Deng, and L\. LeiConformal tail risk control for large language model alignment\.InProceedings of the 42nd International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.267,pp\. 8955–8978\.External Links:[Link](https://proceedings.mlr.press/v267/chen25bd.html)Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p4.1),[§C\.5](https://arxiv.org/html/2609.29549#A3.SS5.SSS0.Px4.p1.1),[§C\.8](https://arxiv.org/html/2609.29549#A3.SS8.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1),[§9\.4](https://arxiv.org/html/2609.29549#S9.SS4.p1.1)\.
- Chen and Goldfarb\-Tarrant \(2025\)H\. Chen and S\. Goldfarb\-TarrantSafer or luckier? LLMs as safety evaluators are not robust to artifacts\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 19750–19766\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.970),[Link](https://aclanthology.org/2025.acl-long.970/)Cited by:[§C\.5](https://arxiv.org/html/2609.29549#A3.SS5.SSS0.Px4.p1.1)\.
- Cuiet al\.\(2024\)J\. Cui, W\. Chiang, I\. Stoica, and C\. HsiehOR\-Bench: an over\-refusal benchmark for large language models\.arXiv preprint arXiv:2405\.20947\.Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.5.2.1.1)\.
- Fuet al\.\(2020\)J\. Fu, A\. Kumar, O\. Nachum, G\. Tucker, and S\. LevineD4RL: datasets for deep data\-driven reinforcement learning\.arXiv preprint arXiv:2004\.07219\.Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Fuet al\.\(2021\)J\. Fu, M\. Norouzi, O\. Nachum, G\. Tucker, Z\. Wang, A\. Novikov, M\. Yang, M\. R\. Zhang, Y\. Chen, A\. Kumar, C\. Paduraru, S\. Levine, and T\. L\. PaineBenchmarks for deep off\-policy evaluation\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Fujimoto and Gu \(2021\)S\. Fujimoto and S\. S\. GuA minimalist approach to offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Fujimotoet al\.\(2019\)S\. Fujimoto, D\. Meger, and D\. PrecupOff\-policy deep reinforcement learning without exploration\.InInternational Conference on Machine Learning,Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Gehmanet al\.\(2020\)S\. Gehman, S\. Gururangan, M\. Sap, Y\. Choi, and N\. A\. SmithRealToxicityPrompts: evaluating neural toxic degeneration in language models\.InFindings of the Association for Computational Linguistics: EMNLP 2020,Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.3.2.1.1)\.
- Goeman and Solari \(2010\)J\. J\. Goeman and A\. SolariThe sequential rejection principle of familywise error control\.The Annals of Statistics38\(6\),pp\. 3782–3810\.Cited by:[§A\.1](https://arxiv.org/html/2609.29549#A1.SS1.p1.2.1),[Appendix B](https://arxiv.org/html/2609.29549#A2.p3.1),[§1](https://arxiv.org/html/2609.29549#S1.p3.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1),[§6](https://arxiv.org/html/2609.29549#S6.SS0.SSS0.Px4.p1.2)\.
- Google DeepMind \(2024\)Google DeepMindFACTS grounding: a new benchmark for evaluating the factuality of large language models\.Note:Technical reportCited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.6.2.1.1)\.
- Hartvigsenet al\.\(2022\)T\. Hartvigsen, S\. Gabriel, H\. Palangi, M\. Sap, D\. Ray, and E\. KamarToxiGen: a large\-scale machine\-generated dataset for adversarial and implicit hate speech detection\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.3.2.1.1)\.
- Holm \(1979\)S\. HolmA simple sequentially rejective multiple test procedure\.Scandinavian Journal of Statistics6\(2\),pp\. 65–70\.Cited by:[§A\.1](https://arxiv.org/html/2609.29549#A1.SS1.p1.2.1),[Appendix B](https://arxiv.org/html/2609.29549#A2.p3.1),[§1](https://arxiv.org/html/2609.29549#S1.p3.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1),[§6](https://arxiv.org/html/2609.29549#S6.SS0.SSS0.Px4.p1.2)\.
- Jiang and Li \(2016\)N\. Jiang and L\. LiDoubly robust off\-policy value evaluation for reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Konyushkovaet al\.\(2021\)K\. Konyushkova, Y\. Chen, T\. L\. Paine, C\. Gulcehre, C\. Paduraru, D\. J\. Mankowitz, M\. Denil, and N\. de FreitasActive offline policy selection\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1)\.
- Kostrikovet al\.\(2022\)I\. Kostrikov, A\. Nair, and S\. LevineOffline reinforcement learning with implicit Q\-learning\.InInternational Conference on Learning Representations,Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Kumaret al\.\(2020\)A\. Kumar, A\. Zhou, G\. Tucker, and S\. LevineConservative Q\-learning for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Leet al\.\(2019\)H\. Le, C\. Voloshin, and Y\. YueBatch policy learning under constraints\.InInternational Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Linet al\.\(2022\)S\. Lin, J\. Hilton, and O\. EvansTruthfulQA: measuring how models mimic human falsehoods\.InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics \(ACL\),Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.4.2.1.1)\.
- Mazeikaet al\.\(2024\)M\. Mazeika, L\. Phan, X\. Yin, A\. Zou, Z\. Wang, N\. Mu, E\. Sakhaee, N\. Li, S\. Basart, B\. Li, D\. Forsyth, and D\. HendrycksHarmBench: a standardized evaluation framework for automated red teaming and robust refusal\.InInternational Conference on Machine Learning \(ICML\),Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.2.2.1.1)\.
- Nakamotoet al\.\(2023\)M\. Nakamoto, Y\. Zhai, A\. Singh, M\. S\. Mark, Y\. Ma, C\. Finn, A\. Kumar, and S\. LevineCal\-QL: calibrated offline RL pre\-training for efficient online fine\-tuning\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[§10](https://arxiv.org/html/2609.29549#S10.p1.1)\.
- Paineet al\.\(2020\)T\. L\. Paine, C\. Paduraru, A\. Michi, C\. Gulcehre, K\. Zolna, A\. Novikov, Z\. Wang, and N\. de FreitasHyperparameter selection for offline reinforcement learning\.arXiv preprint arXiv:2007\.09055\.Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Precupet al\.\(2000\)D\. Precup, R\. S\. Sutton, and S\. SinghEligibility traces for off\-policy policy evaluation\.InInternational Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Röttgeret al\.\(2024\)P\. Röttger, H\. R\. Kirk, B\. Vidgen, G\. Attanasio, F\. Bianchi, and D\. HovyXSTest: a test suite for identifying exaggerated safety behaviours in large language models\.InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics \(NAACL\),Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.5.2.1.1)\.
- Soulyet al\.\(2024\)A\. Souly, Q\. Lu, D\. Bowen, T\. Trinh, E\. Hsieh, S\. Pandey, P\. Abbeel, J\. Svegliato, S\. Emmons, O\. Watkins, and S\. ToyerA strongreject for empty jailbreaks\.Advances in Neural Information Processing Systems \(NeurIPS\) Datasets and Benchmarks\.Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.2.2.1.1)\.
- Stutzet al\.\(2022\)D\. Stutz, K\. Dvijotham, A\. T\. Cemgil, and A\. DoucetLearning optimal conformal classifiers\.InInternational Conference on Learning Representations,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p5.1),[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[Appendix I](https://arxiv.org/html/2609.29549#A9.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Taufiqet al\.\(2022\)M\. F\. Taufiq, J\. Ton, R\. Cornish, Y\. W\. Teh, and A\. DoucetConformal off\-policy prediction in contextual bandits\.InAdvances in Neural Information Processing Systems,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p5.1),[Appendix H](https://arxiv.org/html/2609.29549#A8.p2.1),[Appendix I](https://arxiv.org/html/2609.29549#A9.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Thomas and Brunskill \(2016\)P\. Thomas and E\. BrunskillData\-efficient off\-policy policy evaluation for reinforcement learning\.InInternational Conference on Machine Learning,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Wanget al\.\(2025\)Z\. Wang, Q\. Wang, Y\. Zhang, T\. Chen, X\. Zhu, X\. Shi, and K\. XuSConU: selective conformal uncertainty in large language models\.InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics \(Volume 1: Long Papers\),Vienna, Austria,pp\. 19052–19075\.External Links:[Document](https://dx.doi.org/10.18653/v1/2025.acl-long.934),[Link](https://aclanthology.org/2025.acl-long.934/)Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p4.1),[§C\.5](https://arxiv.org/html/2609.29549#A3.SS5.SSS0.Px4.p1.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1),[§9\.4](https://arxiv.org/html/2609.29549#S9.SS4.p1.1)\.
- Weiet al\.\(2024a\)J\. Wei, N\. Karina, H\. W\. Chung, Y\. J\. Jiao, S\. Papay, A\. Glaese, J\. Schulman, and W\. FedusMeasuring short\-form factuality in large language models \(SimpleQA\)\.arXiv preprint arXiv:2411\.04368\.Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.6.2.1.1)\.
- Weiet al\.\(2024b\)J\. Wei, C\. Yang, X\. Song, Y\. Lu, N\. Hu, J\. Huang, D\. Tran, D\. Peng, R\. Liu, D\. Huang, C\. Du, and Q\. V\. LeLong\-form factuality in large language models \(LongFact\)\.arXiv preprint arXiv:2403\.18802\.Cited by:[Table 8](https://arxiv.org/html/2609.29549#A3.T8.2.1.6.2.1.1)\.
- Xuet al\.\(2022\)H\. Xu, X\. Zhan, and X\. ZhuConstraints penalized Q\-learning for safe offline reinforcement learning\.InAAAI Conference on Artificial Intelligence,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p5.1),[§2](https://arxiv.org/html/2609.29549#S2.p1.1)\.
- Yanget al\.\(2022\)M\. Yang, B\. Dai, O\. Nachum, G\. Tucker, and D\. SchuurmansOffline policy selection under uncertainty\.InInternational Conference on Artificial Intelligence and Statistics,Cited by:[Appendix B](https://arxiv.org/html/2609.29549#A2.p1.1)\.

## Appendix AAlgorithms and Supplementary StepCOPS Material

Algorithm[2](https://arxiv.org/html/2609.29549#A1.F2)makes the split discipline explicit\. If no admissible order statistic exists, returning the empirical minimum would be anti\-conservative; the algorithm instead returns a known support floor or−∞\-\\infty\.

*Input:*logged data𝒟\\mathcal\{D\}, candidates\{πk\}k=1K\\\{\\pi\_\{k\}\\\}\_\{k=1\}^\{K\},α\\alpha,δ\\delta, a score\-construction procedure, and an optional known support floorL0L\_\{0\}\.

1. 1\.Split𝒟\\mathcal\{D\}into𝒟train\\mathcal\{D\}\_\{\\rm train\}and𝒟cal\\mathcal\{D\}\_\{\\rm cal\}at the trajectory level\.
2. 2\.Using only𝒟train\\mathcal\{D\}\_\{\\rm train\}, train and freeze policies, fit nuisance models, definesks\_\{k\}, and fix all hyperparameters\.
3. 3\.Letn=\|𝒟cal\|n=\|\\mathcal\{D\}\_\{\\rm cal\}\|and computer⋆r^\{\\star\}by Eq\. \([1](https://arxiv.org/html/2609.29549#S5.E1)\)\.
4. 4\.For everykk, computeSi​k=sk​\(τi\)S\_\{ik\}=s\_\{k\}\(\\tau\_\{i\}\)\. SetBkα=S\(r⋆\)​kB\_\{k\}^\{\\alpha\}=S\_\{\(r^\{\\star\}\)k\}whenr⋆r^\{\\star\}exists; otherwise useL0L\_\{0\}when available and−∞\-\\inftywhen it is not\.
5. 5\.Returnk^∈arg​maxk⁡Bkα\\widehat\{k\}\\in\\operatorname\*\{arg\\,max\}\_\{k\}B\_\{k\}^\{\\alpha\},πk^\\pi\_\{\\widehat\{k\}\}, andBk^αB\_\{\\widehat\{k\}\}^\{\\alpha\}\.

Figure 2:COPS with exact order\-statistic calibration\.Algorithm 2:StepCOPS \(closed\-testing certification\)\.

Input:frozen candidates\{πk\}k=1K\\\{\\pi\_\{k\}\\\}\_\{k=1\}^\{K\}, jury score map, levelsα,δ\\alpha,\\delta, operating thresholdc0c\_\{0\}; proposal split𝒫\\mathcal\{P\}\(mmunits\), certification split𝒞\\mathcal\{C\}\(nnunits, independent of𝒫\\mathcal\{P\}\)\.

1. 1\.Propose:on𝒫\\mathcal\{P\}, setck=q^0\.075,kpropc\_\{k\}=\\widehat\{q\}^\{\\,\\mathrm\{prop\}\}\_\{0\.075,k\}for everykk\.
2. 2\.Count:on𝒞\\mathcal\{C\},Xk=∑i=1n𝟏\{Si​k<ck\}X\_\{k\}=\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{S\_\{ik\}<c\_\{k\}\\\}\.
3. 3\.Test:pk=ℙ\{Binomial\(n,α\)≤Xk\}p\_\{k\}=\\mathbb\{P\}\\\{\\operatorname\{Binomial\}\(n,\\alpha\)\\leq X\_\{k\}\\\}for everykk\.
4. 4\.Holm step\-down:sortp\(1\)≤⋯≤p\(K\)p\_\{\(1\)\}\\leq\\cdots\\leq p\_\{\(K\)\}; rejectH\(j\)H\_\{\(j\)\}whilep\(j\)≤δ/\(K−j\+1\)p\_\{\(j\)\}\\leq\\delta/\(K\-j\+1\), stopping at the first failure\. Letℛ\\mathcal\{R\}be the rejected set\.
5. 5\.Select:ifℛ≠∅\\mathcal\{R\}\\neq\\emptysetandmaxk∈ℛ⁡ck≥c0\\max\_\{k\\in\\mathcal\{R\}\}c\_\{k\}\\geq c\_\{0\}, returnk^∈arg​maxk∈ℛ⁡ck\\widehat\{k\}\\in\\operatorname\*\{arg\\,max\}\_\{k\\in\\mathcal\{R\}\}c\_\{k\}and floorck^c\_\{\\widehat\{k\}\}; elseabstain\.

Figure 3:StepCOPS certifies independently proposed floors by an exact binomial test and Holm’s step\-down, then deploys the largest certified floor\.### A\.1Proof of Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4)

###### Proof\.

Condition on the proposal split; the floorsc1,…,cKc\_\{1\},\\ldots,c\_\{K\}are then fixed\. If candidatekkhas an invalid proposed floor,ck\>qα,kc\_\{k\}\>q\_\{\\alpha,k\}, then by the definition of the lower quantile,

ℙ⁡\(Sk<ck\)≥ℙ⁡\(Sk≤qα,k\)≥α,\\mathbb\{P\}\(S\_\{k\}<c\_\{k\}\)\\geq\\mathbb\{P\}\(S\_\{k\}\\leq q\_\{\\alpha,k\}\)\\geq\\alpha,soHkH\_\{k\}is true\. UnderHkH\_\{k\}, the countXkX\_\{k\}is stochastically no smaller thanBinomial⁡\(n,α\)\\operatorname\{Binomial\}\(n,\\alpha\), so the lower\-tail binomialpp\-value \([3](https://arxiv.org/html/2609.29549#S6.E3)\) is super\-uniform\. Holm’s procedure applied to\(p1,…,pK\)\(p\_\{1\},\\ldots,p\_\{K\}\)strongly controls the probability of rejecting*any*true null at levelδ\\deltawithout an independence assumption across candidates\([Holm, 1979](https://arxiv.org/html/2609.29549#bib.bib23);[Goeman and Solari, 2010](https://arxiv.org/html/2609.29549#bib.bib24)\)\. On the complementary event—probability at least1−δ1\-\\delta—no true null is rejected, i\.e\. every rejected hypothesis has a valid floorck≤qα,kc\_\{k\}\\leq q\_\{\\alpha,k\}\. The selected indexk^\\widehat\{k\}in \([4](https://arxiv.org/html/2609.29549#S6.E4)\) belongs to the rejected setℛ\\mathcal\{R\}, so on the same event its floor is valid\. Averaging over the proposal split preserves the bound\. ∎

### A\.2Scope table, worked example, and non\-claims

Table 2:What StepCOPS certifies\. Only the first row is distribution\-free under the stated sampling assumptions; the certified floorck^c\_\{\\widehat\{k\}\}is the exact StepCOPS output\.#### Worked certification example\.

WithK=24K=24,δ=0\.05\\delta=0\.05, andn=2,500n=2\{,\}500, suppose the ten smallest exactpp\-values are

0\.000001,0\.000004,0\.000019,0\.000087,\\displaystyle 0\.000001,\\ 0\.000004,\\ 0\.000019,\\ 0\.000087,0\.000341,0\.000568,0\.000926,\\displaystyle 0\.000341,\\ 0\.000568,\\ 0\.000926,0\.001481,0\.001858,0\.002885\.\\displaystyle 0\.001481,\\ 0\.001858,\\ 0\.002885\.Bonferroni’s threshold isδ/K=0\.002083\\delta/K=0\.002083, so it certifies the first nine proposals and stops\. Holm compares the tenth value withδ/\(24−10\+1\)=0\.003333\\delta/\(24\-10\+1\)=0\.003333and certifies it as well\. In the evaluation, that tenth hypothesis belongs to C15, whose proposed floor is62\.462\.4and whose certification split contains209209scores below the proposal; C15 has the largest certified proposal\. This is a concrete dataset on which the step\-down changes the selected policy while retaining the same95%95\\%family\-wise guarantee\.

#### What the theorem does not establish\.

Theorem[4](https://arxiv.org/html/2609.29549#Thmtheorem4)does not establish that the jury score equals human preference, human safety, or true harm; conditional coverage for every task, topic, or demographic group; validity after changing a judge, prompt, parser, candidate, or score normalization; validity under deployment\-distribution shift; that the proposal split chose the optimal floor; a valid floor for candidates whose hypotheses were not rejected; or uniform superiority to exact COPS, DKW, Berk–Jones, or any other confidence\-bound construction\. These limitations remain adjacent to the theorem\.

## Appendix BExtended Related Work

Off\-policy evaluation estimates a target policy from behavior\-policy data\. Importance weighting is unbiased only under support assumptions and can have high variance\([Precup et al\., 2000](https://arxiv.org/html/2609.29549#bib.bib6)\); doubly robust estimators combine weighting with fitted values\([Jiang and Li, 2016](https://arxiv.org/html/2609.29549#bib.bib8);[Thomas and Brunskill, 2016](https://arxiv.org/html/2609.29549#bib.bib7)\); and fitted Q\-evaluation is common in deep offline RL\([Le et al\., 2019](https://arxiv.org/html/2609.29549#bib.bib9)\)\. Offline policy\-selection studies typically use these tools to choose hyperparameters or checkpoints by estimated mean return\([Paine et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib10);[Konyushkova et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib11);[Yang et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib12);[Fu et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib13)\)\. COPS instead targets a lower quantile of a declared score distribution\.

The statistical construction is related to distribution\-free tolerance bounds, conformal calibration, and finite\-sample risk control\. Learn\-Then\-Test \(LTT\) tests risk constraints over a configuration grid\([Angelopoulos et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib17)\), while risk\-controlling prediction sets use related calibration machinery\([Bates et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib18)\)\. COPS uses a classical exact binomial inversion for each quantile and family\-wise control across candidates\. We do not claim order statistics or the binomial CDF as new\. The contribution is their selected\-policy formulation, the explicit separation between calibration and proxy transfer, and the resulting regret decomposition\. This distinction also marks the boundary with standard LTT: an expected\-risk test does not by itself produce the simultaneous candidate\-wise quantile bounds needed to maximize a certified floor\.

StepCOPS additionally imports the closed\-testing machinery of multiple comparisons\. Holm’s sequentially rejective procedure\([Holm, 1979](https://arxiv.org/html/2609.29549#bib.bib23)\)is a shortcut for a Bonferroni\-based closed test and controls family\-wise error under arbitrary dependence;[Goeman and Solari \(2010\)](https://arxiv.org/html/2609.29549#bib.bib24)formalize the general sequential\-rejection principle\. We do not claim these procedures as new\. Our contribution is to pair an*independent*proposal of one floor per candidate with an exact lower\-tail binomial certification and a Holm step\-down, so that the certified floor of the selected policy is valid after selection without any assumption on how candidate scores co\-vary within a prompt, and to show that this uniformly improves the matched\-proposal Bonferroni certificate\. The proposal/certification split follows the data\-splitting logic of selective and post\-selection inference, specialized to lower\-quantile floor certification\.

The closest language\-model work is conformal tail\-risk control for alignment\([Chen et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib20)\), which calibrates monotone response filters against human\-scored distortion risks and develops L\-statistic, DKW, and Berk–Jones constructions\. COPS does not subsume that framework\. Its different target is selection of one member from an arbitrary pool of already trained policies, for which it seeks simultaneous candidate\-wise lower quantile floors and then isolates off\-policy score–return transfer\. SConU instead tests whether an input departs from the calibration uncertainty distribution and targets more conditional uncertainty control\([Wang et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib21)\)\. These distinctions narrow the novelty claim: the contribution is selected\-policy simultaneous certification and the transfer boundary, not conformal control of LLM tails in general\.

Conformal off\-policy prediction has been studied for contextual bandits\([Taufiq et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib15)\), and conformal predictors have been differentiated through during training\([Stutz et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib16)\)\. Risk\-sensitive algorithms such as CPQ modify training itself\([Xu et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib14)\)\. COPS is post hoc: it operates on an already trained pool and can use any predeclared measurable score\. In the language\-model setting, this makes the method a calibrated selection rule over frozen response scores\. It does not remove judge bias or reward\-model gaming; those enter through the separate score\-to\-utility bridge\.

## Appendix CNLP Evaluation Details

### C\.1Full candidate pool

Table[3](https://arxiv.org/html/2609.29549#A3.T3)lists all2424predeclared configurations \(six checkpoints×\\timestwo system prompts×\\timestwo decoding rules\)\. The*standard*prompt is each model’s recommended chat template plus a neutral helpful\-assistant message; the*safety\-aware*prompt is a fixed policy message requiring safe handling of actionable harm while discouraging refusal of benign questions\.*Low entropy*is temperature0\.20\.2, top\-pp0\.900\.90;*moderate entropy*is temperature0\.80\.8, top\-pp0\.950\.95\. Exact repository/tokenizer revisions and chat\-template hashes are recorded in the released manifest\.

Table 3:Predeclared 24\-candidate pool\.
### C\.2Per\-domain reliability

Aggregate ECE \(Table[13](https://arxiv.org/html/2609.29549#A3.T13)\) can hide lower\-tail failure, so per\-domain reliability diagrams and calibration metrics for the range\-penalized primary jury and each individual judge are reported here\. Calibration metrics use out\-of\-sample score probabilities and are not computed on the data used to fit isotonic maps\.

### C\.3Domain\-level native agreement

Table[4](https://arxiv.org/html/2609.29549#A3.T4)gives primary\- and shadow\-jury agreement with benchmark\-native signals per domain, the primary–shadow Spearman correlation, and the primary false\-pass rate\. Truthfulness and factuality are the weakest proxy domains and receive qualitative error analysis rather than being averaged away\.

Table 4:Domain\-level agreement with benchmark\-native evaluators\.Leave\-one\-primary\-judge\-out selection \(Table[5](https://arxiv.org/html/2609.29549#A3.T5)\) changes the selected primaryq0\.10q\_\{0\.10\}by at most1\.21\.2points, and any single judge alone loses2\.72\.7points and considerable rank agreement—evidence that the multi\-judge score is not driven by one member while still exhibiting reported proxy sensitivity\.

Table 5:Leave\-one\-primary\-judge\-out \(reranking only\)\.
### C\.4Selection stability and jury ablation

Table[6](https://arxiv.org/html/2609.29549#A3.T6)reports selection stability across the500500trials; StepCOPS is the most stable selector\. Table[7](https://arxiv.org/html/2609.29549#A3.T7)varies the score construction: the predeclared range\-penalized median is the operating point, and the more conservative minimum\-of\-three jury is slightly worse on the selected shadow tail\.

Table 6:Selection stability across500500trials\.Table 7:Jury\-construction ablation \(selected\-policy diagnostics\)\.
### C\.5Deferred main\-text audit and ablation material

Table 8:Declared five\-domain benchmark mixture \(1111benchmarks, equal domain mass\)\.Table 9:Coverage over500500paired trials, Wilson95%95\\%intervals\. Shadow row is a diagnostic, not the formal target\.Table 10:Selected\-policy primary\-juryq0\.10q\_\{0\.10\}by domain \(Holm\-adjusted across five domains\)\.Table 11:StepCOPS vs\. proposal\-Bonferroni on identical proposals; paired differences\.#### Two held\-out shadow judges\.

Two further evaluator familiesJS​1,JS​2J\_\{S1\},J\_\{S2\}are never used to propose floors, computepp\-values, or select\. Their median is a held\-out proxy used only for primary\-versus\-shadow quantile gaps, selected\-policy agreement, leave\-family\-out robustness, artifact\-flip comparison, and error stratification\. Shadow agreement is a consistency check, not a theorem about human safety: LLM judges can agree through shared training data, rubrics, or artifacts\.

#### Benchmark\-native evaluators and artifact stress\.

Where a benchmark provides references, categorical targets, or an official evaluator \(e\.g\. reference correctness on TruthfulQA/SimpleQA, safe\-vs\-unsafe expectations on XSTest, toxicity labels on ToxiGen, released safety evaluators for HarmBench/StrongREJECT\), we report agreement with that partially independent native signal\. Six predeclared programmatic artifact transformations \(apologetic preface; length\-matched verbosity; Markdown\-only reformatting; refusal\-phrase paraphrase; equivalent answer\-order swap; matched identity substitution\) are applied and retained only when two held\-out semantic\-equivalence models both accept bidirectional entailment and deterministic task checks pass\. This filtering is still automatic and may miss semantic changes, so the analysis is described as an artifact stress test, not a human invariance study\.

#### Data and compute budget\.

A pilot split \(2,0002\{,\}000units\) freezes rubrics, judge prompts, score maps, and the proposal level\. Each trial draws an independent proposal split \(4,0004\{,\}000units\) and certification split \(2,5002\{,\}500units\); a30,00030\{,\}000\-unit primary reference test and a12,00012\{,\}000\-unit shadow\-judge test approximate population quantiles and the oracle selections; an artifact set of1,0001\{,\}000originals plus transformations audits robustness\. We repeat500500paired trials on fixed checkpoints and benchmark population with independent prompt/generation seeds\. When units are resampled from a finite score dump rather than regenerated, trials are labeled*resampling*and uncertainty is clustered by source prompt\. \(This paragraph restates the budget summarized in Section[9\.3](https://arxiv.org/html/2609.29549#S9.SS3)\.\)

Table 12:Predeclared mean–tail conflict \(selected candidates\)\. Full pool in Appendix[C\.1](https://arxiv.org/html/2609.29549#A3.SS1)\.
#### Full baseline definitions\.

Baselines are mean\-jury selection; empirical VaR and CVaR; a task\-stratified studentized bootstrap VaR LCB; an LTT\-style selector\([Angelopoulos et al\., 2021](https://arxiv.org/html/2609.29549#bib.bib17)\)that maximizes accepted\-candidate mean subject to predeclared failure\-threshold tests; SConU\([Wang et al\., 2025](https://arxiv.org/html/2609.29549#bib.bib21)\)applied as an uncertainty\-outlier abstention followed by mean selection; the L\-statistic, DKW, and Berk–Jones constructions of[Chen et al\. \(2025\)](https://arxiv.org/html/2609.29549#bib.bib20)applied to jury disutility \(CDRC\-L/DKW/BJ\); exact COPS; and proposal\-Bonferroni\. A shadow\-jury selector, a primary\-jury oracle \(full30,00030\{,\}000\-unitq0\.10q\_\{0\.10\}\), and a native\-evaluator oracle are analysis\-only references unavailable during selection\. Artifact stress is motivated by[Chen and Goldfarb\-Tarrant \(2025\)](https://arxiv.org/html/2609.29549#bib.bib22)but is fully automatic\.

### C\.6Multi\-LLM and benchmark\-native audit

#### Primary\-judge reliability\.

Against a frozen task\-specific native\-failure threshold, the range\-penalized jury is better calibrated than any single judge and than a plain median, with AUROC0\.9320\.932, ECE0\.0310\.031, and a5\.3%5\.3\\%false\-pass rate \(Table[13](https://arxiv.org/html/2609.29549#A3.T13)\); metrics use out\-of\-sample score probabilities\. Aggregate ECE alone is insufficient because it can hide lower\-tail failure; per\-domain reliability diagrams are in Appendix[C\.2](https://arxiv.org/html/2609.29549#A3.SS2)\.

Table 13:Primary\-judge reliability against benchmark\-native failure labels\.
#### Domain agreement and leave\-one\-judge\-out\.

Native agreement is weakest on truthfulness and factuality \(macro90\.0%90\.0\\%primary,89\.4%89\.4\\%shadow, primary–shadow Spearman0\.810\.81\), which we flag for qualitative error analysis rather than hide in the macro average \(Appendix[C\.3](https://arxiv.org/html/2609.29549#A3.SS3)\)\. Removing any single primary judge changes the selected lower tail by at most1\.21\.2points \(full jury65\.165\.1; worst leave\-one\-out63\.963\.9\), so the result is not driven by one judge but retains reported proxy sensitivity\. In the audit,12\.4%12\.4\\%\[11\.8,13\.0\]\[11\.8,13\.0\]of outputs have a primary\-judge range above2525points; native error is18\.7%18\.7\\%in that disagreement stratum versus6\.2%6\.2\\%elsewhere, and the range penalty lowers the score by7\.17\.1points there versus1\.81\.8elsewhere\.

#### Artifact stress\.

A “flip” is an absolute score change above2020points after automatic semantic\-equivalence filtering\. The robust jury reduces but does not remove artifact sensitivity: apologetic\-preface and refusal\-phrase paraphrase are worst, with individual\-judge flips up to19\.6%19\.6\\%but a primary\-jury flip of at most8\.9%8\.9\\%\(Table[14](https://arxiv.org/html/2609.29549#A3.T14)\)\. Because equivalence is automatically verified, these are diagnostic, not a human invariance study\.

Table 14:Artifact flip rates \(\>20\>20points\) after automatic equivalence filtering\.

### C\.7Selection stability and ablations

StepCOPS is the most stable selector \(modal frequency84%84\\%, pairwise agreement0\.780\.78, selection entropy0\.550\.55bits, top\-two inclusion95%95\\%; Appendix[C\.4](https://arxiv.org/html/2609.29549#A3.SS4)\)\. Sensitivity analyses confirm the predeclared operating point: the pilot\-fixed proposal level0\.0750\.075balances tightness against abstention \(higher levels are tighter when certified but abstain too often\), proposal size mainly affects efficiency \(not coverage, which enters the proof only through independence\), and larger certification samples trade abstention for a higher floor as expected \(Table[15](https://arxiv.org/html/2609.29549#A3.T15)\)\. The range\-penalized jury is the predeclared operating point; a minimum\-of\-three jury is more conservative on the shadow tail\.

Table 15:Key ablations\. Rows inboldare the predeclared operating points\.
### C\.8Faithful Chen\-method response\-filter comparison

[Chen et al\. \(2025\)](https://arxiv.org/html/2609.29549#bib.bib20)tune a monotone filter over generated responses; because no human annotations are available here, this matched experiment uses jury disutility \(lower VaR/CVaR better; cost is mean responses sampled before acceptance\)\. StepCOPS selects among fixed policies at unit sampling cost; a CDRC filter operates within a policy and trades sampling cost for lower tails\. The combined StepCOPS\-policy\-plus\-CDRC\-BJ\-filter row is a systems result, not the core novelty, and does not reproduce the human\-alignment experiment of[Chen et al\. \(2025\)](https://arxiv.org/html/2609.29549#bib.bib20)\(Table[16](https://arxiv.org/html/2609.29549#A3.T16)\)\.

Table 16:Response\-filter comparison on jury disutility\. Cost is mean responses sampled\.
### C\.9Error analysis and reproducibility

Two independent LLM analysis models assign categories from a frozen taxonomy over at least100100examples in each of four predeclared automatic strata \(primary\-high/shadow\-low; primary\-low/shadow\-high; primary\-judge range\>25\>25; and abstention trials\), reporting disagreements rather than resolving them by intuition\. The leading categories are refusal\-cue sensitivity \(22%22\\%\), hidden harmful detail \(17%17\\%\), unsupported factual claim \(15%15\\%\), verbosity/formatting \(12%12\\%\), and safe\-but\-unhelpful refusal \(10%10\\%\); these are LLM\-generated diagnostics, not human error analysis\. Before the final proposal split we freeze and record all candidate identifiers and revisions, prompts/templates/decoding/parsers, benchmark revisions and prompt hashes, the mixture and task weights, all five judge revisions and the range penalty, the proposal/certification sizes andα,δ,c0\\alpha,\\delta,c\_\{0\}, the Holm ordering and abstention rules, every baseline and compute budget, seeds, metrics and multiplicity rules, and a table schema that preserves parser failures and abstentions\. Candidate\-level primary and shadow score dumps are released where licenses permit, and a single reproducible procedure regenerates every table from those dumps\.

## Appendix DExecuted Exact\-COPS Pilot

The predeclared2424\-candidate study of Section[9](https://arxiv.org/html/2609.29549#S9)was motivated by an earlier executed exact\-COPS pilot, which we report because its negative result shaped the pool design\. We ran exact COPS on six Qwen2\.5\-Instruct configurations—sizes\{0\.5​B,1\.5​B,3​B\}\\\{0\.5\\text\{B\},1\.5\\text\{B\},3\\text\{B\}\\\}crossed with temperatures\{0\.7,1\.0\}\\\{0\.7,1\.0\\\}—with300300TruthfulQA prompts split into150150calibration and150150held\-out, and a frozen Claude Sonnet 4\.5 rubric \(larger better\) scoring one response per candidate–prompt pair with no dropped calls\. WithK=6K=6andδ=0\.05\\delta=0\.05, exact calibration usesr⋆=7r^\{\\star\}=7atα=0\.10\\alpha=0\.10andr⋆=19r^\{\\star\}=19atα=0\.20\\alpha=0\.20\. At both levels the selected bound is no larger than the independent held\-out quantile \(0\.300\.30vs\.0\.300\.30and0\.650\.65vs\.0\.700\.70\), an end\-to\-end consistency check rather than a repeated\-coverage estimate\.

The pilot is negative in the way that matters: atα=0\.10\\alpha=0\.10exact COPS selects Qwen2\.5\-1\.5B@0\.7 \(held\-outqq0\.300\.30\) while mean selection picks Qwen2\.5\-3B@1\.0 \(held\-outqq0\.650\.65\)\. This small, single\-family pool does not exhibit the motivating mean–tail tension, and it uses one benchmark, one judge, and one response per pair\. That negative result is exactly why Section[9](https://arxiv.org/html/2609.29549#S9)predeclares a factorial pool across five model families, five domains, and a range\-penalized multi\-judge score, and why StepCOPS replaces the fully simultaneous bound with a proposal/certification split so that a larger deployable floor is certified without a Bonferroni penalty on every candidate\.

## Appendix EScore Construction Details and Score\-Map Constraints

For a finite\-horizon trajectoryτ=\(s0,a0,r0,…,sH\)\\tau=\(s\_\{0\},a\_\{0\},r\_\{0\},\\ldots,s\_\{H\}\), a stochastic target policyπk\\pi\_\{k\}, behavior densityμ\\mu, discount factorγ\\gamma, fitted action\-value functionQ^k\\widehat\{Q\}\_\{k\}, and induced valueV^k​\(s\)=∫πk​\(a∣s\)​Q^k​\(s,a\)​𝑑a\\widehat\{V\}\_\{k\}\(s\)=\\int\\pi\_\{k\}\(a\\mid s\)\\widehat\{Q\}\_\{k\}\(s,a\)\\,da, the cumulative density ratio isρ0:t\(k\)=∏u=0tπk\(au∣su\)/μ\(au∣su\)\\rho\_\{0:t\}^\{\(k\)\}=\\prod\_\{u=0\}^\{t\}\\pi\_\{k\}\(a\_\{u\}\\mid s\_\{u\}\)/\\mu\(a\_\{u\}\\mid s\_\{u\}\)and the unclipped DR score is

SkDR​\(τ\)\\displaystyle S\_\{k\}^\{\\rm DR\}\(\\tau\)=V^k\(s0\)\+∑t=0H−1γtρ0:t\(k\)\\displaystyle=\\widehat\{V\}\_\{k\}\(s\_\{0\}\)\+\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\rho\_\{0:t\}^\{\(k\)\}\(6\)×\[rt\+γ​V^k​\(st\+1\)−Q^k​\(st,at\)\]\.\\displaystyle\\times\\left\[r\_\{t\}\+\\gamma\\widehat\{V\}\_\{k\}\(s\_\{t\+1\}\)\-\\widehat\{Q\}\_\{k\}\(s\_\{t\},a\_\{t\}\)\\right\]\.For numerical stability we use clipped ratiosρ¯0:t\(k\)=min\{ρ0:t\(k\),ρmax\}\\bar\{\\rho\}\_\{0:t\}^\{\(k\)\}=\\min\\\{\\rho\_\{0:t\}^\{\(k\)\},\\rho\_\{\\max\}\\\}and define the clipped score by replacingρ0:t\(k\)\\rho\_\{0:t\}^\{\(k\)\}withρ¯0:t\(k\)\\bar\{\\rho\}\_\{0:t\}^\{\(k\)\}in Eq\. \([6](https://arxiv.org/html/2609.29549#A5.E6)\)\. The score\-level theorem then certifies the quantile of the clipped\-score distribution rather than of the unclipped DR distribution\.

Two implementation choices interact directly with the assumptions\. First, Eq\. \([6](https://arxiv.org/html/2609.29549#A5.E6)\) requires target and behavior densities with respect to the same base measure\. If deployed policies are deterministic, or behavior densities are estimated by a surrogate, the implemented score must be defined explicitly; the theorem then certifies that implemented score law rather than an abstract DR ideal\. Second, global self\-normalization across the calibration set couples scores and can violate the i\.i\.d\. assumption\. Its normalizing constants must therefore be learned on an independent auxiliary split, or the theory must be extended to the induced dependence\. The default result assumes fixed per\-trajectory maps\.

## Appendix FProofs for the Score\-Level Results

### F\.1Deferred statements

The following results are summarized in Section[7](https://arxiv.org/html/2609.29549#S7); we state them in full here, followed by their proofs\.

###### Corollary 8\(Closed\-form DKW coverage\)\.

Under Assumptions[2](https://arxiv.org/html/2609.29549#Thmtheorem2)and[3](https://arxiv.org/html/2609.29549#Thmtheorem3), ifα\>Δn\\alpha\>\\Delta\_\{n\}, then

ℙ\(Bk,DKWα≤qα\(Fk\),∀k∈\[K\]\)≥1−δ\.\\mathbb\{P\}\\\!\\left\(B\_\{k,\{\\rm DKW\}\}^\{\\alpha\}\\leq q\_\{\\alpha\}\(F\_\{k\}\),\\ \\forall k\\in\[K\]\\right\)\\geq 1\-\\delta\.

The simultaneous event also supports comparison with any fixed baseline included in the candidate set\. What it guarantees is dominance of the baseline’s lower confidence floor\. Stronger safe\-improvement language would require showing that the selected policy exceeds the baseline’s*true*quantile, which these one\-sided bounds alone cannot establish\.

###### Theorem 9\(Post\-selection dominance of a certified baseline floor\)\.

Letk0k\_\{0\}be any fixed baseline candidate in\[K\]\[K\]\. On the simultaneous coverage event of Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5),

qα​\(Fk^\)≥Bk^α≥Bk0α\.q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\geq B\_\{\\widehat\{k\}\}^\{\\alpha\}\\geq B\_\{k\_\{0\}\}^\{\\alpha\}\.If additionallyBk^α≤Qα​\(πk^\)B\_\{\\widehat\{k\}\}^\{\\alpha\}\\leq Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)via score\-to\-return transfer, thenQα​\(πk^\)≥Bk0αQ\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\geq B\_\{k\_\{0\}\}^\{\\alpha\}\. If the baseline also satisfies score\-to\-return transfer,Bk0αB\_\{k\_\{0\}\}^\{\\alpha\}is a valid lower confidence bound onQα​\(πk0\)Q\_\{\\alpha\}\(\\pi\_\{k\_\{0\}\}\)\. None of these inequalities impliesQα​\(πk^\)≥Qα​\(πk0\)Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\geq Q\_\{\\alpha\}\(\\pi\_\{k\_\{0\}\}\)\.

For regret we use the DKW selector because the uniform CDF event gives a clean two\-sided quantile localization\. For0≤η<α0\\leq\\eta<\\alpha, the lower\-tail quantile modulusωα​\(η\)=supk∈\[K\]\[qα​\(Fk\)−qα−η​\(Fk\)\]\\omega\_\{\\alpha\}\(\\eta\)=\\sup\_\{k\\in\[K\]\}\[q\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\-\\eta\}\(F\_\{k\}\)\]measures local flatness of the score CDF near the target lower quantile; it is small when the score distribution has nonvanishing density nearqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\)and can be large when there are atoms, plateaus, or sparse lower\-tail samples\.

###### Theorem 10\(Selection regret with score\-to\-return mismatch\)\.

Assumeα\>2​Δn\\alpha\>2\\Delta\_\{n\}and letk^\\widehat\{k\}be selected by the DKW COPS bound\. On the DKW simultaneous event,

qα​\(Fk\)−qα​\(Fk^\)≤ωα​\(2​Δn\),for every​k∈\[K\]\.\\begin\{gathered\}q\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\leq\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\),\\\\ \\text\{for every \}k\\in\[K\]\.\\end\{gathered\}In particular, for the score oraclekS⋆k\_\{S\}^\{\\star\},qα​\(FkS⋆\)−qα​\(Fk^\)≤ωα​\(2​Δn\)q\_\{\\alpha\}\(F\_\{k\_\{S\}^\{\\star\}\}\)\-q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\leq\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)\. Lettingεα=supk∈\[K\]\|Qα​\(πk\)−qα​\(Fk\)\|\\varepsilon\_\{\\alpha\}=\\sup\_\{k\\in\[K\]\}\\left\|Q\_\{\\alpha\}\(\\pi\_\{k\}\)\-q\_\{\\alpha\}\(F\_\{k\}\)\\right\|, the deployment\-return regret satisfies

Qα​\(πkR⋆\)−Qα​\(πk^\)≤ωα​\(2​Δn\)\+2​εα\.Q\_\{\\alpha\}\(\\pi\_\{k\_\{R\}^\{\\star\}\}\)\-Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\leq\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)\+2\\varepsilon\_\{\\alpha\}\.

###### Corollary 11\(Rate under lower density near the target quantile\)\.

If for everykk,FkF\_\{k\}has density at leastc\>0c\>0on the interval betweenqα−2​Δn​\(Fk\)q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)andqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\), thenωα​\(2​Δn\)≤2​Δn/c\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)\\leq 2\\Delta\_\{n\}/cand the score\-oracle regret scales asO⁡\(log⁡\(K/δ\)/n\)O\\\!\\left\(\\sqrt\{\\log\(K/\\delta\)/n\}\\right\)\.

### F\.2Proof of Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)

If Eq\. \([1](https://arxiv.org/html/2609.29549#S5.E1)\) has no admissible index, the result is immediate: a true support floor is no larger than every quantile, and−∞\-\\inftyis trivially valid\. Otherwise, fixkkand writexk=qα​\(Fk\)x\_\{k\}=q\_\{\\alpha\}\(F\_\{k\}\)\. By the definition of the lower quantile,Fk​\(xk\)≥αF\_\{k\}\(x\_\{k\}\)\\geq\\alpha, including whenFkF\_\{k\}has an atom atxkx\_\{k\}\. The eventS\(r\)​k\>xkS\_\{\(r\)k\}\>x\_\{k\}occurs exactly when fewer thanrrcalibration scores are at mostxkx\_\{k\}\. Conditional on the fixed score map,

Nk\(xk\)=∑i=1n𝟏\{Si​k≤xk\}∼Binomial\(n,Fk\(xk\)\)\.N\_\{k\}\(x\_\{k\}\)=\\sum\_\{i=1\}^\{n\}\\mathbf\{1\}\\\{S\_\{ik\}\\leq x\_\{k\}\\\}\\sim\\operatorname\{Binomial\}\(n,F\_\{k\}\(x\_\{k\}\)\)\.The lower tail of a binomial variable is nonincreasing in its success probability, and hence

ℙ\{S\(r\)​k\>xk\}\\displaystyle\\mathbb\{P\}\\\{S\_\{\(r\)k\}\>x\_\{k\}\\\}=ℙ\{Nk\(xk\)<r\}\\displaystyle=\\mathbb\{P\}\\\{N\_\{k\}\(x\_\{k\}\)<r\\\}≤ℙ\{Binomial\(n,α\)<r\}\.\\displaystyle\\leq\\mathbb\{P\}\\\{\\operatorname\{Binomial\}\(n,\\alpha\)<r\\\}\.Forr=r⋆r=r^\{\\star\}, Eq\. \([1](https://arxiv.org/html/2609.29549#S5.E1)\) bounds this probability byδ/K\\delta/K\. A union bound gives simultaneous coverage\. No independence across candidates is used: theKKscores may be arbitrarily dependent within each calibration unit\. On the simultaneous event the inequality holds for every index, so it also holds for the random maximizerk^\\widehat\{k\}\.

### F\.3Proof of Corollary[8](https://arxiv.org/html/2609.29549#Thmtheorem8)

LetF^k\\widehat\{F\}\_\{k\}be the empirical CDF and letq^k,u=inf\{x:F^k​\(x\)≥u\}\\widehat\{q\}\_\{k,u\}=\\inf\\\{x:\\widehat\{F\}\_\{k\}\(x\)\\geq u\\\}\. DKW and a union bound give an event of probability at least1−δ1\-\\deltaon which

supx\|F^k​\(x\)−Fk​\(x\)\|≤Δnfor every​k\.\\sup\_\{x\}\|\\widehat\{F\}\_\{k\}\(x\)\-F\_\{k\}\(x\)\|\\leq\\Delta\_\{n\}\\quad\\text\{for every \}k\.Setβn=α−Δn\\beta\_\{n\}=\\alpha\-\\Delta\_\{n\}\. SinceFk​\(qα​\(Fk\)\)≥αF\_\{k\}\(q\_\{\\alpha\}\(F\_\{k\}\)\)\\geq\\alpha, the DKW event impliesF^k​\(qα​\(Fk\)\)≥βn\\widehat\{F\}\_\{k\}\(q\_\{\\alpha\}\(F\_\{k\}\)\)\\geq\\beta\_\{n\}\. Therefore

Bk,DKWα=q^k,βn≤qα​\(Fk\)\.B\_\{k,\{\\rm DKW\}\}^\{\\alpha\}=\\widehat\{q\}\_\{k,\\beta\_\{n\}\}\\leq q\_\{\\alpha\}\(F\_\{k\}\)\.This generalized\-inverse argument includes discrete distributions and ties\.

### F\.4Proof of Theorem[9](https://arxiv.org/html/2609.29549#Thmtheorem9)

Simultaneous coverage givesqα​\(Fk^\)≥Bk^αq\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\geq B\_\{\\widehat\{k\}\}^\{\\alpha\}\. Maximization givesBk^α≥Bk0αB\_\{\\widehat\{k\}\}^\{\\alpha\}\\geq B\_\{k\_\{0\}\}^\{\\alpha\}\. If the selected candidate satisfies transfer, thenQα​\(πk^\)≥Bk^αQ\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\\geq B\_\{\\widehat\{k\}\}^\{\\alpha\}as well\. Applying transfer to the baseline shows only thatBk0αB\_\{k\_\{0\}\}^\{\\alpha\}is a lower confidence floor for its return quantile; it does not compare the two population return quantiles\.

### F\.5Proof of Theorem[10](https://arxiv.org/html/2609.29549#Thmtheorem10)

On the same DKW event, generalized\-inverse CDF inequalities give

qα−2​Δn​\(Fk\)≤Bk,DKWα≤qα​\(Fk\)for every​k\.q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)\\leq B\_\{k,\{\\rm DKW\}\}^\{\\alpha\}\\leq q\_\{\\alpha\}\(F\_\{k\}\)\\qquad\\text\{for every \}k\.For the lower inequality, everyx<qα−2​Δn​\(Fk\)x<q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)hasFk​\(x\)<α−2​ΔnF\_\{k\}\(x\)<\\alpha\-2\\Delta\_\{n\}, soF^k​\(x\)<α−Δn\\widehat\{F\}\_\{k\}\(x\)<\\alpha\-\\Delta\_\{n\}and the empirical\(α−Δn\)\(\\alpha\-\\Delta\_\{n\}\)\-quantile cannot lie belowqα−2​Δn​\(Fk\)q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)\. Consequently, for anykk,

qα​\(Fk\)−qα​\(Fk^\)\\displaystyle q\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)≤qα​\(Fk\)−Bk^,DKWα\\displaystyle\\leq q\_\{\\alpha\}\(F\_\{k\}\)\-B\_\{\\widehat\{k\},\{\\rm DKW\}\}^\{\\alpha\}≤qα​\(Fk\)−Bk,DKWα\\displaystyle\\leq q\_\{\\alpha\}\(F\_\{k\}\)\-B\_\{k,\{\\rm DKW\}\}^\{\\alpha\}≤qα​\(Fk\)−qα−2​Δn​\(Fk\)\\displaystyle\\leq q\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)≤ωα​\(2​Δn\)\.\\displaystyle\\leq\\omega\_\{\\alpha\}\(2\\Delta\_\{n\}\)\.Apply this inequality tokR⋆k\_\{R\}^\{\\star\}\. The definition ofεα\\varepsilon\_\{\\alpha\}gives

Qα​\(πkR⋆\)≤qα​\(FkR⋆\)\+εα,qα​\(Fk^\)≤Qα​\(πk^\)\+εα,Q\_\{\\alpha\}\(\\pi\_\{k\_\{R\}^\{\\star\}\}\)\\leq q\_\{\\alpha\}\(F\_\{k\_\{R\}^\{\\star\}\}\)\+\\varepsilon\_\{\\alpha\},\\qquad q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\+\\varepsilon\_\{\\alpha\},which proves the return\-regret claim\. Under the density condition in Corollary[11](https://arxiv.org/html/2609.29549#Thmtheorem11), inverse\-CDF Lipschitzness givesqα​\(Fk\)−qα−2​Δn​\(Fk\)≤2​Δn/cq\_\{\\alpha\}\(F\_\{k\}\)\-q\_\{\\alpha\-2\\Delta\_\{n\}\}\(F\_\{k\}\)\\leq 2\\Delta\_\{n\}/c\.

## Appendix GTransfer Geometry and Proofs

The coupling conditions summarized in Section[8](https://arxiv.org/html/2609.29549#S8)are stated in full here\.

###### Proposition 12\(Pointwise pessimism implies quantile conservatism\)\.

Suppose there exists a coupling of the calibration scoreSkS\_\{k\}and deployment returnRkR\_\{k\}withSk≤RkS\_\{k\}\\leq R\_\{k\}almost surely\. Thenqα​\(Fk\)≤Qα​\(πk\)q\_\{\\alpha\}\(F\_\{k\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)for everyα∈\(0,1\)\\alpha\\in\(0,1\)\.

In practice, exact pointwise pessimism is too strong: clipping, importance weights, and finite\-sample fitting introduce small violations\. An approximate condition is enough\.

###### Proposition 13\(Approximate lower\-tail transfer\)\.

Suppose there existηk≥0\\eta\_\{k\}\\geq 0andζk∈\[0,α\)\\zeta\_\{k\}\\in\[0,\\alpha\)such that some coupling ofSkS\_\{k\}andRkR\_\{k\}satisfiesℙ⁡\(Sk≤Rk\+ηk\)≥1−ζk\\mathbb\{P\}\(S\_\{k\}\\leq R\_\{k\}\+\\eta\_\{k\}\)\\geq 1\-\\zeta\_\{k\}\. Thenqα−ζk​\(Fk\)−ηk≤Qα​\(πk\)q\_\{\\alpha\-\\zeta\_\{k\}\}\(F\_\{k\}\)\-\\eta\_\{k\}\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)\.

###### Proposition 15\(No\-overlap obstruction to return certification\)\.

Consider a one\-step MDP with initial states0s\_\{0\}and actionsab,ata\_\{b\},a\_\{t\}\. The behavior policy choosesaba\_\{b\}with probability one and receives reward00\. The target policy choosesata\_\{t\}with probability one\. EnvironmentsM−M\_\{\-\}andM\+M\_\{\+\}agree onaba\_\{b\}but give rewards−1\-1and\+1\+1, respectively, onata\_\{t\}\. Every logged dataset—and hence every logged\-data\-only output—is identical underM−M\_\{\-\}andM\+M\_\{\+\}, while

QαM−\(π\)=−1,QαM\+\(π\)=\+1,for all​α∈\(0,1\)\.\\begin\{gathered\}Q\_\{\\alpha\}^\{M\_\{\-\}\}\(\\pi\)=\-1,\\qquad Q\_\{\\alpha\}^\{M\_\{\+\}\}\(\\pi\)=\+1,\\\\ \\text\{for all \}\\alpha\\in\(0,1\)\.\\end\{gathered\}Any logged\-data\-only procedure therefore returns the same certificate in both environments\. Validity in both permits at most the known lower\-support bound−1\-1; without bounded rewards, no nontrivial finite floor exists\.

Figure[4](https://arxiv.org/html/2609.29549#A7.F4)visualizes the all\-level stochastic ordering that is sufficient for transfer\. The score CDF lies above the return CDF, so its lower quantiles lie to the left\.

xxCDFα\\alphaFkF\_\{k\}\(score\)HkH\_\{k\}\(return\)qα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\)Qα​\(πk\)Q\_\{\\alpha\}\(\\pi\_\{k\}\)transfer gapFigure 4:Geometry of a sufficient transfer condition\. HereFk​\(x\)≥Hk​\(x\)F\_\{k\}\(x\)\\geq H\_\{k\}\(x\)for allxx, so the return law first\-order stochastically dominates the score law andqα​\(Fk\)≤Qα​\(πk\)q\_\{\\alpha\}\(F\_\{k\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)\.### G\.1Proof of Proposition[12](https://arxiv.org/html/2609.29549#Thmtheorem12)

IfSk≤RkS\_\{k\}\\leq R\_\{k\}almost surely, thenℙ⁡\(Sk≤x\)≥ℙ⁡\(Rk≤x\)\\mathbb\{P\}\(S\_\{k\}\\leq x\)\\geq\\mathbb\{P\}\(R\_\{k\}\\leq x\)for everyxx\. ThusFk​\(x\)≥Hk​\(x\)F\_\{k\}\(x\)\\geq H\_\{k\}\(x\), and generalized inverses giveqα​\(Fk\)≤Qα​\(πk\)q\_\{\\alpha\}\(F\_\{k\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)for everyα\\alpha\.

### G\.2Proof of Proposition[13](https://arxiv.org/html/2609.29549#Thmtheorem13)

Letx=Qα​\(πk\)x=Q\_\{\\alpha\}\(\\pi\_\{k\}\)\. On the eventSk≤Rk\+ηkS\_\{k\}\\leq R\_\{k\}\+\\eta\_\{k\}, the implicationRk≤x⇒Sk≤x\+ηkR\_\{k\}\\leq x\\Rightarrow S\_\{k\}\\leq x\+\\eta\_\{k\}holds\. Therefore

ℙ⁡\(Sk≤x\+ηk\)≥ℙ⁡\(Rk≤x\)−ζk≥α−ζk,\\mathbb\{P\}\(S\_\{k\}\\leq x\+\\eta\_\{k\}\)\\geq\\mathbb\{P\}\(R\_\{k\}\\leq x\)\-\\zeta\_\{k\}\\geq\\alpha\-\\zeta\_\{k\},which yieldsqα−ζk​\(Fk\)−ηk≤Qα​\(πk\)q\_\{\\alpha\-\\zeta\_\{k\}\}\(F\_\{k\}\)\-\\eta\_\{k\}\\leq Q\_\{\\alpha\}\(\\pi\_\{k\}\)\.

### G\.3Proof of Proposition[15](https://arxiv.org/html/2609.29549#Thmtheorem15)

The behavior policy never selectsata\_\{t\}, soM−M\_\{\-\}andM\+M\_\{\+\}induce the same logged\-data law\. Every logged\-data\-only procedure consequently has the same output distribution in both environments\. A floor valid in both cannot exceedmin⁡\{QαM−​\(π\),QαM\+​\(π\)\}=−1\\min\\\{Q\_\{\\alpha\}^\{M\_\{\-\}\}\(\\pi\),Q\_\{\\alpha\}^\{M\_\{\+\}\}\(\\pi\)\\\}=\-1\. This is the known support floor; if rewards have no known lower bound, no nontrivial finite floor is uniformly valid\.

## Appendix HDetailed Offline\-Control Experiments

Table 17:Rollout\-estimatedQ0\.10Q\_\{0\.10\}of policies selected without rollout access\. Replay datasets \(†\) are dependence stress tests\. FullQ0\.10Q\_\{0\.10\}andCVaR0\.10\\operatorname\{CVaR\}\_\{0\.10\}results are in Table[20](https://arxiv.org/html/2609.29549#A8.T20)\.The experiments target three separate questions\. The first is score validity: whether the empirical frequency ofBkα≤qα​\(Fk\)B\_\{k\}^\{\\alpha\}\\leq q\_\{\\alpha\}\(F\_\{k\}\)matches the nominal simultaneous guarantee whenqα​\(Fk\)q\_\{\\alpha\}\(F\_\{k\}\)is estimated from an independent large behavior\-policy score sample\. The second is selection quality: whether maximizingBkαB\_\{k\}^\{\\alpha\}selects policies with better deployment lower tails than mean\-OPE, pessimistic mean\-OPE, empirical CVaR, and conformal/OPE interval baselines\. The third is transfer validity: whether, for the policies COPS actually selects,qα​\(Fk^\)≤Qα​\(πk^\)q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\\leq Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)is supported by independent deployment rollouts\. Together these three questions stress the score layer, the selector, and the score\-to\-return bridge separately rather than conflating them\.

The benchmark suite is D4RL continuous\-control\([Fu et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib19)\):halfcheetah\-medium,halfcheetah\-medium\-replay,halfcheetah\-medium\-expert,hopper\-medium,hopper\-medium\-replay,walker2d\-medium, andant\-medium\. Candidate policies are trained using CQL\([Kumar et al\., 2020](https://arxiv.org/html/2609.29549#bib.bib1)\), IQL\([Kostrikov et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib2)\), BCQ\([Fujimoto et al\., 2019](https://arxiv.org/html/2609.29549#bib.bib3)\), TD3\+BC\([Fujimoto and Gu, 2021](https://arxiv.org/html/2609.29549#bib.bib4)\), and Cal\-QL\([Nakamoto et al\., 2023](https://arxiv.org/html/2609.29549#bib.bib5)\), with four hyperparameter settings per algorithm\. All training, score construction, and hyperparameter choices use only the training split before the final calibration run, in keeping with Assumption[2](https://arxiv.org/html/2609.29549#Thmtheorem2)\. The full baseline list, including FQE/DR mean\-OPE, bootstrap and asymptotic mean\-OPE intervals, pessimistic FQE, plug\-in lower\-tail quantile and empirical CVaR over the same calibration scores, conformal/risk\-control\-inspired interval selection\([Taufiq et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib15);[Stutz et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib16)\), and rollout oracles forQαQ\_\{\\alpha\}andCVaRα\\operatorname\{CVaR\}\_\{\\alpha\}used as analysis\-only references, is detailed in Appendix[I](https://arxiv.org/html/2609.29549#A9)\. We report mean return, rollout\-estimatedQαQ\_\{\\alpha\}andCVaRα\\operatorname\{CVaR\}\_\{\\alpha\}, a common\-threshold failure diagnosticℙ^​\(R<ce\)\\widehat\{\\mathbb\{P\}\}\(R<c\_\{e\}\)with the environment\-specificcec\_\{e\}fixed before method comparison, the certified score floor, the additive tightness gapqα​\(Fk^\)−Bk^αq\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\-B\_\{\\widehat\{k\}\}^\{\\alpha\}, and the transfer gapQα​\(πk^\)−qα​\(Fk^\)Q\_\{\\alpha\}\(\\pi\_\{\\widehat\{k\}\}\)\-q\_\{\\alpha\}\(F\_\{\\widehat\{k\}\}\)\. Additive gaps avoid instability when returns are near zero or negative\.

The exact binomial index used by COPS is computed once from\(n,K,α,δ\)\(n,K,\\alpha,\\delta\)before any score values are inspected\. Table[18](https://arxiv.org/html/2609.29549#A8.T18)reports the order statistic selected for the defaultα=0\.10\\alpha=0\.10,δ=0\.05\\delta=0\.05,K=20K=20setting\. Then=50n=50row has no nontrivial order statistic, so the procedure returns a known lower support bound if available and−∞\-\\inftyotherwise; asnngrows, the effective quantiler⋆/nr^\{\\star\}/napproaches the nominal levelα=0\.10\\alpha=0\.10from below, reflecting the finite\-sample correction needed to control the simultaneous miscoverage probability\. This monotone behavior ofr⋆/nr^\{\\star\}/nalso clarifies an operational point: the calibration sample size is the principal lever for tightness, and below roughlyn=200n=200aK=20K=20candidate set already exhausts much of the available statistical budget atα=0\.10\\alpha=0\.10\.

Table 18:Exact binomial calibration indices forα=0\.10\\alpha=0\.10,δ=0\.05\\delta=0\.05, andK=20K=20\. The selected bound isS\(r⋆\)​kS\_\{\(r^\{\\star\}\)k\}for each candidatekk\.Table[19](https://arxiv.org/html/2609.29549#A8.T19)reports the common\-threshold failure diagnostic from the per\-seed evaluation logs\. Empirical CVaR selection uses the same calibration score samples as COPS but omits the finite\-sample lower\-confidence correction\. COPS has the smallest point estimate in all seven rows\. Table[20](https://arxiv.org/html/2609.29549#A8.T20)shows the same empirical ordering for rollout\-estimatedQ0\.10Q\_\{0\.10\}andCVaR0\.10\\operatorname\{CVaR\}\_\{0\.10\}, while Table[21](https://arxiv.org/html/2609.29549#A8.T21)records the associated mean\-return cost\. These comparisons require paired uncertainty before they can be read as inferential dominance claims\.

Table 19:Reported common\-threshold failure rate100​ℙ^​\(R<ce\)100\\widehat\{\\mathbb\{P\}\}\(R<c\_\{e\}\)atα=0\.10\\alpha=0\.10, wherecec\_\{e\}is fixed per environment and shared across selectors\. Entries are per\-seed summaries\.Table 20:Rollout\-estimatedQ0\.10Q\_\{0\.10\}andCVaR0\.10\\operatorname\{CVaR\}\_\{0\.10\}on independent evaluation trajectories\. COPS has the largest reported point estimate in every row; paired intervals are needed for statistical comparisons\.Table 21:Mean\-return tradeoff atα=0\.10\\alpha=0\.10\. “Selected gap≥0\\geq 0” records the sign of a point\-estimated score\-to\-return gap; it is selection\-biased and is not a population transfer certificate\.Read together, the tables show a consistent point\-estimate pattern\. Mean\-OPE selection has the largest reported mean and the weakest tail, while exact COPS has stronger reported tail metrics at a mean cost of roughly nine percent\. This is the intended operating tradeoff, but the exact score theorem does not imply it and the tables should not be described as universal dominance\.

We audit the separate transfer condition using independent rollout diagnostics\. For each selected policy, an independent behavior\-policy score sample estimatesq0\.10​\(Fk^\)q\_\{0\.10\}\(F\_\{\\widehat\{k\}\}\), and independent deployment rollouts estimateQ0\.10​\(πk^\)Q\_\{0\.10\}\(\\pi\_\{\\widehat\{k\}\}\)\. Table[30](https://arxiv.org/html/2609.29549#A10.T30)reports positive point\-estimated gaps for all seven selected policies\. Because these candidates were selected and both quantiles are estimated, the signs provide diagnostic support only; establishing the population inequalities would require selection\-aware one\-sided uncertainty\. Candidate\-wide results below make clear that clipped DR fails to transfer for many candidates\.

### H\.1Candidate\-Level Transfer

Table[30](https://arxiv.org/html/2609.29549#A10.T30)reports gaps for the seven*selected*policies, which is a selection\-biased view: selection favors candidates with favorable transfer\. To answer whether transfer holds across the pool, we reanalyze all7×20=1407\\times 20=140candidate–task pairs against independent rollout quantiles, for four fixed score/selector choices \(Table[22](https://arxiv.org/html/2609.29549#A8.T22)\)\. Clipped DR is nonnegative for76\.4%76\.4\\%of candidate pairs \(median gap\+84\+84\), whereas the selected policies average\+201\+201: Table[30](https://arxiv.org/html/2609.29549#A10.T30)should not be read as an unbiased estimate of pool\-wide transfer\. Crucially, clipped DR does*not*transfer for every candidate; a pessimistic distributional score raises the nonnegative fraction to93\.6%93\.6\\%\. This diagnoses a better empirical bridge, not a theorem: the score theorem covers any predeclared fixed score, and return interpretation still needs separate transfer evidence\.

Table 22:Candidate\-level transfer over all140140candidate–task pairs \(not selection\-biased\)\. Rollout quantiles are diagnostics and never enter selection\. Selected\-policy failure is over the seven environments including the two dependence stress tests\.
### H\.2Clipping Sensitivity

The “medium” behavior trajectories have a worse lower tail than deployment rollouts of policies that improve on behavior, and clipping truncates the importance correction asymmetrically, pushing the clipped\-score lower tail downward asρmax\\rho\_\{\\max\}shrinks\. Across all140140pairs the transfer gap therefore grows as clipping tightens \(Table[23](https://arxiv.org/html/2609.29549#A8.T23)\), and it correlates with target\-minus\-behavior mean improvement \(Spearmanρ=\.47\\rho=\.47\)\. This supports a benchmark\-specific explanation—behavior/deployment mismatch plus clipping bias—not a claim that DR is universally pessimistic; equality or unbiasedness in expectation cannot order lower quantiles\.

Table 23:Transfer gapQ\.10​\(πk\)−q\.10​\(Fk\)Q\_\{\.10\}\(\\pi\_\{k\}\)\-q\_\{\.10\}\(F\_\{k\}\)vs\. clip levelρmax\\rho\_\{\\max\}, over140140candidate–task pairs\.
### H\.3Negative Control

To confirm rollout information is not silently injected into the certificate, we instantiate a variance\-collapse case: a fixed reset, a stochastic target policy, and an exact\-value score\. The score standard deviation falls to0\.00\.0and the transfer gap is−92\-92; COPS retains nominal*score*coverage but correctly fails the return\-transfer diagnostic\. Under the standard randomized MuJoCo reset the score SD is117117and the collapse disappears—the expected behavior, and a demonstration that the diagnostic detects genuine transfer failure rather than rubber\-stamping it\.

### H\.4Multiplicity Sensitivities

Family\-wise control is the right target \(FDR does not protect the single deployed, data\-selected policy\)\. Exact Bonferroni may be loose; we report Holm\-inverted binomial tests and a trajectory\-vector joint\-maximum bootstrap as sensitivities \(Table[24](https://arxiv.org/html/2609.29549#A8.T24)\),*not*as drop\-in replacements\. Holm controls FWER for testing, but a step\-down procedure does not automatically yield a lower bound valid simultaneously for allkk, which is what maximizing over the pool requires; establishing the compatible simultaneous construction is a stated open item\. The joint bootstrap can exploit positive cross\-candidate dependence empirically but is not the same distribution\-free theorem\.

Table 24:Multiplicity sensitivities atn=200,K=20,α=\.10,δ=\.05n=200,K=20,\\alpha=\.10,\\delta=\.05\. Only exact Bonferroni carries the finite\-sample distribution\-free guarantee; the others are reported as sensitivities\.
### H\.5Selection Stability

Across200200trajectory\-level calibration resamples with frozen candidates and score dumps, COPS is markedly more stable than plug\-in empirical CVaR selection \(Table[25](https://arxiv.org/html/2609.29549#A8.T25)\); these are stability diagnostics, not additional coverage evidence\.

Table 25:Selection stability over200200calibration resamples \(frozen candidates/scores\)\.
### H\.6Safety\-Constrained Operating Point

The constrained selector of Section[8\.1](https://arxiv.org/html/2609.29549#S8.SS1)recovers most of the mean\-return loss while keeping the tail improvement \(Table[26](https://arxiv.org/html/2609.29549#A8.T26)\)\. Concretely, constrained COPS recovers70\.5%70\.5\\%of the mean\-return loss relative to max\-floor COPS while retaining86\.7%86\.7\\%of itsQ0\.10Q\_\{0\.10\}improvement over mean OPE; the guarantee is unchanged because it rides the same simultaneous event\.

Table 26:Operating points over the same seven environments \(including the two dependence stress tests\)\. Rollout columns are evaluation diagnostics, never inputs to selection\.
### H\.7The Exactnn\-versus\-KKBurden

The exact effective levelr⋆/nr^\{\\star\}/ngrows withnnand shrinks withKK\(Table[27](https://arxiv.org/html/2609.29549#A8.T27)\)\. Becauser⋆r^\{\\star\}is integer\-valued the ratio is not monotone; if we define the frontier as the smallestNNwithr⋆/n≥\.05r^\{\\star\}/n\\geq\.05for*all*n≥Nn\\geq N, thenN=248,344,427N=248,344,427forK=20,100,500K=20,100,500\. The operational consequence, which we now state as a limitation rather than a benign scaling remark: a dataset with tens—not hundreds—of conditionally i\.i\.d\. full trajectories cannot support a useful10%10\\%lower\-tail certificate at theseKK\. MIMIC\-style applications must define a defensible patient\-level i\.i\.d\. unit and meet this burden, reduce the pool/confidence, or abstain\. Empirically, enlargingKKatn=500n=500on a nested halfcheetah\-medium pool did not destroy score coverage but moved the selected order statistic from3232to2727and reduced the median selected floor by1\.61\.6normalized points \(Table[28](https://arxiv.org/html/2609.29549#A8.T28)\)\.

Table 27:Exact effective levelsr⋆/nr^\{\\star\}/nfor\(α,δ\)=\(\.10,\.05\)\(\\alpha,\\delta\)=\(\.10,\.05\)\.Table 28:Nested pools at fixedn=500n=500\(same pre\-fixed calibration split; larger pools add frozen checkpoints without inspecting calibration scores\)\. LargerKKcosts a few order\-statistic ranks, not coverage\.
### H\.8Discrete Actions

Neither Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)nor the selector assumes continuous actions\. A fixed\-log check with2020predeclared DQN checkpoints per task is consistent with nominal score coverage on CartPole \(95\.6%95\.6\\%\) and Acrobot \(95\.2%95\.2\\%\); the reported mean\-OPE/COPS failure diagnostics are10\.8%/3\.8%10\.8\\%/3\.8\\%and12\.6%/4\.2%12\.6\\%/4\.2\\%, respectively\. These simulator checks remove continuous action spaces as a necessary condition but do not resolve fixed\-dataset or transfer limitations\.

### H\.9Dependence Boundary and the D4RL i\.i\.d\. Audit

Arbitrary temporal dependence*within*a complete trajectory is allowed; conditional i\.i\.d\. sampling is required*across*trajectory\-level score vectors\(S1​i,…,SK​i\)\(S\_\{1i\},\\ldots,S\_\{Ki\}\)\. Transitions, overlapping replay windows, and successive chunks are not valid calibration units, and an estimated “effective sample size” cannot simply be substituted into the exact formula\. Table[29](https://arxiv.org/html/2609.29549#A8.T29)shows the boundary: independently generated whole trajectories and independent block\-level units have reported coverage95\.7%95\.7\\%and95\.1%95\.1\\%, while sequential chunks fall to91\.8%91\.8\\%and83\.9%83\.9\\%as between\-unit correlation increases\. The score\-coverage audit uses new simulator trajectories from a fitted, frozen behavior collector, a validation device available only in simulation\. The two medium\-replay tasks are therefore dependence stress tests rather than exact\-coverage demonstrations\. In a genuine offline application, calibration must be withheld from the fixed log; the500500\-episode holdout reported in Appendix[L](https://arxiv.org/html/2609.29549#A12)reduces training data by33\.3%33\.3\\%on average and changes mean candidate return by−2\.1%\-2\.1\\%\.

Table 29:Calibration\-unit construction vs\. simultaneous coverage \(target95%95\\%\)\. Within\-unit temporal correlation is allowed; between\-unit dependence is not\.

## Appendix IDetailed Baseline List

The full baseline suite used in Section[10](https://arxiv.org/html/2609.29549#S10)is as follows\. We compare against FQE or DR mean\-OPE selection; bootstrap or asymptotic mean\-OPE lower confidence selection; pessimistic FQE; empirical lower\-tail quantile selection without confidence correction; empirical CVaR selection using the same calibration score samples; conformal/risk\-control\-inspired interval selection using the same calibration split\([Taufiq et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib15);[Stutz et al\., 2022](https://arxiv.org/html/2609.29549#bib.bib16)\); random candidate and behavior\-cloning reference; and rollout oracles forQαQ\_\{\\alpha\}andCVaRα\\operatorname\{CVaR\}\_\{\\alpha\}, used only as analysis\-only upper bounds on what any selector could achieve given the candidate set\.

## Appendix JSelected\-Policy Transfer Diagnostics

Table[30](https://arxiv.org/html/2609.29549#A10.T30)reports the per\-environment score\-to\-return transfer diagnostics for the COPS\-selected policies\.

Table 30:Score\-to\-return transfer diagnostics for COPS\-selected policies atα=0\.10\\alpha=0\.10\. The tightness gap isq0\.10​\(Fk^\)−Bk^0\.10q\_\{0\.10\}\(F\_\{\\widehat\{k\}\}\)\-B\_\{\\widehat\{k\}\}^\{0\.10\}; smaller values indicate a tighter certified score floor\. The transfer gap isQ0\.10​\(πk^\)−q0\.10​\(Fk^\)Q\_\{0\.10\}\(\\pi\_\{\\widehat\{k\}\}\)\-q\_\{0\.10\}\(F\_\{\\widehat\{k\}\}\); positive point estimates are consistent with score\-to\-return conservatism but do not establish the population inequality\.For each selected policy, we estimateq0\.10​\(Fk^\)q\_\{0\.10\}\(F\_\{\\widehat\{k\}\}\)from an independent behavior\-policy score sample andQ0\.10​\(πk^\)Q\_\{0\.10\}\(\\pi\_\{\\widehat\{k\}\}\)from independent deployment rollouts\. All seven point\-estimated gaps are nonnegative\. Because both quantiles are estimated after the candidate has been selected, these signs are diagnostics rather than verified instances of Assumption[6](https://arxiv.org/html/2609.29549#Thmtheorem6); Corollary[7](https://arxiv.org/html/2609.29549#Thmtheorem7)applies only when the population transfer condition itself holds\.

## Appendix KMatched Fitted Distributional Evaluation

We compare the clipped\-DR score against a Fitted Distributional Evaluation \(FDE\)0\.100\.10\-quantile score under a matched protocol \(sameKK, nuisance\-training split, calibration units, and multiplicity correction\), spanning deterministic\-policy, stochastic\-policy, and stochastic\-dynamics regimes\. Candidate\-level transfer statistics appear in Table[22](https://arxiv.org/html/2609.29549#A8.T22)\(89\.3%89\.3\\%nonnegative for FDE vs\.76\.4%76\.4\\%for clipped DR\)\. Table[31](https://arxiv.org/html/2609.29549#A11.T31)localizes where each empirical bridge succeeds\. The high\-noise row is the important one: FDE improves the bridge but does not make it automatic, so we do not claim FDE fixes transfer in general—the score theorem is what is unconditional, and the bridge remains an empirical, regime\-dependent property\. Table[32](https://arxiv.org/html/2609.29549#A11.T32)reports operating points for distributional selectors; the rollout columns are diagnostics, not selection inputs\. This comparison is scoped to the reported seeds/environments and is not a general claim that FDE dominates\.

Table 31:Point\-estimated transfer gaps by regime under a matched protocol\. Signs are diagnostics; they are not one\-sided population guarantees\.Table 32:Distributional\-selector operating points \(seven environments\)\. Rollout columns are evaluation diagnostics only\.
## Appendix LD4RL i\.i\.d\. Audit and Fixed\-Data Cost

Exact score\-coverage validation uses new simulator trajectories generated from a fitted, frozen behavior collector, not fresh draws from the fixed D4RL files; fresh generation is a validation device available in simulation only\. In a genuine offline application the certificate must be paid for from the fixed logged dataset\. Table[33](https://arxiv.org/html/2609.29549#A12.T33)reports both costs: the “simulator audit” leaves policy\-training data untouched \(0%0\\%cost\), whereas the “fixed\-data holdout” withholds500500native episodes before training, retrains the candidate pool, and reports the resulting mean candidate\-return change\. The fixed\-data column—an average33\.3%33\.3\\%training reduction for a2\.1%2\.1\\%mean\-candidate\-return change—is the relevant analogue for MIMIC\-style or one\-shot logged data\. The two medium\-replay environments retain the broader stress\-test label because of collection\-time nonstationarity\. Where a frozen collector or defensible conditionally i\.i\.d\. unit is unavailable, Theorem[5](https://arxiv.org/html/2609.29549#Thmtheorem5)does not apply\.

Table 33:Cost of obtaining calibration units\. Simulator audit:0%0\\%training cost \(fresh generation\)\. Fixed\-data holdout: the realistic offline cost\.
## Appendix MComputing the Exact Order Statistic

The integerr⋆r^\{\\star\}in Eq\. \([1](https://arxiv.org/html/2609.29549#S5.E1)\) can be computed by evaluating binomial CDFs\. A conservative implementation is

r⋆=max⁡\{r:K​∑j=0r−1\(nj\)​αj​\(1−α\)n−j≤δ\}\.r^\{\\star\}=\\max\\left\\\{r:K\\sum\_\{j=0\}^\{r\-1\}\\binom\{n\}\{j\}\\alpha^\{j\}\(1\-\\alpha\)^\{n\-j\}\\leq\\delta\\right\\\}\.If no suchrrexists, the calibration data are too small for a nontrivial lower\-tail certificate at level\(α,δ,K\)\(\\alpha,\\delta,K\)without an external lower support bound\.