Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search

# Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search
Source: [https://arxiv.org/html/2607.27422](https://arxiv.org/html/2607.27422)
Ayushman Singhaysingh@stanford\.edu Stanford UniversitySiddharth Aphalesaphale@stanford\.edu Stanford University

###### Abstract

Good action rankings do not make a contrastive critic safe to maximize\. These critics increasingly act as value\-like objectives for best\-of\-KKselection, planning, and critic\-guided generation\. Unbounded bilinear scores can let large embedding norms inflate off\-support values, but cosine bounding does not remove the failure\. A controlled support decomposition attributes most raw bilinear regret to norm drift\. Cosine and hybrid critics nevertheless select off\-support actions from most pools and incur comparable regret\. Contrastive scores are weakly calibrated or inverted in the top score decile across four OGBench navigation tasks, and they fail to order fixed\-query actions by value\. Bellman\-trained TD\-Q succeeds, including in a parameter\-matched function\-class control\. Realized costs depend on the task: simulator rollouts reveal single\-step selection costs on PointMaze and the exact\-Q∗Q^\{\*\}toy but well\-powered nulls on AntMaze and HumanoidMaze, where the controller can self\-correct\. A training/readout decomposition traces the lost ordering to the cosine*training*objective; raw\-trained embeddings retain weak ordering after inference\-time normalization\. Candidate maximization can therefore exploit false positives caused by norm drift, score saturation, or in\-support misranking\. Contrastive critics remain useful compatibility rankers on navigation and manipulation tasks, but action selection requires a value\-calibrated scalar\.

## 1Introduction

Contrastive critics are widely used in goal\-conditioned reinforcement learning\(Eysenbach et al\.,[2022](https://arxiv.org/html/2607.27422#bib.bib4); Zheng et al\.,[2024](https://arxiv.org/html/2607.27422#bib.bib16)\)\. They embed state–action pairs and goals and produce a value\-like score for retrieval, action ranking, planning, and candidate selection\. The standard bilinear critic is

fθ​\(s,a,g\)=ϕθ​\(s,a\)⊤​ψθ​\(g\)\.f\_\{\\theta\}\(s,a,g\)=\\phi\_\{\\theta\}\(s,a\)^\{\\top\}\\psi\_\{\\theta\}\(g\)\.Contrastive training pulls state–action embeddings toward reachable goals and away from negatives, thereby treating goal\-conditioned control as representation learning\.

Contrastive scores now serve roles beyond in\-distribution ranking\. A diffusion, flow, or autoregressive policy can generate many actions and execute the highest\-scoring candidate; a large candidate sampler can do the same,

aK⋆=arg⁡maxai∼π\(⋅∣s,g\),i≤K⁡fθ​\(s,ai,g\)\.a\_\{K\}^\{\\star\}=\\arg\\max\_\{a\_\{i\}\\sim\\pi\(\\cdot\\mid s,g\),\\,i\\leq K\}f\_\{\\theta\}\(s,a\_\{i\},g\)\.The candidate set can extend beyond the data\. The critic then becomes an objective over the full set\. Aggressive search can turn small top\-tail errors into policy errors\.

This shift from ranking to optimization creates a role mismatch: accurate in\-support ranking can coexist with unsafe selection\. The raw bilinear critic exposes one mechanism through the decomposition

fθ​\(s,a,g\)=‖ϕθ​\(s,a\)‖​‖ψθ​\(g\)‖​cos⁡\(ϕθ​\(s,a\),ψθ​\(g\)\)\.f\_\{\\theta\}\(s,a,g\)=\\\|\\phi\_\{\\theta\}\(s,a\)\\\|\\,\\\|\\psi\_\{\\theta\}\(g\)\\\|\\,\\cos\\\!\\bigl\(\\phi\_\{\\theta\}\(s,a\),\\psi\_\{\\theta\}\(g\)\\bigr\)\.The score can increase through closer alignment, a larger state–action embedding norm, or false alignment outside the data support\. An expressive policy may find these high\-score regions even when their true return is poor \([Figure˜1](https://arxiv.org/html/2607.27422#S1.F1)\)\.

The*norm inflation hypothesis*\(H1\) attributes the failure to norm growth\. Our analysis formalizes off\-support nonidentifiability and shows that raw bilinear critics can inflate scores outside the data support \(§[4](https://arxiv.org/html/2607.27422#S4)\); candidate maximization converts this inflation into regret \([Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)\)\. The theorem establishes existence, and our experiments test trained critics\. A bad region only needs to outscore in\-support alternatives, so bounding still fails \([Corollary˜1](https://arxiv.org/html/2607.27422#Thmcorollary1)\)\. Our failures occur below the cap\.*Value decalibration*\(H2\) is the broader problem, and best\-of\-KKsearch exposes it in the score tail\.

##### Contributions\.

We separate*in\-support compatibility ranking*from*value\-calibrated candidate selection*\. Our theory shows how bilinear critics can rank correctly in support yet incur regret under maximization, even when bounded \(§[4](https://arxiv.org/html/2607.27422#S4)\)\. Our experiments find weak or inverted value ordering despite strong compatibility across four OGBench navigation tasks, and strong retrieval on four manipulation tasks \([Table˜S10](https://arxiv.org/html/2607.27422#S5.T10)\)\. Fixed\-query and parameter\-matched controls attribute reliable ordering to the training objective; rollouts reveal harm when one action matters and powered nulls under self\-correction \(§[5](https://arxiv.org/html/2607.27422#S5)\)\.

![Refer to caption](https://arxiv.org/html/2607.27422v1/x1.png)Figure 1:Unified failure mechanism\.Contrastive training separates compatible pairs but need not grade their value\. Candidate maximization finds high\-score false positives caused by in\-support misranking, bounded scores at the cap, or raw norm drift \(§[4](https://arxiv.org/html/2607.27422#S4),[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)\)\. A Bellman\-calibrated scalar restores the ordering in our experiments\.

## 2Related work

##### Contrastive critics and learned critic geometry\.

Contrastive RL embeds state–action pairs and goals so that reachable goals score above negatives\. The inner productfθ​\(s,a,g\)=ϕθ​\(s,a\)⊤​ψθ​\(g\)f\_\{\\theta\}\(s,a,g\)=\\phi\_\{\\theta\}\(s,a\)^\{\\top\}\\psi\_\{\\theta\}\(g\)serves as a goal\-conditioned value\. Bilinear critics perform well on offline and image\-based goal reaching\(Eysenbach et al\.,[2022](https://arxiv.org/html/2607.27422#bib.bib4)\), and later work improves their optimization and stability\(Zheng et al\.,[2024](https://arxiv.org/html/2607.27422#bib.bib16)\)\. These methods primarily train the scalar to distinguish compatible goals from negatives\. Our work studies a different requirement: the same scalar must provide graded value ordering when an action optimizer compares many candidates\. Strong retrieval can satisfy the first requirement without satisfying the second\. We formalize and measure that separation rather than proposing a replacement contrastive learner\.

Structured goal\-reaching critics offer a complementary response\. Quasimetric methods encode asymmetric reachability, while temporal\-distance methods impose a metric\-like notion of progress and support stitching\(Wang et al\.,[2023a](https://arxiv.org/html/2607.27422#bib.bib14); Myers et al\.,[2024](https://arxiv.org/html/2607.27422#biba.bib2)\)\. These structures can supply ordering information that binary compatibility does not\. Our experiments do not compare those objectives directly; they isolate the standard bilinear readout and test whether normalization or auxiliary TD shaping makes it safe to maximize\.

##### Expressive policies and value\-guided action search\.

Expressive policies, including diffusion and flow models\(Chi et al\.,[2023](https://arxiv.org/html/2607.27422#bib.bib1)\), change how a method queries its critic\. A restricted Gaussian actor evaluates relatively few actions, whereas diffusion models, flow models, and large samplers search much larger candidate sets\. Sampling, guidance, or reranking can then magnify high\-score errors\. This risk applies to value\-guided methods such as Diffusion\-QL\(Wang et al\.,[2023](https://arxiv.org/html/2607.27422#biba.bib3)\), EXPO\(Dong et al\.,[2025](https://arxiv.org/html/2607.27422#bib.bib3)\), and flow policy optimization\(Park et al\.,[2025b](https://arxiv.org/html/2607.27422#bib.bib13)\)\. Our best\-of\-KKabstraction captures the common pressure these methods place on a learned scalar: a more expressive proposal exposes more of its upper tail\. We therefore study the selection objective independently of any particular policy parameterization\.

##### Energy guidance and candidate reranking\.

Energy\-guided methods such as QGPO\(Lu et al\.,[2023](https://arxiv.org/html/2607.27422#bib.bib8)\)and Diffusion\-DICE\(Mao et al\.,[2024](https://arxiv.org/html/2607.27422#bib.bib9)\)steer or select samples with a learned signal\. These methods differ in how they construct the proposal and apply guidance, but each depends on the signal remaining meaningful over the actions that search reaches\. Our results identify a failure mode upstream of the sampler: a contrastive compatibility score can preserve retrieval quality while misordering its searched tail\. Support constraints can reduce exposure to that tail, whereas value\-calibrated ordering addresses the deployed scalar\.

##### Offline RL extrapolation, calibration, and ranking\.

Offline RL has long studied policies that exploit value errors outside the data\. Prior work addresses extrapolation error\(Fujimoto et al\.,[2019](https://arxiv.org/html/2607.27422#bib.bib5)\), overestimation under distribution shift\(Kumar et al\.,[2020](https://arxiv.org/html/2607.27422#biba.bib1)\), and queries on unseen actions\(Kostrikov et al\.,[2022](https://arxiv.org/html/2607.27422#bib.bib6)\)\. Calibration\- and ranking\-oriented methods are closest to our setting\. Cal\-QL preserves value lower bounds during offline\-to\-online training\(Nakamoto et al\.,[2023](https://arxiv.org/html/2607.27422#bib.bib11)\), while the concurrent RankQ replaces absolute TD regression with pairwise value\-ranking constraints\(Choi & Xu,[2026](https://arxiv.org/html/2607.27422#bib.bib2)\)\. Both lines treat the deployed scalar as a value or value ordering\. A contrastive critic instead starts from a compatibility objective, so accurate positive–negative separation does not guarantee order among positive or candidate actions\. Our analysis makes this distinction explicit\. Norm growth or false off\-support alignment creates high\-score errors \(§[4](https://arxiv.org/html/2607.27422#S4)\), and candidate maximization turns them into regret \([Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)\)\. The problem is therefore not generic overestimation alone: bounded critics can fail through misordering below their cap, and good compatibility rankers can remain bad selection objectives\.

## 3Problem Setup

We consider a goal\-conditioned MDP with statess∈𝒮s\\in\\mathcal\{S\}, actionsa∈𝒜a\\in\\mathcal\{A\}, goalsg∈𝒢g\\in\\mathcal\{G\}, and an offline dataset\(s,a,g\)∼D\(s,a,g\)\\sim D\. When available, the optimal goal\-conditioned valueQ∗​\(s,a,g\)Q^\{\*\}\(s,a,g\)provides the evaluation target; otherwise, we use a Monte Carlo rollout estimateQ^MC\\hat\{Q\}\_\{\\mathrm\{MC\}\}\. A goal\-conditioned critic assigns a learned scorefθ​\(s,a,g\)f\_\{\\theta\}\(s,a,g\)to each state, action, and goal\. The score has two roles: it orders sampled in\-support actions for*compatibility ranking*and becomes the optimization objective for*candidate selection*\. The standard bilinear contrastive critic is

fθ​\(s,a,g\)=ϕθ​\(s,a\)⊤​ψθ​\(g\)\.f\_\{\\theta\}\(s,a,g\)\\;=\\;\\phi\_\{\\theta\}\(s,a\)^\{\\top\}\\psi\_\{\\theta\}\(g\)\.\(1\)We train this critic with a contrastive objective on sampled positives and negatives\.

An action proposalπη\(⋅∣s,g\)\\pi\_\{\\eta\}\(\\cdot\\mid s,g\)acts as a black\-box sampler, and the candidate countKKproxies for its expressivity\. The setCK=\{a1,…,aK\}C\_\{K\}=\\\{a\_\{1\},\\dots,a\_\{K\}\\\}contains the sampled candidates\.*Candidate maximization*selects

aK⋆=argmaxi≤Kfθ\(s,ai,g\),ai∼πη\(⋅∣s,g\)\.a\_\{K\}^\{\\star\}\\;=\\;\\arg\\max\_\{i\\leq K\}f\_\{\\theta\}\(s,a\_\{i\},g\),\\qquad a\_\{i\}\\sim\\pi\_\{\\eta\}\(\\cdot\\mid s,g\)\.\(2\)We compareaK⋆a\_\{K\}^\{\\star\}with the*candidate pool oracle*aQ⋆=arg⁡maxi≤K⁡Q⋆​\(s,ai,g\)a\_\{Q\}^\{\\star\}=\\arg\\max\_\{i\\leq K\}Q^\{\\star\}\(s,a\_\{i\},g\), which selects the highest\-value action among the*same*KKdraws\. The resulting*candidate regret*is

RegretK=Q⋆​\(s,aQ⋆,g\)−Q⋆​\(s,aK⋆,g\)≥0\.\\mathrm\{Regret\}\_\{K\}\\;=\\;Q^\{\\star\}\(s,a\_\{Q\}^\{\\star\},g\)\-Q^\{\\star\}\(s,a\_\{K\}^\{\\star\},g\)\\;\\geq\\;0\.\(3\)A critic can rank sampled actions by compatibility yet incur large regret under*value\-calibrated selection*\. Selection exposes this gap because it optimizes the critic over the full candidate set instead of evaluating only training data\. We maximize every deployed scalar directly as in \([2](https://arxiv.org/html/2607.27422#S3.E2)\); larger values indicate actions closer to the goal\. TD\-Q uses the in\-support targetγd\\gamma^\{d\}, while the controlled toy usesQ⋆=−‖s\+a−g‖Q^\{\\star\}=\-\\\|s\+a\-g\\\|\. We do not normalize scores separately by critic before selection\. Negative values in later figures only reflectQ⋆≤0Q^\{\\star\}\\leq 0\.

##### Deployed score families\.

We compare four scalar readouts throughout the paper \([Table˜1](https://arxiv.org/html/2607.27422#S3.T1)\)\.*Raw bilinear*uses the unnormalized scorefθ=ϕ⊤​ψf\_\{\\theta\}=\\phi^\{\\top\}\\psifrom \([1](https://arxiv.org/html/2607.27422#S3.E1)\)\.*Cosine*uses the bounded, normalized readout

fcos​\(s,a,g\)=ϕ​\(s,a\)⊤‖ϕ​\(s,a\)‖​ψ​\(g\)‖ψ​\(g\)‖\(\|fcos\|≤1\);f\_\{\\text\{cos\}\}\(s,a,g\)\\;=\\;\\frac\{\\phi\(s,a\)^\{\\top\}\}\{\\\|\\phi\(s,a\)\\\|\}\\,\\frac\{\\psi\(g\)\}\{\\\|\\psi\(g\)\\\|\}\\qquad\(\|f\_\{\\text\{cos\}\}\|\\leq 1\);\(4\)*TD\-Q*uses a scalarQω​\(s,a,g\)Q\_\{\\omega\}\(s,a,g\)trained through Bellman backups\.*Hybrid*trains cosine embeddings with an auxiliary TD residual but uses the bounded cosine readoutfcosf\_\{\\text\{cos\}\}for selection\. The hybrid is a diagnostic, not a proposed method\. It tests whether TD*representation shaping*can rescue a bounded contrastive readout when selection ignores the calibrated scalar\. The TD head provides the positive control in[Section˜S7](https://arxiv.org/html/2607.27422#S7a)\.[Section˜5\.1](https://arxiv.org/html/2607.27422#S5.SS1)gives the training objectives, architectures, and budgets\.

Table 1:The four deployed score families\.All four use the same data and training budget\. Raw, cosine, and hybrid share an encoder architecture; TD\-Q uses a joint\-input twin\-Q MLP\. The families differ in their training signal and deployed scalar\. The cosine and hybrid bounds cap their score ranges without calibrating them in our experiments\.We first analyze why compatibility scores can fail under maximization\. The controlled experiment then makes the dataset support, off\-support value, and candidate pool oracle explicit \([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\.[Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)gives its full setup and learned score surfaces\.

## 4Why Compatibility Scores Can Fail as Value Objectives

We fix a state–goal pair\(s,g\)\(s,g\)and writeφ​\(a\)=ϕθ​\(s,a\)\\varphi\(a\)=\\phi\_\{\\theta\}\(s,a\),ψ=ψθ​\(g\)\\psi=\\psi\_\{\\theta\}\(g\),f​\(a\)=φ​\(a\)⊤​ψf\(a\)=\\varphi\(a\)^\{\\top\}\\psi, andQ​\(a\)=Q⋆​\(s,a,g\)Q\(a\)=Q^\{\\star\}\(s,a,g\)\. We assume bounded true returns,Q​\(a\)∈\[Qmin,Qmax\]Q\(a\)\\in\[Q\_\{\\min\},Q\_\{\\max\}\]\. The bilinear score has the following polar decomposition for anyφ​\(a\)≠0\\varphi\(a\)\\neq 0andψ≠0\\psi\\neq 0:

f​\(a\)=‖φ​\(a\)‖​‖ψ‖​cos⁡\(φ​\(a\),ψ\),f\(a\)\\;=\\;\\\|\\varphi\(a\)\\\|\\,\\\|\\psi\\\|\\,\\cos\\bigl\(\\varphi\(a\),\\,\\psi\\bigr\),\(5\)becausecos⁡\(φ​\(a\),ψ\)=φ​\(a\)⊤​ψ/\(‖φ​\(a\)‖​‖ψ‖\)\\cos\(\\varphi\(a\),\\psi\)=\\varphi\(a\)^\{\\top\}\\psi/\(\\\|\\varphi\(a\)\\\|\\,\\\|\\psi\\\|\)\.

Two mechanisms can increase the score\. Better angular alignment has a natural goal\-reaching interpretation, whereas state–action embedding norm growth need not reflect better control\. We ask whether the contrastive objective limits the second mechanism\.

The variablePPdenotes the contrastive training distribution over tuplesξ=\(s,a,g\+,g1−,…,gm−\)\\xi=\(s,a,g^\{\+\},g\_\{1\}^\{\-\},\\ldots,g\_\{m\}^\{\-\}\), and letℒ​\(f\)=𝔼ξ∼P​\[ℓ​\(f​\(s,a,g\+\),f​\(s,a,g1−\),…,f​\(s,a,gm−\)\)\]\\mathcal\{L\}\(f\)=\\mathbb\{E\}\_\{\\xi\\sim P\}\[\\,\\ell\(f\(s,a,g^\{\+\}\),f\(s,a,g\_\{1\}^\{\-\}\),\\ldots,f\(s,a,g\_\{m\}^\{\-\}\)\)\\,\]be the population contrastive objective\. We useΩP\\Omega\_\{P\}for the support of the triples\(s,a,g\)\(s,a,g\)whose scores enter this objective\.

##### Off\-support nonidentifiability\.

A perturbationhhcan vanish on every evaluated triple:h​\(s,a,g\)=0h\(s,a,g\)=0for all\(s,a,g\)∈ΩP\(s,a,g\)\\in\\Omega\_\{P\}\. The equalityℒ​\(f\+h\)=ℒ​\(f\)\\mathcal\{L\}\(f\+h\)=\\mathcal\{L\}\(f\)then holds because the loss only evaluates triples inΩP\\Omega\_\{P\}\. The functionsffandf\+hf\+htherefore produce identical scores on every training tuple and have the same objective value\. The contrastive objective imposes no constraint outsideΩP\\Omega\_\{P\}, so a critic can be well determined on the training support and arbitrary elsewhere\.

This freedom allows a bilinear critic to have unbounded behavior outside the support\.

### 4\.1Raw norm inflation and finite bounded false positives

The polar decomposition \([5](https://arxiv.org/html/2607.27422#S4.E5)\) separates the two sources of score growth\. The norm channel allows unbounded growth, but the failure does not require norms to diverge: bounded embeddings can also create a finite artificial margin\.

##### Unbounded bilinear extrapolation\.

An off\-support sequence can satisfy‖φ​\(an\)‖→∞\\\|\\varphi\(a\_\{n\}\)\\\|\\to\\inftywith fixed goal alignment,cos⁡\(φ​\(an\),ψ\)≥c\>0\\cos\(\\varphi\(a\_\{n\}\),\\psi\)\\geq c\>0\. Equation \([5](https://arxiv.org/html/2607.27422#S4.E5)\) then givesf​\(an\)≥c​‖ψ‖​‖φ​\(an\)‖→∞f\(a\_\{n\}\)\\geq c\\,\\\|\\psi\\\|\\,\\\|\\varphi\(a\_\{n\}\)\\\|\\to\\infty\. BoundedQQimpliesf​\(an\)−Q​\(an\)→∞f\(a\_\{n\}\)\-Q\(a\_\{n\}\)\\to\\inftyoff support\. The objective does not rule out this divergence, and the unnormalized bilinear form can represent it\. Fixed weights on a compact action space preclude the idealized limit‖φ‖→∞\\\|\\varphi\\\|\\to\\infty, but the finite construction below achieves any artificial margin without norm divergence\.

###### Proposition 1\(Finite off\-support inflation\)\.

LetX=𝒮×𝒜X=\\mathcal\{S\}\\times\\mathcal\{A\}be compact, letΩX⊂X\\Omega\_\{X\}\\subset Xbe the state–action projection of the data support, and letBX⊂XB\_\{X\}\\subset Xbe a closed off\-support state–action region such thatBX∩ΩX=∅B\_\{X\}\\cap\\Omega\_\{X\}=\\emptyset; thus,ΩX\\Omega\_\{X\}andBXB\_\{X\}are separated\. LetG0⊆𝒢G\_\{0\}\\subseteq\\mathcal\{G\}be any compact set of goals of interest, and letf=φ⊤​ψf=\\varphi^\{\\top\}\\psibe a bilinear critic with embeddings bounded onX×𝒢X\\times\\mathcal\{G\}\. For any finite marginMM, there exists a bilinear criticf′=φ′⁣⊤​ψ′f^\{\\prime\}=\\varphi^\{\\prime\\top\}\\psi^\{\\prime\}with bounded embeddings \(sup‖φ′‖<∞\\sup\\\|\\varphi^\{\\prime\}\\\|<\\infty\) such thatf′=ff^\{\\prime\}=fonΩX\\Omega\_\{X\}andf′​\(s,a,g\)≥Mf^\{\\prime\}\(s,a,g\)\\geq Mfor all\(s,a,g\)∈BX×G0\(s,a,g\)\\in B\_\{X\}\\times G\_\{0\}\. The construction does not require norm divergence\.

The proof uses a standard function\-approximation bump \([Section˜S1](https://arxiv.org/html/2607.27422#S1a)\)\. The bump adds one bounded latent coordinate that vanishes on the training support and contributes a fixed finite margin on the separated regionBXB\_\{X\}\. The contrastive loss remains unchanged, and the embeddings remain bounded\. A bilinear critic can therefore represent the construction\. Raw norm growth is theβ→∞\\beta\\to\\inftylimit of the same mechanism, and cosine cap saturation is its bounded\-score version \([Corollary˜1](https://arxiv.org/html/2607.27422#Thmcorollary1)\)\. This proposition is an existence result; it does not imply that training will find the construction\. The experiments test for the resulting selection failure \([Table˜3](https://arxiv.org/html/2607.27422#S7.T3)\) and, for raw bilinear critics, the norm channel \([Figure˜S2](https://arxiv.org/html/2607.27422#S3.F2)\)\.

### 4\.2Candidate maximization and regret

Off\-support nonidentifiability leaves the critic unconstrained outside the data support, and[Section˜4\.1](https://arxiv.org/html/2607.27422#S4.SS1)shows that a bilinear critic can assign inflated scores there\. Candidate maximization connects these facts to regret: even a perfect in\-support ranker can become unsafe over a large candidate set\. The critic returnsaK⋆a\_\{K\}^\{\\star\}from \([2](https://arxiv.org/html/2607.27422#S3.E2)\), the candidate pool oracle returnsaQ⋆a\_\{Q\}^\{\\star\}, andRegretK\\mathrm\{Regret\}\_\{K\}from \([3](https://arxiv.org/html/2607.27422#S3.E3)\) compares their values\.

###### Proposition 2\(Good ranker, bad objective\)\.

Let the candidatesa1,…,aKa\_\{1\},\\ldots,a\_\{K\}be drawn i\.i\.d\. from the proposalμ\\mu, and suppose that the action space partitions into disjoint setsAinA\_\{\\textup\{in\}\}\(in support\) andBB\(out of support\), withQ​\(a\)≥qgoodQ\(a\)\\geq q\_\{\\textup\{good\}\}for alla∈Aina\\in A\_\{\\textup\{in\}\}andQ​\(b\)=qbadQ\(b\)=q\_\{\\textup\{bad\}\}for allb∈Bb\\in B, whereqgood\>qbadq\_\{\\textup\{good\}\}\>q\_\{\\textup\{bad\}\}\. Assume the critic preserves order onAinA\_\{\\textup\{in\}\}but assigns everyb∈Bb\\in Bthe false scoref​\(b\)=Mf\(b\)=M, whereM\>maxa∈Ain⁡f​\(a\)M\>\\max\_\{a\\in A\_\{\\textup\{in\}\}\}f\(a\)\. Ifμ​\(B\)=p∈\(0,1\)\\mu\(B\)=p\\in\(0,1\)andμ​\(Ain\)=1−p\\mu\(A\_\{\\textup\{in\}\}\)=1\-p, then

Pr⁡\(aK⋆∈B\)=1−\(1−p\)K→1as​K→∞,\\Pr\(a\_\{K\}^\{\\star\}\\in B\)\\;=\\;1\-\(1\-p\)^\{K\}\\;\\to\\;1\\qquad\\text\{as \}K\\to\\infty,and

lim infK→∞𝔼​\[RegretK\]≥qgood−qbad\.\\liminf\_\{K\\to\\infty\}\\;\\mathbb\{E\}\[\\textup\{Regret\}\_\{K\}\]\\;\\geq\\;q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\.

The probability that at least one candidate lands inBBis1−\(1−p\)K1\-\(1\-p\)^\{K\}\. The critic selects fromBBwhenever this event occurs because its score exceeds every in\-support score\. The pool also contains an in\-support action with probability approaching one asKKgrows\. The oracle then gets at leastqgoodq\_\{\\textup\{good\}\}, while the critic receivesqbadq\_\{\\textup\{bad\}\}\(proof in[Section˜S1](https://arxiv.org/html/2607.27422#S1a)\)\. The bound does not require order preservation withinAinA\_\{\\textup\{in\}\}; that assumption shows that perfect in\-support ranking cannot prevent failure once an off\-support bad set dominates the score tail\.

The proposition describes an optimizer’s curse: stronger critic optimization can reduce task performance\. Best\-of\-KKsearch becomes increasingly likely to find any bad set that receives the highest scores\. This mechanism suggests three measurements: the proposal mass on the bad set \(pp\), its score advantage, and its value gap \(qgood−qbadq\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\)\.

###### Corollary 1\(Boundedness is not calibration\)\.

[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)does not requireMMto be large\. Suppose the score is bounded with capcmaxc\_\{\\max\}\(e\.g\. cosine,cmax=1c\_\{\\max\}=1\), the bad set saturates at that cap,f​\(b\)=cmaxf\(b\)=c\_\{\\max\}forb∈Bb\\in B, and every in\-support action scores strictly below it:maxa∈Ain⁡f​\(a\)≤cmax−Δ\\max\_\{a\\in A\_\{\\textup\{in\}\}\}f\(a\)\\leq c\_\{\\max\}\-\\Deltafor someΔ\>0\\Delta\>0\. The hypotheses then hold withM=cmaxM=c\_\{\\max\}, soPr⁡\(aK⋆∈B\)=1−\(1−p\)K→1\\Pr\(a\_\{K\}^\{\\star\}\\in B\)=1\-\(1\-p\)^\{K\}\\to 1andlim infK𝔼​\[RegretK\]≥qgood−qbad\\liminf\_\{K\}\\mathbb\{E\}\[\\textup\{Regret\}\_\{K\}\]\\geq q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\.

A bounded score offers no protection once low\-value actions reach the top of its range\.*Boundedness alone cannot guarantee calibration\.*The bad set only needs to outscore the in\-support actions, an ordering that capped and uncapped scores both allow\. Our sweeps do not show saturation\. The cosine and hybrid readouts select low\-value OGBench actions with*negative*cosine values \(median−0\.25\-0\.25to−0\.74\-0\.74\), and no selected score exceeds0\.90\.9\. The measured failure comes from bounded high\-score false positives: the bad set outscores the alternatives without reaching the cap\.

The controlled setting in[Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)provides a concrete example ofBB: a low\-value region outside the support that can score above in\-support actions\.[Figure˜4](https://arxiv.org/html/2607.27422#S7.F4)visualizes this behavior, and[Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)measures it while separating unbounded norm inflation from finite high\-score false positives under bounded readouts\.

The theory identifies a gap in what contrastive training guarantees\. Compatibility can remain high even when graded value ordering is weak, leaving candidate maximization to amplify false positives in the score tail\. We test the score semantics in[Section˜6](https://arxiv.org/html/2607.27422#S6)and the consequences for selection in[Section˜7](https://arxiv.org/html/2607.27422#S7)\.

## 5Experimental Design

The experiments distinguish the two hypotheses from[Section˜1](https://arxiv.org/html/2607.27422#S1)\.H1\(norm inflation\) predicts that bounding the score should make candidate selection safe\.H2\(value decalibration\) predicts that the top score tail remains poorly calibrated to value even when the score is bounded\. The results support H2\.

### 5\.1Benchmarks and critics

Benchmarks\.The controlled 2D example in[Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)provides closed\-formQ⋆Q^\{\\star\}\. We extend the analysis to eight tasks from two domains in OGBench’s long\-horizon, sparse\-reward stitching benchmarks\(Park et al\.,[2025a](https://arxiv.org/html/2607.27422#bib.bib12)\): four navigation tasks \(PointMaze, AntMaze, HumanoidMaze, and ball\-dribbling AntSoccer, with action dimensions2/8/212/8/21\) and four manipulation tasks\. The return\-to\-goγd\\gamma^\{d\}is a valid navigation progress proxy, so the value\-calibration analyses use that domain\. The action\-dimension range also tests whether the failure persists in higher dimensions\. The proxy is invalid for*play*data, so the manipulation tasks contribute only to retrieval \([Table˜S10](https://arxiv.org/html/2607.27422#S5.T10)\)\.

Critics\.We use the four score families from[Section˜3](https://arxiv.org/html/2607.27422#S3): raw, cosine, TD\-Q, and hybrid \([Table˜1](https://arxiv.org/html/2607.27422#S3.T1)\)\. They use matched data and training budgets\. Raw, cosine, and hybrid share an encoder architecture; TD\-Q uses the joint\-input twin\-Q MLP described below\. Raw has an unbounded bilinear score, whereas cosine and hybrid use bounded readouts\.[Section˜S2](https://arxiv.org/html/2607.27422#S2a)describes the objectives and lists the clipped, spectrally normalized, conservative TD, ensemble, and quasimetric variants that the main sweep does not evaluate\.

Scope of the TD\-Q comparison\.Bellman regression receives strictly stronger supervision than the contrastive objective: scalar value targets rather than binary compatibility labels\. TD\-Q also uses a larger joint\-input twin\-Q MLP on\[s;g;a\]\[\\,s;g;a\\,\]with no shared parameters \([Equation˜S1](https://arxiv.org/html/2607.27422#S2.E1)\)\. The four\-critic sweep therefore mixes objective and function\-class effects\. Two controls separate them\. A joint two\-head critic uses one shared representation with two deployed scalars\. A parameter\-matched2×22\{\\times\}2trains each objective in each function class \([Section˜7\.3](https://arxiv.org/html/2607.27422#S7.SS3)\)\. Both controls attribute the failure to the training objective\.

TD\-Q’s success shows that a value\-calibrated scalar is*enough*for safe selection, not that Bellman recursion is*necessary*\. Pairwise value\-ranking losses\(Choi & Xu,[2026](https://arxiv.org/html/2607.27422#bib.bib2)\), ordinal temporal\-distance regression, quasimetric and temporal\-distance critics\(Wang et al\.,[2023a](https://arxiv.org/html/2607.27422#bib.bib14); Myers et al\.,[2024](https://arxiv.org/html/2607.27422#biba.bib2)\), and distance\-conditioned contrastive targets may also calibrate the score tail\. We do not test these alternatives, so our claims cover only the four scalars in[Table˜1](https://arxiv.org/html/2607.27422#S3.T1)\.

### 5\.2Evaluation signals and search protocols

We deploy each critic as a*candidate selection objective*\. The readout selectsaK⋆=arg⁡maxi≤K⁡fselect​\(s,ai,g\)a\_\{K\}^\{\\star\}=\\arg\\max\_\{i\\leq K\}f\_\{\\text\{select\}\}\(s,a\_\{i\},g\)fromCKC\_\{K\}as in \([2](https://arxiv.org/html/2607.27422#S3.E2)\), andKKserves as the expressivity proxy\. A*behavior\-regularized actor*such as DDPG\+\+BC operates differently\. Its on\-support deployment does not probe this failure setting \([Section˜S6\.1](https://arxiv.org/html/2607.27422#S6.SS1a)\)\.

Four protocols test different score properties\. In\-distribution triples reveal score semantics \([Section˜6](https://arxiv.org/html/2607.27422#S6)\), and the controlled environment measures fixed\-query selection error against closed\-formQ⋆Q^\{\\star\}\([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\. Our primary real\-task protocol uses frozen\(s,g\)\(s,g\)queries, cached diffusion candidate pools, and two independent judges: a reference value ensemble and a simulator rollout audit \([Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)\)\. Separate OGBench stress tests measure cross\-query score comparability and local\-support preference \([Sections˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)and[S4\.2](https://arxiv.org/html/2607.27422#S4.SS2a)\)\. These pooled\-triple tests compare scores across contexts; only the fixed\-query protocol compares actions at one decision point\.

WhereQ⋆Q^\{\\star\}is available\.Only the controlled 2D example provides closed\-form off\-supportQ⋆Q^\{\\star\}\([Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)\); OGBench does not support reliable off\-support estimation\. We therefore use the*in\-support*return\-to\-goγd\\gamma^\{d\}as a navigation progress proxy \([Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)\) and evaluate arbitrary actions only with local\-support diagnostics \([Section˜S4\.2](https://arxiv.org/html/2607.27422#S4.SS2a)\)\.[Section˜S2](https://arxiv.org/html/2607.27422#S2a)explains this proxy choice, including why sparse\-reward Monte Carlo estimates fall to zero, and describes the triple\-sampling construction\.

## 6Score Semantics: Compatibility versus Graded Progress

We first ask whether each learned scalar represents goal compatibility, graded progress, or both on shared in\-distribution data\.

### 6\.1Score progress calibration and ordering

Candidate maximization searches the score distribution’s top, so reliability requires high scores to correspond to high return\-to\-goγd\\gamma^\{d\}\. We bin shared in\-distribution\(s,a,g\)\(s,a,g\)triples by critic\-score decile and report meanγd\\gamma^\{d\}in each bin \([Figure˜2](https://arxiv.org/html/2607.27422#S6.F2)\)\. The top\-minus\-bottom decile gapΔ​γd\\Delta\\gamma^\{d\}summarizes the decision\-relevant trend\. This gap is strongly positive for TD\-Q \(0\.180\.18–0\.350\.35\), weak and task\-dependent for raw bilinear \(−0\.01\-0\.01–0\.150\.15\), and near zero or negative for cosine and hybrid\.[Section˜S5\.1](https://arxiv.org/html/2607.27422#S5.SS1a)reports the full values\.

The in\-support proxyγd\\gamma^\{d\}can fall below the optimum when stitching is possible \([Section˜5\.2](https://arxiv.org/html/2607.27422#S5.SS2)\)\. The OGBench results therefore measure score–progress calibration, while the controlled toy provides the closed\-formQ⋆Q^\{\\star\}analysis \([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\.

![Refer to caption](https://arxiv.org/html/2607.27422v1/x2.png)Figure 2:Score progress calibration \(OGBench navigation\)\.In\-distribution\(s,a,g\)\(s,a,g\)triples are binned by critic\-score decile; each bin reports meanγd\\gamma^\{d\}\. TD\-Q is monotone, raw bilinear has a weak slope, and cosine and hybrid are flat or inverted in the high\-score tail\. Curves show seed means with±\\pmstd bands on all four navigation tasks\. Per\-seed std is≤0\.02\\leq 0\.02, so the bands lie within the lines\.##### Global rank ordering\.

Calibration describes the score tail; we also test whether the*full*score order agrees with progress\. Candidate selection depends only on this order, so we report global Kendallτ\\taubetween each deployed readout andγd\\gamma^\{d\}\. Each task uses30003000in\-distribution triples;[Section˜S2](https://arxiv.org/html/2607.27422#S2a)gives sampling and metric details\.111Reported±\\pmvalues are mean±\\pmstd over 3 training seeds \(\{42,43,44\}\)\. The HumanoidMaze TD\-Q seed\-43 checkpoint is a documented retrain with the identical recipe\. A hardware failure erased the original checkpoint, whose archived scores were unusual\. The retrain reproduces the reported behavior \(γd\\gamma^\{d\}τ=\+0\.793\\tau=\+0\.793\)\. See[SectionS2](https://arxiv.org/html/2607.27422#S2a)for the seed policy\.

[Table˜2](https://arxiv.org/html/2607.27422#S6.T2)shows a clear separation\. TD\-Q produces strong progress ordering \(τ=0\.49\\tau=0\.49–0\.790\.79\), whereas raw bilinear has only weak, task\-dependent ordering:0\.220\.22–0\.270\.27on the three core tasks and a negative correlation on AntSoccer\. Cosine and hybrid are near zero or negative despite using the same training data\.

Table 2:In\-support value orderingon four navigation tasks\. Kendallτ\\taubetween deployed score andγd\\gamma^\{d\}, using 3000 in\-distribution triples per task;γd\\gamma^\{d\}is valid only for navigation\. TD\-Q ranks progress, raw bilinear is weak, and cosine and hybrid are near zero\. Values are mean±\\pmstd over training seeds \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.
##### No monotone fix after training\.

Best\-of\-KKselection uses only the*order*of the deployed score in \([2](https://arxiv.org/html/2607.27422#S3.E2)\), so monotone recalibration cannot change it\. Temperature scaling, Platt or isotonic maps, and any other strictly increasing transform leaveaK⋆a\_\{K\}^\{\\star\}unchanged\. The cosine and hybrid readouts have near\-zero or negative rank correlation with value \([Table˜2](https://arxiv.org/html/2607.27422#S6.T2)\), so these post\-training transformations cannot repair selection\. A different training signal, rather than a rescaled score, must repair the ordering\.

##### Reading the TD\-Q reference\.

TD\-Q is the calibrated reference\. Its Bellman fixed point on goal\-directed data isγd\\gamma^\{d\}, so evaluation againstγd\\gamma^\{d\}partly reuses its training target\. We treat its strongΔ​γd\\Delta\\gamma^\{d\}andτ\\tauas a sanity\-check ceiling, not as independent evidence\. The contrastive results do not share this overlap because binary compatibility training never encodesγd\\gamma^\{d\}\. Their flat or inverted score tail is not built into the evaluation target\. The controlled toy shows the same gap against closed\-formQ⋆Q^\{\\star\}\([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\.[Section˜S4\.4](https://arxiv.org/html/2607.27422#S4.SS4)provides a second OGBench anchor: an independent, stitching\-aware GCIQL value reference trained with a different seed\. The contrastive curse remains, so it is not caused byγd\\gamma^\{d\}, including with trained diffusion\-policy candidate pools \([Table˜S6](https://arxiv.org/html/2607.27422#S4.T6)\)\.

### 6\.2Compatibility ranking versus value ordering

Compatibility ranking need not imply value ranking \([Section˜S1](https://arxiv.org/html/2607.27422#S1a)\), so we measure both directly\. The contrastive critics perform well on the task they are trained for:*retrieving*the goal\. We measure retrieval AUC and top\-kkrecall for each in\-distribution tuple\(s,a,g\+\)\(s,a,g^\{\+\}\), whereg\+g^\{\+\}is a future state \([Section˜S5\.3](https://arxiv.org/html/2607.27422#S5.SS3)\)\. Retrieval AUC measures whether the true goal scores above a random goal\.

[Figure˜3](https://arxiv.org/html/2607.27422#S6.F3)\(a\) separates the two forms of ranking\. Raw and cosine reach retrieval AUCs of0\.960\.96–1\.01\.0, while hybrid is lower and more variable at0\.860\.86–0\.990\.99\. The pattern remains with harder negatives built from goals of nearest\-state anchors on other trajectories\. The contrastive readouts remain the strongest retrievers on every task under that construction \([Tables˜S10](https://arxiv.org/html/2607.27422#S5.T10)and[S5\.3](https://arxiv.org/html/2607.27422#S5.SS3)\), yet theirγd\\gamma^\{d\}ordering remains weak or absent\. TD\-Q generally shows the reverse pattern: strongerγd\\gamma^\{d\}ordering and less emphasis on retrieval\. HumanoidMaze is the exception, where TD\-Q also retrieves nearly perfectly\.

[Figure˜3](https://arxiv.org/html/2607.27422#S6.F3)\(b\) explains this separation\. Each contrastive score stays far above the random\-goal baseline at*every*distance but changes little as distance grows\. It is an effective retrieval signal with almost no value gradient\. TD\-Q scores instead decay toward the baseline as the goal recedes\. Compatibility and value ranking can therefore diverge within the same critic\. The retrieval result extends to four manipulation tasks across Cube, Scene, and Puzzle\. Raw and cosine remain near\-perfect retrievers, whereas TD\-Q is much weaker on Cube and Scene \([Table˜S10](https://arxiv.org/html/2607.27422#S5.T10)\)\. These tasks test only compatibility ranking becauseγd\\gamma^\{d\}is invalid on play data\.

![Refer to caption](https://arxiv.org/html/2607.27422v1/x3.png)Figure 3:Good rankers, bad objectiveson navigation tasks\.\(a\)Retrieval AUC \(compatibility\) versus Kendallτ\\tauagainstγd\\gamma^\{d\}\(value ordering\)\. Contrastive critics occupy the high\-AUC, low\-τ\\tauregion; TD\-Q is the calibrated reference\. Markers show mean±\\pmstd over 3 training seeds on all four tasks \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.\(b\)Positive\-pair score versus temporal distance \(1=1=nearest goal,0=0=random baseline\), averaged over the three core tasks \(±\\pmstd band, std≤0\.06\\leq 0\.06\)\. Contrastive scores stay flat, indicating little value gradient, while TD\-Q decays with distance\.
### 6\.3Training objective versus inference readout

[Table˜2](https://arxiv.org/html/2607.27422#S6.T2)shows that cosine and hybrid scores fail to order value but does not identify whether the bounded inference*readout*or cosine*training*objective causes the failure\. We test both explanations by applying several readouts to each trained encoder \([Section˜S5\.1](https://arxiv.org/html/2607.27422#S5.SS1a)\)\. Raw\-trained embeddings retain weak value ordering under a cosine readout, with positiveτ\\tau, while the norm‖ϕ‖\\\|\\phi\\\|alone carries almost none of the signal\. Cosine\-trained embeddings lose the ordering under both raw and normalized readouts\. Cosine*training*therefore removes the ordering; inference\-time normalization preserves the raw\-trained embeddings’ weak ordering\.

The readout comparison shows two roles of magnitude\. The norm‖ϕ‖\\\|\\phi\\\|carries little value signal within support, but raw bilinear magnitude can still provide an off\-support extrapolation channel under candidate maximization\.

The contrastive critics thus keep goal\-compatibility information but lack reliable graded\-progress ordering, especially in the score tail targeted by candidate search\. The next section measures how this gap affects selection\.

## 7Selection Consequences of Score Tail Search

The score\-semantics results show weak progress ordering in the contrastive score tail\. We now vary the candidate budgetKKand measure selection error first against closed\-formQ⋆Q^\{\\star\}and then on OGBench\.

### 7\.1Controlled candidate selection with closed\-form value

The controlled 2D environment hass′=s\+as^\{\\prime\}=s\+aandQ⋆​\(s,a,g\)=−‖s\+a−g‖Q^\{\\star\}\(s,a,g\)=\-\\\|s\+a\-g\\\|\. Dataset actions lie in a support disk of radiusRR\. Each query usess=0s=0and‖g‖≤0\.6​R\\\|g\\\|\\leq 0\.6R, so the optimum lies strictly inside the support\. Clipped Gaussian candidate pools contain actions from both inside and outside the disk\. We compute regret against the exact pool oracle and record whether each selection is off support\.[Figure˜4](https://arxiv.org/html/2607.27422#S7.F4)visualizes one representative run;[Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)gives the full setup\. The main result usesR=0\.40R=0\.40, and the ordering remains stable acrossR∈\{0\.25,0\.40,0\.55\}R\\in\\\{0\.25,0\.40,0\.55\\\}\([Section˜S3](https://arxiv.org/html/2607.27422#S3a)\)\.

![Refer to caption](https://arxiv.org/html/2607.27422v1/x4.png)Figure 4:Qualitative score geometry in the controlled 2D environment\.Columns show the four deployed score families \([Section˜3](https://arxiv.org/html/2607.27422#S3)\)\. Rows show the learned score surface, a contour withK=256K\{=\}256candidates and selected actionaK⋆a\_\{K\}^\{\\star\}\(⋆\\star; orange support disk\), and‖∇af‖\\\|\\nabla\_\{a\}f\\\|\. Raw bilinear, cosine, and hybrid select outside the support disk in this representative run; TD\-Q selects inside\.Table 3:Controlled candidate selection atR=0\.40R\{=\}0\.40, with the optimum in support and exactQ⋆Q^\{\\star\}\. Values are mean±\\pmstd over five seeds\. Cosine is less severe and more variable than raw but still selects off support in most runs; hybrid does so nearly as often as raw\. Only TD\-Q selects inside support with near\-zero regret\.A same\-checkpoint control replaces the cosine readout with a separate TD\-calibrated head on the same network\. This change restores near\-oracle selection: regret falls from0\.900\.90to0\.070\.07, and the off\-support rate falls from80%80\\%to0%0\\%\. The shared representation lets this control locate the failure in the deployed scalar\.[Table˜S2](https://arxiv.org/html/2607.27422#S3.T2)gives the toy results, and[Section˜S7](https://arxiv.org/html/2607.27422#S7a)gives the OGBench version\.

The selected proxy score rises for every contrastive critic asKKgrows, but the selected action’s true value falls\. Only TD\-Q remains near the oracle\. The selected embedding norms isolate the raw bilinear channel: raw norms stay high asKKincreases, while bounded and calibrated critic norms fall\.[Section˜S3](https://arxiv.org/html/2607.27422#S3a)reports the curse curve \([Figure˜S1](https://arxiv.org/html/2607.27422#S3.F1)\) and norm curve \([Figure˜S2](https://arxiv.org/html/2607.27422#S3.F2)\)\.

The controlled study verifies both candidate\-level stages: high\-score false positives exist, and stronger search finds them more often\. We next test the mechanism on OGBench with fixed\-query action selection, our primary selection protocol\.

### 7\.2Fixed\-query action selection on OGBench

This experiment provides our primary real\-task selection evidence\. We freeze15001500validation\-split queries\(s,g\)\(s,g\)per task, with500500in each goal\-distance group \(near, medium, and far\)\. Each query uses five cached pools of6464actions from a goal\-conditioned diffusion behavior proposalπ​\(a\|s,g\)\\pi\(a\\,\|\\,s,g\)plus the logged dataset action as an anchor\. Each generated pool is shared across critics, which pairs the comparisons\.

Two independent judges evaluate each critic’sarg⁡max\\arg\\max\. A*reference ensemble*contains three independently seeded GCIQL value functions\. We use it only after a prescoring check confirms that it resolves within\-pool differences and agrees with itself\. A*simulator rollout audit*executes the selected action from an exactly restored state and then follows a fixed, critic\-independent continuation policy from the same diffusion behavior model\. Common random numbers pair candidates, so realized\-return differences reflect the single selected action\.

*We declared inclusion criteria before scoring\.*A task enters the analysis only if at least one judge is trustworthy\. The reference ensemble passes with resolvability≥50%\\geq 50\\%in a prespecified group and mean pairwise Spearman≥0\.7\\geq 0\.7\. The rollout audit passes when the paired 95% CI half\-width onΔ\\Deltaversus random falls below the smallest effect of interest,emin≈0\.012e\_\{\\min\}\\approx 0\.012, the realized PointMaze margin\.

PointMaze passes both criteria \(Spearman0\.880\.88\)\. AntMaze and HumanoidMaze fail the reference criterion \(Spearman0\.550\.55and0\.420\.42\) because their ensembles do not agree reliably, but both pass the rollout\-power criterion\. We mark their reference cells accordingly\. AntSoccer fails both criteria: it has no reliable value signal, and the value\-trained policy succeeds only∼5%\{\\sim\}5\\%of the time\. We therefore exclude it from selection analysis\. The score\-semantics analysis retains all four tasks because it needs only in\-supportγd\\gamma^\{d\}\([Section˜6](https://arxiv.org/html/2607.27422#S6)\)\. These rules explain the different task counts\.

Table 4:Fixed\-query action selection under one protocol across tasks\.Columns report reference\-judgedτb\\tau\_\{b\}and normalized regret atK=64K\{=\}64, plus rolloutΔ\\Deltaversus random;∗∗∗marks a 95% CI excluding zero\. AntMaze and HumanoidMaze enter through rollouts because their reference estimates are unreliable\. PointMaze reference metrics use mean±\\pmstd over seeds\{42,43,44\}\\\{42,43,44\\\}; other cells use seed 42\. Rollouts use 102 preregistered PointMaze and AntMaze queries and 408 HumanoidMaze queries\.[Table˜4](https://arxiv.org/html/2607.27422#S7.T4)shows a consistent*ordering gap*\. The Bellman\-trained scalar orders within\-pool actions on every included task \(τb=0\.60\\tau\_\{b\}=0\.60–0\.900\.90\)\. Every deployed contrastive scalar is near\-flat \(τb≤0\.41\\tau\_\{b\}\\leq 0\.41\) and has random\-level normalized regret atK=64K\{=\}64\. A same\-checkpoint two\-head control shows the same gap while holding the representation fixed\. The deployed scalar alone moves AntMazeτb\\tau\_\{b\}from0\.730\.73to0\.110\.11\([Section˜S7](https://arxiv.org/html/2607.27422#S7a)\)\.

The realized cost of misordering is task dependent\. One selected action matters on PointMaze: TD\-Q gains\+0\.012\+0\.012over random with a confidence interval excluding zero, raw gains about half as much, and cosine gains nothing\. AntMaze and HumanoidMaze yield well\-powered nulls: no selector, including the reference oracle or logged action, beats a random pool draw\. A self\-correcting controller can therefore flatten single\-step differences, while contrastive selection leaves the available margin unused when one action matters \([Section˜S6\.2](https://arxiv.org/html/2607.27422#S6.SS2a)\)\.

### 7\.3Function\-class control: objective versus capacity

TD\-Q changes both the objective and function class \([Equation˜S1](https://arxiv.org/html/2607.27422#S2.E1)\), so we complete the2×22\{\\times\}2on PointMaze\. A Bellman\-trained bilinear scalar in the raw critic’s function class and a contrastive\-trained MLP parameter\-matched to TD\-Q supply the added cells\. Each uses three seeds\.

Table 5:Objective versus function classunder the PointMaze fixed\-query protocol\. Cells reportτb\\tau\_\{b\}@64 \[NRegret@64\]\. Added cells use mean±\\pmstd over seeds\{42,43,44\}\\\{42,43,44\\\}; existing cells use seed 42\. The capacity\-matched contrastive MLP follows the contrastive bilinear cell\.The objective explains most of the gap \([Table˜5](https://arxiv.org/html/2607.27422#S7.T5)\)\. Bellman training raisesτb\\tau\_\{b\}by0\.300\.30in the bilinear class and0\.440\.44in the MLP class\. The architecture match raises contrastiveτb\\tau\_\{b\}only from0\.400\.40to0\.460\.46\. Function class has a smaller effect: the bilinear bottleneck lowers Bellmanτb\\tau\_\{b\}from0\.910\.91to0\.700\.70\. The worst Bellman cell still beats the best contrastive cell on every metric\.

## 8Discussion and Conclusion

##### Boundedness versus calibration\.

Candidate selection depends on score ordering, not scale\. Norm drift explains most raw regret, but bounded cosine and hybrid critics also choose low\-value actions below the cap\. Value decalibration \(H2\) is the broader problem: maximization exposes false positives caused by norm drift, bounded scores, or in\-support misranking \([Corollary˜1](https://arxiv.org/html/2607.27422#Thmcorollary1)\)\.

##### Where misordering costs realized return\.

Misordering reduces return in the toy and on PointMaze, where one action matters\. AntMaze and HumanoidMaze yield well\-powered nulls because no selector, including the reference oracle, beats random \([Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)\)\. These results suggest that task dynamics mediate the practical cost of misordering\. Future work can test whether immediate action effects and controller self\-correction account for this variation\. Cross\-query comparability can still fail when within\-query choice has little effect \([Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)\)\.

##### Training and deployment scope\.

The training/readout control locates the failure in the objective: raw\-trained embeddings retain weak ordering after normalization, whereas cosine training removes it\. Binary compatibility leaves positives unordered \([Section˜S1](https://arxiv.org/html/2607.27422#S1a)\), but pairwise ranking or temporal\-distance targets may also provide graded supervision \([Section˜5\.1](https://arxiv.org/html/2607.27422#S5.SS1)\)\. A two\-head critic can combine contrastive retrieval with calibrated selection \([Section˜S7](https://arxiv.org/html/2607.27422#S7a)\), and a behavior\-regularized actor can stay near support \([Section˜S6\.1](https://arxiv.org/html/2607.27422#S6.SS1a)\)\. We therefore argue against maximizing an uncalibrated compatibility score, not contrastive representations or expressive policies\.

##### Scope and future work\.

Our experiments establish the ranking–selection distinction in a controlled toy and four navigation tasks, while manipulation play data supplies complementary retrieval evidence\. The findings remain stable across seeds, support radii, and proposals\. Future work can extend fixed\-query value audits to manipulation domains with suitable progress oracles, compare cosine bounding with structured or support\-constrained objectives, and test when controller self\-correction makes score misordering inconsequential\. These directions build on the central result: contrastive critics remain useful rankers, while action selection benefits from value ordering over searched actions\.

## References

- Chi et al\. \(2023\)Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, Russ Tedrake, and Shuran Song\.Diffusion policy: Visuomotor policy learning via action diffusion\.In*Proceedings of Robotics: Science and Systems*, 2023\.URL[https://arxiv\.org/abs/2303\.04137](https://arxiv.org/abs/2303.04137)\.
- Choi & Xu \(2026\)Andrew Choi and Wei Xu\.RankQ: Offline\-to\-online reinforcement learning via self\-supervised action ranking\.*arXiv preprint arXiv:2605\.11151*, 2026\.URL[https://arxiv\.org/abs/2605\.11151](https://arxiv.org/abs/2605.11151)\.
- Dong et al\. \(2025\)Perry Dong, Qiyang Li, Dorsa Sadigh, and Chelsea Finn\.EXPO: Stable reinforcement learning with expressive policies\.*arXiv preprint arXiv:2507\.07986*, 2025\.URL[https://arxiv\.org/abs/2507\.07986](https://arxiv.org/abs/2507.07986)\.
- Eysenbach et al\. \(2022\)Benjamin Eysenbach, Tianjun Zhang, Ruslan Salakhutdinov, and Sergey Levine\.Contrastive learning as goal\-conditioned reinforcement learning\.In*Advances in Neural Information Processing Systems*, 2022\.URL[https://arxiv\.org/abs/2206\.07568](https://arxiv.org/abs/2206.07568)\.
- Fujimoto et al\. \(2019\)Scott Fujimoto, David Meger, and Doina Precup\.Off\-policy deep reinforcement learning without exploration\.In*Proceedings of the International Conference on Machine Learning*, 2019\.URL[https://arxiv\.org/abs/1812\.02900](https://arxiv.org/abs/1812.02900)\.
- Kostrikov et al\. \(2022\)Ilya Kostrikov, Ashvin Nair, and Sergey Levine\.Offline reinforcement learning with implicit q\-learning\.In*International Conference on Learning Representations*, 2022\.URL[https://arxiv\.org/abs/2110\.06169](https://arxiv.org/abs/2110.06169)\.
- Kumar et al\. \(2020\)Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine\.Conservative Q\-learning for offline reinforcement learning\.In*Advances in Neural Information Processing Systems*, 2020\.URL[https://arxiv\.org/abs/2006\.04779](https://arxiv.org/abs/2006.04779)\.
- Lu et al\. \(2023\)Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu\.Contrastive energy prediction for exact energy\-guided diffusion sampling in offline reinforcement learning\.In*Proceedings of the International Conference on Machine Learning*, 2023\.URL[https://arxiv\.org/abs/2304\.12824](https://arxiv.org/abs/2304.12824)\.
- Mao et al\. \(2024\)Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang\.Diffusion\-DICE: In\-sample diffusion guidance for offline reinforcement learning\.In*Advances in Neural Information Processing Systems*, 2024\.URL[https://arxiv\.org/abs/2407\.20109](https://arxiv.org/abs/2407.20109)\.
- Myers et al\. \(2024\)Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach\.Learning temporal distances: Contrastive successor features can provide a metric structure for decision\-making\.In*Proceedings of the International Conference on Machine Learning*, 2024\.URL[https://arxiv\.org/abs/2406\.17098](https://arxiv.org/abs/2406.17098)\.
- Nakamoto et al\. \(2023\)Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine\.Cal\-QL: Calibrated offline RL pre\-training for efficient online fine\-tuning\.In*Advances in Neural Information Processing Systems*, 2023\.URL[https://arxiv\.org/abs/2303\.05479](https://arxiv.org/abs/2303.05479)\.
- Park et al\. \(2025a\)Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine\.OGBench: Benchmarking offline goal\-conditioned RL\.In*International Conference on Learning Representations*, 2025a\.URL[https://arxiv\.org/abs/2410\.20092](https://arxiv.org/abs/2410.20092)\.
- Park et al\. \(2025b\)Seohong Park, Qiyang Li, and Sergey Levine\.Flow Q\-learning\.*arXiv preprint arXiv:2502\.02538*, 2025b\.URL[https://arxiv\.org/abs/2502\.02538](https://arxiv.org/abs/2502.02538)\.
- Wang et al\. \(2023a\)Tongzhou Wang, Antonio Torralba, Phillip Isola, and Amy Zhang\.Optimal goal\-reaching reinforcement learning via quasimetric learning\.In*Proceedings of the International Conference on Machine Learning*, 2023a\.URL[https://arxiv\.org/abs/2304\.01203](https://arxiv.org/abs/2304.01203)\.
- Wang et al\. \(2023b\)Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou\.Diffusion policies as an expressive policy class for offline reinforcement learning\.In*International Conference on Learning Representations*, 2023b\.URL[https://arxiv\.org/abs/2208\.06193](https://arxiv.org/abs/2208.06193)\.
- Zheng et al\. \(2024\)Chongyi Zheng, Benjamin Eysenbach, Homer Walke, Patrick Yin, Kuan Fang, Ruslan Salakhutdinov, and Sergey Levine\.Stabilizing contrastive RL: Techniques for robotic goal reaching from offline data\.In*International Conference on Learning Representations*, 2024\.URL[https://arxiv\.org/abs/2306\.03346](https://arxiv.org/abs/2306.03346)\.

Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search \(Supplementary Material\)

Ayushman Singh††thanks:Also affiliated with Sesame AI \(ayushman@sesame\.com\); work done independently outside Sesame\.aysingh@stanford\.edu Stanford University

Siddharth Aphalesaphale@stanford\.edu Stanford University

##### Roadmap\.

This supplementary material supports the main paper\. Section and table references of the form Snnpoint here; plain numbers refer to the main paper\.[Section˜S1](https://arxiv.org/html/2607.27422#S1a)contains the proofs for[Section˜4](https://arxiv.org/html/2607.27422#S4)\.[Section˜S2](https://arxiv.org/html/2607.27422#S2a)describes the training objectives andγd\\gamma^\{d\}protocol used in[Section˜5](https://arxiv.org/html/2607.27422#S5)\. The controlled setup and robustness checks for[Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)appear in[Section˜S3](https://arxiv.org/html/2607.27422#S3a)\.[Section˜S4](https://arxiv.org/html/2607.27422#S4a)contains the cross\-query and local\-support tests, pool checks, and independent value oracle\.[Section˜S5](https://arxiv.org/html/2607.27422#S5a)contains the full score\-semantics, Bellman\-residual, and retrieval tables for[Section˜6](https://arxiv.org/html/2607.27422#S6)\. The final two sections cover deployment scope \([Section˜S6](https://arxiv.org/html/2607.27422#S6a)\) and the two\-head positive control \([Section˜S7](https://arxiv.org/html/2607.27422#S7a)\)\.

## S1Proofs

This section proves[Proposition˜1](https://arxiv.org/html/2607.27422#Thmproposition1),[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2),[Corollary˜1](https://arxiv.org/html/2607.27422#Thmcorollary1), and the2​ε2\\varepsilonregret bound\. §[4](https://arxiv.org/html/2607.27422#S4)gives the arguments for off\-support nonidentifiability and unbounded bilinear extrapolation\.

### Proof of[Proposition˜1](https://arxiv.org/html/2607.27422#Thmproposition1)

The off\-support pairs inBXB\_\{X\}lie a positive distanceδ=dist​\(ΩX,BX\)\>0\\delta=\\mathrm\{dist\}\(\\Omega\_\{X\},B\_\{X\}\)\>0from the support becauseΩX\\Omega\_\{X\}andBXB\_\{X\}are disjoint and closed in a compact domain\. Define the bump

u​\(s,a\)=max⁡\{0,1−dist​\(\(s,a\),BX\)/δ\}∈\[0,1\]\.u\(s,a\)=\\max\\\!\\left\\\{0,\\;1\-\\mathrm\{dist\}\\\!\\big\(\(s,a\),\\,B\_\{X\}\\big\)/\\delta\\right\\\}\\;\\in\\;\[0,1\]\.This function is continuous and1/δ1/\\delta\-Lipschitz\. It equals11onBXB\_\{X\}and0on the support because every support point is at leastδ\\deltafromBXB\_\{X\}\. Setv≡1v\\equiv 1; more generally, any continuousv​\(g\)∈\[0,1\]v\(g\)\\in\[0,1\]equal to11onG0G\_\{0\}would be enough\. Add one coordinate to each embedding:

φ′​\(s,a\)=\[φ​\(s,a\);β​u​\(s,a\)\],ψ′​\(g\)=\[ψ​\(g\);v​\(g\)\]\.\\varphi^\{\\prime\}\(s,a\)=\[\\varphi\(s,a\);\\;\\beta\\,u\(s,a\)\],\\qquad\\psi^\{\\prime\}\(g\)=\[\\psi\(g\);\\;v\(g\)\]\.Thenf′​\(s,a,g\)=φ​\(s,a\)⊤​ψ​\(g\)\+β​u​\(s,a\)​v​\(g\)f^\{\\prime\}\(s,a,g\)=\\varphi\(s,a\)^\{\\top\}\\psi\(g\)\+\\beta\\,u\(s,a\)\\,v\(g\)\. The identityu=0u=0holds onΩX\\Omega\_\{X\}, sof′=ff^\{\\prime\}=fand off\-support nonidentifiability leaves the contrastive loss unchanged\. Choose the finite weight

β=max⁡\{0,M−minBX×G0⁡φ⊤​ψ\}\.\\beta=\\max\\\{0,\\;M\-\\min\_\{B\_\{X\}\\times G\_\{0\}\}\\varphi^\{\\top\}\\psi\\\}\.The identitiesu=v=1u=v=1hold onBX×G0B\_\{X\}\\times G\_\{0\}\. This choice ofβ\\betagives

f′​\(s,a,g\)=φ⊤​ψ\+β≥minBX×G0⁡f\+β≥M\.f^\{\\prime\}\(s,a,g\)\\;=\\;\\varphi^\{\\top\}\\psi\+\\beta\\;\\geq\\;\\min\_\{B\_\{X\}\\times G\_\{0\}\}f\+\\beta\\;\\geq\\;M\.The embeddings remain bounded on the compact domain:‖φ′‖≤‖φ‖2\+β2<∞\\\|\\varphi^\{\\prime\}\\\|\\leq\\sqrt\{\\\|\\varphi\\\|^\{2\}\+\\beta^\{2\}\}<\\infty\. Thus, the construction requires no norm divergence\.□\\square

### Proof of[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)

The probability that at least one of theKKindependent draws lies inBBis1−μ​\(Ain\)K=1−\(1−p\)K1\-\\mu\(A\_\{\\textup\{in\}\}\)^\{K\}=1\-\(1\-p\)^\{K\}\. The critic selects fromBBon this event because everyb∈Bb\\in BscoresMM, which strictly exceedsmaxa∈Ain⁡f​\(a\)\\max\_\{a\\in A\_\{\\textup\{in\}\}\}f\(a\)\. Therefore,Pr⁡\(aK⋆∈B\)=1−\(1−p\)K→1\\Pr\(a\_\{K\}^\{\\star\}\\in B\)=1\-\(1\-p\)^\{K\}\\to 1asK→∞K\\to\\infty\.

The eventEKE\_\{K\}denotes, for the regret bound, a pool that contains an action from each set:

EK=\{∃i:ai∈B\}∩\{∃j:aj∈Ain\}\.E\_\{K\}=\\\{\\exists i:a\_\{i\}\\in B\\\}\\cap\\\{\\exists j:a\_\{j\}\\in A\_\{\\textup\{in\}\}\\\}\.Its probability is

Pr⁡\(EK\)=1−pK−\(1−p\)K\.\\Pr\(E\_\{K\}\)=1\-p^\{K\}\-\(1\-p\)^\{K\}\.The critic selects fromBBonEKE\_\{K\}and receivesQ​\(aK⋆\)=qbadQ\(a\_\{K\}^\{\\star\}\)=q\_\{\\textup\{bad\}\}, while the oracle selects fromAinA\_\{\\textup\{in\}\}and receivesQ​\(aQ⋆\)≥qgoodQ\(a\_\{Q\}^\{\\star\}\)\\geq q\_\{\\textup\{good\}\}\. Thus,

RegretK≥qgood−qbadon​EK\.\\text\{Regret\}\_\{K\}\\;\\geq\\;q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\\quad\\text\{on \}E\_\{K\}\.Expectations give

𝔼​\[RegretK\]≥\(qgood−qbad\)​Pr⁡\(EK\)=\(qgood−qbad\)​\(1−pK−\(1−p\)K\)\.\\mathbb\{E\}\[\\text\{Regret\}\_\{K\}\]\\;\\geq\\;\(q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\)\\,\\Pr\(E\_\{K\}\)\\;=\\;\(q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\)\\bigl\(1\-p^\{K\}\-\(1\-p\)^\{K\}\\bigr\)\.BothpKp^\{K\}and\(1−p\)K\(1\-p\)^\{K\}converge to zero because0<p<10<p<1\. Therefore,

lim infK→∞𝔼​\[RegretK\]≥qgood−qbad\.\\liminf\_\{K\\to\\infty\}\\mathbb\{E\}\[\\text\{Regret\}\_\{K\}\]\\geq q\_\{\\textup\{good\}\}\-q\_\{\\textup\{bad\}\}\.□\\square

### Proof of[Corollary˜1](https://arxiv.org/html/2607.27422#Thmcorollary1)

SetM=cmaxM=c\_\{\\max\}\. The hypotheses of[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)require only thatM\>maxa∈Ain⁡f​\(a\)M\>\\max\_\{a\\in A\_\{\\textup\{in\}\}\}f\(a\)\. The condition holds because

maxa∈Ain⁡f​\(a\)≤cmax−Δ<cmax=M,\\max\_\{a\\in A\_\{\\textup\{in\}\}\}f\(a\)\\leq c\_\{\\max\}\-\\Delta<c\_\{\\max\}=M,so the condition holds\. The proof of[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)imposes no other limit on the scale ofMM\. Its conclusions therefore apply whenM=cmaxM=c\_\{\\max\}\.□\\square

### Proof of the2​ε2\\varepsilonregret bound

The uniform accuracy bound gives, for everya∈CKa\\in C\_\{K\},Q​\(a\)≤f​\(a\)\+εQ\(a\)\\leq f\(a\)\+\\varepsilonandf​\(a\)≤Q​\(a\)\+εf\(a\)\\leq Q\(a\)\+\\varepsilon\. We apply the first inequality toaQ⋆a\_\{Q\}^\{\\star\}\. The actionaK⋆a\_\{K\}^\{\\star\}maximizesffoverCKC\_\{K\}, sof​\(aQ⋆\)≤f​\(aK⋆\)f\(a\_\{Q\}^\{\\star\}\)\\leq f\(a\_\{K\}^\{\\star\}\), and therefore

Q​\(aQ⋆\)≤f​\(aQ⋆\)\+ε≤f​\(aK⋆\)\+ε\.Q\(a\_\{Q\}^\{\\star\}\)\\;\\leq\\;f\(a\_\{Q\}^\{\\star\}\)\+\\varepsilon\\;\\leq\\;f\(a\_\{K\}^\{\\star\}\)\+\\varepsilon\.The second inequality givesf​\(aK⋆\)≤Q​\(aK⋆\)\+εf\(a\_\{K\}^\{\\star\}\)\\leq Q\(a\_\{K\}^\{\\star\}\)\+\\varepsilon\. The two bounds give

Q​\(aQ⋆\)≤Q​\(aK⋆\)\+2​ε\.Q\(a\_\{Q\}^\{\\star\}\)\\;\\leq\\;Q\(a\_\{K\}^\{\\star\}\)\+2\\varepsilon\.We subtractQ​\(aK⋆\)Q\(a\_\{K\}^\{\\star\}\)from both sides and obtainRegretK≤2​ε\\text\{Regret\}\_\{K\}\\leq 2\\varepsilon\.□\\square

### Deferred remarks and discussion

##### Why the bad set is allowed\.

The contrastive loss places no constraint onffoverBBbecauseBBlies off support\.[Proposition˜1](https://arxiv.org/html/2607.27422#Thmproposition1)shows that a bounded bilinear critic can assign any finite marginMMthere\. The bad set required by[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)can be constructed within the critic class\. Unbounded norm growth is theM→∞M\\to\\inftylimit of the same construction\. The setBBonly needs to be a separated, low\-value, high\-scoring subset of the unsupported actions,B⊆𝒜∖AinB\\subseteq\\mathcal\{A\}\\setminus A\_\{\\textup\{in\}\}\. Other unsupported actions may be harmless or valuable; one bad region is enough\.

## S2Training objectives and evaluation protocols

This section details the training objectives and evaluation protocols used in[Sections˜5\.1](https://arxiv.org/html/2607.27422#S5.SS1),[5\.2](https://arxiv.org/html/2607.27422#S5.SS2)and[6\.1](https://arxiv.org/html/2607.27422#S6.SS1)\. The four score families in[Table˜1](https://arxiv.org/html/2607.27422#S3.T1)use the same data and training budget\. Raw, cosine, and hybrid share an encoder architecture; TD\-Q uses the joint\-input MLP described below\.

##### Seed policy\.

Diagnostic tables report mean±\\pmstd over three training seeds\{42,43,44\}\\\{42,43,44\\\}\. We exclude a run or rerun it*only*after an implementation failure\. Valid reasons are incomplete required steps, a corrupt or unloadable checkpoint, non\-finite parameters or losses, an incorrect task or config hash, or a confirmed bug that changes the intended objective\. Results never trigger exclusion\. Poor performance, unusual learning curves, and disagreement with other seeds are not valid reasons\. We repeat the same seed rather than use another when a run is invalid\.

We applied this policy once\. A hardware failure destroyed the HumanoidMaze TD\-Q seed\-43 checkpoint, and the archived scores were unusual\. A fresh run with the same recipe reproduced the reported behavior \(γd\\gamma^\{d\}Kendallτ=\+0\.793\\tau=\+0\.793, compared with\+0\.792\+0\.792in the earlier report\)\. We use the retrained seed throughout, replacing an earlier after\-the\-fact seed\-45 substitution\.

Paired bootstrap CIs capture uncertainty over evaluation queries and pool replicates in the selection experiments\. Resamples are shared across critics\. The three\-seed spread instead measures training variability; in the fixed\-query headline, its standard deviation is only11–2%2\\%of the TD\-versus\-contrastive gap \([Table˜4](https://arxiv.org/html/2607.27422#S7.T4)\)\.

##### Training objectives\.

Raw and cosine use the same contrastive objective\. Raw uses unnormalized embeddings, while cosine normalizes both embeddings before computing their inner product\. The cosine scorefcosf\_\{\\text\{cos\}\}is therefore restricted to\[−1,1\]\[\-1,1\]\([Equation˜4](https://arxiv.org/html/2607.27422#S3.E4)\)\. This bound prevents unlimited score growth but not high off\-support scores\.

*TD\-Q*is the GCIQL critic, a separate goal\-conditioned value network\. It is a joint\-input twin\-Q MLP over the concatenated input, trained with Bellman backups and deployed as the ensemble mean,

Qω​\(s,a,g\)=12​∑j∈\{1,2\}MLPωj​\(\[s;g;a\]\)\.Q\_\{\\omega\}\(s,a,g\)\\;=\\;\\tfrac\{1\}\{2\}\\\!\\sum\_\{j\\in\\\{1,2\\\}\}\\mathrm\{MLP\}\_\{\\omega\_\{j\}\}\\\!\\bigl\(\[\\,s;\\,g;\\,a\\,\]\\bigr\)\.\(S1\)It shares no parameters with the contrastive encoders\(ϕθ,ψθ\)\(\\phi\_\{\\theta\},\\psi\_\{\\theta\}\)and is*not*a readout over their embeddings\. Its nonlinear joint architecture also has more capacity than the bilinear score\.[Section˜7\.3](https://arxiv.org/html/2607.27422#S7.SS3)controls for this gap with a parameter\-matched2×22\{\\times\}2: a Bellman\-trained bilinear scalar and a contrastive\-trained MLP matched to TD\-Q’s architecture\. The bilinear cell applies Bellman targets toϕ​\(s,a\)⊤​ψ​\(g\)/d\\phi\(s,a\)^\{\\\!\\top\}\\psi\(g\)/\\sqrt\{d\}in the raw critic’s exact function class; the contrastive MLP applies binary NCE to a joint\-input twin\-MLP logit\. The contrastive objective has no Bellman recursion\.

*Hybrid*adds an auxiliary TD residual during representation learning,

Qhybrid=QTD\+α​fcos\.Q\_\{\\text\{hybrid\}\}\\;=\\;Q\_\{\\text\{TD\}\}\\;\+\\;\\alpha\\,f\_\{\\text\{cos\}\}\.\(S2\)The hybrid usesfcosf\_\{\\text\{cos\}\}for selection\. The value\-calibrated sumQhybridQ\_\{\\text\{hybrid\}\}is not used at deployment\.

Raw exposes the norm\-growth channel, while cosine tests score bounding\. TD\-Q tests a Bellman\-calibrated scalar\. Hybrid asks whether TD\-shaped embeddings improve the bounded contrastive score\. The within\-model control in[Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)applies both readouts to one network, isolating how the deployed scalar changes selection\.

The main sweep does not evaluate clipped scoresf=clip⁡\(ϕ⊤​ψ,−c,c\)f=\\operatorname\{clip\}\(\\phi^\{\\top\}\\psi,\-c,c\)or spectrally normalized encoders\. It also does not evaluate conservative TD penalties on out\-of\-support actions\(Kumar et al\.,[2020](https://arxiv.org/html/2607.27422#biba.bib1); Wang et al\.,[2023](https://arxiv.org/html/2607.27422#biba.bib3)\), ensemble\-based selection \(minj⁡Qj\\min\_\{j\}Q\_\{j\}orμQ−β​σQ\\mu\_\{Q\}\-\\beta\\sigma\_\{Q\}\), or quasimetric or temporal\-distance critics\(Myers et al\.,[2024](https://arxiv.org/html/2607.27422#biba.bib2)\)\.

##### WhereQ⋆Q^\{\\star\}is available\.

The controlled 2D example has a closed\-formQ⋆Q^\{\\star\}\([Section˜S3\.1](https://arxiv.org/html/2607.27422#S3.SS1)\)\. OGBench does not provide a reliable off\-support value target because sparse rewards and the long rollout after each candidate action cause Monte Carlo estimates to fall to zero\. We observed this behavior on both Cube and AntMaze\.

The toy measures candidate regret against closed\-formQ⋆Q^\{\\star\}\([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\. OGBench uses in\-support return\-to\-goγd\\gamma^\{d\}as the value proxy \([Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)\)\. A dataset transition has

Q^MC​\(s,a,g\)=γd​\(s,g\)\.\\hat\{Q\}\_\{\\mathrm\{MC\}\}\(s,a,g\)=\\gamma^\{d\(s,g\)\}\.This is a valid goal\-conditioned target on navigation data, where the agent pursues the future goal\. Manipulation instead uses undirected*play*data\. A future state need not be a goal the agent pursued, so we do not useγd\\gamma^\{d\}for those tasks\.

##### Score ordering triples and Kendallτ\\tau\.

We sample30003000triples\(s,a,g\)\(s,a,g\)from each task’s offline data\. We choose a transition\(s=ot,a=at\)\(s\{=\}o\_\{t\},a\{=\}a\_\{t\}\)at a uniformly sampled time step and a future state from the same trajectory,g=ot\+dg\{=\}o\_\{t\+d\}\. The offset isd∼Unif​\{1,…,min⁡\(60,ℓ\)\}d\\sim\\mathrm\{Unif\}\\\{1,\\dots,\\min\(60,\\ell\)\\\}, whereℓ\\ellis the number of remaining episode steps\. This rule keeps every goal in the same episode\. We skip transitions at the final step\.

Each sampled triple has return\-to\-goγd\\gamma^\{d\}, withγ=0\.99\\gamma=0\.99\([Section˜3](https://arxiv.org/html/2607.27422#S3)\), and we score it with the critic’s deployed readout \([Section˜5\.1](https://arxiv.org/html/2607.27422#S5.SS1)\)\. The cosine and hybrid readouts useL2L\_\{2\}\-normalized embeddings to match training\.

We report Kendallτ\\taubetween the deployed score andγd\\gamma^\{d\}\([Table˜2](https://arxiv.org/html/2607.27422#S6.T2)\)\. A rank statistic is appropriate because candidate maximization depends only on score order\. Monotone rescaling leaves Kendallτ\\tauunchanged, allowing comparison of raw, cosine, and TD\-Q despite their different score scales\. We useτb\\tau\_\{b\}to account for repeatedγd\\gamma^\{d\}values\.

We compute this correlation*globally*over all sampled triples, measuring whether higher scores tend to correspond to closer goals\. The controlled toy gives a separate action\-conditional analysis at fixed\(s,g\)\(s,g\)with availableQ⋆Q^\{\\star\}\([Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)\)\.

## S3Controlled environment robustness and controls

### S3\.1Environment and qualitative score geometry

The controlled environment is a two\-dimensional goal\-reaching problem with closed\-form dataset support, off\-support value, and a candidate pool oracle\. We give the full setup here and preview the learned geometry of the four score families from[Section˜3](https://arxiv.org/html/2607.27422#S3)\.[Section˜7\.1](https://arxiv.org/html/2607.27422#S7.SS1)reports the numerical selection results\.

#### S3\.1\.1Environment, support, and closed\-form value

States, actions, and goals are two\-dimensional,s,a,g∈ℝ2s,a,g\\in\\mathbb\{R\}^\{2\}, with actions restricted to\[−1,1\]2\[\-1,1\]^\{2\}\. The deterministic transition iss′=s\+as^\{\\prime\}=s\+a, and the reward is−‖s′−g‖\-\\\|s^\{\\prime\}\-g\\\|\. The exact one\-step value is

Q⋆​\(s,a,g\)=−‖s\+a−g‖\.Q^\{\\star\}\(s,a,g\)\\;=\\;\-\\,\\\|s\+a\-g\\\|\.\(S3\)The optimal action isa⋆=g−sa^\{\\star\}=g\-s\. It lands on the goal and givesQ⋆=0Q^\{\\star\}=0\.

Isotropic Gaussian noise does not change any ordering\-based quantity whens′=s\+a\+ϵs^\{\\prime\}=s\+a\+\\epsilonandϵ∼𝒩​\(0,σ2​I\)\\epsilon\\sim\\mathcal\{N\}\(0,\\sigma^\{2\}I\)\. The true expected reward is𝔼ϵ​\[−‖s\+a\+ϵ−g‖\]=−m​\(‖s\+a−g‖\)\\mathbb\{E\}\_\{\\epsilon\}\[\-\\\|s\+a\+\\epsilon\-g\\\|\]=\-m\(\\\|s\+a\-g\\\|\)withmmthe Rice distribution mean, which is*strictly increasing*in‖s\+a−g‖\\\|s\+a\-g\\\|\. Thus,[Equation˜S3](https://arxiv.org/html/2607.27422#S3.E3a)is an exact monotone transform of the noisy value: selected actions, off\-support rates, and all best\-of\-KKcurves remain identical\. Regret magnitudes shift by at most𝔼​‖ϵ‖=σ​π/2≈0\.063\\mathbb\{E\}\\\|\\epsilon\\\|=\\sigma\\sqrt\{\\pi/2\}\\approx 0\.063atσ=0\.05\\sigma\{=\}0\.05, which is small relative to the≈1\.0\{\\approx\}1\.0contrastive regret\. We report the deterministic environment so that the target is exactlyQ⋆Q^\{\\star\}\.

The offline dataset limits actions to a*support disk*of radiusRRcentered at the origin in action space,Ω=\{a:‖a‖≤R\}\\Omega=\\\{a:\\\|a\\\|\\leq R\\\}\. A goal\-aligned sampling bias overrepresents actions that reach the goal, while states and goals are uniform over\[−2,2\]2\[\-2,2\]^\{2\}\. We call an action*off support*when‖a‖\>R\\\|a\\\|\>R\. We fixs=0s=0and draw a goalg=δg=\\deltafor each decision query, with‖δ‖≤0\.6​R\\\|\\delta\\\|\\leq 0\.6\\,R\. The optimal actiona⋆=δa^\{\\star\}=\\deltatherefore lies strictly*inside*the support disk, ruling out an unreachable on\-support optimum as the source of failure\.

#### S3\.1\.2Candidate proposal, selection, and regret

We drawKKcandidate actions i\.i\.d\. for each query from an isotropic Gaussian proposal with scale1\.2​R1\.2\\,R\. We clip the samples to the action bounds, which produces pools with actions inside and outside the support disk\. These draws match the hypothesis of[Proposition˜2](https://arxiv.org/html/2607.27422#Thmproposition2)\. The deployed critic selectsaK⋆=arg⁡maxi≤K⁡f​\(s,ai,g\)a\_\{K\}^\{\\star\}=\\arg\\max\_\{i\\leq K\}f\(s,a\_\{i\},g\)as in \([2](https://arxiv.org/html/2607.27422#S3.E2)\), while the candidate pool oracleaQ⋆a\_\{Q\}^\{\\star\}maximizes the closed\-form valueQ⋆Q^\{\\star\}from \([S3](https://arxiv.org/html/2607.27422#S3.E3a)\) over the same draws\. Their value gap is the candidate regretRegretK=Q⋆​\(s,aQ⋆,g\)−Q⋆​\(s,aK⋆,g\)\\mathrm\{Regret\}\_\{K\}=Q^\{\\star\}\(s,a\_\{Q\}^\{\\star\},g\)\-Q^\{\\star\}\(s,a\_\{K\}^\{\\star\},g\)from \([3](https://arxiv.org/html/2607.27422#S3.E3)\)\. The exactQ⋆Q^\{\\star\}also lets us label each selection as on\- or off\-support and decompose regret accordingly\.

#### S3\.1\.3Qualitative preview of learned score geometry

[Figure˜4](https://arxiv.org/html/2607.27422#S7.F4)shows an example failure\. The candidate pool oracle favors high\-value actions inside the support disk, while the raw, cosine, and hybrid critics place their maxima on lower\-value actions outside it\. TD\-Q selects an action inside the disk\. The bounded cosine and hybrid scores therefore do not recover value ordering on their own\.[Sections˜4](https://arxiv.org/html/2607.27422#S4)and[7\.1](https://arxiv.org/html/2607.27422#S7.SS1)show and measure this result\.

### S3\.2Support\-radius robustness

The main toy experiment uses a support radius ofR=0\.40R=0\.40\. We repeat it atR∈\{0\.25,0\.40,0\.55\}R\\in\\\{0\.25,0\.40,0\.55\\\}with the same candidate budget,K=256K=256\([Table˜S1](https://arxiv.org/html/2607.27422#S3.T1a)\)\. The ordering is unchanged across radii\. Every contrastive critic selects off\-support actions on at least80%80\\%of queries and has far more regret than TD\-Q, which remains within the support and has near\-zero regret\. TheR=0\.40R=0\.40results reproduce[Table˜3](https://arxiv.org/html/2607.27422#S7.T3)\.

The candidate proposal scales withRR, so each pool covers actions inside and outside the support at each radius\. The sweep tests changes in scale and boundary location; it does not hold the candidate pool fixed while moving the boundary\.

Table S1:Support\-radius robustness\.Regret \(mean±\\pmstd over 3 seeds\) and off\-support rate at three radii withK=256K=256\. The critic ordering is preserved at every radius\. Results atR=0\.40R=0\.40match[Table˜3](https://arxiv.org/html/2607.27422#S7.T3)\.![Refer to caption](https://arxiv.org/html/2607.27422v1/x5.png)Figure S1:Optimizer’s curse under best ofKKin the controlled toy, whereQ⋆​\(s,a,g\)=−‖s\+a−g‖Q^\{\\star\}\(s,a,g\)=\-\\\|s\+a\-g\\\|\. Every contrastive critic’s selected score rises withKK\(left\) while the selected action’s true value falls \(right\)\. TD\-Q remains near the candidate pool oracle\. Scores use each critic’s native scale, so only within\-critic changes are meaningful\.![Refer to caption](https://arxiv.org/html/2607.27422v1/x6.png)Figure S2:Raw norm\-drift channelin the controlled toy, whereQ⋆​\(s,a,g\)=−‖s\+a−g‖Q^\{\\star\}\(s,a,g\)=\-\\\|s\+a\-g\\\|\. The panels show true return and selected embedding norm‖ϕ​\(s,aK⋆\)‖\\\|\\phi\(s,a\_\{K\}^\{\\star\}\)\\\|versusKK\. The raw bilinear norm stays high as its return*worsens*; norms fall for the bounded and calibrated critics\. The raw pattern matches the norm\-extrapolation mechanism\.##### Within\-model readout control \(controlled toy\)\.

We compare several readouts from one network\. The network contains contrastive embeddings\[ϕ^,ψ^\]\[\\hat\{\\phi\},\\hat\{\\psi\}\], a bounded cosine scorefcosf\_\{\\text\{cos\}\}, and a separate TD headqtdq\_\{\\text\{td\}\}\. Gradients from the TD head also update the contrastive encoder\. We apply three selection scores from this network to the same toy candidate pools \([Table˜S2](https://arxiv.org/html/2607.27422#S3.T2)\)\.

The cosine readout selects off\-support actions on80%80\\%of runs and has regret0\.90±0\.430\.90\{\\pm\}0\.43\. The calibrated TD head selects entirely within the support and reduces regret to0\.07±0\.020\.07\{\\pm\}0\.02\. The full sumQhybrid=qtd\+α​fcosQ\_\{\\text\{hybrid\}\}=q\_\{\\text\{td\}\}\+\\alpha f\_\{\\text\{cos\}\}from \([S2](https://arxiv.org/html/2607.27422#S2.E2)\) gives nearly identical results\. These readouts use the same learned embeddings, so their different outcomes show that the failure comes from the deployed scalar\.

Table S2:Within\-model readout controlin the controlled toy atR=0\.40R\{=\}0\.40\. Three selection scalars from one two\-head network are applied to the same candidate pools\. Values are mean±\\pmstd over 5 seeds\. The bounded cosine readout fails, while the calibratedqtdq\_\{\\text\{td\}\}andQhybridQ\_\{\\text\{hybrid\}\}readouts select near\-optimally\.

## S4OGBench robustness

### S4\.1Cross\-query score comparability under pooled triples

![Refer to caption](https://arxiv.org/html/2607.27422v1/x7.png)Figure S3:OGBench best\-of\-KKselection on four navigation tasks: value curse and local support drift\.Top:Selectedγd\\gamma^\{d\}versusKKover random in\-support pools, matching[Figure˜S1](https://arxiv.org/html/2607.27422#S3.F1)\. TD\-Q tracks the sampled pool oracle \(dashed\); every contrastive critic first rises, then falls toward or below random \(dotted\)\.Bottom:Excess rate of selecting candidates outside local support relative to the pool base rate\. Zero means no preference, not equal value\. Contrastive critics rise above zero withKK, while TD\-Q stays at or below zero\. Curves show mean±\\pmstd over 3 training seeds \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.This experiment measures*cross\-query score comparability*, not within\-query action selection\. The pools mix triples with different\(s,g\)\(s,g\), so a selector can fail by comparing scores poorly across queries even if it orders actions correctly within each query\. A centering analysis confirms that the effect is almost entirely cross\-query\. Per\-query mean\-centering lowers the pooled globalτ\\tauto≈0\.01\{\\approx\}0\.01for every critic, while the within\-query ordering in[Table˜4](https://arxiv.org/html/2607.27422#S7.T4)is unchanged\. Cross\-context comparability matters when one scalar is compared across states, as in ranking\-based filtering or dataset curation\. We reserve all within\-query selection claims for[Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)\.

Sparse rewards wash out the signal needed to estimate per\-actionQ⋆Q^\{\\star\}on OGBench \([Section˜3](https://arxiv.org/html/2607.27422#S3)\)\. The in\-support return\-to\-goγd\\gamma^\{d\}is available, and TD\-Q ranks it well \([Table˜2](https://arxiv.org/html/2607.27422#S6.T2)\), so we use it as the value signal\. We score30003000in\-distribution triples from[Section˜6\.1](https://arxiv.org/html/2607.27422#S6.SS1)for each critic and sample random pools of sizeKK\. We record theγd\\gamma^\{d\}of the highest\-scoring triple and compare it with the sampled pool oracle \([Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)\)\.[Table˜S4](https://arxiv.org/html/2607.27422#S4.T4)gives exact regret atK=64K\{=\}64\. This experiment is the OGBench version of[Figure˜S1](https://arxiv.org/html/2607.27422#S3.F1)\.

The result matches the toy across all four navigation tasks\. TD\-Q’s selected value approaches the pool oracle asKKgrows because its score orders value\. TD\-Q regret atK=64K\{=\}64is at most0\.030\.03on the three core tasks and0\.060\.06on AntSoccer\. Every contrastive critic instead*peaks early and falls toward or below random*\. Candidate maximization targets the score tail, which[Tables˜2](https://arxiv.org/html/2607.27422#S6.T2)and[2](https://arxiv.org/html/2607.27422#S6.F2)show is flat or inverted in value\. LargerKKtherefore worsens selection, producing regret of0\.180\.18–0\.260\.26atK=64K\{=\}64, an order of magnitude above TD\-Q\.

TD\-Q faces the same increase in search pressure, yet its performance does not worsen withKK\. The sweep separates search pressure from miscalibration: selection worsens only when the score tail lacks calibration\. This sweep and[Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)together reveal both sides of the good\-ranker/bad\-objective gap\. The contrastive scalar retrieves well, but its scores are not comparable across contexts under maximization\. Its*realized*within\-query selection cost remains the task\-dependent cost measured in[Table˜4](https://arxiv.org/html/2607.27422#S7.T4)\. The curse remains when an independent, stitching\-aware value replacesγd\\gamma^\{d\}, so the proxy does not cause the result \([Section˜S4\.4](https://arxiv.org/html/2607.27422#S4.SS4)\)\.

### S4\.2Local behavior support preference under action search

The previous experiment measures value within the support\. We now test whether fixed\-state action search pulls selection away from the*local behavior support*\. Off\-supportQ⋆Q^\{\\star\}is unavailable, and simple distance metrics are hard to interpret\. A noise proposal makes the candidate pool spread withKK, which makes every critic appear to drift\. Thekk\-NN distances also concentrate and lose information in high dimensions, as on HumanoidMaze\.

We use a labeled design that remains useful across dimensions\. Each in\-support\(s,g\)\(s,g\)uses*real*dataset actions with valid magnitudes as candidates, which prevents the critic from exploiting norm alone\. Half come from states nearssand are locally on support; the other half come from random states and are off support\. We report the*excess*rate at which best\-of\-KKselects an off\-support candidate relative to the pool base rate\. A critic with no off\-support preference remains at zero for everyKK, while a positive rate that grows withKKshows such a preference\.

[Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)shows this separation across all four navigation tasks\. The excess rate rises above zero withKKfor every contrastive critic\. TD\-Q remains at or below zero, indicating a mild preference for on\-support actions\.

Drift measures*support preference*, not value\. Local support and reference value are nearly uncorrelated in the fixed\-query pools \(ρ≈0\.03\\rho\\approx 0\.03–0\.050\.05\), so selecting a low\-support action does not by itself imply low value\.[Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)measures the value consequences directly\. The amount of drift is also task\-dependent\. Contrastive critics promote off\-local\-support candidates at∼2\.2×\{\\sim\}2\.2\\timesthe pool base rate on AntMaze; they select such candidates near the base rate on PointMaze\.

The two OGBench panels measure different but related effects\. The top panel extends[Figure˜2](https://arxiv.org/html/2607.27422#S6.F2)by measuring in\-support score\-tail value underγd\\gamma^\{d\}\. The bottom panel measures local\-support preference at a fixed\(s,g\)\(s,g\)\. TD\-Q’s safety in this setting comes from coarse*ordering*rather than fine action resolution \([Section˜S6\.2](https://arxiv.org/html/2607.27422#S6.SS2a)\)\.

### S4\.3Candidate\-pool construction robustness

We test whether the OGBench results depend on how candidate pools are constructed\. Both checks evaluate only the seed\-0 checkpoints \(run\_ogbench\_robustness\.py\)\.[Table˜S3](https://arxiv.org/html/2607.27422#S4.T3)reportsK=64K\{=\}64results for all four navigation tasks\.

*Nearby state radius\.*The local support drift \([Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)\) defines on\-support candidates using thennearn\_\{\\text\{near\}\}dataset states closest toss\. We sweepnnear∈\{50,200,800\}n\_\{\\text\{near\}\}\\in\\\{50,200,800\\\}\. Every contrastive critic has a positive excess off\-support rate throughout the sweep, although the rate decreases slightly as the neighborhood grows\. TD\-Q shows no off\-support preference on the maze or AntSoccer tasks\. Its HumanoidMaze excess is small and positive, with\|excess\|≲0\.09\|\\text\{excess\}\|\\lesssim 0\.09\. The result is stable across nearby\-state thresholds\.

*Pools within distance buckets\.*We bin the in\-support triples byγd\\gamma^\{d\}quartile to rule out variation in goal distance\. We then run best\-of\-KKselection within each quartile, which restricts every pool to a narrow range of goal distances\. Regret falls for all critics because little value variation remains within a bucket\. One\-step values are also coarse over such a narrow range \([Section˜S6\.2](https://arxiv.org/html/2607.27422#S6.SS2a)\)\. Even so, contrastive critics have1\.51\.5–2\.5×2\.5\\timesas much regret as TD\-Q\. Goal\-distance control weakens the curse without removing it\.

Table S3:Candidate\-proposal robustnessatK=64K\{=\}64\. We report*excess off\-support rate*fornnear=50/200/800n\_\{\\text\{near\}\}=50/200/800\([Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)\) and*regret*for mixed\-distance pools and pools restricted to oneγd\\gamma^\{d\}quartile \([Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)\)\. Cells are means over 3 training seeds\. Per\-seed std is at most0\.040\.04, except for AntMaze hybrid excess, where it is at most0\.120\.12\(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.##### Exact best\-of\-KKpoint values\.

[Table˜S4](https://arxiv.org/html/2607.27422#S4.T4)gives the exact in\-supportγd\\gamma^\{d\}regret atK=64K\{=\}64behind the pooled\-triple curves in[Figure˜S3](https://arxiv.org/html/2607.27422#S4.F3)\([Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)\)\.

Table S4:In\-supportγd\\gamma^\{d\}regret under OGBench best\-of\-KKselection\.Regret is the pool\-oracleγd\\gamma^\{d\}minus selectedγd\\gamma^\{d\}atK=64K\{=\}64; lower is better\. TD\-Q stays near the oracle, while contrastive critics have roughly an order of magnitude more regret\. Random selection is about0\.750\.75, and the oracle is about0\.990\.99\. Values are mean±\\pmstd over training seeds \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.

### S4\.4Independent value oracle \(cross\-query diagnostic\)

*Scope\.*Both tables use the pooled\-triple,*cross\-query*construction from[Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)\. The pools mix triples with different\(s,g\)\(s,g\)and score them against a single\-seed GCIQL DP reference\. They therefore measure score comparability across contexts under an independent value, not within\-query selection\.

The distinction matters on AntMaze\. There, the DP reference has*no within\-query validity*: its paired correlation with realized rollout return isρ=0\.07\\rho=0\.07, compared withρ=0\.38\\rho=0\.38across queries\. The simulator audit in[Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)likewise finds that single\-step selection does not affect value on this task\. The matching within\-query test is the reference\-ensemble axis of[Table˜4](https://arxiv.org/html/2607.27422#S7.T4), which uses a DP oracle to judge candidates at a fixed query\. That test is reliable on PointMaze, where the three\-seed ensemble has Spearman0\.880\.88and agrees with rollout ground truth\.

The in\-support return\-to\-goγd\\gamma^\{d\}is a progress proxy rather thanQ⋆Q^\{\\star\}and can fall below the optimum when a task requires stitching\. We repeat the analysis with an independent value reference\. We train a seed\-77GCIQL reference for each maze task separately from the deployed critics\. This Bellman/DP value accounts for stitching\. We measure best\-of\-KKregret against bothγd\\gamma^\{d\}and this DP value \([Table˜S5](https://arxiv.org/html/2607.27422#S4.T5)\)\.

The cross\-query effect remains under the DP value\. The correlationρ=Spearman​\(γd,DP\)\\rho=\\mathrm\{Spearman\}\(\\gamma^\{d\},\\text\{DP\}\)stays well below11on PointMaze and AntMaze, revealing a stitching gap\. Normalized contrastive regret roughly doubles on both tasks\. The values mostly agree on HumanoidMaze \(ρ=0\.90\\rho\{=\}0\.90\), and contrastive regret remains near random\. The comparability failure therefore does not depend on theγd\\gamma^\{d\}proxy\.

The TD\-Q row is partly circular because the deployedtd\_qcritic and the reference both use GCIQL\. Its low regret is therefore not independent evidence; the contrastive rows provide the relevant comparison\. We evaluate deployed critics from three training seeds \(\{42,43,44\}\), and every seed shows much higher contrastive regret than TD\-Q\.

Table S5:Cross\-query comparability against an independent OGBench value oracle\.We report pooled\-triple normalized regret atK=64K\{=\}64under the in\-supportγd\\gamma^\{d\}proxy and an independent, stitching\-aware GCIQL value\. This experiment measures score comparability across contexts, not within\-query selection, as[Section˜S4\.4](https://arxiv.org/html/2607.27422#S4.SS4)explains\. A value of1\.01\.0equals random selection; larger values are worse than random\. Contrastive regret remains under the independent value\.ρ=Spearman​\(γd,DP\)\\rho=\\mathrm\{Spearman\}\(\\gamma^\{d\},\\text\{DP\}\)measures each task’s stitching gap\. Theγd\\gamma^\{d\}column rescales raw regret from[Table˜S4](https://arxiv.org/html/2607.27422#S4.T4)by oracle minus random\.†TD\-Q belongs to the same GCIQL family as the reference, making its low regret partly circular\.The previous experiment draws candidates from in\-support dataset actions\. We also train a DDPM behavior model on AntMaze to evaluate candidates from an expressive policy\. Its train/holdout NLL gap is about zero\. We then replace dataset actions in the candidate pools with DDPM samples\. This comparison uses only the DP value becauseγd\\gamma^\{d\}is undefined outside the dataset\.

The pattern is unchanged \([Table˜S6](https://arxiv.org/html/2607.27422#S4.T6)\)\. Contrastive best\-of\-KKregret under diffusion samples closely matches the in\-support result\. The comparison is stable across three independently trained diffusion policies, with cross\-seed standard deviation at most0\.0050\.005\. Cross\-query comparability therefore also fails under a trained expressive sampler\. This experiment has the same scope as[Table˜S5](https://arxiv.org/html/2607.27422#S4.T5): it uses a pooled\-triple, cross\-query construction on AntMaze\. The single\-seed DP reference has no within\-query validity \(ρwithin=0\.07\\rho\_\{\\text\{within\}\}=0\.07\), and realized single\-step selection does not affect value \([Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)\)\. The experiment measures score comparability, not within\-query selection harm\.

Table S6:Cross\-query score comparability under a trained expressive sampler on AntMaze\.We report normalized best\-of\-KKregret atK=64K\{=\}64against the independent GCIQL value\. Candidate pools use either in\-support dataset actions or samples from a trained diffusion policy\. The diffusion column is mean±\\pmstd over three policies \(seeds42/7/1342/7/13,n=3000n\{=\}3000\), with cross\-seed std at most0\.0050\.005\.†TD\-Q belongs to the same GCIQL family as the reference, making its low regret partly circular\.

## S5Additional diagnostics

### S5\.1Score semantics tables

[Tables˜S7](https://arxiv.org/html/2607.27422#S5.T7)and[S8](https://arxiv.org/html/2607.27422#S5.T8)report the full per\-task values behind the score\-semantics diagnostics in[Sections˜6\.1](https://arxiv.org/html/2607.27422#S6.SS1)and[6\.3](https://arxiv.org/html/2607.27422#S6.SS3)\.

Table S7:Top\-tail calibration measured by the top\-minus\-bottom score decile gapΔ​γd\\Delta\\gamma^\{d\}\.The gap is strongly positive for TD\-Q, weak for raw bilinear, and close to zero or negative for cosine and hybrid\. Navigation cells are seed means with per\-seed std at most0\.020\.02\(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.[Figure˜2](https://arxiv.org/html/2607.27422#S6.F2)shows the complete decile curves\.Table S8:Training/readout decomposition\.Kendallτ\\tauagainstγd\\gamma^\{d\}\. Raw\-trained embeddings keep weak ordering with a cosine readout, while their norm‖ϕ‖\\\|\\phi\\\|carries almost no ordering signal\. Cosine\-trained embeddings lose ordering with both raw and cosine readouts\. Values are mean±\\pmstd over training seeds \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.
### S5\.2Pathwise Bellman consistency of the deployed score

A value function must satisfy Bellman recursion, whereas a compatibility score need not\. We measure*pathwise*Bellman consistency along actions in the dataset\. This test differs from applying the optimal Bellman operatormaxa′⁡Q​\(s′,a′,g\)\\max\_\{a^\{\\prime\}\}Q\(s^\{\\prime\},a^\{\\prime\},g\)\.

Consider an in\-trajectory triple\(st,at,g=st\+d\)\(s\_\{t\},a\_\{t\},g\{=\}s\_\{t\+d\}\)withd≥2d\\geq 2\. The goal has not been reached, so the sparse reward is zero and the episode continues\. The deployed score has one\-step residual

δ=f​\(st,at,g\)−γ​f​\(st\+1,at\+1,g\)\.\\delta=f\(s\_\{t\},a\_\{t\},g\)\-\\gamma f\(s\_\{t\+1\},a\_\{t\+1\},g\)\.Along this path, the return\-to\-go values areQ^MC​\(st,at,g\)=γd\\hat\{Q\}\_\{\\mathrm\{MC\}\}\(s\_\{t\},a\_\{t\},g\)=\\gamma^\{d\}andQ^MC​\(st\+1,at\+1,g\)=γd−1\\hat\{Q\}\_\{\\mathrm\{MC\}\}\(s\_\{t\+1\},a\_\{t\+1\},g\)=\\gamma^\{d\-1\}\. Therefore,

Q^MC​\(st,at,g\)−γ​Q^MC​\(st\+1,at\+1,g\)=γd−γ​γd−1=0\.\\hat\{Q\}\_\{\\mathrm\{MC\}\}\(s\_\{t\},a\_\{t\},g\)\-\\gamma\\,\\hat\{Q\}\_\{\\mathrm\{MC\}\}\(s\_\{t\+1\},a\_\{t\+1\},g\)\\;=\\;\\gamma^\{d\}\-\\gamma\\,\\gamma^\{d\-1\}\\;=\\;0\.Thus, pathwise return\-to\-go has zero residual\.[Table˜S9](https://arxiv.org/html/2607.27422#S5.T9)reports the scale\-free errorRMSE​\(δ\)/std​\(f\)\\mathrm\{RMSE\}\(\\delta\)/\\mathrm\{std\}\(f\)\.

TD\-Q has normalized error near0\.050\.05\. The contrastive scores are44–8×8\\timeslarger, ranging from0\.20\.2to0\.380\.38\. Hybrid lies between them because TD shapes its embeddings, although selection still uses the cosine readout\. Raw residuals are not directly comparable across critics because their score scales differ\. TD\-Q’s low error is expected from its Bellman training, so the contrastive errors are the relevant comparison\.

We omit MSE againstQ⋆Q^\{\\star\}, ECE, and Brier score because OGBench lacks off\-supportQ⋆Q^\{\\star\}\([Section˜3](https://arxiv.org/html/2607.27422#S3)\) and contrastive scores have no natural value scale\. Computing these metrics would require fitting a monotone map from score to value or success\. The resulting values would mostly reflect the ordering already measured in[Tables˜2](https://arxiv.org/html/2607.27422#S6.T2)and[S7](https://arxiv.org/html/2607.27422#S5.T7)\.

Table S9:Normalized Bellman errorRMSE​\(δ\)/std​\(f\)\\mathrm\{RMSE\}\(\\delta\)/\\mathrm\{std\}\(f\)for the deployed score; lower values mean greater value consistency\. TD\-Q is nearly consistent, whereas the contrastive scores are not\. Values are mean±\\pmstd over training seeds \(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\.
### S5\.3Retrieval \(compatibility ranking\) metrics

[Table˜S10](https://arxiv.org/html/2607.27422#S5.T10)reports per\-task retrieval AUC and top\-kkrecall across eight tasks: four navigation and four manipulation tasks\. The navigation rows supply the data for[Figure˜3](https://arxiv.org/html/2607.27422#S6.F3)\. Positive goalsg\+g^\{\+\}are future states on the same trajectory, and negative goals come from other trajectories\. We evaluate each deployed score on30003000in\-distribution triples, using pools of5050goals for recall\.

*Hard\-negative control\.*Random\-goal negatives may make retrieval look too easy\. The Hard\-neg\. AUC column therefore uses a stronger construction\. Each anchor uses as its negative the goal from a sampled triple on a*different*trajectory whose state is closest to the anchor state\. This produces a realistic same\-region goal rather than a global shuffle; we score it as a paired win rate\. Absolute values are lower in densely sampled regions where states overlap, especially on PointMaze\. PointMaze’s 2\-D observations make the nearest other\-trajectory goal almost a valid alternative target\. Even with these harder negatives, the critic ordering is unchanged on every task\. Contrastive readouts remain the strongest retrievers, with manipulation AUC at0\.940\.94–1\.01\.0, and TD\-Q remains the weakest\.

A second construction uses same\-trajectory goals at different temporal distances\. Every critic performs at chance \(≈0\.5\\approx 0\.5\) because both candidates are valid futures of the anchor\. This construction measures temporal precision rather than compatibility and does not separate the critics\.

Table S10:Retrieval \(compatibility\-ranking\) metrics across all eight tasks\.The table reports retrieval AUC and recall@\{1,5\}\\\{1,5\\\}for each deployed score\. Hard\-neg\. AUC replaces goal\-shuffled negatives with goals from nearest\-state anchors on other trajectories and reports the paired win rate \([Section˜S5\.3](https://arxiv.org/html/2607.27422#S5.SS3)\)\. Critic ordering is unchanged under this harder construction\. Darker cells mean larger values; AUC shading spans0\.50\.5–1\.01\.0, and recall shading spans0–11\. Contrastive critics retrieve well despite weakγd\\gamma^\{d\}ordering \([Table˜2](https://arxiv.org/html/2607.27422#S6.T2)\), while TD\-Q generally retrieves less well\. Raw and cosine retrieve nearly perfectly in both domains\. TD\-Q is much weaker on Cube and Scene \(AUC0\.530\.53–0\.560\.56\) but performs well on Puzzle \(AUC0\.890\.89\)\. Hybrid has the lowest Puzzle AUC at0\.790\.79\. Results use30003000in\-distribution triples and recall pools of5050goals\. Cells are seed means\. Per\-seed std is at most0\.060\.06, except for AntSoccer TD\-Q recall, where it is at most0\.270\.27\(Footnote[1](https://arxiv.org/html/2607.27422#footnote1)\)\. We reportγd\\gamma^\{d\}value ordering only for navigation, where the proxy is valid \([Table˜2](https://arxiv.org/html/2607.27422#S6.T2)\)\.

## S6Deployment scope checks

### S6\.1Behavior regularized actor

We evaluate each critic with a*behavior\-regularized actor*\(DDPG\+\+BC\)\. The actor follows the critic locally, while a behavior\-cloning term keeps its actions near the data support\. It does not search over a large candidate pool\.

The actor reaches∼\\sim20–32% AntMaze success with each of the four critics\. The local optimization avoids candidate\-selection failure because it remains near the support\. This single\-task experiment does not compare actor performance; it identifies the deployment setting in which score errors become harmful\.

### S6\.2Limits of per\-step closed\-loop best\-of\-KK

One possible protocol runs best\-of\-KKselection at every environment step\. The method drawsKKactions from a behavior\-plus\-noise proposal, selects the critic’s maximum, and executes it\. We tested this protocol using clipped isotropic Gaussian noise withσ=0\.3\\sigma\{=\}0\.3\.

This protocol is difficult to interpret because one\-step values are nearly flat for distant goals\. TD\-Q’s score spread across the action cube at a fixed state is only about0\.8%0\.8\\%of its magnitude, versus77–23%23\\%for the bilinear critics\. A one\-step change in a distant goal’s distance changes its value by only about1%1\\%whenγ=0\.99\\gamma\{=\}0\.99\.

Critic noise can dominate such small value differences\. Even TD\-Q therefore worsens asKKgrows\. Its PointMaze success falls from36%36\\%atK=1K\{=\}1to0%0\\%atK=64K\{=\}64\. We do not treat this failure as evidence of decalibration because it follows from a flat one\-step objective\. The controlled toy has a strongQ⋆Q^\{\\star\}action gradient, and the in\-support triples in[Section˜S4\.1](https://arxiv.org/html/2607.27422#S4.SS1a)have meaningfulγd\\gamma^\{d\}variation\. Both evaluations avoid this problem\.

The fixed\-query rollout audits in[Section˜7\.2](https://arxiv.org/html/2607.27422#S7.SS2)directly measure this flatness\. No single\-step selector, including the reference\-ensemble oracle, beats a random candidate on AntMaze or HumanoidMaze; both are well\-powered nulls\. A point\-mass action moves the PointMaze agent directly, so single\-step selection matters and the ordering gap affects realized return\. The realized selection harm therefore varies with the measured flatness of the per\-step closed\-loop objective\.

## S7Two\-head positive control

This control tests whether one checkpoint can support retrieval and value\-calibrated selection\. We train a network with two heads: a bounded cosine headfcosf\_\{\\cos\}for retrieval and a separate Bellman/IQL headqTDq\_\{\\text\{TD\}\}for candidate selection\.

We compare two ways of connecting the encoders\. The*separate*variant givesqTDq\_\{\\text\{TD\}\}its own encoder\. Both heads in the*joint*variant share the contrastive backbone, which also receives the value gradient\.

[Table˜S11](https://arxiv.org/html/2607.27422#S7.T11)evaluates both heads from the same checkpoint\. The separate variant matches TD\-Q’sγd\\gamma^\{d\}ordering on all three tasks, while its cosine head keeps retrieval AUC near0\.990\.99\. A single network can therefore provide both calibrated selection and near\-perfect retrieval\.

The joint variant works on PointMaze and AntMaze, but its value ordering is weak on HumanoidMaze\. Backbone sharing therefore works less well in higher dimensions\. A separate TD head avoids this problem and restores value ordering without reducing the cosine retrieval score\.

The toy control in[Table˜S2](https://arxiv.org/html/2607.27422#S3.T2)shows the same pattern\. BothqTDq\_\{\\text\{TD\}\}selectors achieve regret close to TD\-Q and far below the contrastive readouts\. These controls show that the selection failure lies in the deployed score\.

Table S11:Two\-head critic with calibration and retrieval in one network\.We reportγd\\gamma^\{d\}Kendallτ\\taufor the deployedqTDq\_\{\\text\{TD\}\}head and retrieval AUC for the cosine head\. The*separate*variant matches TD\-Q ordering on all three tasks while keeping cosine\-level retrieval\. The joint variant has weak HumanoidMaze ordering\.
## References

- Kumar et al\. \(2020\)Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine\.Conservative Q\-learning for offline reinforcement learning\.In*Advances in Neural Information Processing Systems*, 2020\.URL[https://arxiv\.org/abs/2006\.04779](https://arxiv.org/abs/2006.04779)\.
- Myers et al\. \(2024\)Vivek Myers, Chongyi Zheng, Anca Dragan, Sergey Levine, and Benjamin Eysenbach\.Learning temporal distances: Contrastive successor features can provide a metric structure for decision\-making\.In*Proceedings of the International Conference on Machine Learning*, 2024\.URL[https://arxiv\.org/abs/2406\.17098](https://arxiv.org/abs/2406.17098)\.
- Wang et al\. \(2023\)Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou\.Diffusion policies as an expressive policy class for offline reinforcement learning\.In*International Conference on Learning Representations*, 2023\.URL[https://arxiv\.org/abs/2208\.06193](https://arxiv.org/abs/2208.06193)\.