Dual Advantage Fields

# Dual Advantage Fields
Source: [https://arxiv.org/html/2606.04188](https://arxiv.org/html/2606.04188)
\\acceptedworkshopnotice

Accepted at ICML 2026 Workshop on Decision Making from Offline Datasets to Online Adaptation, Black\-Box Optimization to Reinforcement Learning\.

Alexey Zemtsov1,2, Maxim Bobrin3, Alexander Nikulin2,5, Dmitry V\. Dylov3, Fakhri Karray4, Vladislav Kurenkov5,6, Martin Takáč4, Arip Asadulaev4 1NUST MISIS2MSU3Computational Imaging Lab 4MBZUAI5dunnolab6Innopolis University

###### Abstract

Offline goal\-conditioned reinforcement learning requires both long\-horizon reachability estimates and local action comparisons\. Dual goal representations provide value fields that capture global goal reachability, but they do not directly specify which action should be preferred at a given state\. We propose Dual Advantage Fields, a policy\-extraction method that turns a bilinear dual value model into a local advantage signal\. Under bilinear dual parameterization, the goal embedding is the gradient of the value field with respect to the state representation\. DAF learns an action\-effect model that predicts the discounted feature displacement induced by an action and scores actions by the alignment between this displacement and the goal direction\. In the realizable case, this score equals the goal\-conditioned Bellman advantage, yielding a standard local policy\-improvement guarantee\. On OGBench locomotion, manipulation, and puzzle tasks, DAF improves aggregate RLiable metrics and performs strongly in settings where locally correct actions differ from direct movement toward the final goal\.

![Refer to caption](https://arxiv.org/html/2606.04188v1/Images/Fields_2.png)Figure 1:A dual goal\-conditioned value model defines a global value surface over state representations\. DAF converts this global surface into a local action\-comparative signal by predicting how each action moves the state representation and measuring whether this movement aligns with the goal direction\.## 1Introduction

Goal\-conditioned reinforcement learning \(GCRL\) aims to learn policies that reach arbitrary goals from a fixed dataset of prior experience\. This requires solving two different problems at once\. First, the agent must reason globally: it must infer how states are connected over long horizons so that behavior observed in one part of the dataset can be stitched together with behavior observed elsewhere\. Second, the agent must act locally: at the current state, it must decide which available action makes the most progress toward the requested goal\. A good goal\-conditioned agent therefore needs both a global map of reachability and a local compass for action selection\.

Recent dual goal representations provide a strong answer to the first problem\. They parameterize a goal\-conditioned value function as a bilinear interaction between a state embedding and a goal embeddingVθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\)V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\)\. This structure induces a value surface for each goal, where states that are more reachable or desirable for the goal receive higher values\. Such value surfaces are well suited for long\-horizon reasoning: they encode temporal structure, support stitching across offline trajectories, and generalize across state\-goal pairs\. However, a value surface alone does not directly answer the local control question\. It says how good the current state is for a goal, but not which action should be preferred among the actions available at that state\.

This distinction is central in offline GCRL\. Policy extraction requires an action\-comparative signal\. Two actions can start from the same state and therefore share the same valueVθ​\(s,g\)V\_\{\\theta\}\(s,g\), while only one of them may move the agent toward the goal\. What is missing is not another global estimate of reachability, but a local advantage\-like quantity: a way to score whether an action changes the state in a direction that improves goal\-conditioned value\.

Our key observation is that this local signal is already implicit in the geometry of dual representations\. Under the bilinear parameterization above, the goal embeddingϕθ​\(g\)\\phi\_\{\\theta\}\(g\)is the direction in state\-representation space along which the goal\-conditioned value increases:

∇ψVθ​\(s,g\)=ϕθ​\(g\)\.\\nabla\_\{\\psi\}V\_\{\\theta\}\(s,g\)=\\phi\_\{\\theta\}\(g\)\.Thus, if an action induces a displacement in the state representation, its usefulness for the goal can be evaluated by a simple geometric test: does the predicted displacement align with the goal direction? This turns goal\-conditioned policy improvement into a local alignment problem in the dual representation space\.

We introduce*Dual Advantage Fields*\(DAF\), a policy\-extraction method that makes this geometry explicit; see Figure[1](https://arxiv.org/html/2606.04188#S0.F1)\. DAF learns an action\-effect model that predicts the discounted change in the state representation caused by an action\. It then scores actions by the inner product between this predicted action effect and the goal embedding\. The resulting score is local, goal\-conditioned, and action\-comparative: it prefers actions whose predicted latent effect points in the direction of increasing value for the goal\.

This perspective leads to a simple principle for offline GCRL:*global value fields should be paired with local advantage fields*\. Dual representations provide the global map; DAF extracts from the same representation space the local compass needed for policy improvement\. This yields an efficient actor\-free mechanism for policy extraction: rather than learning a separate goal\-conditioned action\-value function, DAF reuses the geometry of the dual critic to obtain an advantage\-like score for weighting offline actions\.

Our contributions are:

- •We show that, under the standard dual goal representation parameterization, the goal embedding can be interpreted as the gradient direction of the goal\-conditioned value field with respect to the learned state representation\.
- •We introduce*Dual Advantage Fields*, which learn action\-effect vectors and score actions by their alignment with this goal direction, producing a local advantage\-like signal for goal\-conditioned policy extraction\.
- •We use this signal to extract policies from offline data without training a separate goal\-conditioned action\-value function, and evaluate the resulting method across challenging offline GCRL benchmarks\.

## 2Preliminaries

##### Goal\-conditioned Reinforcement Learning\.

We study*offline*goal\-conditioned reinforcement learning \(GCRL\)\[[14](https://arxiv.org/html/2606.04188#bib.bib5),[5](https://arxiv.org/html/2606.04188#bib.bib15),[15](https://arxiv.org/html/2606.04188#bib.bib4),[17](https://arxiv.org/html/2606.04188#bib.bib3)\]: the learner has access to a fixed offline dataset of transitions but cannot collect new experience in the environment\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\. The objective is to infer a an optimal goal\-conditioned policy even for unseen during training combinations of state\-goal pairs\.

Let𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote state and action spaces, and let𝒢⊆𝒮\\mathcal\{G\}\\subseteq\\mathcal\{S\}\(or an abstract goal space\) denote goals\. At each step the environment emits a transition\(s,a,s′\)\(s,a,s^\{\\prime\}\)according to an unknown Markov kernelP​\(s′∣s,a\)P\(s^\{\\prime\}\\\!\\mid\\\!s,a\)\. A goalg∈𝒢g\\in\\mathcal\{G\}induces a reward signalr​\(s,a,g\)r\(s,a,g\): in sparse goal\-reaching problems this is often zero until a success condition holds\. A stochastic policyπ​\(a∣s,g\)\\pi\(a\\\!\\mid\\\!s,g\)induces the usual discounted return with discountγ∈\(0,1\)\\gamma\\in\(0,1\)\. The goal\-conditioned value and action\-value functions are

Qπ​\(s,a,g\):=𝔼π​\[∑t=0∞γt​r​\(st,at,g\)\|s0=s,a0=a\],Vπ​\(s,g\):=𝔼a∼π\(⋅∣s,g\)​\[Qπ​\(s,a,g\)\]\.Q^\{\\pi\}\(s,a,g\):=\\mathbb\{E\}\_\{\\pi\}\\Bigl\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\,r\(s\_\{t\},a\_\{t\},g\)\\,\\Bigm\|\\,s\_\{0\}=s,\\,a\_\{0\}=a\\Bigr\],\\quad V^\{\\pi\}\(s,g\):=\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\\mid s,g\)\}\\bigl\[Q^\{\\pi\}\(s,a,g\)\\bigr\]\.Qπ​\(s,a,g\)=𝔼s′∼P\(⋅∣s,a\)​\[r​\(s,a,g\)\+γ​Vπ​\(s′,g\)\],Vπ​\(s,g\)=𝔼a∼π\(⋅∣s,g\)​\[Qπ​\(s,a,g\)\]\.Q^\{\\pi\}\(s,a,g\)=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\bigl\[r\(s,a,g\)\+\\gamma\\,V^\{\\pi\}\(s^\{\\prime\},g\)\\bigr\],\\quad V^\{\\pi\}\(s,g\)=\\mathbb\{E\}\_\{a\\sim\\pi\(\\cdot\\mid s,g\)\}\\bigl\[Q^\{\\pi\}\(s,a,g\)\\bigr\]\.\(1\)Recent GCRL methods combine several ideas, including representation learning, quasimetric objectives\[[25](https://arxiv.org/html/2606.04188#bib.bib11),[11](https://arxiv.org/html/2606.04188#bib.bib17),[17](https://arxiv.org/html/2606.04188#bib.bib3)\], and hierarchical horizon reduction\[[21](https://arxiv.org/html/2606.04188#bib.bib7),[7](https://arxiv.org/html/2606.04188#bib.bib16),[18](https://arxiv.org/html/2606.04188#bib.bib20)\]over value functions,QQ\-functions, and actors\. These design choices are often complementary, but existing methods still show domain\-specific strengths: hierarchical methods tend to excel in long\-horizon locomotion, while quasimetric representations often work well for manipulation\. In contrast, DAF emphasizes local policy improvement during training while retaining long\-horizon reasoning, leading to more consistent performance across both domains\.

Hierarchical Implicit Q\-Learning \(HIQL\)\.In GCRL, accurately estimating the value function for distant goals is the main challenge in solving complex long\-horizon tasks\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\]\. To address this issue, HIQL\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\]proposed a hierarchical policy structure that utilizes a value function learned with IQL\[[13](https://arxiv.org/html/2606.04188#bib.bib8)\]\. This hierarchical design enables the agent to produce effective actions even when value estimates for distant goals are noisy or unreliable\. More specifically, HIQL trains a goal\-conditioned state\-value functionVVwith the following loss:

ℒ​\(V\)=𝔼\(s,s′\)∼𝒟,g∼p​\(g\)​\[L2τ​\(r​\(s,g\)\+γ​V¯​\(s′,g\)−V​\(s,g\)\)\],\\displaystyle\\mathcal\{L\}\(V\)=\\mathbb\{E\}\_\{\(s,s^\{\\prime\}\)\\sim\\mathcal\{D\},\\;g\\sim p\(g\)\}\\left\[L\_\{2\}^\{\\tau\}\\left\(r\(s,g\)\+\\gamma\\bar\{V\}\(s^\{\\prime\},g\)\-V\(s,g\)\\right\)\\right\],\(2\)where the expectile loss is defined asL2τ​\(u\)=\|τ−𝟏​\(u<0\)\|​u2L\_\{2\}^\{\\tau\}\(u\)=\|\\tau\-\\mathbf\{1\}\(u<0\)\|u^\{2\}, withτ\>0\.5\\tau\>0\.5, andV¯\\bar\{V\}denotes the targetVVnetwork\.111Since the inherent over\-estimation problem of IQL, we assume that the environment dynamics is deterministic\.Following prior works\[[3](https://arxiv.org/html/2606.04188#bib.bib10),[25](https://arxiv.org/html/2606.04188#bib.bib11),[21](https://arxiv.org/html/2606.04188#bib.bib7)\], we adopt the sparse rewardr​\(s,g\)=−𝟏​\{s≠g\}r\(s,g\)=\-\\mathbf\{1\}\\\{s\\neq g\\\}\. Under this reward, the optimal value\|V⋆​\(s,g\)\|\|V^\{\\star\}\(s,g\)\|corresponds to thediscounted temporal distance,i\.e\., a discounted measure of the minimum number of environment steps required to reach the goalggfrom statess\. HIQL separates policy extraction222Policy extraction refers to learning a policy from a learned value function, emphasizing the separation between value learning and policy learning\.into two levels: a high\-level policyπh​\(st\+k\|st,g\)\\pi^\{h\}\(s\_\{t\+k\}\|s\_\{t\},g\)generates akk\-step subgoal to guide progress toward the goal, while a low\-level policyπℓ​\(at\|st,st\+k\)\\pi^\{\\ell\}\(a\_\{t\}\|s\_\{t\},s\_\{t\+k\}\)produces primitive actions to reach the subgoal\. Both policies are extracted using advantage\-weighted regression \(AWR\)\[[23](https://arxiv.org/html/2606.04188#bib.bib14),[26](https://arxiv.org/html/2606.04188#bib.bib12)\]with the following objective:

𝒥​\(πh\)\\displaystyle\\mathcal\{J\}\(\\pi^\{h\}\)=𝔼\(st,st\+k,g\)∼𝒟​\[exp⁡\(βh⋅Ah​\(st,st\+k,g\)\)​log⁡πh​\(st\+k\|st,g\)\],\\displaystyle=\\mathbb\{E\}\_\{\(s\_\{t\},s\_\{t\+k\},g\)\\sim\\mathcal\{D\}\}\\left\[\\exp\\left\(\\beta^\{h\}\\cdot A^\{h\}\(s\_\{t\},s\_\{t\+k\},g\)\\right\)\\log\\pi^\{h\}\(s\_\{t\+k\}\|s\_\{t\},g\)\\right\],\(3\)𝒥​\(πℓ\)\\displaystyle\\mathcal\{J\}\(\\pi^\{\\ell\}\)=𝔼\(st,at,st\+1,st\+k\)∼𝒟​\[exp⁡\(βℓ⋅Aℓ​\(st,st\+1,st\+k\)\)​log⁡πℓ​\(at\|st,st\+k\)\],\\displaystyle=\\mathbb\{E\}\_\{\(s\_\{t\},a\_\{t\},s\_\{t\+1\},s\_\{t\+k\}\)\\sim\\mathcal\{D\}\}\\left\[\\exp\\left\(\\beta^\{\\ell\}\\cdot A^\{\\ell\}\(s\_\{t\},s\_\{t\+1\},s\_\{t\+k\}\)\\right\)\\log\\pi^\{\\ell\}\(a\_\{t\}\|s\_\{t\},s\_\{t\+k\}\)\\right\],\(4\)whereβh\\beta^\{h\}andβl\\beta^\{l\}are inverse temperature parameters,Ah​\(st,st\+k,g\)=Vh​\(st\+k,g\)−Vh​\(st,g\)A^\{h\}\(s\_\{t\},s\_\{t\+k\},g\)=V^\{h\}\(s\_\{t\+k\},g\)\-V^\{h\}\(s\_\{t\},g\)denotes the high\-level policy advantage, andAℓ​\(st,st\+1,st\+k\)=Vℓ​\(st\+1,st\+k\)−Vℓ​\(st,st\+k\)A^\{\\ell\}\(s\_\{t\},s\_\{t\+1\},s\_\{t\+k\}\)=V^\{\\ell\}\(s\_\{t\+1\},s\_\{t\+k\}\)\-V^\{\\ell\}\(s\_\{t\},s\_\{t\+k\}\)denotes the low\-level policy advantage\. HIQL uses a single goal\-conditioned value functionVV, which is shared between bothπh\\pi^\{h\}andπℓ\\pi^\{\\ell\}\(i\.e\.,Vh=Vℓ=VV^\{h\}=V^\{\\ell\}=V\)\. However, despite this design, HIQL still struggles with long\-horizon, complex tasks, as shown in the GCRL benchmark, OGBench\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\.

Dual Goal Representations\[[22](https://arxiv.org/html/2606.04188#bib.bib6)\]\.In goal\-conditioned RL, the goal representation determines what information the policy and value function use about the target state\. Rather than conditioning directly on the raw goal observation, which may contain irrelevant or exogenous factors, dual goal representations encode a goal by its reachability relation to other states\. A goalggis represented by

ϕ∨​\(g\):s↦d⋆​\(s,g\),\\phi^\{\\vee\}\(g\):s\\mapsto d^\{\\star\}\(s,g\),whered⋆​\(s,g\)d^\{\\star\}\(s,g\)denotes the optimal temporal distance from statessto goalgg\. In practice, we approximate this functional through a bilinear goal\-conditioned potential\[[9](https://arxiv.org/html/2606.04188#bib.bib22)\]:

Vθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\),V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\),\(5\)whereψθ:𝒮→ℝd\\psi\_\{\\theta\}:\\mathcal\{S\}\\to\\mathbb\{R\}^\{d\}andϕθ:𝒢→ℝd\\phi\_\{\\theta\}:\\mathcal\{G\}\\to\\mathbb\{R\}^\{d\}are state and goal embeddings\. The goal embeddingϕθ​\(g\)\\phi\_\{\\theta\}\(g\)then serves as a finite\-dimensional dual representation: when paired withψθ​\(s\)\\psi\_\{\\theta\}\(s\), it predicts a value or distance\-like quantity that reflects the environment’s reachability structure\.

## 3Dual Advantage Fields

![Refer to caption](https://arxiv.org/html/2606.04188v1/Images/Dual_fields.png)Figure 2:Dual Advantage Fields\.Under a bilinear goal\-conditioned value model, the goal embedding defines a direction in representation space\. DAF scores an action by projecting its induced feature displacement onto this goal direction, yielding a local advantage\-like signal for policy improvement\.Our method is based on a simple insight from bilinear value decomposition in Eq\. \([5](https://arxiv.org/html/2606.04188#S2.E5)\)\. Holding the goal fixed and viewing the value as a function of the state embeddingψ\\psi, we have

###### Proposition 3\.1\.

Under the bilinear goal\-conditioned value modelVθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\)V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\), the gradient of the value with respect to the state embedding is the goal embedding:

∇ψVθ​\(s,g\)=∇ψ\(ψ⊤​ϕθ​\(g\)\)=ϕθ​\(g\)\.\\nabla\_\{\\psi\}V\_\{\\theta\}\(s,g\)=\\nabla\_\{\\psi\}\\bigl\(\\psi^\{\\top\}\\phi\_\{\\theta\}\(g\)\\bigr\)=\\phi\_\{\\theta\}\(g\)\.\(6\)

Thus, the goal embeddingϕθ​\(g\)\\phi\_\{\\theta\}\(g\)is the value\-gradient direction in representation space under the Euclidean geometry of the learned embedding\. Please, see Figure[2](https://arxiv.org/html/2606.04188#S3.F2)for intuition\. For any transition fromsstos′s^\{\\prime\}, the change in the bilinear value is exactly

Vθ​\(s′,g\)−Vθ​\(s,g\)=ϕθ​\(g\)⊤​\(ψθ​\(s′\)−ψθ​\(s\)\)\.V\_\{\\theta\}\(s^\{\\prime\},g\)\-V\_\{\\theta\}\(s,g\)=\\phi\_\{\\theta\}\(g\)^\{\\top\}\\bigl\(\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\bigr\)\.\(7\)
We use this identity to construct an advantage\-like local policy improvement signal\. For a policyπ\\pi, the standard goal\-conditioned advantage is

Aπ​\(s,a,g\)=𝔼s′∼p\(⋅\|s,a\)​\[r​\(s,a,g\)\+γ​Vπ​\(s′,g\)−Vπ​\(s,g\)\]\.A^\{\\pi\}\(s,a,g\)=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim p\(\\cdot\|s,a\)\}\\left\[r\(s,a,g\)\+\\gamma V^\{\\pi\}\(s^\{\\prime\},g\)\-V^\{\\pi\}\(s,g\)\\right\]\.\(8\)ReplacingVπV^\{\\pi\}with the learned bilinear fieldVθV\_\{\\theta\}gives the model\-induced Bellman advantage

Aθ​\(s,a,g\)=𝔼s′∼p\(⋅\|s,a\)​\[r​\(s,a,g\)\+ϕθ​\(g\)⊤​\(γ​ψθ​\(s′\)−ψθ​\(s\)\)\]\.A\_\{\\theta\}\(s,a,g\)=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim p\(\\cdot\|s,a\)\}\\left\[r\(s,a,g\)\+\\phi\_\{\\theta\}\(g\)^\{\\top\}\\bigl\(\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\bigr\)\\right\]\.\(9\)In offline learning, each dataset transition\(s,a,s′\)\(s,a,s^\{\\prime\}\)provides a sample estimate of this quantity:

###### Corollary 3\.2\.

The sample\-level Dual Advantage Field score is

A^θ\(s,a,s′,g\)=r\(s,a,g\)\+ϕθ\(g\)⊤\(γψθ\(s′\)−ψθ\(s\)\)\.\\boxed\{\\widehat\{A\}\_\{\\theta\}\(s,a,s^\{\\prime\},g\)=r\(s,a,g\)\+\\phi\_\{\\theta\}\(g\)^\{\\top\}\\bigl\(\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\bigr\)\.\}\(10\)

##### Local policy improvement\.

In the realizable case, the DAF score is exactly the goal\-conditioned Bellman advantage\. Specifically, ifVπ​\(s,g\)=ψ​\(s\)⊤​ϕ​\(g\)V^\{\\pi\}\(s,g\)=\\psi\(s\)^\{\\top\}\\phi\(g\)andu​\(s,a\)=𝔼s′∼P\(⋅∣s,a\)​\[γ​ψ​\(s′\)−ψ​\(s\)\]u\(s,a\)=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\[\\gamma\\psi\(s^\{\\prime\}\)\-\\psi\(s\)\], then

r​\(s,a,g\)\+u​\(s,a\)⊤​ϕ​\(g\)=Aπ​\(s,a,g\)\.r\(s,a,g\)\+u\(s,a\)^\{\\top\}\\phi\(g\)=A^\{\\pi\}\(s,a,g\)\.Thus, increasing the probability of actions \(alignment\) with positive DAF score is a standard policy\-improvement step\. Repeated exact DAF improvement therefore recovers an optimal primitive goal\-conditioned policy; in particular, its limiting policy is at least as good as any policy restricted to a fixed hierarchical class\. We provide the formal statement and proof in Appendix[F\.1](https://arxiv.org/html/2606.04188#A6.Thmtheorem1)\.

Equation \([10](https://arxiv.org/html/2606.04188#S3.E10)\) defines the DAF score\. The termγ​ψθ​\(s′\)−ψθ​\(s\)\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)is the discounted feature displacement caused by actionaa, andϕθ​\(g\)\\phi\_\{\\theta\}\(g\)is the value\-gradient direction toward goalgg\. Their inner product measures the one\-step increase in the bilinear value field, with the reward term completing the Bellman advantage\. Thus,A^θ\\widehat\{A\}\_\{\\theta\}provides a local, goal\-conditioned action\-ranking signal derived from the learned dual value geometry\. This follows the comparative view of policy improvement, where actions are improved by relative advantages rather than absolute value estimates\[[4](https://arxiv.org/html/2606.04188#bib.bib1)\]\.

### 3\.1Motivational Example

We illustrate the local geometry captured by Dual Advantage Fields on thecube\-single\-play\-v0\-task1manipulation task from OGBench\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\. This task highlights a common failure mode in goal\-conditioned control: before the cube can be placed at the final target, the agent must first move the gripper into a pre\-grasp configuration\. Thus, a direction that points directly toward the terminal object location may be globally plausible but locally unhelpful\.

DAF addresses this by scoring actions according to their local improvement of the learned goal\-conditioned potential\. By Eq\. \([6](https://arxiv.org/html/2606.04188#S3.E6)\), the goal embeddingϕθ​\(g\)\\phi\_\{\\theta\}\(g\)is the representation\-space gradient of the bilinear value field\. We define an action\-effect modeluξ​\(s,a\)u\_\{\\xi\}\(s,a\)that estimates the discounted feature displacement induced by actionaa,

uξ​\(s,a\)≈𝔼s′∼p\(⋅\|s,a\)​\[γ​ψθ​\(s′\)−ψθ​\(s\)\]\.u\_\{\\xi\}\(s,a\)\\approx\\mathbb\{E\}\_\{s^\{\\prime\}\\sim p\(\\cdot\|s,a\)\}\\left\[\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\right\]\.Ignoring reward terms that are constant across actions in the pre\-grasp region, DAF scores actions by

zθ​\(s,a,g\)=uξ​\(s,a\)⊤​ϕθ​\(g\)\.z\_\{\\theta\}\(s,a,g\)=u\_\{\\xi\}\(s,a\)^\{\\top\}\\phi\_\{\\theta\}\(g\)\.\(11\)This score favors actions whose predicted feature displacement is aligned with the local direction of value increase toward the goal\.

![Refer to caption](https://arxiv.org/html/2606.04188v1/x1.png)Figure 3:Pre\-grasp vector field incube\-single\.Arrows show decoded high\-level directions from sampled gripper positions around the cube, with the cube and final goal fixed\. DAF points locally toward the cube before grasping, while OTA points toward the terminal placement goal\. The yellow marker denotes the mean decoded target\.Figure[3](https://arxiv.org/html/2606.04188#S3.F3)visualizes this effect\. We sample query states\{s~i\}i=1N\\\{\\tilde\{s\}\_\{i\}\\\}\_\{i=1\}^\{N\}by perturbing only the gripper position around the cube, while keeping the object state and final goal fixed\. For each methodm∈\{OTA,DAF\}m\\in\\\{\\mathrm\{OTA\},\\mathrm\{DAF\}\\\}, we decode its high\-level subgoal prediction into an X\-Y coordinates via probing,

x^im=Dm​\(hm​\(s~i,g\)\),\\hat\{x\}\_\{i\}^\{\\,m\}=D\_\{m\}\\\!\\left\(h\_\{m\}\(\\tilde\{s\}\_\{i\},g\)\\right\),\(12\)wherehmh\_\{m\}is the method\-specific latent output andDmD\_\{m\}is a linear probe fitted on demonstration states\. The plotted direction is

dim=x^im−xee​\(s~i\)‖x^im−xee​\(s~i\)‖2,d\_\{i\}^\{\\,m\}=\\frac\{\\hat\{x\}\_\{i\}^\{\\,m\}\-x\_\{\\mathrm\{ee\}\}\(\\tilde\{s\}\_\{i\}\)\}\{\\\|\\hat\{x\}\_\{i\}^\{\\,m\}\-x\_\{\\mathrm\{ee\}\}\(\\tilde\{s\}\_\{i\}\)\\\|\_\{2\}\},\(13\)drawn from the gripper positionxee​\(s~i\)x\_\{\\mathrm\{ee\}\}\(\\tilde\{s\}\_\{i\}\)\. Near the cube, DAF produces directions that point toward the object, matching the immediate pre\-grasp behavior required before transport\. OTA instead points toward the terminal placement region in this example, which is appropriate only after grasping\. The example shows why local advantage fields can be more useful than a high\-level subgoals alone: they select actions by whether they locally improve the goal\-conditioned potential\.

## 4Training and goal\-conditioned policy extraction

Dayan and Singh \[[4](https://arxiv.org/html/2606.04188#bib.bib1)\]showed that policy improvement can be organized around*relative*measures of how actions compare at a state\-*merits*that need not reduce to a fully trusted global value oracle\. In the goal\-conditioned setting, the Bellman advantageAπ​\(s,a,g\)A^\{\\pi\}\(s,a,g\)in \([8](https://arxiv.org/html/2606.04188#S3.E8)\) is exactly such an object: it ranksaaby the expected one\-step gain in return, isolating the effect of the transition from the baselineVπ​\(s,g\)V^\{\\pi\}\(s,g\)\. Our bilinear potential \([5](https://arxiv.org/html/2606.04188#S2.E5)\) turns this comparison into explicit geometry inψ\\psi\. Under the modelVθV\_\{\\theta\}, the backup contributionγ​Vθ​\(s′,g\)−Vθ​\(s,g\)\\gamma V\_\{\\theta\}\(s^\{\\prime\},g\)\-V\_\{\\theta\}\(s,g\)equalsϕθ​\(g\)⊤​\(γ​ψθ​\(s′\)−ψθ​\(s\)\)\\phi\_\{\\theta\}\(g\)^\{\\top\}\\bigl\(\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\bigr\)by \([7](https://arxiv.org/html/2606.04188#S3.E7)\), so the analogue of the advantage \([8](https://arxiv.org/html/2606.04188#S3.E8)\) withVπV^\{\\pi\}replaced byVθV\_\{\\theta\}is the closed form \([9](https://arxiv.org/html/2606.04188#S3.E9)\)\-\([10](https://arxiv.org/html/2606.04188#S3.E10)\)\. The goal embeddingϕθ​\(g\)\\phi\_\{\\theta\}\(g\)acts as∇ψVθ\\nabla\_\{\\psi\}V\_\{\\theta\}\(Eq\. \([6](https://arxiv.org/html/2606.04188#S3.E6)\)\): the inner product in \([10](https://arxiv.org/html/2606.04188#S3.E10)\) measures whether the*local*feature displacement induced byaais aligned with steepest increase of the learned potential towardgg\. Thus Dayan’s comparative view of improvement is instantiated here as projection of one\-stepψ\\psi\-dynamics onto the value\-gradient direction\.

In practice we estimate the discounted incrementγ​ψθ​\(s′\)−ψθ​\(s\)\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)with a mapuξ​\(s,a\)u\_\{\\xi\}\(s,a\)trained on offline transitions \(Sec\.[4\.1](https://arxiv.org/html/2606.04188#S4.SS1)\), and absorbrrin the critic stack where noted\. The raw dual score is

zθ​\(s,a,g\):=uξ​\(s,a\)⊤​ϕθ​\(g\),z\_\{\\theta\}\(s,a,g\)\\;:=\\;u\_\{\\xi\}\(s,a\)^\{\\top\}\\phi\_\{\\theta\}\(g\)\\,,\(14\)which agrees with \([10](https://arxiv.org/html/2606.04188#S3.E10)\) whenuξ​\(s,a\)≈γ​ψθ​\(s′\)−ψθ​\(s\)u\_\{\\xi\}\(s,a\)\\approx\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)and rewards are handled by the value heads feeding the same Bellman targets\.

### 4\.1Offline critic and feature dynamics

We learn\(ψθ,ϕθ\)\(\\psi\_\{\\theta\},\\phi\_\{\\theta\}\), and the displacement mapuξu\_\{\\xi\}from offline tuples\(s,a,s′,g\)\(s,a,s^\{\\prime\},g\)\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\. For stability we used a common approach in offline RL\[[13](https://arxiv.org/html/2606.04188#bib.bib8)\]that learns twin criticsQθ\(1\),Qθ\(2\)Q^\{\(1\)\}\_\{\\theta\},Q^\{\(2\)\}\_\{\\theta\}, and the bilinear headVθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\)V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\)is tied to pessimisticQQ\-estimates via expectile regression and to Bellman backups onQθ\(j\)Q^\{\(j\)\}\_\{\\theta\}\. To avoid brittlemaxa⁡Q\\max\_\{a\}Qoperators in continuous control\[[16](https://arxiv.org/html/2606.04188#bib.bib24)\], we add an*actor\-free*coupling betweenVθV\_\{\\theta\}and the scalar dual scorezθz\_\{\\theta\}from \([14](https://arxiv.org/html/2606.04188#S4.E14)\), followingPerrin\-Gilbert \[[24](https://arxiv.org/html/2606.04188#bib.bib23)\]; the explicit construction is deferred to Appendix[E](https://arxiv.org/html/2606.04188#A5)\. Finally, we grounduξu\_\{\\xi\}with the auxiliary loss

ℒae=𝔼​\[‖uξ​\(s,a\)−sg​\(γ​ψθ​\(s′\)−ψθ​\(s\)\)‖22\],\\mathcal\{L\}\_\{\\mathrm\{ae\}\}\\;=\\;\\mathbb\{E\}\\Bigl\[\\bigl\\\|u\_\{\\xi\}\(s,a\)\-\\mathrm\{sg\}\\bigl\(\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\)\\bigr\)\\bigr\\\|\_\{2\}^\{2\}\\Bigr\]\\,,\(15\)withsg\\mathrm\{sg\}stopping gradients through the target, souξu\_\{\\xi\}tracks one\-step feature dynamics on𝒟\\mathcal\{D\}\.

### 4\.2Policy extraction

Letπω​\(a∣s,c\)\\pi\_\{\\omega\}\(a\\mid s,c\)denote the policy with conditioningcconggthroughϕθ​\(g\)\\phi\_\{\\theta\}\(g\)\(and optionallyss\)\. Advantage\-weighted regression\[[23](https://arxiv.org/html/2606.04188#bib.bib14)\]uses weights

wθ​\(s,a,g\)=min⁡\{exp⁡\(α​zθ​\(s,a,g\)\),Wmax\}w\_\{\\theta\}\(s,a,g\)\\;=\\;\\min\\Bigl\\\{\\exp\\bigl\(\\alpha\\,z\_\{\\theta\}\(s,a,g\)\\bigr\),\\,W\_\{\\max\}\\Bigr\\\}\(16\)with temperatureα\>0\\alpha\>0and capWmaxW\_\{\\max\}, and minimizes−𝔼𝒟​\[wθ​log⁡πω​\(a∣s,c\)\]\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[w\_\{\\theta\}\\log\\pi\_\{\\omega\}\(a\\mid s,c\)\]\. Becausezθz\_\{\\theta\}does not depend onω\\omega, this is weighted behavior cloning that up\-weights actions whose localψ\\psi\-displacement aligns with the goal directionϕθ​\(g\)\\phi\_\{\\theta\}\(g\), i\.e\. actions that the bilinear model classifies as improving the goal\-conditioned potential in the sense of \([10](https://arxiv.org/html/2606.04188#S3.E10)\)\.

Hierarchical goals\.For long horizons, a high\-level policy over subgoals can be trained alongside the low\-level stack above, with value differences along options as in hierarchical offline GCRL\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\]; option\-aware temporally abstracted value learning offers a related hierarchical baseline\[[2](https://arxiv.org/html/2606.04188#bib.bib26)\]\.

Algorithm 1DAF training\.1:Input:offline dataset

𝒟\\mathcal\{D\}of

\(s,a,s′,g\)\(s,a,s^\{\\prime\},g\);

2:Initialize:

ψθ,ϕθ\\psi\_\{\\theta\},\\phi\_\{\\theta\}, displacement map

uξu\_\{\\xi\}, policy

πω\\pi\_\{\\omega\}, target networks

\(Qtgt,Vtgt\)\(Q^\{\\mathrm\{tgt\}\},V^\{\\mathrm\{tgt\}\}\)\.

3:whilenot convergeddo

4:Sample a minibatch from

𝒟\\mathcal\{D\}\.

5:Critic:update

ψθ,ϕθ\\psi\_\{\\theta\},\\phi\_\{\\theta\}so

Vθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\)V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\)\([5](https://arxiv.org/html/2606.04188#S2.E5)\) using target networks\.

6:AFU coupling:minimize the actor\-free loss coupling

VθV\_\{\\theta\}to

zθz\_\{\\theta\}\([14](https://arxiv.org/html/2606.04188#S4.E14)\) \{Appendix[E](https://arxiv.org/html/2606.04188#A5)\} and minimize

ℒae\\mathcal\{L\}\_\{\\mathrm\{ae\}\}\([15](https://arxiv.org/html/2606.04188#S4.E15)\) for

uξu\_\{\\xi\}\.

7:Policy:

wθ←min⁡\{exp⁡\(α​zθ​\(s,a,g\)\),Wmax\}w\_\{\\theta\}\\leftarrow\\min\\\{\\exp\(\\alpha z\_\{\\theta\}\(s,a,g\)\),W\_\{\\max\}\\\}; minimize

−𝔼𝒟​\[wθ​log⁡πω​\(a∣s,c\)\]\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[w\_\{\\theta\}\\log\\pi\_\{\\omega\}\(a\\mid s,c\)\]over

ω\\omega\.

8:Update target networks\.

9:endwhile

## 5Experiments

In this section, we empirically validate the findings developed in the previous sections on the OGBench benchmark\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\. OGBench is designed to evaluate several core capabilities required by offline goal\-conditioned reinforcement learning, including long\-horizon reasoning, trajectory stitching, generalization to unseen goals, robustness to suboptimal data, and control under imperfect offline coverage\. We focus on the state\-based locomotion and manipulation tasks used in prior work, which allows us to test whether DAF provides consistent improvements across domains with substantially different control structure\.

All methods are trained purely offline on the provided datasets and are evaluated without additional environment interaction during training\. We report success\-based performance in\[0,1\]\[0,1\], where higher values indicate better goal reaching\. For each environment, we evaluate the corresponding OGBench dataset regimes\. In maze\-style locomotion, we usenavigateandstitchdatasets:navigatedata is collected from noisy expert policies that traverse the environment, whilestitchdata contains shorter trajectory segments and therefore requires composing partial behaviors into longer goal\-reaching solutions\. In manipulation, we useplayandnoisydatasets:playdata contains natural temporally correlated interactions generated by scripted policies, whereasnoisydata increases state\-action coverage through less structured exploration noise, making the offline data more suboptimal\.

##### Baselines\.

We compare against a representative set of recent and relevant methods for offline GCRL, including HIQL\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\], OTA\[[2](https://arxiv.org/html/2606.04188#bib.bib26)\], MQE\[[18](https://arxiv.org/html/2606.04188#bib.bib20)\], CRL\[[5](https://arxiv.org/html/2606.04188#bib.bib15)\], GCIQL\[[13](https://arxiv.org/html/2606.04188#bib.bib8)\], and GCIVL\[[10](https://arxiv.org/html/2606.04188#bib.bib19)\]\. When applicable, we also include their corresponding variants that learn representations in the form of dual\-goal representations\[[22](https://arxiv.org/html/2606.04188#bib.bib6)\]\. These baselines cover the main families of methods used in offline GCRL, including horizon\-reduction methods\[[20](https://arxiv.org/html/2606.04188#bib.bib18)\]and methods based on representation priors such as quasimetrics\.

##### What DAF does in each dataset\.

Across all datasets, DAF uses the same policy\-extraction principle: it scores offline actions by the alignment between their predicted local feature displacement and the goal direction induced by the dual value representation\. Concretely, the action\-effect model estimatesγ​ψθ​\(s′\)−ψθ​\(s\)\\gamma\\psi\_\{\\theta\}\(s^\{\\prime\}\)\-\\psi\_\{\\theta\}\(s\), and the dual score projects this displacement ontoϕθ​\(g\)\\phi\_\{\\theta\}\(g\)\. Thus, DAF uses the learned value field not only as a global map of reachability, but also as a local compass for choosing among actions available in the offline dataset\.

![Refer to caption](https://arxiv.org/html/2606.04188v1/x2.png)Figure 4:Performance profile across all tasks and environments\. DAF achieves a better distribution of scores than the baselines across the OGBench evaluation suite\.
##### Maze locomotion: long\-horizon navigation and stitching\.

We first evaluate onhumanoidmazeandantmaze, shown in Table[1](https://arxiv.org/html/2606.04188#S5.T1)\. These environments isolate the long\-horizon navigation aspect of offline GCRL\. The agent must reach target states from diverse initial states using only fixed offline data\. Theantmazetasks require quadruped locomotion through maze layouts, whilehumanoidmazeis more challenging because it combines full\-body humanoid control with long\-horizon goal reaching\. We include bothnavigateandstitchvariants because they test complementary capabilities:navigateevaluates whether the method can exploit noisy expert trajectories, whilestitchevaluates whether the method can compose shorter trajectory fragments into successful long\-horizon behavior\.

These tasks are important because many prior offline GCRL methods are designed around horizon reduction or hierarchical subgoal prediction\. DAF is not primarily a hierarchical method: instead, it extracts local action preferences from a dual value field\. Strong performance on these mazes therefore tests whether local advantage\-field extraction can preserve the long\-horizon structure needed for navigation\. DAF is competitive with the strongest horizon\-reduction baselines onnavigatedatasets and obtains the best results on the harderstitchcases where composing partial trajectories is essential\.

Table 1:Maze locomotion results onhumanoidmazeandantmaze\. These tasks test long\-horizon goal reaching from fixed offline data\. Thenavigatedatasets evaluate learning from noisy expert trajectories, while thestitchdatasets evaluate whether a method can compose shorter trajectory segments into successful goal\-reaching behavior\. Results within 95% of the best value are written inbold\. Best values are highlighted inblue\.
##### Object manipulation: local control from imperfect demonstrations\.

Next, we evaluate oncubeandscene, shown in Table[2](https://arxiv.org/html/2606.04188#S5.T2)\. Unlike maze navigation, these tasks require precise object\-centric control\. Thecubetasks include pick\-and\-place, stacking, swapping, and multi\-object rearrangement, whilescenetasks require sequencing interactions with objects such as cubes, drawers, windows, and buttons\.

These datasets test DAF’s central motivation: globally plausible behavior can be locally wrong\. For example, moving toward a final object placement may be inappropriate before reaching a pre\-grasp state\. DAF addresses this by ranking dataset actions according to whether their predicted feature displacement aligns with the goal direction\. This is especially useful inplayandnoisydatasets, where demonstrations contain useful local skills but also incomplete or suboptimal trajectories\.

Table 2:Object\-manipulation results oncubeandscene\. These datasets test whether an offline GCRL method can extract precise local skills fromplaydata and remain robust under the less structurednoisyregime\. Results within 95% of the best value are written inbold\. Best values are highlighted inblue\.Env\.DatasetDimensionDAFHIQLOTAMQECRLCRL DUALGCIQLGCIVLGCIVL DUALcubeplaydouble0\.41±0\.04\\pm 0\.040\.13±0\.01\\pm 0\.010\.05±0\.01\\pm 0\.010\.03±0\.00\\pm 0\.000\.16±0\.01\\pm 0\.010\.38±0\.06\\pm 0\.060\.35±0\.06\\pm 0\.060\.33±0\.05\\pm 0\.050\.58±0\.04\\pm 0\.04triple0\.17±0\.03\\pm 0\.030\.05±0\.02\\pm 0\.020\.02±0\.00\\pm 0\.000\.01±0\.00\\pm 0\.000\.06±0\.02\\pm 0\.020\.05±0\.05\\pm 0\.050\.02±0\.01\\pm 0\.010\.01±0\.01\\pm 0\.010\.01±0\.00\\pm 0\.00quadruple0\.03±0\.01\\pm 0\.010\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.00noisydouble0\.33±0\.05\\pm 0\.050\.03±0\.01\\pm 0\.010\.05±0\.03\\pm 0\.030\.07±0\.01\\pm 0\.010\.04±0\.02\\pm 0\.020\.08±0\.02\\pm 0\.020\.24±0\.06\\pm 0\.060\.17±0\.04\\pm 0\.040\.26±0\.02\\pm 0\.02triple0\.23±0\.01\\pm 0\.010\.04±0\.01\\pm 0\.010\.01±0\.00\\pm 0\.000\.04±0\.02\\pm 0\.020\.03±0\.01\\pm 0\.010\.06±0\.02\\pm 0\.020\.05±0\.01\\pm 0\.010\.11±0\.02\\pm 0\.020\.09±0\.03\\pm 0\.03quadruple0\.02±0\.01\\pm 0\.010\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.01±0\.01\\pm 0\.010\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.000\.00±0\.00\\pm 0\.00sceneplay0\.81±0\.04\\pm 0\.040\.55±0\.09\\pm 0\.090\.34±0\.04\\pm 0\.040\.20±0\.03\\pm 0\.030\.29±0\.02\\pm 0\.020\.56±0\.06\\pm 0\.060\.53±0\.02\\pm 0\.020\.51±0\.05\\pm 0\.050\.78±0\.07\\pm 0\.07noisy0\.43±0\.03\\pm 0\.030\.27±0\.02\\pm 0\.020\.10±0\.02\\pm 0\.020\.07±0\.02\\pm 0\.020\.02±0\.01\\pm 0\.010\.06±0\.01\\pm 0\.010\.29±0\.02\\pm 0\.020\.31±0\.05\\pm 0\.050\.45±0\.02\\pm 0\.02

##### Puzzle rearrangement: continuous control with combinatorial structure\.

Finally, we evaluate onpuzzle, shown in Table[3](https://arxiv.org/html/2606.04188#S5.T3)\. These environments are robotic versions of Lights Out: pressing one button changes the state of neighboring buttons\. They therefore combine continuous control with combinatorial generalization over discrete configurations\. The3x3and4x4variants further increase the configuration space, testing whether goal representations generalize beyond simple object reaching\.

Puzzle tasks stress a failure mode not captured by maze navigation or standard manipulation\. Here, each local action can affect a larger configuration, so policy extraction must compare actions by their downstream effect on the goal\. DAF is suited to this setting because it scores actions by whether their predicted local transition improves the goal\-conditioned value field\.

Table 3:Puzzle rearrangement results onpuzzle\. These tasks test structured spatial reasoning: each local button press changes neighboring button states, so the policy must combine continuous control with combinatorial goal generalization\. Results within 95% of the best value are written inbold\. Best values are highlighted inblue\.
##### Aggregate comparison\.

Tables[1](https://arxiv.org/html/2606.04188#S5.T1),[2](https://arxiv.org/html/2606.04188#S5.T2), and[3](https://arxiv.org/html/2606.04188#S5.T3)show that DAF performs strongly across different kinds of offline coverage and control structure\. The aggregate comparison in Figure[5](https://arxiv.org/html/2606.04188#S5.F5)further summarizes performance across all tasks using RLiable metrics\[[1](https://arxiv.org/html/2606.04188#bib.bib27)\]\. We report Median, interquartile mean \(IQM\), Mean, and Optimality Gap with stratified\-bootstrap confidence intervals\. The IQM reduces sensitivity to outlier tasks, while the optimality gap measures the average remaining shortfall from perfect success\. Overall, DAF improves the aggregate metrics while also achieving strong per\-task performance, indicating that the gains are not driven by a single environment family\.

![Refer to caption](https://arxiv.org/html/2606.04188v1/x3.png)Figure 5:Performance comparison\.Following the protocol proposed byAgarwalet al\.\[[1](https://arxiv.org/html/2606.04188#bib.bib27)\], we report aggregate RLiable metrics, including Median, IQM, Mean, and Optimality Gap, with stratified\-bootstrap confidence intervals across the offline GCRL environments\. The colored horizontal segments denote confidence intervals, and the dark vertical markers denote point estimates\.

## 6Broader Impact and Limitations

DAF extracts goal\-conditioned policies from offline data without additional environment interaction\. Like other offline RL methods, it is reliable only when the dataset sufficiently covers the actions needed for improvement; poor coverage can produce incorrect action rankings\. DAF also relies on the learned dual representation and action\-effect model\.

Although∇ψVθ​\(s,g\)=ϕθ​\(g\)\\nabla\_\{\\psi\}V\_\{\\theta\}\(s,g\)=\\phi\_\{\\theta\}\(g\)holds exactly for the bilinear head, this direction is useful only if the representation encodes reachability\. In stochastic or poorly covered regions,uξ​\(s,a\)u\_\{\\xi\}\(s,a\)may predict inaccurate feature displacements\. Future work should study uncertainty\-aware or distributional action\-effect models and extend DAF to image\-based and more stochastic goal\-reaching settings\.

## 7Conclusion

We introduced Dual Advantage Fields, a method that turns dual goal representations into local policy\-improvement signals\. Under the bilinear value parameterization, the goal embedding acts as the gradient of the goal\-conditioned value field with respect to the state representation\. DAF uses this observation to score actions by the alignment between their predicted feature displacement and the goal direction\.

Empirically, DAF improves aggregate performance across offline GCRL benchmarks and is especially effective in manipulation tasks where local directional choices are important\. Overall, the results suggest that dual representations should be used not only as global value maps, but also as local advantage fields for goal\-conditioned policy extraction\.

## References

- \[1\]\(2021\)Deep reinforcement learning at the edge of the statistical precipice\.InAdvances in Neural Information Processing Systems \(NeurIPS\),Cited by:[Appendix C](https://arxiv.org/html/2606.04188#A3.SS0.SSS0.Px4.p1.1),[Appendix D](https://arxiv.org/html/2606.04188#A4.p1.1),[Figure 5](https://arxiv.org/html/2606.04188#S5.F5),[Figure 5](https://arxiv.org/html/2606.04188#S5.F5.4.2),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px6.p1.1)\.
- \[2\]H\. Ahn, H\. Choi, J\. Han, and T\. Moon\(2025\)Option\-aware temporally abstracted value for offline goal\-conditioned reinforcement learning\.External Links:2505\.12737,[Link](https://arxiv.org/abs/2505.12737)Cited by:[§B\.2](https://arxiv.org/html/2606.04188#A2.SS2.p1.2),[§4\.2](https://arxiv.org/html/2606.04188#S4.SS2.p2.1),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[3\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, O\. Pieter Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.Advances in neural information processing systems30\.Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.12)\.
- \[4\]P\. Dayan and S\. P\. Singh\(1995\)Improving policies without measuring merits\.InAdvances in Neural Information Processing Systems,G\. Tesauro, D\. Touretzky, and T\. Leen \(Eds\.\),Vol\.8\.External Links:[Link](https://proceedings.neurips.cc/paper/1995/hash/208e43f0e45c4c78cafadb83d2888cb6-Abstract.html)Cited by:[§3](https://arxiv.org/html/2606.04188#S3.SS0.SSS0.Px1.p2.5),[§4](https://arxiv.org/html/2606.04188#S4.p1.14)\.
- \[5\]B\. Eysenbach, R\. Salakhutdinov, and S\. Levine\(2022\)Contrastive learning as goal\-conditioned reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.35\.Cited by:[§B\.5](https://arxiv.org/html/2606.04188#A2.SS5.p1.3),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p1.1),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[6\]D\. Ghosh, C\. A\. Bhateja, and S\. Levine\(2023\)Reinforcement learning from passive data via latent intentions\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 11321–11339\.Cited by:[§B\.1](https://arxiv.org/html/2606.04188#A2.SS1.p7.1)\.
- \[7\]V\. Giammarino and A\. H\. Qureshi\(2026\)Goal reaching with eikonal\-constrained hierarchical quasimetric reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=5WhsCB0Vty)Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10)\.
- \[8\]D\. Hendrycks and K\. Gimpel\(2016\)Gaussian error linear units \(gelus\)\.arXiv preprint arXiv:1606\.08415\.Cited by:[Table 4](https://arxiv.org/html/2606.04188#A1.T4.9.12.3.2)\.
- \[9\]Z\. Hong, G\. Yang, and P\. Agrawal\(2022\)Bilinear value networks\.InInternational Conference on Learning Representations,Note:arXiv:2204\.13695Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p4.4)\.
- \[10\]K\. Ke, Q\. Lin, Z\. Liu, S\. He, and C\. Yu\(2025\)Conservative offline goal\-conditioned implicit v\-learning\.InForty\-second International Conference on Machine Learning,External Links:[Link](https://openreview.net/forum?id=5ryn8tYWHL)Cited by:[§B\.4](https://arxiv.org/html/2606.04188#A2.SS4.p1.3),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[11\]K\. Ke, Z\. Ruan, S\. Tan, and W\. Wu\(2025\)Hierarchical quasimetric reinforcement learning\.InProceedings of the 2025 International Conference on Machine Learning and Neural Networks,pp\. 34–41\.Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10)\.
- \[12\]D\. P\. Kingma and J\. Ba\(2015\)Adam: a method for stochastic optimization\.InInternational Conference on Learning Representations \(ICLR\),Cited by:[Table 4](https://arxiv.org/html/2606.04188#A1.T4.9.11.2.2)\.
- \[13\]I\. Kostrikov, A\. Nair, and S\. Levine\(2022\)Offline reinforcement learning with implicit Q\-learning\.InInternational Conference on Learning Representations,Cited by:[§B\.1](https://arxiv.org/html/2606.04188#A2.SS1.p1.2),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.1),[§4\.1](https://arxiv.org/html/2606.04188#S4.SS1.p1.11),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[14\]M\. Liu, M\. Zhu, and W\. Zhang\(2022\)Goal\-conditioned reinforcement learning: problems and solutions\.arXiv preprint arXiv:2201\.08299\.Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p1.1)\.
- \[15\]J\. Y\. Ma, J\. Yan, D\. Jayaraman, and O\. Bastani\(2022\)Offline goal\-conditioned reinforcement learning viaff\-advantage regression\.Advances in neural information processing systems35,pp\. 310–323\.Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p1.1)\.
- \[16\]G\. Matheron, N\. Perrin, and O\. Sigaud\(2020\)Understanding the impact of the max operation in value\-based deep reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.33\.Cited by:[§4\.1](https://arxiv.org/html/2606.04188#S4.SS1.p1.11)\.
- \[17\]V\. Myers, B\. C\. Zheng, B\. Eysenbach, and S\. Levine\(2025\)Offline goal\-conditioned reinforcement learning with quasimetric representations\.arXiv preprint arXiv:2509\.20478\.Cited by:[§B\.3](https://arxiv.org/html/2606.04188#A2.SS3.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10)\.
- \[18\]V\. Myers, C\. Zheng, A\. Dragan, S\. Levine, and B\. Eysenbach\(2024\)Learning temporal distances: contrastive successor features can provide a metric structure for decision\-making\.arXiv preprint arXiv:2406\.17098\.Cited by:[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[19\]S\. Park, K\. Frans, B\. Eysenbach, and S\. Levine\(2025\)OGBench: benchmarking offline goal\-conditioned RL\.InInternational Conference on Learning Representations,Note:arXiv:2410\.20092Cited by:[Appendix A](https://arxiv.org/html/2606.04188#A1.p1.1),[Appendix C](https://arxiv.org/html/2606.04188#A3.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.20),[§3\.1](https://arxiv.org/html/2606.04188#S3.SS1.p1.1),[§4\.1](https://arxiv.org/html/2606.04188#S4.SS1.p1.11),[§5](https://arxiv.org/html/2606.04188#S5.p1.1)\.
- \[20\]S\. Park, K\. Frans, D\. Mann, B\. Eysenbach, A\. Kumar, and S\. Levine\(2025\)Horizon reduction makes rl scalable\.arXiv preprint arXiv:2506\.04168\.Cited by:[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[21\]S\. Park, D\. Ghosh, B\. Eysenbach, and S\. Levine\(2023\)HIQL: offline goal\-conditioned RL with latent states as actions\.InAdvances in Neural Information Processing Systems,Note:arXiv:2307\.11949Cited by:[Appendix A](https://arxiv.org/html/2606.04188#A1.p2.1),[§B\.2](https://arxiv.org/html/2606.04188#A2.SS2.p1.2),[item 3](https://arxiv.org/html/2606.04188#A6.I1.i3.p1.4),[§F\.2\.4](https://arxiv.org/html/2606.04188#A6.SS2.SSS4.1.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.12),[§4\.2](https://arxiv.org/html/2606.04188#S4.SS2.p2.1),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[22\]S\. Park, D\. Mann, and S\. Levine\(2025\)Dual goal representations\.arXiv preprint arXiv:2510\.06714\.Cited by:[Appendix A](https://arxiv.org/html/2606.04188#A1.p1.1),[Appendix A](https://arxiv.org/html/2606.04188#A1.p2.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p4.1.1),[§5](https://arxiv.org/html/2606.04188#S5.SS0.SSS0.Px1.p1.1)\.
- \[23\]X\. B\. Peng, A\. Kumar, G\. Zhang, and S\. Levine\(2019\)Advantage\-weighted regression: simple and scalable off\-policy reinforcement learning\.External Links:1910\.00177,[Link](https://arxiv.org/abs/1910.00177)Cited by:[§B\.1](https://arxiv.org/html/2606.04188#A2.SS1.p4.5),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.12),[§4\.2](https://arxiv.org/html/2606.04188#S4.SS2.p1.5)\.
- \[24\]N\. Perrin\-Gilbert\(2024\)AFU: actor\-free critic updates in off\-policy RL for continuous control\.External Links:2404\.16159,[Link](https://arxiv.org/abs/2404.16159)Cited by:[Appendix A](https://arxiv.org/html/2606.04188#A1.p2.1),[Appendix E](https://arxiv.org/html/2606.04188#A5.p1.1),[§4\.1](https://arxiv.org/html/2606.04188#S4.SS1.p1.11)\.
- \[25\]T\. Wang, A\. Torralba, P\. Isola, and A\. Zhang\(2023\)Optimal goal\-reaching reinforcement learning via quasimetric learning\.InInternational Conference on Machine Learning,pp\. 36411–36430\.Cited by:[§B\.3](https://arxiv.org/html/2606.04188#A2.SS3.p1.1),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p2.10),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.12)\.
- \[26\]Z\. Wang, A\. Novikov, K\. Zolna, J\. S\. Merel, J\. T\. Springenberg, S\. E\. Reed, B\. Shahriari, N\. Siegel, C\. Gulcehre, N\. Heess,et al\.\(2020\)Critic regularized regression\.Advances in Neural Information Processing Systems33,pp\. 7768–7778\.Cited by:[§B\.1](https://arxiv.org/html/2606.04188#A2.SS1.p4.5),[§2](https://arxiv.org/html/2606.04188#S2.SS0.SSS0.Px1.p3.12)\.

## Appendix AImplementation and Reproducibility

Our method and baselines are implemented on top of the implementations given in OGBench\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]and Dual Goal Representations\[[22](https://arxiv.org/html/2606.04188#bib.bib6)\]codebases\. Our method is employed upon hierachy of actors, with low actor being updated by dual score \([Equation˜14](https://arxiv.org/html/2606.04188#S4.E14)\) and high actor by AWR \([Equation˜3](https://arxiv.org/html/2606.04188#S2.E3)\)\.

[Table˜4](https://arxiv.org/html/2606.04188#A1.T4)details the common hyperparameters for all methods on OGBench\.[Table˜5](https://arxiv.org/html/2606.04188#A1.T5)shows theα\\alpharegularization hyperparameter that was found to be the best for performance of DAF\. We also report the ablation studies on important architectural aspects of our proposed method: AFU Coupling\[[24](https://arxiv.org/html/2606.04188#bib.bib23)\], presence of action\-effect module \([Equation˜15](https://arxiv.org/html/2606.04188#S4.E15)\), hierarchical actor\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\]and integrating dual representations\[[22](https://arxiv.org/html/2606.04188#bib.bib6)\]upon the hierarchical backbone\.

Table 4:Network configuration for DAF on OGBench\.Table 5:Coefficientα\\alphafor each environment\\newcolumntype

R\>r

Table 6:Ablation on OGBench\. We report full DAF and four requested ablations: removing AFU coupling, removing the action\-effect model \(using direct one\-step value\-difference scoring\), removing hierarchy, and using a dual\-representation hierarchical baseline\. Results within 95% of the best value are written inbold\. Best values are highlighted inblue\.
## Appendix BRelated Works

### B\.1Goal\-conditioned Implicit Q\-Learning \(GCIQL\)

Implicit Q\-Learning \(IQL\)\[[13](https://arxiv.org/html/2606.04188#bib.bib8)\]stabilizes offline RL by avoiding queries to out\-of\-distribution \(OOD\) actions through two key components: a state\-value functionVψ​\(s\)V\_\{\\psi\}\(s\)and an action\-value functionQθ​\(s,a\)Q\_\{\\theta\}\(s,a\)\. The value functions are trained via:

ℒQ​\(θ\)=𝔼\(s,a,s′\)∼𝒟​\[\(r​\(s,a\)\+γ​Vψ​\(s′\)−Qθ​\(s,a\)\)2\],\\mathcal\{L\}\_\{Q\}\(\\theta\)=\\mathbb\{E\}\_\{\(s,a,s^\{\\prime\}\)\\sim\\mathcal\{D\}\}\\left\[\\left\(r\(s,a\)\+\\gamma V\_\{\\psi\}\(s^\{\\prime\}\)\-Q\_\{\\theta\}\(s,a\)\\right\)^\{2\}\\right\],\(17\)
ℒV​\(ψ\)=𝔼\(s,a\)∼𝒟​\[L2τ​\(Qθ¯​\(s,a\)−Vψ​\(s\)\)\],\\mathcal\{L\}\_\{V\}\(\\psi\)=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{D\}\}\\left\[L\_\{2\}^\{\\tau\}\\left\(Q\_\{\\bar\{\\theta\}\}\(s,a\)\-V\_\{\\psi\}\(s\)\\right\)\\right\],\(18\)
whereL2τ​\(x\)=\|τ−𝟙​\(x<0\)\|​x2L\_\{2\}^\{\\tau\}\(x\)=\|\\tau\-\\mathds\{1\}\(x<0\)\|x^\{2\}andτ∈\[0\.5,1\)\\tau\\in\[0\.5,1\)controls conservatism \(higherτ\\tauprioritizes optimistic returns\), andθ¯\\bar\{\\theta\}are the parameters of the target Q network\. The policyπϕ​\(a\|s\)\\pi\_\{\\phi\}\(a\|s\)is then extracted via advantage\-weighted regression \(AWR\)\[[23](https://arxiv.org/html/2606.04188#bib.bib14),[26](https://arxiv.org/html/2606.04188#bib.bib12)\]:

Jπ​\(ϕ\)=𝔼\(s,a\)∼𝒟​\[exp⁡\(β⋅A​\(s,a\)\)​log⁡πϕ​\(a\|s\)\],J\_\{\\pi\}\(\\phi\)=\\mathbb\{E\}\_\{\(s,a\)\\sim\\mathcal\{D\}\}\\left\[\\exp\\left\(\\beta\\cdot A\(s,a\)\\right\)\\log\\pi\_\{\\phi\}\(a\|s\)\\right\],\(19\)
withA​\(s,a\)=Qθ​\(s,a\)−Vψ​\(s\)A\(s,a\)=Q\_\{\\theta\}\(s,a\)\-V\_\{\\psi\}\(s\), andβ\\betais the inverse temperature parameter\.

For goal\-conditioned RL, IQL is extended to learn a goal\-conditioned state\-value functionVψ​\(s,g\)V\_\{\\psi\}\(s,g\), preserving IQL’s key advantage of stable value learning without requiring explicit Q\-function evaluations on out\-of\-distribution actions\[[6](https://arxiv.org/html/2606.04188#bib.bib13)\]\.

### B\.2Option\-aware Temporally Abstracted Value \(OTA\)

HIQL\[[21](https://arxiv.org/html/2606.04188#bib.bib7)\]addresses long horizons by introducing a hierarchy over subgoals, but still relies on flat temporal\-difference updates to a high\-level value\. OTA\[[2](https://arxiv.org/html/2606.04188#bib.bib26)\]instead bakes temporal abstraction directly into the Bellman operator by learning*option\-aware*values: for an optionoothat lastsk​\(o\)k\(o\)steps, the high\-level Bellman target becomes

V​\(s,g\)≈𝔼​\[r\(o\)​\(s,g\)\+γk​\(o\)​V​\(s′,g\)\],V\(s,g\)\\approx\\mathbb\{E\}\\bigl\[r^\{\(o\)\}\(s,g\)\+\\gamma^\{k\(o\)\}V\(s^\{\\prime\},g\)\\bigr\],\(20\)wherer\(o\)r^\{\(o\)\}is the cumulative option reward ands′s^\{\\prime\}is the option\-termination state\. Each update contracts the effective horizon fromd⋆​\(s,g\)d^\{\\star\}\(s,g\)to roughlyd⋆​\(s,g\)/k​\(o\)d^\{\\star\}\(s,g\)/k\(o\), so value differences and the corresponding high\-level advantages are computed over multi\-step options rather than single primitive actions\. This leads to more stable high\-level signals and better long\-horizon stitching on OGBench, at the cost of committing to a particular temporal abstraction schedule\.

### B\.3Quasimetric representations and MQE\-style methods

Recent work views goal\-conditioned value learning as estimating an asymmetric “distance”d​\(s,g\)d\(s,g\)between states and goals\. Quasimetric approaches\[[25](https://arxiv.org/html/2606.04188#bib.bib11),[17](https://arxiv.org/html/2606.04188#bib.bib3)\]directly fit such distances with multistep returns: instead of bootstrapping only from immediate successors, they regress

dθ​\(s,g\)≈𝔼​\[∑t=0K−1c​\(st,g\)\+dθ​\(sK,g\)\|s0=s\]d\_\{\\theta\}\(s,g\)\\approx\\mathbb\{E\}\\Bigl\[\\sum\_\{t=0\}^\{K\-1\}c\(s\_\{t\},g\)\+d\_\{\\theta\}\(s\_\{K\},g\)\\,\\Bigm\|\\,s\_\{0\}=s\\Bigr\]\(21\)for random horizonsKK, while encouraging triangle\-like inequalitiesdθ​\(s,g\)≤dθ​\(s,g~\)\+dθ​\(g~,g\)d\_\{\\theta\}\(s,g\)\\leq d\_\{\\theta\}\(s,\\tilde\{g\}\)\+d\_\{\\theta\}\(\\tilde\{g\},g\)for sampled pivotsg~\\tilde\{g\}\. This multistep quasimetric estimation \(MQE\) improves horizon generalization—including long\-horizon stitching in visual domains—but typically requires stronger structural assumptions on the value landscape than local TD methods and can be sensitive to misspecification of the quasimetric prior\.

### B\.4Conservative goal\-conditioned implicit V\-learning \(GCIVL\)

GCIQL\-style methods extend IQL to goal\-conditioned settings but can overestimate values for*unconnected*state–goal pairs produced by cross\-trajectory pairing\. GCIVL\[[10](https://arxiv.org/html/2606.04188#bib.bib19)\]introduces conservative penalties on such pairs together with a quasimetric formulation\. Concretely, for a learned value or distancevθ​\(s,g\)v\_\{\\theta\}\(s,g\)and a connectivity indicatorc​\(s,g\)∈\{0,1\}c\(s,g\)\\in\\\{0,1\\\}\(reachable from𝒟\\mathcal\{D\}\), the GCIVL loss augments Bellman terms with

ℒcons​\(θ\)=λ​𝔼\(s,g\)∼ppair​\[\(1−c​\(s,g\)\)​\(max⁡\{0,vθ​\(s,g\)−δ\}\)2\],\\mathcal\{L\}\_\{\\mathrm\{cons\}\}\(\\theta\)=\\lambda\\,\\mathbb\{E\}\_\{\(s,g\)\\sim p\_\{\\mathrm\{pair\}\}\}\\bigl\[\(1\-c\(s,g\)\)\\,\\bigl\(\\max\\\{0,v\_\{\\theta\}\(s,g\)\-\\delta\\\}\\bigr\)^\{2\}\\bigr\],\(22\)penalizing large estimates on likely\-unreachable pairs\. This improves robustness on goal\-stitching tasks in OGBench, but depends on correctly identifying or regularizing unreachable pairs and still operates on scalar values rather than local action\-effect structure\.

### B\.5Contrastive representation learning \(CRL\)

Contrastive RL methods treat goal\-conditioned control as a representation learning problem: they learn embeddings so that inner products between state\(\-action\) and goal features approximate a goal\-conditioned value or reachability score\[[5](https://arxiv.org/html/2606.04188#bib.bib15)\]\. A typical loss takes the form

ℒCRL=−𝔼​\[log⁡exp⁡\(ϕ​\(s,a\)⊤​ψ​\(g\+\)/τ\)∑g′∈𝒩exp⁡\(ϕ​\(s,a\)⊤​ψ​\(g′\)/τ\)\],\\mathcal\{L\}\_\{\\mathrm\{CRL\}\}=\-\\mathbb\{E\}\\Biggl\[\\log\\frac\{\\exp\\bigl\(\\phi\(s,a\)^\{\\top\}\\psi\(g^\{\+\}\)/\\tau\\bigr\)\}\{\\sum\_\{g^\{\\prime\}\\in\\mathcal\{N\}\}\\exp\\bigl\(\\phi\(s,a\)^\{\\top\}\\psi\(g^\{\\prime\}\)/\\tau\\bigr\)\}\\Biggr\],\(23\)where\(s,a,g\+\)\(s,a,g^\{\+\}\)is a positive triple and𝒩\\mathcal\{N\}is a set of negatives\. Policies then act by choosing actions whose embeddings are closest to the goal embedding\. These approaches can learn powerful, task\-agnostic representations from unlabeled trajectories, but the contrastive loss is global rather than local in the sense of our dual advantage field: it encourages correct ordering over large batches of positive and negative pairs without explicitly privileging one\-step action\-induced displacements in representation space\.

## Appendix CAdditional Environment and Evaluation Details

##### OGBench environments\.

We evaluate on goal\-conditioned offline reinforcement learning tasks from OGBench\[[19](https://arxiv.org/html/2606.04188#bib.bib2)\]\. OGBench is designed to test several capabilities that are central to offline GCRL, including long\-horizon reasoning, trajectory stitching, generalization to unseen goals, robustness to suboptimal data, and control under stochasticity\. In our main experiments, we focus on the state\-based locomotion and manipulation tasks used in prior work\.

The locomotion tasks include maze\-style navigation domains such aspointmaze,antmaze, andhumanoidmaze, as well asantsoccer\. These tasks require the agent to reach target goal states from diverse initial configurations using only offline data\. The difficulty varies with maze size, agent morphology, and dataset coverage\. In particular,humanoidmazerequires full\-body control and therefore combines low\-level locomotion with long\-horizon navigation, whileantsocceradditionally requires controlling a ball while navigating\.

The manipulation tasks includecube,scene, andpuzzle\. Thecubeenvironments test basic object manipulation through pick\-and\-place, stacking, swapping, and rearrangement of colored cubes\. Thesceneenvironment contains multiple interacting objects, such as a cube, drawer, window, and buttons, and therefore requires sequencing several atomic behaviors to achieve the desired goal configuration\. Thepuzzleenvironments instantiate a robotic version of the Lights Out puzzle, where pressing one button changes the state of neighboring buttons\. These tasks are particularly challenging because the agent must combine continuous robotic control with combinatorial generalization over many possible configurations\.

##### Dataset variants\.

For each environment, OGBench provides multiple dataset variants that differ in coverage, trajectory quality, and the extent to which successful behavior can be recovered directly from the dataset\. In maze\-style locomotion tasks,navigatedatasets are collected from noisy expert policies that traverse the environment, whilestitchdatasets contain shorter trajectory segments and require the policy to compose partial behaviors into longer goal\-reaching trajectories\. Some locomotion domains also provideexploredatasets, which contain highly exploratory and substantially suboptimal trajectories\.

For manipulation tasks, OGBench providesplayandnoisydatasets\. Theplaydatasets contain natural interaction trajectories generated by scripted policies with temporally correlated behavior\. These datasets often contain useful local skills but do not necessarily demonstrate each evaluation task end\-to\-end\. Thenoisydatasets are collected with larger, less structured exploration noise, which increases state\-action coverage but also makes the data more suboptimal\. Together, these dataset variants test whether an offline GCRL method can learn useful local behaviors, compose them over long horizons, and remain robust when the data are imperfect or only partially aligned with the evaluation goals\.

##### Evaluation protocol\.

We follow the standard OGBench protocol and report success\-based performance on each task\. For a methodmm, environmentee, and random seedrr, letsm,e,r∈\[0,1\]s\_\{m,e,r\}\\in\[0,1\]denote the resulting success rate, averaged over the evaluation episodes and goals for that environment\. Higher values indicate better goal\-reaching performance\. Unless otherwise stated, all methods are trained purely offline on the provided datasets and are evaluated without additional environment interaction during training\.

##### Aggregate metrics with RLiable\.

In addition to per\-environment results, we report aggregate statistics using the RLiable evaluation framework\[[1](https://arxiv.org/html/2606.04188#bib.bib27)\]\. RLiable is useful in the few\-seed regime because it summarizes performance across tasks while also quantifying uncertainty with stratified\-bootstrap confidence intervals\. Importantly, RLiable does not discard “noisy” runs or remove experiments\. Instead, it estimates how sensitive aggregate conclusions are to the finite set of tasks and random seeds\.

LetSm=\{sm,e,r\}e,rS\_\{m\}=\\\{s\_\{m,e,r\}\\\}\_\{e,r\}denote the collection of scores for methodmmacross environments and seeds\. We report the following aggregate metrics:

Mean​\(m\)\\displaystyle\\mathrm\{Mean\}\(m\)=1\|ℰ\|​\|ℛ\|​∑e∈ℰ∑r∈ℛsm,e,r,\\displaystyle=\\frac\{1\}\{\|\\mathcal\{E\}\|\|\\mathcal\{R\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\}\\sum\_\{r\\in\\mathcal\{R\}\}s\_\{m,e,r\},\(24\)Median​\(m\)\\displaystyle\\mathrm\{Median\}\(m\)=median⁡\(\{sm,e,r\}e,r\),\\displaystyle=\\operatorname\{median\}\\left\(\\\{s\_\{m,e,r\}\\\}\_\{e,r\}\\right\),\(25\)IQM​\(m\)\\displaystyle\\mathrm\{IQM\}\(m\)=mean⁡\(\{sm,e,r:sm,e,r​lies between the​25​th and​75​th percentiles\}\),\\displaystyle=\\operatorname\{mean\}\\left\(\\\{s\_\{m,e,r\}:s\_\{m,e,r\}\\text\{ lies between the \}25\\text\{th and \}75\\text\{th percentiles\}\\\}\\right\),\(26\)OptimalityGap​\(m\)\\displaystyle\\mathrm\{OptimalityGap\}\(m\)=1\|ℰ\|​\|ℛ\|​∑e∈ℰ∑r∈ℛmax⁡\(0,1−sm,e,r\)\.\\displaystyle=\\frac\{1\}\{\|\\mathcal\{E\}\|\|\\mathcal\{R\}\|\}\\sum\_\{e\\in\\mathcal\{E\}\}\\sum\_\{r\\in\\mathcal\{R\}\}\\max\(0,1\-s\_\{m,e,r\}\)\.\(27\)The interquartile mean \(IQM\) averages the middle50%50\\%of outcomes, making it less sensitive to extreme outlier tasks than the mean, while being more statistically efficient than the median\. The optimality gap measures the average shortfall from the maximum normalized score of11; thus, lower values are better\. Since our scores are success rates in\[0,1\]\[0,1\], the optimality gap is directly interpretable as the average remaining failure mass\. If scores are reported as percentages, they are first divided by100100before computing the RLiable metrics\.

For confidence intervals, we use stratified bootstrap resampling over tasks and seeds\. Each bootstrap replicate preserves the task structure: for every environment, we resample seeds with replacement and then recompute the aggregate metric on the resampled score matrix\. The reported intervals correspond to the empirical percentiles of the bootstrap distribution\. This procedure avoids treating all scores as exchangeable independent samples and prevents environments with more runs from dominating the uncertainty estimate\.

## Appendix DAdditional Results

We include the additional Rliable\[[1](https://arxiv.org/html/2606.04188#bib.bib27)\]plots in[Figure˜6](https://arxiv.org/html/2606.04188#A4.F6)\.

![Refer to caption](https://arxiv.org/html/2606.04188v1/x4.png)Figure 6:Rliable Probability of Improvement\.
## Appendix EAFU\-style coupling of the bilinear value and dual score

This section provides the actor\-free coupling we use between the bilinear valueVθ​\(s,g\)=ψθ​\(s\)⊤​ϕθ​\(g\)V\_\{\\theta\}\(s,g\)=\\psi\_\{\\theta\}\(s\)^\{\\top\}\\phi\_\{\\theta\}\(g\)and the dual score, following the separation of roles emphasized byPerrin\-Gilbert \[[24](https://arxiv.org/html/2606.04188#bib.bib23)\]\. The policy parameters do not receive gradients through this objective; policy learning uses only the weighted regression step\.

##### Surrogate dual score for the coupling\.

The main text defines the raw dual scorezθz\_\{\\theta\}in \([14](https://arxiv.org/html/2606.04188#S4.E14)\)\. In the AFU objective below it is convenient to use a non\-positive surrogate

A~θ​\(s,a,g\):=h​\(zθ​\(s,a,g\)\),h:ℝ→\(−∞,0\],\\widetilde\{A\}\_\{\\theta\}\(s,a,g\)\\;:=\\;h\\\!\\left\(z\_\{\\theta\}\(s,a,g\)\\right\),\\qquad h:\\mathbb\{R\}\\to\(\-\\infty,0\],\(28\)wherehhis any monotone transformation used in implementation to keep the coupling term bounded on the optimistic side while preserving action ordering\. In our experiments we usesoftplusfunction\. The samezθz\_\{\\theta\}can still be used directly in advantage\-weighted regression, as in the main text; \([28](https://arxiv.org/html/2606.04188#A5.E28)\) is only required for the piecewise coupling withVθV\_\{\\theta\}\.

##### Scalar Bellman target\.

Let

T​\(s,a,g\):=r​\(s,g\)\+γ​Vθtgt​\(s′,g\),T\(s,a,g\)\\;:=\\;r\(s,g\)\+\\gamma\\,V^\{\\mathrm\{tgt\}\}\_\{\\theta\}\(s^\{\\prime\},g\)\\,,\(29\)withVθtgtV^\{\\mathrm\{tgt\}\}\_\{\\theta\}a slowly updated target network for the bilinear head\.

##### Conditional scaling ofVθV\_\{\\theta\}\.

LetU=𝟙​\[Vθ\+A~θ<T\]U=\\mathbbm\{1\}\[V\_\{\\theta\}\+\\widetilde\{A\}\_\{\\theta\}<T\]andρ∈\(0,1\)\\rho\\in\(0,1\)\. Define

V~:=\(1−ρ​U\)​Vθ\+ρ​U​stopgrad​\(Vθ\)\.\\widetilde\{V\}\\;:=\\;\(1\-\\rho U\)\\,V\_\{\\theta\}\+\\rho U\\,\\mathrm\{stopgrad\}\(V\_\{\\theta\}\)\\,\.\(30\)When the optimistic sumVθ\+A~θV\_\{\\theta\}\+\\widetilde\{A\}\_\{\\theta\}falls short of the Bellman targetTT, the mask down\-weights direct updates toVθV\_\{\\theta\}so thatA~θ\\widetilde\{A\}\_\{\\theta\}can absorb slack in the near\-optimistic regime\.

##### Piecewise coupling loss\.

Withx=V~−Tx=\\widetilde\{V\}\-Tandy=A~θy=\\widetilde\{A\}\_\{\\theta\}, set

Z​\(x,y\)=\{\(x\+y\)2,x≥0,x2\+y2,x<0\.Z\(x,y\)\\;=\\;\\begin\{cases\}\(x\+y\)^\{2\}\\,,&x\\geq 0\\,,\\\\ x^\{2\}\+y^\{2\}\\,,&x<0\\,\.\\end\{cases\}\(31\)Training minimizes𝔼𝒟​\[Z​\(x,y\)\]\\mathbb\{E\}\_\{\\mathcal\{D\}\}\[Z\(x,y\)\]jointly over the parameters ofVθV\_\{\\theta\}\(equivalentlyψθ\\psi\_\{\\theta\}and, where tied,ϕθ\\phi\_\{\\theta\}\) and of the heads that defineA~θ\\widetilde\{A\}\_\{\\theta\}\(includinguξu\_\{\\xi\}andϕθ\\phi\_\{\\theta\}as used inzθz\_\{\\theta\}\)\. The asymmetric split betweenx≥0x\\geq 0andx<0x<0mirrors the AFU construction: pessimistic errors onVVand the dual score are not forced to cancel spuriously when the backup is optimistic\.

##### Feature dynamics auxiliary loss\.

The lossℒae\\mathcal\{L\}\_\{\\mathrm\{ae\}\}in \([15](https://arxiv.org/html/2606.04188#S4.E15)\) complements the coupling above: the AFU\-style term enforces Bellman consistency betweenVθV\_\{\\theta\}andA~θ\\widetilde\{A\}\_\{\\theta\}, whileℒae\\mathcal\{L\}\_\{\\mathrm\{ae\}\}groundsuξu\_\{\\xi\}in explicit one\-step feature dynamics on the offline dataset\.

## Appendix FTheoretical Analysis

This section establishes two complementary properties of Dual Advantage Fields \(DAF\)\. First, we show that under exact representability DAF recovers the true Bellman advantage and therefore constitutes a valid policy‑improvement operator \(Section[F\.1](https://arxiv.org/html/2606.04188#A6.SS1)\)\. Second, we analyse a didactic 1‑D example and prove that, even when the learned goal embedding is corrupted by noise in irrelevant directions, DAF’s local advantage remains significantly more robust than both flat and hierarchical value‑difference extraction \(Section[F\.2](https://arxiv.org/html/2606.04188#A6.SS2)\)\.

### F\.1DAF as exact policy‑improvement signal

Fix a goalggand consider the goal‑conditioned MDP with rewardrg​\(s,a\):=r​\(s,a,g\)r\_\{g\}\(s,a\):=r\(s,a,g\)\. For a policyπ\\pi, define the usual Bellman advantage

Aπ​\(s,a,g\):=𝔼s′∼P\(⋅∣s,a\)​\[r​\(s,a,g\)\+γ​Vπ​\(s′,g\)−Vπ​\(s,g\)\]\.A^\{\\pi\}\(s,a,g\):=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\left\[r\(s,a,g\)\+\\gamma V^\{\\pi\}\(s^\{\\prime\},g\)\-V^\{\\pi\}\(s,g\)\\right\]\.This is the relative quantity that drives policy improvement: only the ordering of actions at a given state matters, not the absolute level ofVπV^\{\\pi\}\.

Assume that the policy value is realisable by the bilinear dual field,

Vπ​\(s,g\)=ψ​\(s\)⊤​ϕ​\(g\),V^\{\\pi\}\(s,g\)=\\psi\(s\)^\{\\top\}\\phi\(g\),and that the action‑effect model is exact,

u​\(s,a,g\)=𝔼s′∼P\(⋅∣s,a\)​\[γ​ψ​\(s′\)−ψ​\(s\)\]\.u\(s,a,g\)=\\mathbb\{E\}\_\{s^\{\\prime\}\\sim P\(\\cdot\\mid s,a\)\}\\left\[\\gamma\\psi\(s^\{\\prime\}\)\-\\psi\(s\)\\right\]\.Then the DAF score

Dπ​\(s,a,g\):=r​\(s,a,g\)\+u​\(s,a,g\)⊤​ϕ​\(g\)D^\{\\pi\}\(s,a,g\):=r\(s,a,g\)\+u\(s,a,g\)^\{\\top\}\\phi\(g\)equals the true goal‑conditioned advantage:

Dπ​\(s,a,g\)=Aπ​\(s,a,g\)\.D^\{\\pi\}\(s,a,g\)=A^\{\\pi\}\(s,a,g\)\.
###### Proposition F\.1\(DAF local policy improvement\)\.

Letπ\+\\pi^\{\+\}be any goal‑conditioned policy satisfying

𝔼a∼π\+\(⋅∣s,g\)​\[Dπ​\(s,a,g\)\]≥0for all​s,g\.\\mathbb\{E\}\_\{a\\sim\\pi^\{\+\}\(\\cdot\\mid s,g\)\}\\left\[D^\{\\pi\}\(s,a,g\)\\right\]\\geq 0\\qquad\\text\{for all \}s,g\.Under the realizability and exact action‑effect assumptions above,

Vπ\+​\(s,g\)≥Vπ​\(s,g\)for all​s,g\.V^\{\\pi^\{\+\}\}\(s,g\)\\geq V^\{\\pi\}\(s,g\)\\qquad\\text\{for all \}s,g\.

###### Proof\.

SinceDπ=AπD^\{\\pi\}=A^\{\\pi\}, the assumption gives𝔼a∼π\+\(⋅∣s,g\)​\[Aπ​\(s,a,g\)\]≥0\\mathbb\{E\}\_\{a\\sim\\pi^\{\+\}\(\\cdot\\mid s,g\)\}\[A^\{\\pi\}\(s,a,g\)\]\\geq 0\. This is exactly

\(Tπ\+​Vπ\)​\(s,g\)−Vπ​\(s,g\)=𝔼a∼π\+\(⋅∣s,g\)​\[Aπ​\(s,a,g\)\]≥0,\(T\_\{\\pi^\{\+\}\}V^\{\\pi\}\)\(s,g\)\-V^\{\\pi\}\(s,g\)=\\mathbb\{E\}\_\{a\\sim\\pi^\{\+\}\(\\cdot\\mid s,g\)\}\[A^\{\\pi\}\(s,a,g\)\]\\geq 0,whereTπ\+T\_\{\\pi^\{\+\}\}is the Bellman operator for policyπ\+\\pi^\{\+\}\. HenceTπ\+​Vπ≥VπT\_\{\\pi^\{\+\}\}V^\{\\pi\}\\geq V^\{\\pi\}pointwise, and by monotonicity of the Bellman operator,Tπ\+k​Vπ≥VπT\_\{\\pi^\{\+\}\}^\{k\}V^\{\\pi\}\\geq V^\{\\pi\}for everyk≥1k\\geq 1\. Takingk→∞k\\to\\inftyand using the contraction property ofTπ\+T\_\{\\pi^\{\+\}\}yieldsVπ\+≥VπV^\{\\pi^\{\+\}\}\\geq V^\{\\pi\}\. ∎

The advantage‑weighted regression \(AWR\) update used by DAF is one such improvement in the exact on‑policy case\. If

πα\+​\(a∣s,g\)=π​\(a∣s,g\)​exp⁡\(α​Dπ​\(s,a,g\)\)∑bπ​\(b∣s,g\)​exp⁡\(α​Dπ​\(s,b,g\)\),α≥0,\\pi\_\{\\alpha\}^\{\+\}\(a\\mid s,g\)=\\frac\{\\pi\(a\\mid s,g\)\\exp\(\\alpha D^\{\\pi\}\(s,a,g\)\)\}\{\\sum\_\{b\}\\pi\(b\\mid s,g\)\\exp\(\\alpha D^\{\\pi\}\(s,b,g\)\)\},\\qquad\\alpha\\geq 0,then a standard argument shows𝔼a∼πα\+​\[Dπ​\(s,a,g\)\]≥𝔼a∼π​\[Dπ​\(s,a,g\)\]=0\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\alpha\}^\{\+\}\}\[D^\{\\pi\}\(s,a,g\)\]\\geq\\mathbb\{E\}\_\{a\\sim\\pi\}\[D^\{\\pi\}\(s,a,g\)\]=0, so the AWR policy satisfies the condition of Proposition[F\.1](https://arxiv.org/html/2606.04188#A6.Thmtheorem1)\.

###### Corollary F\.2\(Exact DAF policy iteration\)\.

In a finite discounted goal‑conditioned MDP, suppose each iterationkkuses exact representations forVπkV^\{\\pi\_\{k\}\}and an exact action‑effect model, and define

πk\+1\(⋅∣s,g\)∈arg​maxπ′𝔼a∼π′\(⋅∣s,g\)\[Dπk\(s,a,g\)\]\.\\pi\_\{k\+1\}\(\\cdot\\mid s,g\)\\in\\operatorname\*\{arg\\,max\}\_\{\\pi^\{\\prime\}\}\\mathbb\{E\}\_\{a\\sim\\pi^\{\\prime\}\(\\cdot\\mid s,g\)\}\[D^\{\\pi\_\{k\}\}\(s,a,g\)\]\.Thenπk\+1\\pi\_\{k\+1\}is the standard greedy policy‑improvement step with respect toQπkQ^\{\\pi\_\{k\}\}\. Consequently, repeated exact DAF improvement is policy iteration and converges to an optimal goal‑conditioned policy\.

###### Proof\.

BecauseDπk=Aπk=Qπk−VπkD^\{\\pi\_\{k\}\}=A^\{\\pi\_\{k\}\}=Q^\{\\pi\_\{k\}\}\-V^\{\\pi\_\{k\}\}, maximisingDπkD^\{\\pi\_\{k\}\}over actions is equivalent to maximisingQπkQ^\{\\pi\_\{k\}\}\. The result follows from classical policy iteration for finite discounted MDPs, applied separately for each goalgg\. ∎

##### Relation to hierarchical policies\.

LetΠ\\Pidenote the class of all stationary goal‑conditioned primitive‑action policies, and letΠhier⊆Π\\Pi\_\{\\mathrm\{hier\}\}\\subseteq\\Pibe any hierarchically constrained class \(e\.g\. subgoal or option policies\)\. The optimal primitive‑action policyπ⋆∈arg​maxπ∈Π⁡Vπ\\pi^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{\\pi\\in\\Pi\}V^\{\\pi\}satisfies

Vπ⋆​\(s,g\)≥supπ∈ΠhierVπ​\(s,g\)for all​s,g\.V^\{\\pi^\{\\star\}\}\(s,g\)\\geq\\sup\_\{\\pi\\in\\Pi\_\{\\mathrm\{hier\}\}\}V^\{\\pi\}\(s,g\)\\quad\\text\{for all \}s,g\.Thus, in the exact realisable limit, DAF policy iteration reaches a policy that is at least as good as the best policy in any fixed hierarchical class\.

This comparison is a representational statement: hierarchy may improve learning by reducing the effective horizon, but a fixed hierarchy can also introduce subgoal‑level constraints that exclude the true optimal primitive‑action policy\. DAF instead performs improvement directly at the primitive‑action level using the local dual advantage, while preserving the long‑horizon reachability information encoded in the dual value field\.

### F\.2Robustness to learned embedding noise: a didactic example

We now turn to a more practical regime where the representation is learned from finite data and inevitably contains noise\.

#### F\.2\.1Environment and representation model

##### Line‑world dynamics\.

Consider deterministic statess∈\{0,1,…,T\}s\\in\\\{0,1,\\dots,T\\\}with a fixed goalg=T\>0g=T\>0\. Two actions are available: right \(a=\+1a=\+1,s→s\+1s\\to s\+1\) and left \(a=−1a=\-1,s→s−1s\\to s\-1\)\. The episode terminates upon reachinggg; the reward is0at the goal and−1\-1otherwise\. Hence the optimal policy always moves right fors<Ts<T, and the optimal \(negative\) value function is

V⋆​\(s,g\)=s−T,s≤T\.V^\{\\star\}\(s,g\)=s\-T,\\qquad s\\leq T\.

##### Fixed state embedding\.

The environment provides a feature mapψ:ℤ→ℝd\\psi:\\mathbb\{Z\}\\to\\mathbb\{R\}^\{d\}withd=m\+2​\(m≥0\)d=m\+2\\;\(m\\geq 0\):

ψ​\(s\)=\[s,1,f1​\(s\),…,fm​\(s\)\]⊤,\\psi\(s\)=\\bigl\[s,\\;1,\\;f\_\{1\}\(s\),\\dots,f\_\{m\}\(s\)\\bigr\]^\{\\\!\\top\},where\{fi\}i=1m\\\{f\_\{i\}\\\}\_\{i=1\}^\{m\}are boundedC2C^\{2\}functions \(or, in the discrete case, functions with well‑defined first and second differences\)\. The first two coordinates are “essential” for representing the linear optimal value; the remaining ones are*nuisance*dimensions that are irrelevant for the control task \(e\.g\., visual textures, lighting gradients\)\.

##### True goal embedding\.

The optimal value can be expressed via an inner product:

ϕ⋆​\(g\)=\[1,−T,0,…,0\]⊤⟹ψ​\(s\)⊤​ϕ⋆​\(g\)=s−T=V⋆​\(s,g\)\.\\phi^\{\\star\}\(g\)=\\bigl\[1,\\,\-T,\\,0,\\dots,0\\bigr\]^\{\\\!\\top\}\\quad\\Longrightarrow\\quad\\psi\(s\)^\{\\\!\\top\}\\phi^\{\\star\}\(g\)=s\-T=V^\{\\star\}\(s,g\)\.

#### F\.2\.2Noise model for the learned goal embedding

In offline training, the goal embeddingϕ​\(g\)\\phi\(g\)is estimated from a finite dataset\. Because the temporal‑difference loss only weakly constrains the coefficients of the nuisance coordinates \(especially if those coordinates vary slowly\), the learned embedding can accumulate significant noise along those directions\. We model this by an additive perturbation confined to the nuisance components:

ϕ​\(g\)=ϕ⋆​\(g\)\+𝜺,𝜺=\[0,0,η1,…,ηm\]⊤,\\phi\(g\)=\\phi^\{\\star\}\(g\)\+\\boldsymbol\{\\varepsilon\},\\qquad\\boldsymbol\{\\varepsilon\}=\\bigl\[0,\\,0,\\,\\eta\_\{1\},\\dots,\\eta\_\{m\}\\bigr\]^\{\\\!\\top\},whereηi∼𝒩​\(0,σi2\)\\eta\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{i\}^\{2\}\)are independent\. The essential coordinates are assumed to be learned accurately for simplicity; allowing noise there would not change the qualitative conclusions\.

Consequently the noisy value estimate at any statessis

V^​\(s,g\)=ψ​\(s\)⊤​ϕ​\(g\)=s−T\+∑i=1mηi​fi​\(s\)\.\\widehat\{V\}\(s,g\)=\\psi\(s\)^\{\\\!\\top\}\\phi\(g\)=s\-T\+\\sum\_\{i=1\}^\{m\}\\eta\_\{i\}\\,f\_\{i\}\(s\)\.
For the subgoalssubs\_\{\\text\{sub\}\}we assume the same embedding functionϕ​\(⋅\)\\phi\(\\cdot\)is applied and that its noise is independent ofϕ​\(g\)\\phi\(g\):

ϕ​\(ssub\)=ϕ⋆​\(ssub\)\+𝜺~,η~i∼𝒩​\(0,σi2\)​independent of​ηi\.\\phi\(s\_\{\\text\{sub\}\}\)=\\phi^\{\\star\}\(s\_\{\\text\{sub\}\}\)\+\\tilde\{\\boldsymbol\{\\varepsilon\}\},\\qquad\\tilde\{\\eta\}\_\{i\}\\sim\\mathcal\{N\}\(0,\\sigma\_\{i\}^\{2\}\)\\;\\text\{independent of \}\\eta\_\{i\}\.

#### F\.2\.3Action‑effect model and policy extraction rules

We assume that a separate action‑effect modelu​\(s,a\)u\(s,a\)has been trained to regress to the true one‑step feature changeψ​\(s\+a\)−ψ​\(s\)\\psi\(s\+a\)\-\\psi\(s\)and has converged to the exact quantity \(realistic because the model sees abundant transitions and the dynamics are deterministic\)\.

Thus

u​\(s,\+1\)=ψ​\(s\+1\)−ψ​\(s\),u​\(s,−1\)=ψ​\(s−1\)−ψ​\(s\)\.u\(s,\+1\)=\\psi\(s\+1\)\-\\psi\(s\),\\qquad u\(s,\-1\)=\\psi\(s\-1\)\-\\psi\(s\)\.
We compare three policy extraction methods, all built upon the same learned bilinear valueV^\\widehat\{V\}and the sameuu\.

1. 1\.Flat value‑difference\.Choose the action that leads to the highest estimated next‑state value: aV​\(s\)=arg​maxa∈\{−1,\+1\}⁡V^​\(s\+a,g\)\.a\_\{\\mathrm\{V\}\}\(s\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\\{\-1,\+1\\\}\}\\widehat\{V\}\(s\+a,g\)\.This corresponds to the implicit advantage used in HIQL’s flat baseline \(comparingV​\(s\+1,g\)V\(s\+1,g\)andV​\(s−1,g\)V\(s\-1,g\)\)\.
2. 2\.DAF local advantage\.Score each action by the inner product of its predicted feature displacement and the goal embedding \(Eq\.[14](https://arxiv.org/html/2606.04188#S4.E14)in the main paper\): aDAF​\(s\)=arg​maxa∈\{−1,\+1\}⁡u​\(s,a\)⊤​ϕ​\(g\)\.a\_\{\\mathrm\{DAF\}\}\(s\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\\{\-1,\+1\\\}\}u\(s,a\)^\{\\\!\\top\}\\phi\(g\)\.\(The sparse reward, identical for both actions, is omitted from the comparison\.\)
3. 3\.Hierarchical HIQL\.The hierarchical policy first selects a subgoal at distancek≥2k\\geq 2\(to the right,ssub=s\+ks\_\{\\text\{sub\}\}=s\+k\) by comparing values of the candidate subgoals: ssub=arg​maxx∈\{s\+k,s−k\}⁡\[V^​\(x,g\)−V^​\(s,g\)\]\.s\_\{\\text\{sub\}\}=\\operatorname\*\{arg\\,max\}\_\{x\\in\\\{s\+k,s\-k\\\}\}\\bigl\[\\widehat\{V\}\(x,g\)\-\\widehat\{V\}\(s,g\)\\bigr\]\.Subsequently a low‑level controller attempts to reach that subgoal, using the subgoal’s own embeddingϕ​\(ssub\)\\phi\(s\_\{\\text\{sub\}\}\)and the same flat value‑difference rule: aℓ​\(s\)=arg​maxa∈\{−1,\+1\}⁡V^​\(s\+a,ssub\)\.a\_\{\\ell\}\(s\)=\\operatorname\*\{arg\\,max\}\_\{a\\in\\\{\-1,\+1\\\}\}\\widehat\{V\}\(s\+a,s\_\{\\text\{sub\}\}\)\.An error occurs if either the subgoal choice is wrong or the low‑level action is wrong; we bound this with a union argument as inParket al\.\[[21](https://arxiv.org/html/2606.04188#bib.bib7), Proposition 4\.1\]\.

#### F\.2\.4Error probabilities

For any nuisance functionff, define the first and second discrete differences at statess:

Δ​f​\(s\):=f​\(s\+1\)−f​\(s−1\),Δ2​f​\(s\):=f​\(s\+1\)\+f​\(s−1\)−2​f​\(s\)\.\\Delta f\(s\):=f\(s\+1\)\-f\(s\-1\),\\qquad\\Delta^\{2\}f\(s\):=f\(s\+1\)\+f\(s\-1\)\-2f\(s\)\.
Flat value‑difference\.

ΔV​\(s\)=V^​\(s\+1,g\)−V^​\(s−1,g\)=2\+∑i=1mηi​Δ​fi​\(s\)\.\\Delta\_\{\\mathrm\{V\}\}\(s\)=\\widehat\{V\}\(s\+1,g\)\-\\widehat\{V\}\(s\-1,g\)=2\+\\sum\_\{i=1\}^\{m\}\\eta\_\{i\}\\,\\Delta f\_\{i\}\(s\)\.
DAF\.

ΔDAF​\(s\)=\(u​\(s,\+1\)−u​\(s,−1\)\)⊤​ϕ​\(g\)=2\+∑i=1mηi​Δ2​fi​\(s\)\.\\Delta\_\{\\mathrm\{DAF\}\}\(s\)=\\bigl\(u\(s,\+1\)\-u\(s,\-1\)\\bigr\)^\{\\\!\\top\}\\phi\(g\)=2\+\\sum\_\{i=1\}^\{m\}\\eta\_\{i\}\\,\\Delta^\{2\}f\_\{i\}\(s\)\.
Hierarchical high‑level\.

Δhigh​\(s\)=V^​\(s\+k,g\)−V^​\(s−k,g\)=2​k\+∑i=1mηi​\(fi​\(s\+k\)−fi​\(s−k\)\)\.\\Delta\_\{\\mathrm\{high\}\}\(s\)=\\widehat\{V\}\(s\+k,g\)\-\\widehat\{V\}\(s\-k,g\)=2k\+\\sum\_\{i=1\}^\{m\}\\eta\_\{i\}\\,\\bigl\(f\_\{i\}\(s\+k\)\-f\_\{i\}\(s\-k\)\\bigr\)\.
Hierarchical low‑level\.Conditioned on the subgoals\+ks\+kbeing selected,

Δlow​\(s\)=V^​\(s\+1,s\+k\)−V^​\(s−1,s\+k\)=2\+∑i=1mη~i​Δ​fi​\(s\)\.\\Delta\_\{\\mathrm\{low\}\}\(s\)=\\widehat\{V\}\(s\+1,s\+k\)\-\\widehat\{V\}\(s\-1,s\+k\)=2\+\\sum\_\{i=1\}^\{m\}\\tilde\{\\eta\}\_\{i\}\\,\\Delta f\_\{i\}\(s\)\.
All decision statistics are Gaussian\. LetΦ\\Phibe the standard normal c\.d\.f\.

###### Proposition F\.3\(Error probabilities\)\.

For any states∈\{1,…,T−1\}s\\in\\\{1,\\dots,T\-1\\\}and subgoal stepkk,

εflat​\(s\)\\displaystyle\\varepsilon\_\{\\mathrm\{flat\}\}\(s\)=Φ​\(−2∑iσi2​\(Δ​fi​\(s\)\)2\),\\displaystyle=\\Phi\\\!\\left\(\-\\frac\{2\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\,\\bigl\(\\Delta f\_\{i\}\(s\)\\bigr\)^\{2\}\}\}\\right\),εDAF​\(s\)\\displaystyle\\varepsilon\_\{\\mathrm\{DAF\}\}\(s\)=Φ​\(−2∑iσi2​\(Δ2​fi​\(s\)\)2\),\\displaystyle=\\Phi\\\!\\left\(\-\\frac\{2\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\,\\bigl\(\\Delta^\{2\}f\_\{i\}\(s\)\\bigr\)^\{2\}\}\}\\right\),εhigh​\(s\)\\displaystyle\\varepsilon\_\{\\mathrm\{high\}\}\(s\)=Φ​\(−2​k∑iσi2​\(fi​\(s\+k\)−fi​\(s−k\)\)2\),\\displaystyle=\\Phi\\\!\\left\(\-\\frac\{2k\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\,\\bigl\(f\_\{i\}\(s\+k\)\-f\_\{i\}\(s\-k\)\\bigr\)^\{2\}\}\}\\right\),εlow​\(s\)\\displaystyle\\varepsilon\_\{\\mathrm\{low\}\}\(s\)=Φ​\(−2∑iσi2​\(Δ​fi​\(s\)\)2\)\.\\displaystyle=\\Phi\\\!\\left\(\-\\frac\{2\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\,\\bigl\(\\Delta f\_\{i\}\(s\)\\bigr\)^\{2\}\}\}\\right\)\.The overall hierarchical error is bounded by

εhier​\(s\)≤εhigh​\(s\)\+εlow​\(s\)\.\\varepsilon\_\{\\mathrm\{hier\}\}\(s\)\\;\\leq\\;\\varepsilon\_\{\\mathrm\{high\}\}\(s\)\\;\+\\;\\varepsilon\_\{\\mathrm\{low\}\}\(s\)\.

###### Proof\.

Each decision margin is a normal random variable with the stated mean and variance; misclassification is the event “margin<0<0”\. The hierarchical bound follows from a union bound over the two decision stages, exactly as inParket al\.\[[21](https://arxiv.org/html/2606.04188#bib.bib7), Proposition 4\.1\]\. ∎

#### F\.2\.5Why DAF can be more robust

The formulas in Proposition[F\.3](https://arxiv.org/html/2606.04188#A6.Thmtheorem3)show that DAF’s noise enters through the*second differences*Δ2​fi​\(s\)\\Delta^\{2\}f\_\{i\}\(s\), whereas all value‑difference methods \(flat and low‑level\) involve the*first differences*Δ​fi​\(s\)\\Delta f\_\{i\}\(s\)\. The high‑level comparison involves the even larger spanfi​\(s\+k\)−fi​\(s−k\)f\_\{i\}\(s\+k\)\-f\_\{i\}\(s\-k\)\.

For many realistic nuisance functions, the second difference is much smaller than the first difference\. Two concrete regimes make this quantitative\.

###### Corollary F\.4\(Affine nuisance coordinates are eliminated by DAF\)\.

Iffi​\(s\)=αi​s\+βif\_\{i\}\(s\)=\\alpha\_\{i\}s\+\\beta\_\{i\}for allii, thenΔ2​fi​\(s\)=0\\Delta^\{2\}f\_\{i\}\(s\)=0for everyss; henceΔDAF​\(s\)≡2\\Delta\_\{\\mathrm\{DAF\}\}\(s\)\\equiv 2andεDAF​\(s\)=0\\varepsilon\_\{\\mathrm\{DAF\}\}\(s\)=0\. In contrast,

εflat​\(s\)=εlow​\(s\)=Φ​\(−1∑iσi2​αi2\),εhigh​\(s\)=Φ​\(−1∑iσi2​αi2\)\.\\varepsilon\_\{\\mathrm\{flat\}\}\(s\)=\\varepsilon\_\{\\mathrm\{low\}\}\(s\)=\\Phi\\\!\\left\(\-\\frac\{1\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\alpha\_\{i\}^\{2\}\}\}\\right\),\\qquad\\varepsilon\_\{\\mathrm\{high\}\}\(s\)=\\Phi\\\!\\left\(\-\\frac\{1\}\{\\sqrt\{\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\alpha\_\{i\}^\{2\}\}\}\\right\)\.Thus DAF makes*zero*mistakes regardless of the horizon, while the flat and hierarchical baselines can suffer significant error whenever∑iσi2​αi2\\sum\_\{i\}\\sigma\_\{i\}^\{2\}\\alpha\_\{i\}^\{2\}is large\.

###### Corollary F\.5\(Low‑curvature nuisance coordinates\)\.

Suppose eachfif\_\{i\}is twice differentiable with\|fi′′​\(s\)\|≤C\|f\_\{i\}^\{\\prime\\prime\}\(s\)\|\\leq Cand that over a short interval the first difference can be expressed asΔ​fi​\(s\)=2​fi′​\(s\)\+O​\(C\)\\Delta f\_\{i\}\(s\)=2f\_\{i\}^\{\\prime\}\(s\)\+O\(C\),Δ2​fi​\(s\)=2​fi′′​\(s\)\+O​\(C\)\\Delta^\{2\}f\_\{i\}\(s\)=2f\_\{i\}^\{\\prime\\prime\}\(s\)\+O\(C\)\. If the local slopefi′​\(s\)f\_\{i\}^\{\\prime\}\(s\)is large \(e\.g\., a strong linear trend\) while the curvature remains bounded, thenσDAF2​\(s\)=O​\(C2\)\\sigma\_\{\\mathrm\{DAF\}\}^\{2\}\(s\)=O\(C^\{2\}\)whereasσflat2​\(s\)=4​∑iσi2​fi′​\(s\)2\\sigma\_\{\\mathrm\{flat\}\}^\{2\}\(s\)=4\\sum\_\{i\}\\sigma\_\{i\}^\{2\}f\_\{i\}^\{\\prime\}\(s\)^\{2\}can be arbitrarily large\. ConsequentlyεDAF​\(s\)\\varepsilon\_\{\\mathrm\{DAF\}\}\(s\)stays close to zero whileεflat​\(s\)\\varepsilon\_\{\\mathrm\{flat\}\}\(s\)andεlow​\(s\)\\varepsilon\_\{\\mathrm\{low\}\}\(s\)may approach12\\frac\{1\}\{2\}\.

##### Comparison with the hierarchical baseline\.

Even with a well‑chosen subgoal stepkk, the low‑level controller still relies on first differences \(Proposition[F\.3](https://arxiv.org/html/2606.04188#A6.Thmtheorem3)\), inheriting the same vulnerability as the flat extraction\. Moreover, the high‑level stage introduces an additional source of error that scales with the span of the nuisance functions\. As a result, a*single*DAF flat policy can achieve a lower error rate than a hierarchical policy that employs two value‑difference decisions\.

##### Illustrative quantitative example\.

Letm=1m=1andf1​\(s\)=s2f\_\{1\}\(s\)=s^\{2\}\. ThenΔ​f1​\(s\)=4​s,Δ2​f1​\(s\)=2\\Delta f\_\{1\}\(s\)=4s,\\;\\Delta^\{2\}f\_\{1\}\(s\)=2, and

εDAF​\(s\)=Φ​\(−2σ1⋅2\),εflat​\(s\)=εlow​\(s\)=Φ​\(−2σ1⋅4​s\)\.\\varepsilon\_\{\\mathrm\{DAF\}\}\(s\)=\\Phi\\\!\\left\(\-\\frac\{2\}\{\\sigma\_\{1\}\\cdot 2\}\\right\),\\qquad\\varepsilon\_\{\\mathrm\{flat\}\}\(s\)=\\varepsilon\_\{\\mathrm\{low\}\}\(s\)=\\Phi\\\!\\left\(\-\\frac\{2\}\{\\sigma\_\{1\}\\cdot 4s\}\\right\)\.For a state far from the goal \(s=T−1≫1s=T\-1\\gg 1\),εflat\\varepsilon\_\{\\mathrm\{flat\}\}andεlow\\varepsilon\_\{\\mathrm\{low\}\}are close to0\.50\.5ifσ1\\sigma\_\{1\}is large, whileεDAF\\varepsilon\_\{\\mathrm\{DAF\}\}remains bounded by a constant that does not grow withTT\.

## Appendix GCompute Resources

All experiments were performed on servers with a single H100 GPU with 80 GB of GPU memory, 12 CPU cores, and 244 GB of RAM\. All metrics for the experiments were logged using the Weights & Biases platform\. Overall, the Weights & Biases project of the paper had 17,359 tracked experiments at the time of submission and used an estimated∼407\\sim 407days of GPU compute in total\.