生成式价值分类器

# Generative Value Classifier
Source: [https://arxiv.org/html/2610.22497](https://arxiv.org/html/2610.22497)
Correspondence toaswerdlo@stanford\.edu\\teaser

Figure 1:Action data provides additional supervision\.A conventional critic fitsQ⁡\(s,a\)Q\(s,a\)to a scalar targetyy\. GVC instead learns a return\-conditioned action modelp⁡\(a∣s,z\)p\(a\\mid s,z\)and a return priorp⁡\(z∣s\)p\(z\\mid s\), i\.e\., a distributional state\-value function, and normalizes their product over returns to obtainQ⁡\(s,a\)Q\(s,a\)\.###### Abstract

Q\-learning is premised on learning a function to predict the value of a state\-action pair, with the goal of using such a function to guide policy improvement\. Most approaches learn this function from scratch using scalar or distributional return targets\. This discriminative approach leaves another source of supervision for learning representations unused: the actions themselves\. Rather than directly predict the return of a state\-action pair, we ask the reverse question: how likely is the action under each possible return? This perspective provides a critic whose generative component is a return\-conditioned policy, which can share the actor’s architecture\. Training this policy supervises the critic with every action rather than with returns alone, and a squared\-error loss anchors the critic’s Q\-value to the bootstrapped target\. We evaluate our approach on both offline and online RL tasks and show gains over leading discriminative methods, including distributional RL methods\. Code:[https://github\.com/alexanderswerdlow/gvc](https://github.com/alexanderswerdlow/gvc) Project page:[https://alexanderswerdlow\.github\.io/gvc](https://alexanderswerdlow.github.io/gvc)

## 1Introduction

Value functions play a key role in sample\-efficient learning, enabling policy improvement to use past rollouts to better score newer rollouts\. However, learning good value functions particularly in sparse\-reward settings has remained a significant challenge\[[2](https://arxiv.org/html/2610.22497#bib.bib2),[24](https://arxiv.org/html/2610.22497#bib.bib24)\]\. Standard Monte\-Carlo \(MC\) approaches result in high variance and lack the ability to stitch information across trajectories whereas Temporal\-Difference \(TD\) bootstrapping addresses this problem but at the cost of compounding errors that propagate across timesteps\. Sparse\-reward settings exacerbate these issues with longer\-horizon trajectories and necessitate additional rollouts to provide accurate value estimation\.

Prior work has sought to improve value learning under sparse supervision through richer prediction tasks\. Distributional RL\[[4](https://arxiv.org/html/2610.22497#bib.bib4)\]learns the return distribution rather than a single scalar\. Other approaches use contrastive learning to predict future\-state occupancy by contrasting known future states with noisy negatives\[[16](https://arxiv.org/html/2610.22497#bib.bib16),[17](https://arxiv.org/html/2610.22497#bib.bib17),[36](https://arxiv.org/html/2610.22497#bib.bib36)\]\. These methods reformulate value learning through additional representational capacity or inductive biases; we instead ask whether modeling actions can supply additional supervision\.

Our insight is that traditional value learning fails to take advantage of a key source of supervision: the actions themselves\. Value functions areconditionedon state\-action pairs, yet are only supervised by a scalar—or at best distributional—return\. To use the actions as supervision, we learnp⁡\(a∣s,z\)p\(a\\mid s,z\)rather than directly predictingp⁡\(z∣s,a\)p\(z\\mid s,a\), wherezzdenotes the return\. With the state held fixed, Bayes’ rule combines this likelihood with a return priorp⁡\(z∣s\)p\(z\\mid s\), the return distribution given the state alone\. Normalizing their product over return bins gives a predicted return distribution whose mean isQ⁡\(s,a\)Q\(s,a\)\. We also minimize the squared error betweenQ⁡\(s,a\)Q\(s,a\)and the bootstrapped target return, which we callvalue anchoring\. This uses a discriminative loss to fit the value while retaining the generative supervision and structure\.

In summary, our main contributions are:

- •We introduce a new formulation for Q\-learning that leverages a return\-conditioned generative likelihood to implicitly model the return distribution\.
- •We develop a hybrid generative–discriminative objective that combines return\-conditioned flow matching with value anchoring\. This aligns the return predictions with the standard Bellman backup targets while retaining the generative supervision\.
- •We evaluateGVCon challenging continuous\-control tasks, including online RL benchmarks on RoboMimic and offline and offline\-to\-online RL benchmarks on OGBench, demonstrating strong performance relative to prior methods\. Our analysis demonstrates that removing this generative supervision correlates with lower performance on the same architecture\.

## 2Related Work

#### Discriminative Tasks Using Generative Models\.

Using generative models for discriminative tasks has a long history, with\[[38](https://arxiv.org/html/2610.22497#bib.bib38)\]first comparing generative and discriminative classifiers and subsequent work learning a single model with both generative and discriminative objectives\[[26](https://arxiv.org/html/2610.22497#bib.bib26)\]\. Driven by the strong performance of conditional generative models, recent work has built upon this, demonstrating that pretrained image and text models act as strong zero\-shot classifiers, such as with text\-to\-image models\[[28](https://arxiv.org/html/2610.22497#bib.bib28),[10](https://arxiv.org/html/2610.22497#bib.bib10)\]or with large language models\[[43](https://arxiv.org/html/2610.22497#bib.bib43),[25](https://arxiv.org/html/2610.22497#bib.bib25)\]\. Finally, another set of works, such as JEM\[[19](https://arxiv.org/html/2610.22497#bib.bib19)\]and HDGE\[[31](https://arxiv.org/html/2610.22497#bib.bib31)\], reinterprets standard discriminative classifiers as energy\-based models \(EBMs\) and adds maximum likelihood or contrastive losses\. Our work brings this view to reinforcement learning, where the “class” is a discretized return and the generative model is an action generation policy, thus constructing an implicit Q\-function\.

#### Generative Policies as Value Functions\.

At the same time, both reward modeling and value learning have made significant use of generative policies\. In the language domain, reward models are commonly finetuned generative language models used to provide scalar rewards\[[39](https://arxiv.org/html/2610.22497#bib.bib39)\], or to provide text\-based responses to grade actions based on a set of criteria\[[47](https://arxiv.org/html/2610.22497#bib.bib47),[49](https://arxiv.org/html/2610.22497#bib.bib49)\]\. Similar methods have been applied to robotics, leveraging pretrainedlanguagepolicies\[[34](https://arxiv.org/html/2610.22497#bib.bib34),[9](https://arxiv.org/html/2610.22497#bib.bib9),[27](https://arxiv.org/html/2610.22497#bib.bib27)\]for reward or value models, but these methods are unable to judge policyoutputs, i\.e\., actions, unlike their language domain counterparts\. Finally, a recent line of work on RL for Vision\-Language\-Action \(VLA\) models has adopted the pretrainedstaterepresentation from policies as an initialization for value functions\[[48](https://arxiv.org/html/2610.22497#bib.bib48),[32](https://arxiv.org/html/2610.22497#bib.bib32)\]\. Similar to the language reward models above—though unlike these robotics methods—GVCscores the policy’s actions directly\. A robot policy \(e\.g\., one that generates joint positions\) has no direct way to output a value that is compatible with its pretraining, unlike a language model, which can simply state one in text\. This motivatesGVC: we instead condition a generative policy on the return and score each action by its likelihood under different returns\.

#### Value Function Learning\.

A separate line of work maintains standard value\-based RL but increases the expressivity of the value function itself\. Distributional RL models the full return distribution rather than its mean\[[4](https://arxiv.org/html/2610.22497#bib.bib4)\], with quantile regression\[[12](https://arxiv.org/html/2610.22497#bib.bib12)\]and implicit quantile networks\[[11](https://arxiv.org/html/2610.22497#bib.bib11)\]increasing distributional flexibility\. Recent work replaces these heads with stronger generative architectures for added expressivity\[[14](https://arxiv.org/html/2610.22497#bib.bib14),[1](https://arxiv.org/html/2610.22497#bib.bib1)\], with Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]modeling the full return distribution with flow matching\. Another set of works instead uses a transformer backbone with self\-attention over state\-action tokens in place of an MLP: Q\-Transformer\[[5](https://arxiv.org/html/2610.22497#bib.bib5)\]exploits this to causally predict Q\-values across action dimensions and timesteps, while PAC\[[44](https://arxiv.org/html/2610.22497#bib.bib44)\]scales actor\-critic training to large models; however, naively scaling such transformer critics induces attention entropy collapse, which TQL\[[13](https://arxiv.org/html/2610.22497#bib.bib13)\]addresses by regularizing attention entropy toward a target value\. This differs from sequence\-modeling methods like Decision Transformer\[[8](https://arxiv.org/html/2610.22497#bib.bib8)\], which learn value\-like representations implicitly via return\-conditioned prediction rather than Bellman backups\. These methods differ in how they represent the value and in the network that computes it, but all of them predict a return from a state–action pair\.GVCworks in the opposite direction: it models the action given the state and a return bin, and recoversQ⁡\(s,a\)Q\(s,a\)with Bayes’ rule\. Decision Transformer\[[8](https://arxiv.org/html/2610.22497#bib.bib8)\]also models actions given a return, but it learns from MC returns\-to\-go and acts by conditioning on a desired return, whereasGVClearns from bootstrapped Bellman targets and uses its action model to score candidate actions\.

## 3Preliminaries

#### Reinforcement learning\.

We consider a discounted Markov decision process \(MDP\) specified by the tuple\{𝒮,𝒜,P,r,ρ0,γ\}\\\{\\mathcal\{S\},\\mathcal\{A\},P,r,\\rho\_\{0\},\\gamma\\\}, where𝒮\\mathcal\{S\}denotes the state space,𝒜\\mathcal\{A\}the action space,P⁡\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)the state\-transition distribution,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}the one\-step reward function,ρ0​\(s\)\\rho\_\{0\}\(s\)the distribution over initial states, andγ∈\[0,1\)\\gamma\\in\[0,1\)the discount factor\. A policyπ⁡\(a∣s\)\\pi\(a\\mid s\)induces the discounted returnGt=∑k≥0γk​r​\(st\+k,at\+k\)G\_\{t\}=\\sum\_\{k\\geq 0\}\\gamma^\{k\}r\(s\_\{t\+k\},a\_\{t\+k\}\), and the goal of RL is to find a policy that maximizes𝔼π​\[G0\]\\mathbb\{E\}\_\{\\pi\}\\left\[G\_\{0\}\\right\]\. The associated action\-value function isQπ\(s,a\)=𝔼π\[Gt∣st=s,at=a\]Q^\{\\pi\}\(s,a\)=\\mathbb\{E\}\_\{\\pi\}\\left\[G\_\{t\}\\mid s\_\{t\}=s,a\_\{t\}=a\\right\]\. Distributional RL models the conditional distribution of the returnGtG\_\{t\}, rather than its expectationQπ​\(s,a\)Q^\{\\pi\}\(s,a\)\. We use a categorical approximation with support𝒵=\{z1,…,zK\}\\mathcal\{Z\}=\\\{z\_\{1\},\\ldots,z\_\{K\}\\\}and we refer to each support pointziz\_\{i\}as a return bin\. Forz∈𝒵z\\in\\mathcal\{Z\},p⁡\(z∣s,a\)p\(z\\mid s,a\)denotes the probability mass atzz\.

In this paper, we study how to learn better value functions, asking whether the actions themselves, rather than only the scalar return, can serve as supervision forQQ, and how this affects downstream control performance\. We focus on off\-policy RL, where the Q\-function estimates the discounted return of the policy given a state and action, and is learned from a replay buffer𝒟\\mathcal\{D\}of one\-step transitions\(s,a,r,s′,m\)\(s,a,r,s^\{\\prime\},m\), withm∈\{0,1\}m\\in\\\{0,1\\\}masking bootstrapping at terminal transitions\. The critic is trained with TD learning:

ℒ=𝔼\(s,a,r,s′,m\)∼𝒟​\[\(r\+γ​m​Q¯​\(s′,a~′\)−Q⁡\(s,a\)\)2\],\\mathcal\{L\}=\\mathbb\{E\}\_\{\(s,a,r,s^\{\\prime\},m\)\\sim\\mathcal\{D\}\}\\left\[\\left\(r\+\\gamma\\,m\\,\\bar\{Q\}\(s^\{\\prime\},\\tilde\{a\}^\{\\prime\}\)\-Q\(s,a\)\\right\)^\{2\}\\right\],\(1\)whereQ¯\\bar\{Q\}is a frozen target copy of the criticQQanda~′∼π\(⋅∣s′\)\\tilde\{a\}^\{\\prime\}\\sim\\pi\(\\cdot\\mid s^\{\\prime\}\)is the next action selected by the RL policy\.

#### Flow matching\.

A conditional flow transforms a simple base variablex0∼𝒩⁡\(0,I\)x\_\{0\}\\sim\\mathcal\{N\}\(0,I\)into datax1∼pdata\(⋅∣c\)x\_\{1\}\\sim p\_\{\\mathrm\{data\}\}\(\\cdot\\mid c\)\. With flow timeλ∼𝒰⁡\[0,1\]\\lambda\\sim\\mathcal\{U\}\[0,1\], we use the straight path

xλ=\(1−λ\)​x0\+λ​x1,uλ=x1−x0,x\_\{\\lambda\}=\(1\-\\lambda\)x\_\{0\}\+\\lambda x\_\{1\},\\qquad u\_\{\\lambda\}=x\_\{1\}\-x\_\{0\},\(2\)and train a vector fieldvθ​\(c,xλ,λ\)v\_\{\\theta\}\(c,x\_\{\\lambda\},\\lambda\)with

ℒFM​\(θ\)=𝔼⁡\[‖vθ​\(c,xλ,λ\)−uλ‖22\]\.\\mathcal\{L\}\_\{\\mathrm\{FM\}\}\(\\theta\)=\\mathbb\{E\}\\\!\\left\[\\left\\\|v\_\{\\theta\}\(c,x\_\{\\lambda\},\\lambda\)\-u\_\{\\lambda\}\\right\\\|\_\{2\}^\{2\}\\right\]\.\(3\)Sampling integratesx˙λ=vθ​\(c,xλ,λ\)\\dot\{x\}\_\{\\lambda\}=v\_\{\\theta\}\(c,x\_\{\\lambda\},\\lambda\)from Gaussian noise atλ=0\\lambda=0to data atλ=1\\lambda=1\.

#### Generative Flow Classifier\.

A class\-conditional flow can also classify a fixed observationxx\. For each classcc, we noisexxalong the same path and estimate its flow residual

Eθ​\(x,c\)=1\|ℬ\|​∑\(ϵ,λ\)∈ℬ1D​‖vθ​\(c,\(1−λ\)​ϵ\+λ​x,λ\)−\(x−ϵ\)‖22\.E\_\{\\theta\}\(x,c\)=\\frac\{1\}\{\|\\mathcal\{B\}\|\}\\sum\_\{\(\\epsilon,\\lambda\)\\in\\mathcal\{B\}\}\\frac\{1\}\{D\}\\left\\\|v\_\{\\theta\}\\\!\\left\(c,\(1\-\\lambda\)\\epsilon\+\\lambda x,\\lambda\\right\)\-\(x\-\\epsilon\)\\right\\\|\_\{2\}^\{2\}\.\(4\)This term is a likelihood\-like energy and not an exact normalizing\-flow density\. Reusing the same noise–time pairsℬ\\mathcal\{B\}for every class gives the proxy, with inverse temperatureβ\>0\\beta\>0:

p~θ​\(c∣x\)=softmaxc​\(−β​Eθ​\(x,c\)\+log⁡p⁡\(c\)\)\.\\widetilde\{p\}\_\{\\theta\}\(c\\mid x\)=\\mathrm\{softmax\}\_\{c\}\\\!\\left\(\-\\beta E\_\{\\theta\}\(x,c\)\+\\log p\(c\)\\right\)\.\(5\)
Given a uniform class prior, the classifier selects the class with the smallest noising error\. This is the flow\-matching analogue of diffusion classifiers, which compare denoising errors across conditions\[[28](https://arxiv.org/html/2610.22497#bib.bib28),[10](https://arxiv.org/html/2610.22497#bib.bib10)\]\.

## 4Generative Value Classifier

Figure 2:Generative Value Classifier at inference time\.The actor proposes candidate actions \(left\)\. For each action, we compute the return\-conditioned action likelihood, along with the return prior\. Normalizing their product over return bins \(center\) provides a return distribution whose mean givesQ⁡\(s,a\)Q\(s,a\), which we use to rank the action candidates \(right\)\.#### Overview\.

GVCis an implicit generative critic: rather than predicting the return of a state–action pair directly, it models the likelihood of an action under a set of discretized returns and inverts this relationship with Bayes’ rule\. Conceptually, this changes the question from “What is the expected return given this state–action pair?” to “How likely is this action given this state and return?”

The critic consists of two models:pθ​\(a∣s,z\)p\_\{\\theta\}\(a\\mid s,z\)assigns a likelihood to a given action under each return bin, andpψ​\(z∣s\)p\_\{\\psi\}\(z\\mid s\)predicts the return distribution from the state alone\. The latter is thereturn prior: a distributional state\-value function\. For each return bin, we multiply the action likelihood by its prior probability, normalize across bins, and take the mean to obtainQ⁡\(s,a\)Q\(s,a\)\.

A separate flow\-matching actor, trained by behavior cloning on dataset or replay actions, proposes candidate actions \([Section4\.3](https://arxiv.org/html/2610.22497#S4.SS3)\)\. The critic scores these candidates, and we execute the one with the highestQ⁡\(s,a\)Q\(s,a\)\.[Section4\.1](https://arxiv.org/html/2610.22497#S4.SS1)summarizes scoring, training, and action selection\.

### 4\.1From discriminative values to a generative critic

A conventional distributional critic directly modelsp⁡\(z∣s,a\)p\(z\\mid s,a\)over𝒵\\mathcal\{Z\}\. We instead hold the state fixed and ask the reverse question: how likely is the actionaaunder each returnzz? Withssas context, Bayes’ rule relates the two directly:

p⁡\(z∣s,a\)=p⁡\(a∣s,z\)​p​\(z∣s\)p⁡\(a∣s\)∝p⁡\(a∣s,z\)​p​\(z∣s\),p\(z\\mid s,a\)=\\frac\{p\(a\\mid s,z\)\\,p\(z\\mid s\)\}\{p\(a\\mid s\)\}\\propto p\(a\\mid s,z\)\\,p\(z\\mid s\),\(6\)where the denominatorp⁡\(a∣s\)p\(a\\mid s\)does not depend onzz, so normalizingp⁡\(a∣s,z\)​p​\(z∣s\)p\(a\\mid s,z\)\\,p\(z\\mid s\)overzzrecoversp⁡\(z∣s,a\)p\(z\\mid s,a\)\. The critic therefore needs only two models: the return\-conditioned action modelpθ​\(a∣s,z\)p\_\{\\theta\}\(a\\mid s,z\)and the return priorpψ​\(z∣s\)p\_\{\\psi\}\(z\\mid s\)\. Treating the state as part of the observation instead,p⁡\(s,a∣z\)p\(s,a\\mid z\), would require a generative model of states; conditioning onssavoids this, and we found it to perform better empirically\.

LetppandQQdenote the learned return posterior and its mean under\(θ,ψ\)\(\\theta,\\psi\), with target copiesp¯,Q¯\\bar\{p\},\\bar\{Q\}:

p⁡\(zi∣s,a\)=pθ​\(a∣s,zi\)​pψ​\(zi∣s\)∑jpθ​\(a∣s,zj\)​pψ​\(zj∣s\),Q⁡\(s,a\)=∑ip⁡\(zi∣s,a\)​zi\.p\(z\_\{i\}\\mid s,a\)=\\frac\{p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)p\_\{\\psi\}\(z\_\{i\}\\mid s\)\}\{\\sum\_\{j\}p\_\{\\theta\}\(a\\mid s,z\_\{j\}\)p\_\{\\psi\}\(z\_\{j\}\\mid s\)\},\\qquad Q\(s,a\)=\\sum\_\{i\}p\(z\_\{i\}\\mid s,a\)z\_\{i\}\.\(7\)
To evaluate Eq\. \([7](https://arxiv.org/html/2610.22497#S4.E7)\), for each binziz\_\{i\}we takepψ​\(zi∣s\)p\_\{\\psi\}\(z\_\{i\}\\mid s\)from the prior, estimatepθ​\(a∣s,zi\)p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)from the flow\-matching error ofaaconditioned on\(s,zi\)\(s,z\_\{i\}\), and then normalize over bins\. We first introduce this flow\-matching surrogate, then the objectives that train both models and align the resulting return\.

Algorithm 1\.Generative Value Classifier

1functionScoreθ,ψ\(s,as,a\)

2ϵ∼𝒩⁡\(0,Id\)\\epsilon\\sim\\mathcal\{N\}\(0,I\_\{d\}\),λ∼𝒰⁡\[0,1\]\\lambda\\sim\\mathcal\{U\}\[0,1\]⊳\\trianglerightOne draw for clarity; shared across bins

3pi←softmaxi⁡\[βret​log⁡pθ​\(a∣s,zi\)\+βprior​log⁡pψ​\(zi∣s\)\]p\_\{i\}\\leftarrow\\operatorname\{softmax\}\_\{i\}\[\\beta\_\{\\mathrm\{ret\}\}\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)\+\\beta\_\{\\mathrm\{prior\}\}\\log p\_\{\\psi\}\(z\_\{i\}\\mid s\)\]⊳\\trianglerightNLL surrogate from Eq\. \([8](https://arxiv.org/html/2610.22497#S4.E8)\)

4return\(p,∑ipi​zi\)\(p,\\sum\_\{i\}p\_\{i\}z\_\{i\}\)⊳\\trianglerightp⁡\(z∣s,a\)p\(z\\mid s,a\)andQ⁡\(s,a\)Q\(s,a\)

5

6functionTrain\(\(s,a,r,s′,m\)∼𝒟\(s,a,r,s^\{\\prime\},m\)\\sim\\mathcal\{D\}\)⊳\\triangleright𝒟\\mathcal\{D\}: offline dataset or replay buffer

7Samplea′a^\{\\prime\}givens′s^\{\\prime\}\.⊳\\trianglerighte\.g\., Best\-of\-NNsampling using policyϕ\\phi

8\(p,Q\)←\(p,Q\)\\leftarrow\{\}Scoreθ,ψ\(s,as,a\);\(p′,Q′\)←\(p^\{\\prime\},Q^\{\\prime\}\)\\leftarrow\{\}Scoreθ¯,ψ¯\{\}\_\{\\bar\{\\theta\},\\bar\{\\psi\}\}\(s′,a′s^\{\\prime\},a^\{\\prime\}\)

9p^←sg⁡\[Π𝒵​\(r\+γ​m​Z′\)\]\\widehat\{p\}\\leftarrow\\operatorname\{sg\}\[\\Pi\_\{\\mathcal\{Z\}\}\(r\+\\gamma mZ^\{\\prime\}\)\];y←sg⁡\(∑ip^i​zi\)y\\leftarrow\\operatorname\{sg\}\(\\sum\_\{i\}\\widehat\{p\}\_\{i\}z\_\{i\}\)⊳\\trianglerightPr⁡\(Z′=zi\)=pi′\\Pr\(Z^\{\\prime\}=z\_\{i\}\)=p^\{\\prime\}\_\{i\}

10ℒflow←−∑ip^ilogpθ\(a∣s,zi\)\\mathcal\{L\}\_\{\\mathrm\{flow\}\}\\leftarrow\-\\\!\\sum\_\{i\}\\widehat\{p\}\_\{i\}\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)⊳\\trianglerightCritic: Conditional Flow loss

11ℒreturn←−∑ip^ilogpψ\(zi∣s\)\\mathcal\{L\}\_\{\\mathrm\{return\}\}\\leftarrow\-\\\!\\sum\_\{i\}\\widehat\{p\}\_\{i\}\\log p\_\{\\psi\}\(z\_\{i\}\\mid s\)⊳\\trianglerightCritic: Categorical Value loss

12ℒanchor←\(Q−y\)2\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\\leftarrow\(Q\-y\)^\{2\}⊳\\trianglerightCritic: Value Anchoring loss

13ℒactor←−log⁡pϕ​\(a∣s\)\\mathcal\{L\}\_\{\\mathrm\{actor\}\}\\leftarrow\-\\\!\\log p\_\{\\phi\}\(a\\mid s\)⊳\\trianglerightActor: BC Flow loss; Eq\. \([13](https://arxiv.org/html/2610.22497#S4.E13)\)

14Minimizeℒ=ℒactor\+λF​ℒflow\+λR​ℒreturn\+λV​ℒanchor\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{actor\}\}\+\\lambda\_\{F\}\\mathcal\{L\}\_\{\\mathrm\{flow\}\}\+\\lambda\_\{R\}\\mathcal\{L\}\_\{\\mathrm\{return\}\}\+\\lambda\_\{V\}\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\.

15Update target copies\(θ¯,ψ¯\)\(\\bar\{\\theta\},\\bar\{\\psi\}\)with EMA\.

16

17functionAct\(ss\)

18Integratefϕf\_\{\\phi\}givenNNseeds,ϵ∼𝒩⁡\(0,Id\)\\epsilon\\sim\\mathcal\{N\}\(0,I\_\{d\}\), to obtain𝒜N\\mathcal\{A\}\_\{N\}\.

19returnarg⁡maxa∈𝒜N⁡Q⁡\(s,a\)\\arg\\max\_\{a\\in\\mathcal\{A\}\_\{N\}\}Q\(s,a\), withQQfromScoreθ,ψ\(s,as,a\)\.⊳\\trianglerightExecute the best action

### 4\.2Flow likelihood

We use the conditional flow\-matching error as a surrogate for the negative log\-likelihood\[[22](https://arxiv.org/html/2610.22497#bib.bib22),[37](https://arxiv.org/html/2610.22497#bib.bib37)\], writing its negative aslog⁡pθ​\(a∣s,zi\)\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)below\. The vector fieldgθ​\(s,x,λ,zi\)g\_\{\\theta\}\(s,x,\\lambda,z\_\{i\}\)is evaluated for every return bin in a single forward pass: a shared trunk encodes\(s,x,λ\)\(s,x,\\lambda\), and each bin has its corresponding learned embedding added to this before a shared head\.

−log⁡pθ​\(a∣s,zi\)=𝔼ϵ,λ​\[1d​‖gθ​\(s,\(1−λ\)​ϵ\+λ​a,λ,zi\)−\(a−ϵ\)‖22\]\.\-\\\!\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)=\\mathbb\{E\}\_\{\\epsilon,\\lambda\}\\\!\\left\[\\frac\{1\}\{d\}\\left\\\|g\_\{\\theta\}\\\!\\left\(s,\(1\-\\lambda\)\\epsilon\+\\lambda a,\\lambda,z\_\{i\}\\right\)\-\(a\-\\epsilon\)\\right\\\|\_\{2\}^\{2\}\\right\]\.\(8\)
Substituting this surrogate into Eq\. \([7](https://arxiv.org/html/2610.22497#S4.E7)\), with an exponent on each factor, gives

p⁡\(zi∣s,a\)∝pθ​\(a∣s,zi\)βret​pψ​\(zi∣s\)βprior,p\(z\_\{i\}\\mid s,a\)\\propto p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)^\{\\beta\_\{\\mathrm\{ret\}\}\}\\,p\_\{\\psi\}\(z\_\{i\}\\mid s\)^\{\\beta\_\{\\mathrm\{prior\}\}\},\(9\)whereβret\\beta\_\{\\mathrm\{ret\}\}andβprior\\beta\_\{\\mathrm\{prior\}\}are hyperparameters that weight the two factors; both would be11for exact likelihoods\. Because the flow error is only a likelihood\-like energy, we tune them instead and use a value anchor \([Section4\.3](https://arxiv.org/html/2610.22497#S4.SS3)\) to fit the meanQ⁡\(s,a\)Q\(s,a\)to the TD target, since the mean is what ranks actions\.[Section4\.1](https://arxiv.org/html/2610.22497#S4.SS1)shows a single noise–time draw for clarity; when scoring, we average over multiple stratified flow timesteps with antithetic noise, shared across return bins and action candidates to reduce variance\[[28](https://arxiv.org/html/2610.22497#bib.bib28),[6](https://arxiv.org/html/2610.22497#bib.bib6)\]\.

### 4\.3Objectives

#### Critic objectives\.

For each transition, we choose a next actiona′a^\{\\prime\}, for example via best\-of\-NNsampling\. Letp^\\widehat\{p\}be the distribution ofr\+γ​Z′r\+\\gamma Z^\{\\prime\}projected onto𝒵\\mathcal\{Z\}, whereZ′∼p¯\(⋅∣s′,a′\)Z^\{\\prime\}\\sim\\bar\{p\}\(\\cdot\\mid s^\{\\prime\},a^\{\\prime\}\)\. The projection clips shifted atoms to\[z1,zK\]\[z\_\{1\},z\_\{K\}\]\([SectionC\.1](https://arxiv.org/html/2610.22497#A3.SS1)\)\.

We use this distribution as a soft target, weighting each bin byp^i\\widehat\{p\}\_\{i\}to supervise both the return\-conditioned generative loss and the return prior:

ℒflow\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{flow\}\}=𝔼𝒟\[−∑ip^ilogpθ\(a∣s,zi\)\],\\displaystyle=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\-\\\!\\sum\_\{i\}\\widehat\{p\}\_\{i\}\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)\\right\],\(10\)ℒreturn\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{return\}\}=𝔼𝒟\[−∑ip^ilogpψ\(zi∣s\)\],\\displaystyle=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\-\\\!\\sum\_\{i\}\\widehat\{p\}\_\{i\}\\log p\_\{\\psi\}\(z\_\{i\}\\mid s\)\\right\],\(11\)whereℒflow\\mathcal\{L\}\_\{\\mathrm\{flow\}\}is just the flow\-matching loss of Eq\. \([8](https://arxiv.org/html/2610.22497#S4.E8)\) on the transition’s actionaa\(a fresh noise–time sample per transition\), with each return bin weighted by its TD\-target probabilityp^i\\widehat\{p\}\_\{i\}, andℒreturn\\mathcal\{L\}\_\{\\mathrm\{return\}\}fits the prior to the same target\. The flow error is only a surrogate, with unknown scale and bin\-dependent offsets, so these losses alone do not ensure thatQ⁡\(s,a\)Q\(s,a\)matches the target meany=∑ip^i​ziy=\\sum\_\{i\}\\widehat\{p\}\_\{i\}z\_\{i\}\. To penalize the value error directly, we add a squared\-error value anchor:

ℒanchor\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}=𝔼𝒟​\[\(Q⁡\(s,a\)−sg⁡\(y\)\)2\]\.\\displaystyle=\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\\!\\left\[\\left\(Q\(s,a\)\-\\mathrm\{sg\}\(y\)\\right\)^\{2\}\\right\]\.\(12\)The value anchor constrains the return which is used to rank actions\.

#### Actor Objective\.

GVCis independent of the policy extraction mechanism and can work with both implicit and explicit extraction methods\. For all experiments, except the ablation in[SectionA\.3](https://arxiv.org/html/2610.22497#A1.SS3), we use best\-of\-NNsampling for implicit extraction alongside a flow\-matching behavior policy trained on replay actions, both to act and to selecta′a^\{\\prime\}for the TD target;[SectionC\.1](https://arxiv.org/html/2610.22497#A3.SS1)listsNNfor each setting\.

The actor is a behavior\-cloning flow with vector fieldfϕf\_\{\\phi\}\. Its loss is−𝔼𝒟​log⁡pϕ​\(a∣s\)\-\\mathbb\{E\}\_\{\\mathcal\{D\}\}\\log p\_\{\\phi\}\(a\\mid s\), using the flow\-matching surrogate

−log⁡pϕ​\(a∣s\)=𝔼ϵ,λ​\[1d​‖fϕ​\(s,\(1−λ\)​ϵ\+λ​a,λ\)−\(a−ϵ\)‖22\]\.\-\\\!\\log p\_\{\\phi\}\(a\\mid s\)=\\mathbb\{E\}\_\{\\epsilon,\\lambda\}\\\!\\left\[\\frac\{1\}\{d\}\\left\\\|f\_\{\\phi\}\\\!\\left\(s,\(1\-\\lambda\)\\epsilon\+\\lambda a,\\lambda\\right\)\-\(a\-\\epsilon\)\\right\\\|\_\{2\}^\{2\}\\right\]\.\(13\)

#### Complete Objective\.

The complete update minimizes

ℒ=ℒactor\+λF​ℒflow\+λR​ℒreturn\+λV​ℒanchor\.\\mathcal\{L\}=\\mathcal\{L\}\_\{\\mathrm\{actor\}\}\+\\lambda\_\{F\}\\mathcal\{L\}\_\{\\mathrm\{flow\}\}\+\\lambda\_\{R\}\\mathcal\{L\}\_\{\\mathrm\{return\}\}\+\\lambda\_\{V\}\\mathcal\{L\}\_\{\\mathrm\{anchor\}\}\.\(14\)At evaluation, we sampleNNdistinct actions from the actor, score them using Eq\. \([9](https://arxiv.org/html/2610.22497#S4.E9)\), execute the action with the largestQ⁡\(s,a\)Q\(s,a\), and replan at the next environment step\.

## 5Experiments

The goal of our experiments is to answer the following key questions:

1. \(Q1\)Does replacing a discriminative critic withGVCimprove online RL when the actor and training recipe are held fixed?
2. \(Q2\)Does the gain hold in offline RL and during offline\-to\-online finetuning, including against distributional critics?
3. \(Q3\)Which parts of theGVCobjective are responsible for the gains?

Environments\.We evaluateGVCin two settings\. For online RL, we use four robotic manipulation tasks—Lift, Can, Square, and Tool Hang—from RoboMimic\[[35](https://arxiv.org/html/2610.22497#bib.bib35)\], where a 7\-DoF robot arm must perform diverse manipulation behaviors under sparse task\-completion rewards\. Lift requires the robot to grasp and lift up an object; Can requires grasping a cylinder and placing it at a specified location; Square involves precisely inserting a square nut onto a peg; Tool Hang is a long\-horizon, multi\-stage assembly task in which the robot must insert a hook into a base to make a frame and then hang a wrench from it\. For offline and offline\-to\-online RL, we evaluate on OGBench\[[40](https://arxiv.org/html/2610.22497#bib.bib40)\], which provides offline datasets paired with online finetuning for long\-horizon, sparse\-reward control\. Offline, we use thecube,puzzle, and sparse\-rewardscenedomains listed in[Table1](https://arxiv.org/html/2610.22497#S5.T1); for offline\-to\-online finetuning we use the six tasks from Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]\.

Figure 3:Online RL on RoboMimic\.Success rate versus environment steps \(in millions\) on Lift, Can, Square, and Tool Hang, from left to right\. We compareGVCwith QC, RLPD, IDQL, FQL, QAM, and QSM\. Shaded regions show one standard deviation across seeds\.Baselines\.We organize the baselines based on what they change relative toGVC\. BFN\[[30](https://arxiv.org/html/2610.22497#bib.bib30)\]keeps the flow\-matching actor and the best\-of\-NNselection ofGVCand swaps the critic for a regression head, so the two differ only in the critic\. IDQL\[[20](https://arxiv.org/html/2610.22497#bib.bib20)\]and its flow\-policy variant IFQL\[[41](https://arxiv.org/html/2610.22497#bib.bib41)\]also select actions by best\-of\-NNfrom a behavior\-cloned generative policy but fit their critic by expectile regression, without querying the policy\. C51\[[4](https://arxiv.org/html/2610.22497#bib.bib4)\], IQN\[[11](https://arxiv.org/html/2610.22497#bib.bib11)\], CODAC\[[33](https://arxiv.org/html/2610.22497#bib.bib33)\], and Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]keep a discriminative head but model the full return distribution, with categorical, quantile, conservative quantile, and flow\-matching parameterizations respectively; they test whether distributional targets alone explain our gains\. FQL\[[41](https://arxiv.org/html/2610.22497#bib.bib41)\], QSM\[[42](https://arxiv.org/html/2610.22497#bib.bib42)\], and QAM\[[29](https://arxiv.org/html/2610.22497#bib.bib29)\]instead keep a standard critic and train an explicit flow or diffusion policy against it, by distillation with a BC penalty, score matching, or adjoint matching, while QC\[[30](https://arxiv.org/html/2610.22497#bib.bib30)\]runs TD learning over action chunks to shorten the effective horizon\. Finally, we include RLPD\[[3](https://arxiv.org/html/2610.22497#bib.bib3)\], IQL\[[23](https://arxiv.org/html/2610.22497#bib.bib23)\], and, offline only, BC, ReBRAC\[[45](https://arxiv.org/html/2610.22497#bib.bib45)\], and FBRAC\[[46](https://arxiv.org/html/2610.22497#bib.bib46),[41](https://arxiv.org/html/2610.22497#bib.bib41)\]as standard references; offline baseline numbers are taken from Value Flows or reproduced directly where unavailable \([SectionC\.3](https://arxiv.org/html/2610.22497#A3.SS3)\)\. RLPD and FQL are also run offline\-to\-online, initialized from the offline dataset and finetuned online\.

Implementation details\.To fairly compareGVCto prior methods, we follow the same training recipe as QC\[[30](https://arxiv.org/html/2610.22497#bib.bib30)\]for the RoboMimic online experiments and Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]for the OGBench offline and offline\-to\-online experiments\. We utilize the same BFN actor architecture, only modifying the critic architecture to implement our method\. For RoboMimic and OGBenchcubeenvironments, we predict an advantage instead of the absolute Q\-value to reduce binning error\.[SectionC\.1](https://arxiv.org/html/2610.22497#A3.SS1)lists theGVChyperparameters for each setting,[SectionC\.2](https://arxiv.org/html/2610.22497#A3.SS2)the training and evaluation protocol, and[SectionC\.3](https://arxiv.org/html/2610.22497#A3.SS3)the baseline tuning\.

### 5\.1Online RL

[Figure3](https://arxiv.org/html/2610.22497#S5.F3)shows learning curves on the four RoboMimic tasks\. Lift, Can, and Square are eventually solved byGVC, QC, and IDQL, althoughGVCdemonstrates small sample\-efficiency gains over all curves\. Tool Hang is where the critic methods diverge, withGVCreaching roughly 85% success within 0\.5M steps whereas QC, whose recipeGVCshares apart from the critic, and RLPD only reach about 50% at 1M steps\. Meanwhile, IDQL, which uses the same best\-of\-NNselection but a regression critic, does not learn the task\.

Table 1:Offline evaluation on OGBench\.Success rate \(%\) per task\. We bold values within95%95\\%of the best performance for each task\.Gaussian PoliciesFlow PoliciesEnvironmentTaskBCIQLReBRACFBRACIFQLFQLC51IQNCODACValue FlowsBFNGVCscene\-sparsetask119​±619\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}9​±89\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟗𝟔​±3\\mathbf\{96\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}65​±1665\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 16$\}\}19​±619\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}𝟗𝟗​±2\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}task21​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}66​±1966\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}86​±1086\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}83​±583\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}52​±1252\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟗𝟖​±2\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟗𝟖​±3\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟗𝟗​±2\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}task31​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟗𝟒​±8\\mathbf\{94\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}56​±856\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}60​±260\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}93​±693\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}72​±3872\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 38$\}\}89​±589\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟗𝟒​±6\\mathbf\{94\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}𝟗𝟗​±2\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}task42​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}77​±3477\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 34$\}\}72​±1272\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}69​±1669\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 16$\}\}4​±44\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}82​±1582\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 15$\}\}76​±3076\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 30$\}\}92​±1092\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}90​±490\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟗𝟖​±3\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}task50​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}75​±1975\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}52​±852\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}19​±519\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}1​±21\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}57​±3057\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 30$\}\}66​±2066\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 20$\}\}91​±791\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}45​±1445\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 14$\}\}𝟗𝟕​±3\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}avg\.550015158686727253536686868282𝟗𝟒\\mathbf\{94\}8686𝟗𝟖\\mathbf\{98\}puzzle3x3\-sparsetask15​±25\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟗𝟖​±2\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}75​±1975\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}6​±36\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}37​±2937\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 29$\}\}5​±45\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}78​±1578\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 15$\}\}𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task21​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}66​±1566\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 15$\}\}35​±3735\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 37$\}\}𝟗𝟗​±2\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}3​±33\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}13​±1813\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 18$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}5​±75\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}𝟗𝟕​±5\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task31​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}25​±1625\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 16$\}\}21​±2221\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 22$\}\}𝟗𝟖​±2\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟗𝟕​±4\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}41​±4841\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 48$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}10​±510\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟗𝟓​±5\\mathbf\{95\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟗𝟕​±4\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}task41​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}55​±1355\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 13$\}\}41​±3441\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 34$\}\}𝟗𝟖​±2\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟗𝟖​±2\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}1​±21\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}8​±68\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}𝟗𝟖​±3\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}𝟗𝟖​±1\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}task51​±01\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}68​±1868\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 18$\}\}43​±4043\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 40$\}\}𝟗𝟕​±4\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}1​±21\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}4​±44\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟗𝟔​±2\\mathbf\{96\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟗𝟔​±4\\mathbf\{96\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}avg\.220062624343𝟗𝟖\\mathbf\{98\}𝟗𝟗\\mathbf\{99\}221919222121𝟗𝟕\\mathbf\{97\}𝟗𝟖\\mathbf\{98\}puzzle4x4task11​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}12​±212\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}26​±426\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}32​±932\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}49​±949\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}34​±834\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}41​±241\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}37​±3237\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 32$\}\}36​±336\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}62​±1062\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}𝟕𝟔​±4\\mathbf\{76\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}task20​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}7​±47\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}12​±412\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}5​±35\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}4​±44\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}16​±516\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}12​±412\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}10​±1010\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}𝟐𝟕​±5\\mathbf\{27\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}20​±720\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}19​±1219\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}task30​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}9​±39\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}15​±315\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}20​±1020\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}50​±1450\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 14$\}\}18​±518\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}45​±745\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}33​±2933\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 29$\}\}30​±430\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}41​±841\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}𝟔𝟑​±12\\mathbf\{63\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}task40​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}5​±25\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}10​±310\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}5​±15\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}21​±1121\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}11​±311\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}23​±223\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}12​±1012\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}𝟐𝟖​±5\\mathbf\{28\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}21​±621\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}𝟐𝟖​±11\\mathbf\{28\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}task50​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}4​±14\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}7​±37\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}7​±37\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}16​±616\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}10​±810\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}13​±213\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}14​±614\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}𝟏𝟗​±9\\mathbf\{19\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}avg\.00771414131325251717002727202027273232𝟒𝟏\\mathbf\{41\}puzzle4x5task10​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}14​±714\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}32​±3032\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 30$\}\}86​±886\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}53​±1953\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟗𝟎​±5\\mathbf\{90\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}51​±1051\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}67​±467\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}83​±483\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟗𝟐​±4\\mathbf\{92\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}task20​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟏​±2\\mathbf\{1\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task30​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±1\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟎​±1\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task4𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎\\mathbf\{0\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task5𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}avg\.003300661717111100𝟏𝟖\\mathbf\{18\}101013131717𝟏𝟖\\mathbf\{18\}puzzle4x6task10​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}2​±32\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}8​±78\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}16​±2016\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 20$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}22​±522\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}30​±1430\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 14$\}\}10​±1110\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}22​±2522\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 25$\}\}22​±822\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}𝟒𝟖​±16\\mathbf\{48\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 16$\}\}task20​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟐​±2\\mathbf\{2\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task3𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task4𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task5𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎\\mathbf\{0\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}avg\.0000223300440066224455𝟏𝟎\\mathbf\{10\}cube\-doubletask18​±38\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}27​±527\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}45​±645\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}47​±1147\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}35​±935\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}61​±961\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}9​±09\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}70​±1470\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 14$\}\}80​±1180\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}𝟗𝟕​±1\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}73​±473\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}88​±2188\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 21$\}\}task20​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}7​±37\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}22​±1222\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}9​±59\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}36​±636\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}24​±924\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}63​±463\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟕𝟔​±7\\mathbf\{76\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}𝟕𝟖​±4\\mathbf\{78\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}71​±1271\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}task30​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}4​±14\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}4​±24\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}8​±58\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}22​±522\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}25​±625\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}66​±966\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}73​±473\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟕𝟕​±7\\mathbf\{77\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}𝟕𝟒​±26\\mathbf\{74\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 26$\}\}task40​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±10\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}5​±25\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}10​±110\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}13​±213\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟑𝟎​±5\\mathbf\{30\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}23​±623\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}19​±619\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}task50​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}4​±34\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}4​±24\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}17​±617\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}19​±1019\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟖𝟏​±8\\mathbf\{81\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}𝟖𝟐​±4\\mathbf\{82\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}69​±569\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟕𝟗​±6\\mathbf\{79\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}75​±1975\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}avg\.226612121515141429292242426161𝟔𝟗\\mathbf\{69\}𝟔𝟔\\mathbf\{66\}6565cube\-tripletask11​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}4​±44\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}1​±21\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}2​±22\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}20​±620\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}1​±01\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}29​±229\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}9​±59\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟓𝟗​±12\\mathbf\{59\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}28​±1328\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 13$\}\}49​±3049\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 30$\}\}task20​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟏​±2\\mathbf\{1\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟏​±0\\mathbf\{1\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}task30​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±01\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟕​±3\\mathbf\{7\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}2​±02\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}2​±32\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}task4𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}task50​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}1​±11\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟐​±1\\mathbf\{2\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}avg\.001100000044006622𝟏𝟒\\mathbf\{14\}661010

### 5\.2Offline RL

[Table1](https://arxiv.org/html/2610.22497#S5.T1)reports per\-task success on 35 tasks across seven OGBench domains\. On the two largest puzzles,puzzle\-4x5andpuzzle\-4x6,GVCreaches 92% and 48% on the first task, compared to BFN—the closest comparison, differing fromGVConly in the critic—which achieves 83% and 22%, respectively\. The two are nearly tied oncube\-double\(65%\-66%\) and are both near ceiling on the sparsepuzzle\-3x3tasks;GVCleads by 12 points on the sparsescenetasks \(98% versus 86%\), mostly ontask5where BFN reaches 45%, and by 4 oncube\-triple\(10% versus 6%\)\. IFQL, which also selects by best\-of\-NNbut trains its critic by expectile regression, is further behind:GVCis ahead on six of the seven domains\.GVCalso improves over the leading distributional baseline, Value Flows, onpuzzle\-4x4,puzzle\-4x5, andpuzzle\-4x6by an average of 8 points, while trailing slightly oncube\-doubleandcube\-tripleby an average of 4 points\.

Table 2:Offline\-to\-online evaluations on OGBench tasks\.Success rate \(%\) on 6 OGBench tasks after offline training→\\rightarrowafter online finetuning\. Mean±\\pmstandard deviation over 8 seeds, with values within95%95\\%of the best final performance for each task in bold\.TaskIQNIFQLFQLRLPDIQLValue FlowsBFNGVCantmaze\-large55​±555\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}→\\rightarrow91​±191\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}11​±311\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}→\\rightarrow3​±23\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}𝟖𝟕​±7\\mathbf\{87\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}→\\rightarrow𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow𝟗𝟗​±2\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}41​±541\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}→\\rightarrow45​±545\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}𝟕𝟐​±4\\mathbf\{72\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}→\\rightarrow𝟗𝟔​±1\\mathbf\{96\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟔𝟔​±19\\mathbf\{66\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 19$\}\}→\\rightarrow𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟕𝟕​±9\\mathbf\{77\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}→\\rightarrow𝟗𝟕​±2\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}humanoidmaze\-medium23​±323\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}→\\rightarrow14​±214\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}56​±1256\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}→\\rightarrow82​±782\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}12​±712\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}→\\rightarrow22​±1222\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow8​±38\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}21​±521\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}→\\rightarrow16​±316\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}29​±629\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}→\\rightarrow86​±386\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}32​±832\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}→\\rightarrow89​±489\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟗𝟒​±2\\mathbf\{94\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}→\\rightarrow𝟗𝟗​±1\\mathbf\{99\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}cube\-double29​±429\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}→\\rightarrow42​±742\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}12​±312\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}→\\rightarrow41​±241\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}40​±1140\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}→\\rightarrow92​±392\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}65​±765\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}→\\rightarrow79​±679\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}79​±779\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}→\\rightarrow90​±790\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}𝟔𝟒​±21\\mathbf\{64\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 21$\}\}→\\rightarrow𝟗𝟖​±1\\mathbf\{98\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}cube\-triple25​±525\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}→\\rightarrow14​±614\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}2​±12\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}→\\rightarrow7​±57\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 5$\}\}4​±14\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}→\\rightarrow83​±1283\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}3​±13\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}29​±829\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}→\\rightarrow70​±770\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 7$\}\}21​±1021\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 10$\}\}→\\rightarrow90​±1790\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 17$\}\}𝟒𝟔​±21\\mathbf\{46\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 21$\}\}→\\rightarrow𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}puzzle\-4x422​±222\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}→\\rightarrow6​±16\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}23​±223\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 2$\}\}→\\rightarrow19​±1219\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}8​±38\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}→\\rightarrow38​±5238\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 52$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}5​±15\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}14​±314\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}→\\rightarrow51​±1251\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 12$\}\}24​±624\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 6$\}\}→\\rightarrow68​±3968\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 39$\}\}𝟐𝟓​±8\\mathbf\{25\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 8$\}\}→\\rightarrow𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}scene0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}0​±00\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow60​±1460\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 14$\}\}𝟖𝟐​±11\\mathbf\{82\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 11$\}\}→\\rightarrow𝟏𝟎𝟎​±1\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 1$\}\}𝟎​±0\\mathbf\{0\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}→\\rightarrow𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}15​±415\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}→\\rightarrow10​±310\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 3$\}\}𝟗𝟐​±23\\mathbf\{92\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 23$\}\}→\\rightarrow𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}𝟓𝟏​±16\\mathbf\{51\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 16$\}\}→\\rightarrow𝟗𝟕​±4\\mathbf\{97\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 4$\}\}𝟗𝟔​±9\\mathbf\{96\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 9$\}\}→\\rightarrow𝟏𝟎𝟎​±0\\mathbf\{100\}\\,\\text\{\\scalebox\{\.80\}\{\\color\[rgb\]\{0\.35,0\.35,0\.35\}$\\pm 0$\}\}

### 5\.3Offline\-to\-online RL

[Figure4](https://arxiv.org/html/2610.22497#S5.F4)and[Table2](https://arxiv.org/html/2610.22497#S5.T2)show offline\-to\-online RL performance ofGVCon six OGBench tasks\.GVChas the highest average success at the end of both phases and finishes within 95% of the best method on all six tasks\. Its average success rises from 67\.1% to 98\.9%, about 19 points above Value Flows and 26 above FQL in[Table2](https://arxiv.org/html/2610.22497#S5.T2)\. BFN finishes second at 88\.7% but scores 45\.4% after offline training, below Value Flows\. It edges ahead ofGVConantmaze\-largebut reaches only 90% on the two cube tasks and 89% onhumanoidmaze\-medium, and two of its eight seeds collapse onpuzzle\-4x4\. RLPD, which learns only online, reaches 99–100% on three tasks but stays near zero on the other three\.

Figure 4:Offline\-to\-online RL on OGBench\.Average success rate \(%\) across six OGBench environments\. The first10610^\{6\}steps are offline, followed by online finetuning for10610^\{6\}steps\. We compareGVCwith BFN, Value Flows, FQL, IFQL, IQL, IQN, and RLPD\.

### 5\.4Ablations

Our ablations look at the generative loss, which distinguishesGVCfrom a discriminative critic, as well as the impact of timestep sampling to approximate the action likelihood\. First,[Figure5](https://arxiv.org/html/2610.22497#S5.F5)varies the coefficientλF\\lambda\_\{F\}onℒflow\\mathcal\{L\}\_\{\\mathrm\{flow\}\}from its default of11down to00\. On Tool Hang, we find that the ordering is monotone inλF\\lambda\_\{F\}, and success drops from about 85% atλF=1\\lambda\_\{F\}=1to 80% at0\.10\.1, 65% at0\.010\.01, and 45% either at0\.0010\.001or with the loss removed\. Next,[Figure5](https://arxiv.org/html/2610.22497#S5.F5)looks at which part of the loss matters\. On Tool Hang, we find that weighting all return bins uniformly—so that the flow loss trains the action model but no longer links actions to returns—recovers about 70%, between the full loss at 85% and no flow loss at about 50%: roughly half the benefit comes from the generative supervision as such, and the rest from conditioning it on the return\. Finally,[Figure5](https://arxiv.org/html/2610.22497#S5.F5)varies the number of noise timesteps,TT, used to score actions, as well as how the noise is sampled across action candidates\. With noise shared across candidates,T=1T=1reaches about 80% on Tool Hang, and usingT≥2T\\geq 2adds a few points; without sharing,T=1T=1falls to about 60%\.[AppendixA](https://arxiv.org/html/2610.22497#A1)shows results for the same set of ablations on the easier Square task, as well as ablations over the value\-anchor weightλV\\lambda\_\{V\}, and a comparison of best\-of\-NNselection with a one\-step actor distilled from critic gradients\.

Figure 5:Ablations on Tool Hang\.Success rate \(%\) vs\. environment steps\. \(a\) Action loss: varying flow loss coefficients,λF\\lambda\_\{F\}\. \(b\) Flow loss:GVCwhere the flow loss is weighted by the target return distributionp^i\\widehat\{p\}\_\{i\}\(blue\), uniform weighting across return bins \(coral\), and the flow loss removed \(teal\)\. \(c\) Scoring: differentTTvalues, with and without shared noise\. Shaded regions show one standard deviation across seeds\.

## 6Discussion

We have presentedGVC, a method for value learning that leverages a return\-conditioned generative likelihood to implicitly model the return distribution\. Our work builds off of recent work on generative classifiers and extends it to a domain where sample\-efficiency is particularly important\.

#### Limitations\.

GVCincreases the theoretical compute costs of the critic, requiringTTforward passes to compute the likelihood, overKKbins, possibly over ensembled copies\. In practice, we reduce this to only two forward passes per timestep and find that the compute costs are comparable to those of other performant distributional RL methods \([AppendixB](https://arxiv.org/html/2610.22497#A2)\), but this remains a consideration\. Finally, in the form presented,GVCis factorized into two model components and requires discretizing the return space, both of which increase the implementation complexity\.

## 7Acknowledgments

We thank Armaan Abraham and Lars Ankile for helpful feedback on drafts of this manuscript\.

This work was supported by an NSF CAREER award\. This work was made possible by the Cloud TPUs provided through Google’s TPU Research Cloud \(TRC\) and TPU Builder programs\. This also used the Delta system at the National Center for Supercomputing Applications \[award OAC 2005572\] through allocation CIS260152 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support \(ACCESS\) program, which is supported by U\.S\. National Science Foundation grants \#2138259, \#2138286, \#2138307, \#2137603, and \#2138296\.

## References

- \[1\]Bhavya Agrawalla, Michal Nauman, and Aviral Kumar\.What does flow\-matching bring to TD\-learning?In*International Conference on Machine Learning*, 2026\.URL[https://openreview\.net/forum?id=e7EKA7Y9Im](https://openreview.net/forum?id=e7EKA7Y9Im)\.
- \[2\]Jose A\. Arjona\-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, Johannes Brandstetter, and Sepp Hochreiter\.RUDDER: Return decomposition for delayed rewards\.In*Advances in Neural Information Processing Systems*, volume 32, 2019\.
- \[3\]Philip J\. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine\.Efficient online reinforcement learning with offline data\.In*Proceedings of the 40th International Conference on Machine Learning*, volume 202 of*Proceedings of Machine Learning Research*, pages 1577–1594\. PMLR, 2023\.URL[https://proceedings\.mlr\.press/v202/ball23a\.html](https://proceedings.mlr.press/v202/ball23a.html)\.
- \[4\]Marc G\. Bellemare, Will Dabney, and Rémi Munos\.A distributional perspective on reinforcement learning\.In*Proceedings of the 34th International Conference on Machine Learning*, volume 70 of*Proceedings of Machine Learning Research*, pages 449–458\. PMLR, 2017\.URL[https://proceedings\.mlr\.press/v70/bellemare17a\.html](https://proceedings.mlr.press/v70/bellemare17a.html)\.
- \[5\]Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, Keerthana Gopalakrishnan, Julian Ibarz, Ofir Nachum, Sumedh Anand Sontakke, Grecia Salazar, Huong T\. Tran, Jodilyn Peralta, Clayton Tan, Deeksha Manjunath, Jaspiar Singh, Brianna Zitkovich, Tomas Jackson, Kanishka Rao, Chelsea Finn, and Sergey Levine\.Q\-Transformer: Scalable offline reinforcement learning via autoregressive Q\-functions\.In*Proceedings of The 7th Conference on Robot Learning*, volume 229 of*Proceedings of Machine Learning Research*, pages 3909–3928\. PMLR, 2023\.URL[https://proceedings\.mlr\.press/v229/chebotar23a\.html](https://proceedings.mlr.press/v229/chebotar23a.html)\.
- \[6\]Huanran Chen, Yinpeng Dong, Shitong Shao, Zhongkai Hao, Xiao Yang, Hang Su, and Jun Zhu\.Diffusion models are certifiably robust classifiers\.In*Advances in Neural Information Processing Systems*, volume 37, pages 50062–50097\. Curran Associates, Inc\., 2024a\.[10\.52202/079017\-1584](https://doi.org/10.52202/079017-1584)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2024/file/59a3444d39b97ba01a17994f938e1ccc\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2024/file/59a3444d39b97ba01a17994f938e1ccc-Paper-Conference.pdf)\.
- \[7\]Huanran Chen, Yinpeng Dong, Zhengyi Wang, Xiao Yang, Chengqi Duan, Hang Su, and Jun Zhu\.Robust classification via a single diffusion model\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 6643–6665\. PMLR, 2024b\.URL[https://proceedings\.mlr\.press/v235/chen24k\.html](https://proceedings.mlr.press/v235/chen24k.html)\.
- \[8\]Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch\.Decision transformer: Reinforcement learning via sequence modeling\.In*Advances in Neural Information Processing Systems*, volume 34, pages 15084–15097\. Curran Associates, Inc\., 2021\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2021/hash/7f489f642a0ddb10272b5c31057f0663\-Abstract\.html](https://proceedings.neurips.cc/paper_files/paper/2021/hash/7f489f642a0ddb10272b5c31057f0663-Abstract.html)\.
- \[9\]Shirui Chen, Cole Harrison, Ying\-Chun Lee, Angela Jin Yang, Zhongzheng Ren, Lillian J\. Ratliff, Jiafei Duan, Dieter Fox, and Ranjay Krishna\.TOPReward: Token Probabilities as Hidden Zero\-Shot Rewards for Robotics, February 2026\.URL[http://arxiv\.org/abs/2602\.19313](http://arxiv.org/abs/2602.19313)\.arXiv:2602\.19313 \[cs\.RO\]\.
- \[10\]Kevin Clark and Priyank Jaini\.Text\-to\-Image Diffusion Models are Zero Shot Classifiers\.In*Advances in Neural Information Processing Systems*, volume 36, pages 58921–58937\. Curran Associates, Inc\., December 2023\.[10\.52202/075280\-2571](https://doi.org/10.52202/075280-2571)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/hash/b87bdcf963cad3d0b265fcb78ae7d11e\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/b87bdcf963cad3d0b265fcb78ae7d11e-Abstract-Conference.html)\.
- \[11\]Will Dabney, Georg Ostrovski, David Silver, and Remi Munos\.Implicit quantile networks for distributional reinforcement learning\.In Jennifer Dy and Andreas Krause, editors,*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pages 1096–1105\. PMLR, 10–15 Jul 2018a\.URL[https://proceedings\.mlr\.press/v80/dabney18a\.html](https://proceedings.mlr.press/v80/dabney18a.html)\.
- \[12\]Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos\.Distributional reinforcement learning with quantile regression\.*Proceedings of the AAAI Conference on Artificial Intelligence*, 32\(1\), 2018b\.[10\.1609/aaai\.v32i1\.11791](https://doi.org/10.1609/aaai.v32i1.11791)\.URL[https://doi\.org/10\.1609/aaai\.v32i1\.11791](https://doi.org/10.1609/aaai.v32i1.11791)\.
- \[13\]Perry Dong, Kuo\-Han Hung, Alexander Swerdlow, Dorsa Sadigh, and Chelsea Finn\.TQL: Scaling Q\-functions with transformers by preventing attention collapse\.In*International Conference on Machine Learning*, 2026a\.URL[https://openreview\.net/forum?id=lwHSE1xYHH](https://openreview.net/forum?id=lwHSE1xYHH)\.
- \[14\]Perry Dong, Chongyi Zheng, Chelsea Finn, Dorsa Sadigh, and Benjamin Eysenbach\.Value flows\.In*International Conference on Learning Representations*, 2026b\.URL[https://arxiv\.org/abs/2510\.07650](https://arxiv.org/abs/2510.07650)\.
- \[15\]Gabriel Dulac\-Arnold, Nir Levine, Daniel J\. Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester\.Challenges of real\-world reinforcement learning: Definitions, benchmarks and analysis\.*Machine Learning*, 110\(9\):2419–2468, 2021\.[10\.1007/s10994\-021\-05961\-4](https://doi.org/10.1007/s10994-021-05961-4)\.URL[https://doi\.org/10\.1007/s10994\-021\-05961\-4](https://doi.org/10.1007/s10994-021-05961-4)\.
- \[16\]Benjamin Eysenbach, Ruslan Salakhutdinov, and Sergey Levine\.C\-Learning: Learning to achieve goals via recursive classification\.In*International Conference on Learning Representations*, 2021\.URL[https://openreview\.net/forum?id=tc5qisoB\-C](https://openreview.net/forum?id=tc5qisoB-C)\.
- \[17\]Benjamin Eysenbach, Tianjun Zhang, Sergey Levine, and Ruslan Salakhutdinov\.Contrastive learning as goal\-conditioned reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 35, pages 35603–35620\. Curran Associates, Inc\., 2022\.[10\.52202/068431\-2580](https://doi.org/10.52202/068431-2580)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/hash/e7663e974c4ee7a2b475a4775201ce1f\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/e7663e974c4ee7a2b475a4775201ce1f-Abstract-Conference.html)\.
- \[18\]Jesse Farebrother, Jordi Orbay, Quan Vuong, Adrien Ali Taïga, Yevgen Chebotar, Ted Xiao, Alex Irpan, Sergey Levine, Pablo Samuel Castro, Aleksandra Faust, Aviral Kumar, and Rishabh Agarwal\.Stop regressing: Training value functions via classification for scalable deep RL\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 13049–13071\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/farebrother24a\.html](https://proceedings.mlr.press/v235/farebrother24a.html)\.
- \[19\]Will Grathwohl, Kuan\-Chieh Wang, Jörn\-Henrik Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky\.Your classifier is secretly an energy based model and you should treat it like one\.In*International Conference on Learning Representations*, 2020\.URL[https://openreview\.net/forum?id=Hkxzx0NtDB](https://openreview.net/forum?id=Hkxzx0NtDB)\.
- \[20\]Philippe Hansen\-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine\.IDQL: Implicit Q\-learning as an actor\-critic method with diffusion policies, 2023\.URL[https://arxiv\.org/abs/2304\.10573](https://arxiv.org/abs/2304.10573)\.
- \[21\]Ehsan Imani and Martha White\.Improving regression performance with distributional losses\.In*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pages 2157–2166\. PMLR, 2018\.URL[https://proceedings\.mlr\.press/v80/imani18a\.html](https://proceedings.mlr.press/v80/imani18a.html)\.
- \[22\]Diederik Kingma and Ruiqi Gao\.Understanding diffusion objectives as the ELBO with simple data augmentation\.In*Advances in Neural Information Processing Systems*, volume 36, pages 65484–65516\. Curran Associates, Inc\., 2023\.[10\.52202/075280\-2858](https://doi.org/10.52202/075280-2858)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/file/ce79fbf9baef726645bc2337abb0ade2\-Paper\-Conference\.pdf](https://proceedings.neurips.cc/paper_files/paper/2023/file/ce79fbf9baef726645bc2337abb0ade2-Paper-Conference.pdf)\.
- \[23\]Ilya Kostrikov, Ashvin Nair, and Sergey Levine\.Offline reinforcement learning with implicit Q\-learning\.In*International Conference on Learning Representations*, 2022\.URL[https://openreview\.net/forum?id=68n2s9ZJWF8](https://openreview.net/forum?id=68n2s9ZJWF8)\.
- \[24\]Aviral Kumar, Abhishek Gupta, and Sergey Levine\.DisCor: Corrective feedback in reinforcement learning via distribution correction\.In*Advances in Neural Information Processing Systems*, volume 33, pages 18560–18572\. Curran Associates, Inc\., 2020\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2020/hash/d7f426ccbc6db7e235c57958c21c5dfa\-Abstract\.html](https://proceedings.neurips.cc/paper_files/paper/2020/hash/d7f426ccbc6db7e235c57958c21c5dfa-Abstract.html)\.
- \[25\]Sachin Kumar, Chan Young Park, and Yulia Tsvetkov\.Gen\-Z: Generative zero\-shot text classification with contextualized label descriptions\.In*International Conference on Learning Representations*, 2024\.URL[https://openreview\.net/forum?id=rkplYfqUr0](https://openreview.net/forum?id=rkplYfqUr0)\.
- \[26\]Julia A\. Lasserre, Christopher M\. Bishop, and Thomas P\. Minka\.Principled hybrids of generative and discriminative models\.In*2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition*, volume 1, pages 87–94, 2006\.[10\.1109/CVPR\.2006\.227](https://doi.org/10.1109/CVPR.2006.227)\.URL[https://www\.microsoft\.com/en\-us/research/publication/principled\-hybrids\-of\-generative\-and\-discriminative\-models/](https://www.microsoft.com/en-us/research/publication/principled-hybrids-of-generative-and-discriminative-models/)\.
- \[27\]Tony Lee, Andrew Wagenmaker, Karl Pertsch, Percy Liang, Sergey Levine, and Chelsea Finn\.RoboReward: General\-purpose vision\-language reward models for robotics, 2026\.URL[https://arxiv\.org/abs/2601\.00675](https://arxiv.org/abs/2601.00675)\.
- \[28\]Alexander C\. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak\.Your diffusion model is secretly a zero\-shot classifier\.In*2023 IEEE/CVF International Conference on Computer Vision \(ICCV\)*, pages 2206–2217\. IEEE, 2023\.[10\.1109/ICCV51070\.2023\.00210](https://doi.org/10.1109/ICCV51070.2023.00210)\.URL[https://doi\.org/10\.1109/ICCV51070\.2023\.00210](https://doi.org/10.1109/ICCV51070.2023.00210)\.
- \[29\]Qiyang Li and Sergey Levine\.Q\-learning with adjoint matching\.In*International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=vd4eNAdtO6](https://openreview.net/forum?id=vd4eNAdtO6)\.
- \[30\]Qiyang Li, Zhiyuan Zhou, and Sergey Levine\.Reinforcement learning with action chunking\.In*Advances in Neural Information Processing Systems*, volume 38, pages 55518–55553, 2025\.[10\.52202/085713\-1856](https://doi.org/10.52202/085713-1856)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2025/hash/50348e8f9aef984abe0ea1ec2a326f78\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2025/hash/50348e8f9aef984abe0ea1ec2a326f78-Abstract-Conference.html)\.Results from[https://github\.com/ColinQiyangLi/qc/blob/48283b4f662bd1c127ec9fa80647b49759f10653/plot\_data/ogbench\-individual\.pkl](https://github.com/ColinQiyangLi/qc/blob/48283b4f662bd1c127ec9fa80647b49759f10653/plot_data/ogbench-individual.pkl)\.
- \[31\]Hao Liu and Pieter Abbeel\.Hybrid discriminative\-generative training via contrastive learning, 2020\.URL[https://arxiv\.org/abs/2007\.09070](https://arxiv.org/abs/2007.09070)\.
- \[32\]Jijia Liu, Feng Gao, Bingwen Wei, Xinlei Chen, Qingmin Liao, Yi Wu, Chao Yu, and Yu Wang\.What can RL bring to VLA generalization? an empirical study\.In*Advances in Neural Information Processing Systems*, volume 38, pages 97121–97151, 2025\.[10\.52202/085713\-3248](https://doi.org/10.52202/085713-3248)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2025/hash/8c0fabe372177d2aded596be2d3b4544\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2025/hash/8c0fabe372177d2aded596be2d3b4544-Abstract-Conference.html)\.
- \[33\]Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani\.Conservative offline distributional reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 34, pages 19235–19247\. Curran Associates, Inc\., 2021\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2021/hash/a05d886123a54de3ca4b0985b718fb9b\-Abstract\.html](https://proceedings.neurips.cc/paper_files/paper/2021/hash/a05d886123a54de3ca4b0985b718fb9b-Abstract.html)\.
- \[34\]Yecheng Jason Ma, Joey Hejna, Chuyuan Fu, Dhruv Shah, Jacky Liang, Zhuo Xu, Sean Kirmani, Peng Xu, Danny Driess, Ted Xiao, Osbert Bastani, Dinesh Jayaraman, Wenhao Yu, Tingnan Zhang, Dorsa Sadigh, and Fei Xia\.Vision language models are in\-context value learners\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=friHAl5ofG](https://openreview.net/forum?id=friHAl5ofG)\.
- \[35\]Ajay Mandlekar, Danfei Xu, Josiah Wong, Soroush Nasiriany, Chen Wang, Rohun Kulkarni, Li Fei\-Fei, Silvio Savarese, Yuke Zhu, and Roberto Martín\-Martín\.What matters in learning from offline human demonstrations for robot manipulation\.In*Proceedings of the 5th Conference on Robot Learning*, volume 164 of*Proceedings of Machine Learning Research*, pages 1678–1690\. PMLR, 2021\.URL[https://proceedings\.mlr\.press/v164/mandlekar22a\.html](https://proceedings.mlr.press/v164/mandlekar22a.html)\.
- \[36\]Bogdan Mazoure, Benjamin Eysenbach, Ofir Nachum, and Jonathan Tompson\.Contrastive value learning: Implicit models for simple offline RL\.In*Proceedings of The 7th Conference on Robot Learning*, volume 229 of*Proceedings of Machine Learning Research*, pages 1257–1267\. PMLR, 2023\.URL[https://proceedings\.mlr\.press/v229/mazoure23a\.html](https://proceedings.mlr.press/v229/mazoure23a.html)\.
- \[37\]David McAllister, Songwei Ge, Brent Yi, Chung Min Kim, Ethan Weber, Hongsuk Choi, Haiwen Feng, and Angjoo Kanazawa\.Flow matching policy gradients\.In*International Conference on Learning Representations*, 2026\.URL[https://openreview\.net/forum?id=eoEmoKoQpJ](https://openreview.net/forum?id=eoEmoKoQpJ)\.
- \[38\]Andrew Y\. Ng and Michael I\. Jordan\.On Discriminative vs\. Generative Classifiers: A comparison of logistic regression and naive Bayes\.In*Advances in Neural Information Processing Systems*, volume 14\. MIT Press, 2001\.URL[https://papers\.nips\.cc/paper\_files/paper/2001/hash/7b7a53e239400a13bd6be6c91c4f6c4e\-Abstract\.html](https://papers.nips.cc/paper_files/paper/2001/hash/7b7a53e239400a13bd6be6c91c4f6c4e-Abstract.html)\.
- \[39\]Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe\.Training language models to follow instructions with human feedback\.In*Advances in Neural Information Processing Systems*, volume 35, pages 27730–27744\. Curran Associates, Inc\., 2022\.[10\.52202/068431\-2011](https://doi.org/10.52202/068431-2011)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2022/hash/b1efde53be364a73914f58805a001731\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2022/hash/b1efde53be364a73914f58805a001731-Abstract-Conference.html)\.
- \[40\]Seohong Park, Kevin Frans, Benjamin Eysenbach, and Sergey Levine\.OGBench: Benchmarking offline goal\-conditioned RL\.In*International Conference on Learning Representations*, 2025a\.URL[https://openreview\.net/forum?id=M992mjgKzI](https://openreview.net/forum?id=M992mjgKzI)\.
- \[41\]Seohong Park, Qiyang Li, and Sergey Levine\.Flow Q\-learning\.In*Proceedings of the 42nd International Conference on Machine Learning*, volume 267 of*Proceedings of Machine Learning Research*, pages 48104–48127\. PMLR, 2025b\.URL[https://proceedings\.mlr\.press/v267/park25f\.html](https://proceedings.mlr.press/v267/park25f.html)\.
- \[42\]Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma\.Learning a diffusion model policy from rewards via Q\-score matching\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 41163–41182\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/psenka24a\.html](https://proceedings.mlr.press/v235/psenka24a.html)\.
- \[43\]Raul Puri and Bryan Catanzaro\.Zero\-shot text classification with generative language models, 2019\.URL[https://arxiv\.org/abs/1912\.10165](https://arxiv.org/abs/1912.10165)\.
- \[44\]Jost Tobias Springenberg, Abbas Abdolmaleki, Jingwei Zhang, Oliver Groth, Michael Bloesch, Thomas Lampe, Philemon Brakel, Sarah Maria Elisabeth Bechtle, Steven Kapturowski, Roland Hafner, Nicolas Heess, and Martin Riedmiller\.Offline actor\-critic reinforcement learning scales to large models\.In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors,*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 46323–46350\. PMLR, 21–27 Jul 2024\.URL[https://proceedings\.mlr\.press/v235/springenberg24a\.html](https://proceedings.mlr.press/v235/springenberg24a.html)\.
- \[45\]Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov\.Revisiting the minimalist approach to offline reinforcement learning\.In*Advances in Neural Information Processing Systems*, volume 36, pages 11592–11620\. Curran Associates, Inc\., 2023\.[10\.52202/075280\-0511](https://doi.org/10.52202/075280-0511)\.URL[https://proceedings\.neurips\.cc/paper\_files/paper/2023/hash/26cce1e512793f2072fd27c391e04652\-Abstract\-Conference\.html](https://proceedings.neurips.cc/paper_files/paper/2023/hash/26cce1e512793f2072fd27c391e04652-Abstract-Conference.html)\.
- \[46\]Yifan Wu, George Tucker, and Ofir Nachum\.Behavior regularized offline reinforcement learning, 2019\.URL[https://arxiv\.org/abs/1911\.11361](https://arxiv.org/abs/1911.11361)\.
- \[47\]Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston\.Self\-rewarding language models\.In*Proceedings of the 41st International Conference on Machine Learning*, volume 235 of*Proceedings of Machine Learning Research*, pages 57905–57923\. PMLR, 2024\.URL[https://proceedings\.mlr\.press/v235/yuan24d\.html](https://proceedings.mlr.press/v235/yuan24d.html)\.
- \[48\]Shaopeng Zhai, Qi Zhang, Tianyi Zhang, Fuxian Huang, Haoran Zhang, Ming Zhou, Shengzhe Zhang, Litao Liu, Sixu Lin, and Jiangmiao Pang\.A Vision\-Language\-Action\-Critic model for robotic real\-world reinforcement learning, 2025\.URL[https://arxiv\.org/abs/2509\.15937](https://arxiv.org/abs/2509.15937)\.
- \[49\]Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal\.Generative verifiers: Reward modeling as next\-token prediction\.In*International Conference on Learning Representations*, 2025\.URL[https://openreview\.net/forum?id=Ccwp4tFEtE](https://openreview.net/forum?id=Ccwp4tFEtE)\.

## Appendix AAdditional Ablations

[Section5](https://arxiv.org/html/2610.22497#S5)reports our ablations on Tool Hang\. This appendix adds the same ablations on Square \([SectionA\.1](https://arxiv.org/html/2610.22497#A1.SS1)\), a sweep over the weightλV\\lambda\_\{V\}of the value anchor \([SectionA\.2](https://arxiv.org/html/2610.22497#A1.SS2)\), and a comparison of best\-of\-NNselection with a one\-step actor trained by gradient ascent onQQ\([SectionA\.3](https://arxiv.org/html/2610.22497#A1.SS3)\)\. Unless stated otherwise, every run uses the settings of[Table4](https://arxiv.org/html/2610.22497#A3.T4)\.

### A\.1Ablations on Square

Figure 6:Ablations on Square\.Success rate \(%\) versus environment steps\. \(a\) Action loss: different flow\-loss coefficientsλF\\lambda\_\{F\}, compared with the default\. \(b\) Flow loss:GVCwith the default flow loss weighted by the target return distributionp^i\\widehat\{p\}\_\{i\}\(blue\), uniform weighting across return bins \(coral\), and the flow loss removed \(teal\)\. \(c\) Scoring: differentTTvalues, with and without shared noise\. Shaded regions show one standard deviation across seeds\.[Figure6](https://arxiv.org/html/2610.22497#A1.F6)repeats the three ablations of[Figure5](https://arxiv.org/html/2610.22497#S5.F5)on Square, with the same settings and88seeds per variant\. Square is the easier task and separates the variants far less\. Every flow\-loss coefficient in \(a\) and every flow\-loss variant in \(b\), including removing the loss, ends near95%95\\%; the default learns slightly faster early on, but the curves otherwise nearly coincide\. Scoring \(c\) tells the same story as on Tool Hang\. With shared noise,T=1T=1learns a little more slowly thanT≥2T\\geq 2but catches up; without sharing,T=1T=1learns more slowly still and ends several points lower\.

### A\.2Value\-anchor coefficient

Figure 7:Value\-anchor ablation on Square, Tool Hang, andpuzzle\-4x4\.Success rate versus environment steps, or offline updates onpuzzle\-4x4\(task4\), with the anchor coefficientλV\\lambda\_\{V\}scaled to1010,0\.10\.1,0\.010\.01, and00times its default \(1×1\\times,[Table4](https://arxiv.org/html/2610.22497#A3.T4)\),88seeds per setting\.Value anchoring \(Eq\. \([12](https://arxiv.org/html/2610.22497#S4.E12)\)\) fitsQ⁡\(s,a\)Q\(s,a\)—the mean of the return distribution—to the TD target\.[Figure7](https://arxiv.org/html/2610.22497#A1.F7)varies its weightλV\\lambda\_\{V\}on Square, Tool Hang, and offlinepuzzle\-4x4, with everything else as in[Table4](https://arxiv.org/html/2610.22497#A3.T4)\. Without the anchor, success stays below10%10\\%on all three tasks\. With it, Square is insensitive to the weight, and any non\-zero value matches the default\. Tool Hang is more sensitive but tolerates a wide range:0\.1×0\.1\\timesends near the default1×1\\times\(88%88\\%vs\.86%86\\%\),10×10\\timeslearns more slowly and ends at77%77\\%, and0\.01×0\.01\\timesis slower still \(57%57\\%\)\. Onpuzzle\-4x4the weight matters little once it is non\-zero, and all four settings end between30%30\\%and38%38\\%after10610^\{6\}updates\.

### A\.3Best\-of\-NNselection versus a distilled actor

Figure 8:Best\-of\-NNselection versus a distilled one\-step actor\.Success rate ofGVCwith its default best\-of\-NNextraction \(solid\) and with a one\-step actor \(shortcut model\) trained by direct Q\-gradients underGVC, for a range of distillation coefficientsα\\alpha\(dashed\)\. Left and center: online RL on Square and Tool Hang\. Right: offline RL onpuzzle\-4x4, averaged over its five tasks\.α=0\\alpha=0removes the behavior anchor from the one\-step actor\.GVCscores actions, so it can also drive an explicit actor that maximizesQQby gradient ascent\. Here we test the extraction recipe of FQL\[[41](https://arxiv.org/html/2610.22497#bib.bib41)\]: a one\-step policyμη​\(s,ϵ\)\\mu\_\{\\eta\}\(s,\\epsilon\)that maps noise to an action is trained with

ℒdistill​\(η\)=𝔼s∼𝒟,ϵ∼𝒩⁡\(0,I\)​\[α​‖μη​\(s,ϵ\)−Flowϕ​\(s,ϵ\)‖22−Q⁡\(s,μη​\(s,ϵ\)\)\],\\mathcal\{L\}\_\{\\mathrm\{distill\}\}\(\\eta\)=\\mathbb\{E\}\_\{s\\sim\\mathcal\{D\},\\,\\epsilon\\sim\\mathcal\{N\}\(0,I\)\}\\\!\\left\[\\alpha\\left\\\|\\mu\_\{\\eta\}\(s,\\epsilon\)\-\\mathrm\{Flow\}\_\{\\phi\}\(s,\\epsilon\)\\right\\\|\_\{2\}^\{2\}\-Q\\\!\\left\(s,\\mu\_\{\\eta\}\(s,\\epsilon\)\\right\)\\right\],\(15\)whereFlowϕ​\(s,ϵ\)\\mathrm\{Flow\}\_\{\\phi\}\(s,\\epsilon\)is the action obtained by integrating the BC flowfϕf\_\{\\phi\}fromϵ\\epsilonwith1010Euler steps,QQis theGVCcritic, and its gradient with respect to the action is taken through the flow surrogate of Eq\. \([8](https://arxiv.org/html/2610.22497#S4.E8)\)\. The BC flow actor is trained as before, and the one\-step actor is used instead of best\-of\-NNselection\. This follows FQL\[[41](https://arxiv.org/html/2610.22497#bib.bib41)\]where theQQterm is not normalized and other hyperparameters follow[Table4](https://arxiv.org/html/2610.22497#A3.T4)\. We sweepα\\alphaover\{0,0\.1,0\.3,1,3,10,30,100,300,1000\}\\\{0,0\.1,0\.3,1,3,10,30,100,300,1000\\\}on Tool Hang,\{500,750,1000,1500\}\\\{500,750,1000,1500\\\}on Square, and\{500,1000,1500,2000\}\\\{500,1000,1500,2000\\\}onpuzzle\-4x4\.

[Figure8](https://arxiv.org/html/2610.22497#A1.F8)shows the results\. On Square, the distilled actor eventually matches the performance of best\-of\-NNby10610^\{6\}steps \(96%96\\%for the bestα\\alpha\) but learns more slowly, reaching7878–90%90\\%at5×1055\\times 10^\{5\}steps vs\.96%96\\%for best\-of\-NN\. On Tool Hang, experiments do not find any coefficient that works: best\-of\-NNreaches91%91\\%, while the distilled actors stay near zero\. Onpuzzle\-4x4, the distilled actor\(s\) peak at∼30%\\sim 30\\%around4×1054\\times 10^\{5\}steps and then decline, while best\-of\-NNimproves throughout to40%40\\%\.

Best\-of\-NNsampling only considers actions that have been sampled from the behavior flow, so it does not leave the support of the data\. The distilled actor follows∇aQ\\nabla\_\{a\}Qinstead, increasing the risk of instability\. We find that only coefficients in the hundreds and above make progress, and on Tool Hang none does\. The main results therefore use best\-of\-NNthroughout\. Extraction methods that use the action gradient while staying in support, such as adjoint matching\[[29](https://arxiv.org/html/2610.22497#bib.bib29)\], may narrow this gap in the future\.

## Appendix BCompute Cost

GVCspends more compute per scored action than a discriminative critic\. Instead of predicting a value in a single forward pass, it evaluates the return\-conditioned likelihood of the action under every return bin\. In principle, the action model could be any generative model with an exact likelihood in a single forward pass, such as a discrete autoregressive or Gaussian model\. Diffusion and flow\-matching policies are the strongest architectures on the tasks we evaluate, however, so we use the same architecture for the critic and approximate the likelihood over noise timesteps as described in[Section4\.2](https://arxiv.org/html/2610.22497#S4.SS2)\.

We also reuse the policy architecture for the critic in every experiment, whereas most strong value\-learning algorithms use a critic far smaller than the policy\[[13](https://arxiv.org/html/2610.22497#bib.bib13)\]\. We see this as a strength, since most prior methods cannot benefit from naively scaling the critic and in fact suffer from it\[[13](https://arxiv.org/html/2610.22497#bib.bib13)\]\. In practice, moreover, value learning is usually bottlenecked by environment interaction rather than by critic compute\[[15](https://arxiv.org/html/2610.22497#bib.bib15)\]\.

We keep the cost down by predicting the action likelihood for all return bins in a single forward pass through a lightweight head, following[Chen et al\. \[7\]](https://arxiv.org/html/2610.22497#bib.bib7)\. Most of the network capacity therefore goes into learningpθ​\(a∣s\)p\_\{\\theta\}\(a\\mid s\), and the head performs the final conversion topθ​\(a∣s,z\)p\_\{\\theta\}\(a\\mid s,z\)\. Measured in forward evaluations \(NFEs\),GVCis then comparable to any diffusion or flow\-based critic, such as Value Flows, that must integrate over timesteps\.

Concretely,[Figure9](https://arxiv.org/html/2610.22497#A2.F9)reports the wall\-clock time per gradient update forGVCand the offline baselines onpuzzle\-4x4, each run with its reference implementation and published hyperparameters on a single NVIDIA H100\.GVCtakes15\.615\.6ms per update, comparable to Value Flows \(13\.513\.5ms\) and IQN \(13\.113\.1ms\), the two distributional critics that also evaluate many return bins or quantiles per transition, and55–9×9\\timesthe cost of the scalar regression critics of IQL, ReBRAC, FBRAC, IFQL, and FQL \(1\.81\.8–3\.43\.4ms\)\.

Figure 9:Offline RL training step time\.Wall\-clock time per gradient update onpuzzle\-4x4on a single H100, mean and standard deviation over33seeds\.
## Appendix CImplementation Details

### C\.1GVC hyperparameters

[Table3](https://arxiv.org/html/2610.22497#A3.T3)lists the hyperparameters shared by all experiments and[Table4](https://arxiv.org/html/2610.22497#A3.T4)lists those that differ by domain\. The paragraphs below expand on both\.

Table 3:Common GVC hyperparameters\.Values shared by all experiments\. Domain\-specific values are given in[Table4](https://arxiv.org/html/2610.22497#A3.T4)\.HyperparameterValueOptimizerAdamLearning rate3×10−43\\times 10^\{\-4\}Batch size256256Target network update rate0\.0050\.005Actorfϕf\_\{\\phi\}MLP,44hidden layers of width512512, no layer normalizationAction modelgθg\_\{\\theta\}trunkMLP,33hidden layers of width512512, layer normalization, GELUReturn\-bin headper\-bin512512\-d embedding, one hidden layer of width512512, linear outputReturn priorpψp\_\{\\psi\}MLP onss,33hidden layers of width512512, layer normalization, GELUFlow\-time embedding \(critic\)sinusoidal,3232\-dFlow steps \(Euler\)1010Flow time forℒflow\\mathcal\{L\}\_\{\\mathrm\{flow\}\}andℒactor\\mathcal\{L\}\_\{\\mathrm\{actor\}\}λ∼𝒰⁡\[0,1\]\\lambda\\sim\\mathcal\{U\}\[0,1\], one sample per transitionScoring flow timesTT44, stratified on\[0\.1,0\.9\]\[0\.1,0\.9\], antithetic noise pair per timeLoss and prior weightsλF=λR=1\\lambda\_\{F\}=\\lambda\_\{R\}=1,βprior=0\.1\\beta\_\{\\mathrm\{prior\}\}=0\.1Per\-domain weightsβret\\beta\_\{\\mathrm\{ret\}\},λV\\lambda\_\{V\}\([Table4](https://arxiv.org/html/2610.22497#A3.T4)\)Update\-to\-data ratio \(online\)11Offline gradient steps \(OGBench\)10610^\{6\}Online environment steps10610^\{6\}Evaluation episodes100100Table 4:GVC hyperparameters\.KKis the number of return bins;NNis the number of candidates for best\-of\-NNselection; “Critics” gives the ensemble size and aggregation\.Domainγ\\gammaKKβret\\beta\_\{\\mathrm\{ret\}\}λV\\lambda\_\{V\}NNCriticsRoboMimic, online \(h=5h=5\)lift,can,square\(MH\)0\.990\.995540400\.150\.158822, mintool\-hang\(PH\)0\.990\.995540400\.050\.058822, minOGBench, offline \(h=1h=1\)cube\-double\-play0\.9950\.995212140400\.20\.28822, meancube\-triple\-play0\.9950\.995212110100\.20\.28822, meanpuzzle\-3x3\-play\-sparse0\.990\.99515140400\.20\.28822, meanpuzzle\-4x4\-play0\.990\.99111140400\.20\.28822, meanpuzzle\-4x5\-play0\.990\.99212140400\.50\.58822, meanpuzzle\-4x6\-play0\.990\.99212140400\.50\.58822, meanscene\-play\-sparse0\.9950\.995111140400\.050\.054422, meanOGBench, offline\-to\-online \(h=1h=1\)antmaze\-large\-navigate0\.990\.99212140400\.20\.24422, meanhumanoidmaze\-medium\-navigate0\.9950\.995212140400\.20\.28822, meancube\-double\-play0\.9950\.995212140400\.20\.24422, meancube\-triple\-play0\.9950\.995212140400\.20\.24422, meanpuzzle\-4x4\-play0\.990\.99515140400\.20\.28822, meanscene\-play0\.9950\.995515140400\.050\.058822, mean#### Networks\.

The actorfϕf\_\{\\phi\}is the flow policy of FQL and QC, an MLP on\(s,xλ,λ\)\(s,x\_\{\\lambda\},\\lambda\)\. The action modelgθg\_\{\\theta\}takes the same inputs, withλ\\lambdapassed through a sinusoidal embedding\. The action model conditions on the return bin only after the trunk, and we add a learned embedding of each bin to the trunk features, and apply a small head, shared across bins\. One forward pass therefore covers every bin, and these embeddings are the only bin\-specific parameters\. The return priorpψ​\(z∣s\)p\_\{\\psi\}\(z\\mid s\)is a separate MLP onsswith the same trunk architecture andKKoutput logits\. For ensembling \(as denoted in[Table4](https://arxiv.org/html/2610.22497#A3.T4)\), we train independent copies of the action model and prior\. With “mean” we average the two return distributions bin by bin\. With “min” we keep whichever one has the lower mean\. This combined distribution is what ranks actions and whatp¯\\bar\{p\}denotes in[Section4\.1](https://arxiv.org/html/2610.22497#S4.SS1)\. Target networks are exponential moving averages of the online ones\.

#### Scoring\.

We estimate Eq\. \([8](https://arxiv.org/html/2610.22497#S4.E8)\) atT=4T=4flow times, stratified over\[λmin,λmax\]\[\\lambda\_\{\\min\},\\lambda\_\{\\max\}\]\(typicallyλmin=0\.1\\lambda\_\{\\min\}=0\.1,λmax=0\.9\\lambda\_\{\\max\}=0\.9\), with an antithetic noise pair\(ϵ,−ϵ\)\(\\epsilon,\-\\epsilon\)at each time\. Scoring a candidate therefore costs2​T2Tforward passes ofgθg\_\{\\theta\}, each of which returns allKKbins\. The same noise and times are reused across bins and across theNNcandidates at a state\. We use this estimator everywhere the critic is used for scoring\. The training lossℒflow\\mathcal\{L\}\_\{\\mathrm\{flow\}\}instead draws one fresh\(ϵ,λ\)\(\\epsilon,\\lambda\)per transition, withλ∼𝒰⁡\[0,1\]\\lambda\\sim\\mathcal\{U\}\[0,1\]\.[Figure5](https://arxiv.org/html/2610.22497#S5.F5)variesTTwhile keeping the antithetic pairs; its variants with non\-shared noise draw independent noise and times for each candidate\. For RoboMimiclift,can, andsquare, as well as OGBenchcubeenvironments, we find that scoring over each action dimension separately produces more accurate likelihood estimates\.

#### Return bins and loss weights\.

We discretize returns intoKKuniformly spaced bins over\[z1,zK\]\[z\_\{1\},z\_\{K\}\]\. This is typically set from the analytical minimum and maximum return given the environment reward and discount\. The projectionΠ𝒵\\Pi\_\{\\mathcal\{Z\}\}in[Section4\.1](https://arxiv.org/html/2610.22497#S4.SS1)is defined in[Bellemare et al\. \[4\]](https://arxiv.org/html/2610.22497#bib.bib4)and the shifted atomsr\+γ​m​zjr\+\\gamma mz\_\{j\}are first clipped to\[z1,zK\]\[z\_\{1\},z\_\{K\}\]\. The mass is then split between the two closest bins by linear interpolation\. To keep the predicted distribution from collapsing to a single bin, we smooth the projected targetp^\\widehat\{p\}with a Gaussian kernel similar to HL\-Gauss\[[21](https://arxiv.org/html/2610.22497#bib.bib21),[18](https://arxiv.org/html/2610.22497#bib.bib18)\]\.

For Eq\. \([9](https://arxiv.org/html/2610.22497#S4.E9)\), we weight these two terms separately, withβret\\beta\_\{\\mathrm\{ret\}\}onlog⁡pθ​\(a∣s,zi\)\\log p\_\{\\theta\}\(a\\mid s,z\_\{i\}\)andβprior\\beta\_\{\\mathrm\{prior\}\}onlog⁡pψ​\(zi∣s\)\\log p\_\{\\psi\}\(z\_\{i\}\\mid s\)\([Table3](https://arxiv.org/html/2610.22497#A3.T3)\)\. Unless otherwise noted, we setλF=λR=1\\lambda\_\{F\}=\\lambda\_\{R\}=1and pickλV\\lambda\_\{V\}for each domain \([Table4](https://arxiv.org/html/2610.22497#A3.T4)\)\.[Figure5](https://arxiv.org/html/2610.22497#S5.F5)variesλF\\lambda\_\{F\}, and[Figure7](https://arxiv.org/html/2610.22497#A1.F7)variesλV\\lambda\_\{V\}\.

#### Actor and extraction\.

In all settings, the actor is the behavior\-cloning flow of Eq\. \([13](https://arxiv.org/html/2610.22497#S4.E13)\), trained on the actions in the batch \(dataset actions offline and replay actions online\)\. We extract a policy by best\-of\-NNsampling: sampleNNactions from the flow with1010Euler steps, score them with the critic, and execute the one with the highestQ⁡\(s,a\)Q\(s,a\)\.

We re\-sample at every environment step, or after each chunk for RoboMimic\. We choosea′a^\{\\prime\}for the TD target the same way and use the online critic to rank the candidates and the target critic to score the selected one\. We use direct Q\-gradients only in the ablation in[SectionA\.3](https://arxiv.org/html/2610.22497#A1.SS3)\.

### C\.2Training and evaluation protocol

#### RoboMimic\.

We follow the QC recipe\. There is no offline phase: updates start after5,0005\{,\}000environment steps with an update\-to\-data \(UTD\) ratio of11, and each batch is a 50/50 mix of demonstration transitions \(multi\-human for Lift, Can, and Square; proficient\-human for Tool Hang\) and replay transitions\. At inference, the policy predicts chunks ofh=5h=5actions and executes them open\-loop\. As in QC, the critic scores whole chunks\. A transition is\(st,at:t\+h,∑k<hγkrt\+k,st\+h\)\(s\_\{t\},a\_\{t:t\+h\},\\sum\_\{k<h\}\\gamma^\{k\}r\_\{t\+k\},s\_\{t\+h\}\)and the target usesγh\\gamma^\{h\}in place ofγ\\gamma\. We use a sparse reward of−1\-1at every step until success and00at success\. Episodes end at success, so a chunk that ends the episode hasm=0m=0, and time\-outs still bootstrap\. We use33seeds and evaluate for100100episodes every25,00025\{,\}000steps\. The ablations in[Section5](https://arxiv.org/html/2610.22497#S5)and[SectionA\.1](https://arxiv.org/html/2610.22497#A1.SS1)use the Square and Tool Hang settings of[Table4](https://arxiv.org/html/2610.22497#A3.T4)with88seeds\.

#### OGBench\.

We follow the Value Flows recipe\. Offline results use10610^\{6\}gradient steps on the offline dataset withh=1h=1, whereas offline\-to\-online results follow the same recipe but add an additional10610^\{6\}environment steps at an update\-to\-data \(UTD\) ratio of11, with updates resuming after1,0001\{,\}000online steps\. The sparsesceneandpuzzle\-3x3variants follow QC\. We relabel the dataset rewards to−1\-1, except at steps where the task is complete, which get00\. For offline\-to\-online we usetask1ofantmaze\-large\-navigate,humanoidmaze\-medium\-navigate, andcube\-triple\-play,task2ofcube\-double\-playandscene\-play, andtask4ofpuzzle\-4x4\-play, all with the default rewards\. The replay buffer starts out as the offline dataset\. Online transitions are added to it as they are collected, and batches are always sampled uniformly from the entire buffer\. We use88seeds and evaluate for100100episodes every10510^\{5\}steps;[Table1](https://arxiv.org/html/2610.22497#S5.T1)reports the final checkpoint\.

#### Tables\.

Entries in[Tables1](https://arxiv.org/html/2610.22497#S5.T1)and[2](https://arxiv.org/html/2610.22497#S5.T2)are means±\\pmsample standard deviations over88seeds, in percentage points\.

#### Learning curves\.

In[Figure3](https://arxiv.org/html/2610.22497#S5.F3)we smooth each seed’s curve with a Gaussian kernel \(bandwidth2%2\\%of the observations\) and then take the mean and sample standard deviation across seeds\. In[Figure4](https://arxiv.org/html/2610.22497#S5.F4)theGVCand BFN curves are unsmoothed averages of the six per\-task means and their bands are averages of the six per\-task standard deviations, with88seeds per task\. The other baseline curves and bands are taken from Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\], except RLPD, which we ran ourselves\.[Figures5](https://arxiv.org/html/2610.22497#S5.F5),[6](https://arxiv.org/html/2610.22497#A1.F6)and[7](https://arxiv.org/html/2610.22497#A1.F7)use88seeds per variant, and shading shows one standard deviation\.

### C\.3Baseline tuning

#### Reference results\.

For offline OGBench results onpuzzle\-4x4\-play,cube\-double\-play, andcube\-triple\-play, we report the baseline numbers from Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]\. We run the remaining baselines ourselves and tune them as described below\. For offline\-to\-online OGBench results, we report the numbers from Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]for all baselines except BFN\.

#### Tuning procedure\.

We follow the tuning protocol of Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]: each hyperparameter is swept on the domain’s default task over the values in[Table5](https://arxiv.org/html/2610.22497#A3.T5)\. The BC policy is Gaussian with four hidden layers of width512512, and CODAC’s conservative penalty is fixed at0\.10\.1\.

#### BFN\.

BFN shares the actor, discount, critic ensemble, and training schedule ofGVC\([Table4](https://arxiv.org/html/2610.22497#A3.T4)\) and replaces the generative critic with the regression critic of QC\[[30](https://arxiv.org/html/2610.22497#bib.bib30)\]\.

Table 5:Hyperparameter search spaces for tuning our baselines, following Value Flows\[[14](https://arxiv.org/html/2610.22497#bib.bib14)\]\. BC denotes behavioral cloning; BCFM denotes bootstrapped conditional flow matching\.MethodHyperparameterSearch valuesIQLAWR inverse temperature\{0\.3,1,3,10\}\\\{0\.3,1,3,10\\\}ReBRACActor BC coefficient\{0\.003,0\.01,0\.03,0\.1,0\.3,1\}\\\{0\.003,0\.01,0\.03,0\.1,0\.3,1\\\}Critic BC coefficient\{0,0\.001,0\.01,0\.1\}\\\{0,0\.001,0\.01,0\.1\\\}IFQLAction candidates\{32,64,128\}\\\{32,64,128\\\}FBRACBC coefficient\{1,3,10,30,100,300\}\\\{1,3,10,30,100,300\\\}FQLBC distillation coefficient\{3,10,30,100,300,1000\}\\\{3,10,30,100,300,1000\\\}C51Distributional atoms\{51,101\}\\\{51,101\\\}IQN, CODACQuantile\-lossκ\\kappa\{0\.7,0\.8,0\.9,0\.95\}\\\{0\.7,0\.8,0\.9,0\.95\\\}CODACBC coefficient\{100,300,1000,3000,10000,30000\}\\\{100,300,1000,3000,10000,30000\\\}Value FlowsBCFM coefficientλ\\lambda\{0\.3,0\.5,1,3,10\}\\\{0\.3,0\.5,1,3,10\\\}Confidence temperatureτ\\tau\{0\.01,0\.03,0\.1,0\.3,1,3\}\\\{0\.01,0\.03,0\.1,0\.3,1,3\\\}Online distillation coefficient\{10,30,100,300\}\\\{10,30,100,300\\\}