Learning More from Less: Reinforcement Learning from Hindsight

# Learning More from Less: Reinforcement Learning from Hindsight
Source: [https://arxiv.org/html/2607.09042](https://arxiv.org/html/2607.09042)
Iris Xu1,2,\*,Sunshine Jiang1,John Marangola1,Nitish Dashora1,Richard Li1,Thomas Liu1,Zexue He3,Yuheng Zhi4,Alex Pentland3,Pulkit Agrawal1,Zhang\-Wei Hong1,21Massachusetts Institute of Technology2MIT\-IBM Computing Research Lab3Stanford University4University of California, San Diego

###### Abstract

Reinforcement learning \(RL\) is increasingly used to post\-train vision\-language\-action \(VLA\) models, but every update consumes robot rollouts that are slow and costly to collect, making sample efficiency a central concern\. Manipulation tasks typically provide only sparse rewards, so a weak policy fails almost every rollout early in training and has little to learn from, even when those failures execute coherent behavior\. Such a failure, however, is a success at a different task\. We present Learning from Hindsight \(LfH\), which brings hindsight relabeling to RL post\-training of VLAs by scoring failed rollouts against the tasks they actually achieved\. A single vision\-language model relabels both the instruction and the reward, proposing a hindsight instruction for a group of failed rollouts and scoring how well each satisfies it, and the policy trains on the relabeled and original rollouts jointly\. Because VLAs generalize across language, relabeling in language lets the policy learn more from the same trajectories\. On out\-of\-distribution LIBERO\-PRO tasks, where standard RL improves only slowly, LfH achieves5×5\\timesimprovement in sample efficiency, and outperforms a dense progress\-reward baseline\. The gains hold across VLA backbones and on a physical Franka robot\.

††footnotetext:\*Correspondence:irisxu@mit\.edu## 1Introduction

Reinforcement learning \(RL\)\[[10](https://arxiv.org/html/2607.09042#bib.bib20)\]has become a standard tool for post\-training large language models\[[19](https://arxiv.org/html/2607.09042#bib.bib21),[1](https://arxiv.org/html/2607.09042#bib.bib22)\], and is increasingly being used to fine\-tune vision\-language\-action \(VLA\) models for robot control\[[6](https://arxiv.org/html/2607.09042#bib.bib23)\]\. In robotics, however, post\-training is constrained by a much harsher data bottleneck: each RL update depends on rollouts collected from a physical robot, which are slow, expensive, and often difficult to scale\. Improving sample efficiency is therefore important for making RL post\-training practical for VLAs\.

This challenge is especially acute in manipulation, where VLAs are commonly fine\-tuned with sparse rewards\. A rollout receives reward only if the robot completes the commanded instruction, leaving all other behavior uncredited\. Sparse rewards are a long\-standing obstacle in RL\[[4](https://arxiv.org/html/2607.09042#bib.bib42),[20](https://arxiv.org/html/2607.09042#bib.bib41)\]: before a weak policy can improve, it must first discover a successful trajectory through exploration\. Early in training, nearly all rollouts therefore appear useless\. For example, when instructed to “close the microwave,” the robot may instead pick up a cup \(Figure[1](https://arxiv.org/html/2607.09042#S1.F1)\)\. This behavior is coherent and task\-relevant in a broad manipulation sense, but it does not satisfy the commanded instruction\. Standard RL treats the entire rollout as a failure, ignoring the fact that the robot successfully executed another meaningful skill\.

Our key observation is that many failed rollouts are failures only relative to the original instruction\. Hindsight relabeling\[[2](https://arxiv.org/html/2607.09042#bib.bib8),[22](https://arxiv.org/html/2607.09042#bib.bib1),[21](https://arxiv.org/html/2607.09042#bib.bib96),[8](https://arxiv.org/html/2607.09042#bib.bib10),[23](https://arxiv.org/html/2607.09042#bib.bib24)\]exploits this by evaluating a trajectory against the task it actually achieved rather than the task it was commanded to solve\. The cup\-picking rollout, for instance, can be relabeled as an instance of “pick up the cup\.” This is distinct from dense progress rewards\[[13](https://arxiv.org/html/2607.09042#bib.bib59)\], which provide finer feedback for the same commanded instruction\. Hindsight changes the task assignment itself, converting an otherwise unrewarded failure into a successful demonstration of a different, related instruction\. Since VLAs are conditioned on language and can generalize across instructions, such relabeled rollouts provide useful supervision from data that standard RL would ignore\. The remaining challenge is to infer, in language, what the robot actually did; a pretrained vision\-language model \(VLM\)\[[3](https://arxiv.org/html/2607.09042#bib.bib19)\]can provide this description directly from the rollout observations\.

We proposeLearning from Hindsight \(LfH\), a method for RL post\-training of VLAs that turns failed rollouts into additional training signal\. LfH uses a single VLM to relabel both instructions and rewards\. Given a group of rollouts that fail the commanded instruction, the VLM examines one rollout and proposes a hindsight instruction describing the behavior the robot achieved\. It then scores how well each rollout in the group satisfies that hindsight instruction\. The VLA is trained jointly on the original rollouts and their hindsight\-relabeled counterparts, allowing the policy to learn both from intended successes and from unintended but meaningful behaviors\.

We evaluate LfH by fine\-tuning VLAs on out\-of\-distribution LIBERO\-PRO tasks\[[32](https://arxiv.org/html/2607.09042#bib.bib45)\], where the initial policy has near\-zero success and standard RL improves only slowly\. LfH recovers usable signal from most failed rollouts, reaches the final success rate of standard RL in roughly one fifth of the training steps, and yields a5×5\\timesimprovement in sample efficiency\. It also outperforms a dense progress\-reward baseline, indicating that in sparse, low\-success regimes, changing which task a rollout is credited for can be more valuable than assigning denser feedback to the original command alone\. These gains hold across VLA backbones and transfer to a physical Franka robot\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x1.png)Figure 1:Learning from Hindsight\.A trajectory that fails the commanded instruction can still exhibit meaningful behavior\. LfH uses a VLM to relabel the achieved behavior in language, turning otherwise unused failures into training signal for VLA fine\-tuning\.
## 2Related Work

Hindsight Relabeling\.Hindsight Experience Replay \(HER\)\[[2](https://arxiv.org/html/2607.09042#bib.bib8)\]established this idea by relabeling goals with the states that were actually achieved by the agent as opposed to the original goal\. HER has also been extended to image goals with imagined goal generators\[[23](https://arxiv.org/html/2607.09042#bib.bib24),[33](https://arxiv.org/html/2607.09042#bib.bib93)\], goal\-conditioned value functions\[[8](https://arxiv.org/html/2607.09042#bib.bib10)\], and self\-supervised approaches\[[21](https://arxiv.org/html/2607.09042#bib.bib96)\]\. Concurrent work on open\-ended instruction relabeling\[[31](https://arxiv.org/html/2607.09042#bib.bib14)\]demonstrates language\-based hindsight in game\-like environments, while we provide a unique algorithm and implementation of language\-based hindsight that we prove is effective in the robotics setting\.

Reward Shaping and Foundation Reward Models\.Instead of hindsight relabeling, a classic way to address sparse\-reward RL is reward shaping, which provides intermediate feedback that guides the agent toward the desired behavior\[[18](https://arxiv.org/html/2607.09042#bib.bib95)\]\. Since such shaping can be prohibitively expensive to design by hand, recent work has proposed large\-scale dense reward models for transferable reward learning\. Prior work trains text\-conditioned progress models, either as goal\-reaching value functions\[[17](https://arxiv.org/html/2607.09042#bib.bib54),[16](https://arxiv.org/html/2607.09042#bib.bib55)\], with interpolation heuristics over demonstrations\[[15](https://arxiv.org/html/2607.09042#bib.bib56),[13](https://arxiv.org/html/2607.09042#bib.bib59),[29](https://arxiv.org/html/2607.09042#bib.bib82)\], or using preference labels\[[25](https://arxiv.org/html/2607.09042#bib.bib57)\]\. LfH is complementary to dense progress rewards: rather than densifying feedback for fixed tasks, LfH generates new diverse tasks with relabelling and bridges success to the original tasks via generalization\.

Improving Language Steerability\.Besides reinforcement learning, another way to raise VLA task completion is to strengthen language steerability\. A common strategy is data augmentation under an imitation learning or offline RL paradigm, using VLMs to relabel or synthesize instructions over an offline dataset to expand task diversity\[[27](https://arxiv.org/html/2607.09042#bib.bib16),[30](https://arxiv.org/html/2607.09042#bib.bib15),[9](https://arxiv.org/html/2607.09042#bib.bib5),[28](https://arxiv.org/html/2607.09042#bib.bib6)\]\. A complementary direction steers at inference time:Wuet al\.\[[26](https://arxiv.org/html/2607.09042#bib.bib7)\]use a VLM to pick the sampled action best matching the VLA’s original text instruction\. Our method differs from these methods as it is 1\) online and 2\) bakes the signal from the relabelled tasks into the weights, instead of doing test\-time steering\.

## 3Preliminaries

We study RL fine\-tuning of vision\-language\-action \(VLA\) policies\[[6](https://arxiv.org/html/2607.09042#bib.bib23),[11](https://arxiv.org/html/2607.09042#bib.bib32)\]\. At the start of each episode the environment samples an instructiong∼Pgg\\sim P\_\{g\}and an initial RGB observationo0o\_\{0\}\. A policyπθ\\pi\_\{\\theta\}rolls out actionsat∼πθ\(⋅∣ot,g\)a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{t\},g\)forTTsteps, yielding a trajectoryτ=\(o0,a0,…,aT−1,oT\)\\tau=\(o\_\{0\},a\_\{0\},\\ldots,a\_\{T\-1\},o\_\{T\}\)that receives a binary success rewardR​\(τ,g\)R\(\\tau,g\), equal to11whenτ\\taucompletesggand0otherwise\. The objective is to maximize expected successJ​\(θ\)=𝔼g∼Pg,τ∼πθ\(⋅∣g\)​\[R​\(τ,g\)\]J\(\\theta\)=\\mathbb\{E\}\_\{g\\sim P\_\{g\},\\,\\tau\\sim\\pi\_\{\\theta\}\(\\cdot\\mid g\)\}\[R\(\\tau,g\)\]\.

We instantiate LfH with group relative policy optimization \(GRPO\)\[[24](https://arxiv.org/html/2607.09042#bib.bib26)\]\. For each instructiongg, GRPO samples a group ofKKtrajectories\{τi\}i=1K\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{K\}fromπθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}, scores them withRi=R​\(τi,g\)R\_\{i\}=R\(\\tau\_\{i\},g\), and normalizes advantages within the group:

A^i=Ri−μRσR\+δ,μR=1K​∑j=1KRj,\\displaystyle\\hat\{A\}\_\{i\}=\\frac\{R\_\{i\}\-\\mu\_\{R\}\}\{\\sigma\_\{R\}\+\\delta\},\\qquad\\mu\_\{R\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}R\_\{j\},\(1\)whereσR\\sigma\_\{R\}is the group standard deviation andδ\\deltaa small constant\. With the importance ratio

ri,t​\(θ\)=πθ​\(ai,t∣oi,t,g\)πθold​\(ai,t∣oi,t,g\),\\displaystyle r\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{i,t\}\\mid o\_\{i,t\},g\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{i,t\}\\mid o\_\{i,t\},g\)\},\(2\)and a KL penaltyDi,t​\(θ\)D\_\{i,t\}\(\\theta\)to a reference policy, GRPO maximizes

ℒGRPO​\(θ\)=𝔼g,\{τi\}​\[1K​∑i=1K∑t=0T−1ℓi,t​\(θ\)\],ℓi,t=min⁡\(ri,t​A^i,r¯i,t​A^i\)−β​Di,t,\\displaystyle\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{g,\\\{\\tau\_\{i\}\\\}\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\sum\_\{t=0\}^\{T\-1\}\\ell\_\{i,t\}\(\\theta\)\\right\],\\qquad\\ell\_\{i,t\}=\\min\(r\_\{i,t\}\\hat\{A\}\_\{i\},\\,\\bar\{r\}\_\{i,t\}\\hat\{A\}\_\{i\}\)\-\\beta D\_\{i,t\},\(3\)withr¯i,t=clip​\(ri,t,1−ϵ,1\+ϵ\)\\bar\{r\}\_\{i,t\}=\\mathrm\{clip\}\(r\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\. Groups with zero reward varianceσR\\sigma\_\{R\}provide no learning signal and are discarded\.

## 4Method: Learning from Hindsight \(LfH\)

Problem\.A GRPO group provides no learning signal when all its trajectories receive the same reward\. All\-one groups\{τi∣Ri=1\}\\\{\\tau\_\{i\}\\mid R\_\{i\}=1\\\}already contain successful behavior; the bottleneck is all\-zero groups\{τi∣Ri=0\}\\\{\\tau\_\{i\}\\mid R\_\{i\}=0\\\}, which have zero normalized advantage in Eq\.[1](https://arxiv.org/html/2607.09042#S3.E1)and contribute no gradient\. Since the reward depends on both the trajectoryτ\\tauand the commanded instructiongg, a weak initial policy fails most rollouts, leaving many groups all\-zero and fine\-tuning makes little progress\. LfH recovers learning signal from these otherwise discarded trajectories\.

Approach\.A trajectory that fails under one instruction may succeed under another\. A robot commanded to “close the microwave” may instead pick up a mug: this rollout fails forg=“close the microwave”g=\\text\{\`\`close the microwave''\}but succeeds for the hindsight instructiong′=“pick up the mug”g^\{\\prime\}=\\text\{\`\`pick up the mug''\}\. LfH relabels failed trajectories with instructions describing what the robot actually accomplished, converting zero\-reward samples\(τ,g\)\(\\tau,g\)into successful examples\(τ,g′\)\(\\tau,g^\{\\prime\}\)\. The hindsight instructiong′g^\{\\prime\}need not be drawn fromPgP\_\{g\}: in the low\-success regime a failure under the original instruction gives no signal, while a hindsight\-labeled success does\. LfH lets the policy learn from what it already manages to do before it can reliably solve the commanded tasks\. Two questions remain: \(1\) how to generate hindsight\-labeled data \(Section[4\.1](https://arxiv.org/html/2607.09042#S4.SS1)\), and \(2\) how to incorporate it into GRPO \(Section[4\.2](https://arxiv.org/html/2607.09042#S4.SS2)\)\.

### 4\.1Training Signals from Hindsight

For each commanded instructiongg, GRPO samples a group\{τi\}i=1K\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{K\}and scores it withRi=R​\(τi,g\)R\_\{i\}=R\(\\tau\_\{i\},g\)\. LfH activates only on low\-signal groups, i\.e\. those with mean reward1K​∑iRi<η\\frac\{1\}\{K\}\\sum\_\{i\}R\_\{i\}<\\eta; high\-reward groups already reinforce the commanded behavior, so relabeling them would replace a correct training signal with an alternative one\. Given a low\-reward group, LfH converts it into a hindsight\-labeled group in two steps\.

Instruction relabeling\.We infer an alternative prompt under which a failed rollout still provides supervision\. Since robot rollouts may lack privileged object states or task annotations, the VLM relabelerℳψ\\mathcal\{M\}\_\{\\psi\}works from available trajectory information such as RGB observations and actions\. LfH samples an anchor from the failed trajectories,i⋆∼Unif​\(\{i:Ri=0\}\)i^\{\\star\}\\sim\\mathrm\{Unif\}\(\\\{i:R\_\{i\}=0\\\}\), since successful trajectories already carry a reliable label\. The VLM then generates a hindsight instructiong′g^\{\\prime\}describing what the anchor did:

g′∼ℳψinst\(⋅∣τi⋆\)\.g^\{\\prime\}\\sim\\mathcal\{M\}^\{\\mathrm\{inst\}\}\_\{\\psi\}\(\\cdot\\mid\\tau\_\{i^\{\\star\}\}\)\.\(4\)Not every failure is useful: some trajectories contain only accidental motion\. To avoid noisy prompts, we first ask the VLM to classify whether the anchor contains meaningful behavior and discard those labeled uninteresting\.

Reward relabeling\.LfH then evaluates every trajectory in the group under the shared hindsight instructiong′g^\{\\prime\}:

R~i=ℳψrew​\(τi,g′\)∈\{0,0\.5,1\},\\tilde\{R\}\_\{i\}=\\mathcal\{M\}^\{\\mathrm\{rew\}\}\_\{\\psi\}\(\\tau\_\{i\},g^\{\\prime\}\)\\in\\\{0,0\.5,1\\\},\(5\)where11denotes completion ofg′g^\{\\prime\},0failure, and0\.50\.5an ambiguous outcome\. A shared instruction is essential because GRPO computes advantages relative to other trajectories in the group; relabeling each with a different prompt would make their rewards incomparable\. This yields the hindsight group𝒢~=\{\(τi,g′,R~i\)\}i=1K\\tilde\{\\mathcal\{G\}\}=\\\{\(\\tau\_\{i\},g^\{\\prime\},\\tilde\{R\}\_\{i\}\)\\\}\_\{i=1\}^\{K\}, which GRPO consumes like an ordinary rollout group, computing advantages from\{R~i\}\\\{\\tilde\{R\}\_\{i\}\\\}in place of\{Ri\}\\\{R\_\{i\}\\\}\. Because relabeling fires only on low\-reward groups, LfH leaves the standard GRPO update untouched wherever the commanded instruction already provides signal, and acts only on failed groups that would otherwise yield no gradient\.

### 4\.2GRPO with Hindsight Relabeling

For a hindsight\-labeled group𝒢~\\tilde\{\\mathcal\{G\}\}, we compute group\-normalized hindsight advantages

A~i=R~i−μ~Rσ~R\+δ,μ~R=1K​∑j=1KR~j,\\tilde\{A\}\_\{i\}=\\frac\{\\tilde\{R\}\_\{i\}\-\\tilde\{\\mu\}\_\{R\}\}\{\\tilde\{\\sigma\}\_\{R\}\+\\delta\},\\qquad\\tilde\{\\mu\}\_\{R\}=\\frac\{1\}\{K\}\\sum\_\{j=1\}^\{K\}\\tilde\{R\}\_\{j\},\(6\)withσ~R\\tilde\{\\sigma\}\_\{R\}the standard deviation of\{R~j\}j=1K\\\{\\tilde\{R\}\_\{j\}\\\}\_\{j=1\}^\{K\}\. Since the trajectories were rolled out under the commanded instructionggbut we now optimize the policy at the hindsight instructiong′g^\{\\prime\}, we apply an importance correction following hindsight policy gradients\[[22](https://arxiv.org/html/2607.09042#bib.bib1)\], replacing the standard ratio with

r~i,t​\(θ\)=πθ​\(ai,t∣oi,t,g′\)πθold​\(ai,t∣oi,t,g\),\\tilde\{r\}\_\{i,t\}\(\\theta\)=\\frac\{\\pi\_\{\\theta\}\(a\_\{i,t\}\\mid o\_\{i,t\},g^\{\\prime\}\)\}\{\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{i,t\}\\mid o\_\{i,t\},g\)\},\(7\)which evaluates the action under the hindsight instructiong′g^\{\\prime\}while correcting for its having been sampled undergg\. WithD~i,t​\(θ\)\\tilde\{D\}\_\{i,t\}\(\\theta\)the KL penalty betweenπθ\(⋅∣oi,t,g′\)\\pi\_\{\\theta\}\(\\cdot\\mid o\_\{i,t\},g^\{\\prime\}\)and the reference policy, the hindsight objective is

ℒH​\-​GRPO​\(θ\)=𝔼𝒢~​\[1K​∑i=1K∑t=0T−1ℓ~i,t​\(θ\)\],ℓ~i,t=min⁡\(r~i,t​A~i,r¯~i,t​A~i\)−β​D~i,t,\\mathcal\{L\}\_\{\\mathrm\{H\\text\{\-\}GRPO\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tilde\{\\mathcal\{G\}\}\}\\\!\\left\[\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}\\sum\_\{t=0\}^\{T\-1\}\\tilde\{\\ell\}\_\{i,t\}\(\\theta\)\\right\],\\quad\\tilde\{\\ell\}\_\{i,t\}=\\min\(\\tilde\{r\}\_\{i,t\}\\tilde\{A\}\_\{i\},\\,\\tilde\{\\bar\{r\}\}\_\{i,t\}\\tilde\{A\}\_\{i\}\)\-\\beta\\tilde\{D\}\_\{i,t\},\(8\)wherer¯~i,t=clip​\(r~i,t,1−ϵ,1\+ϵ\)\\tilde\{\\bar\{r\}\}\_\{i,t\}=\\mathrm\{clip\}\(\\tilde\{r\}\_\{i,t\},1\-\\epsilon,1\+\\epsilon\)\. We train on both commanded and hindsight groups,

ℒLfH​\(θ\)=ℒGRPO​\(θ\)\+λ​ℒH​\-​GRPO​\(θ\),\\mathcal\{L\}\_\{\\mathrm\{LfH\}\}\(\\theta\)=\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\theta\)\+\\lambda\\,\\mathcal\{L\}\_\{\\mathrm\{H\\text\{\-\}GRPO\}\}\(\\theta\),\(9\)so the GRPO term keeps training aligned withPgP\_\{g\}while the hindsight term, weighted byλ\\lambda, recovers signal from low\-reward groups\.

Algorithm 1Learning from Hindsight \(LfH\)0:Policy

πθ\\pi\_\{\\theta\}, VLM relabeler

ℳψ\\mathcal\{M\}\_\{\\psi\}, instruction distribution

PgP\_\{g\}, group size

KK, threshold

η\\eta, hindsight weight

λ\\lambda
1:repeat

2:

ℬ←∅\\mathcal\{B\}\\leftarrow\\emptyset,

ℬ~←∅\\tilde\{\\mathcal\{B\}\}\\leftarrow\\emptyset,

πθold←πθ\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\\leftarrow\\pi\_\{\\theta\}
3:foreach sampled instruction

g∼Pgg\\sim P\_\{g\}do

4:Roll out

𝒢=\{\(τi,g,Ri\)\}i=1K\\mathcal\{G\}=\\\{\(\\tau\_\{i\},g,R\_\{i\}\)\\\}\_\{i=1\}^\{K\}with

πθold\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\};

ℬ←ℬ∪𝒢\\mathcal\{B\}\\leftarrow\\mathcal\{B\}\\cup\\mathcal\{G\}
5:if

1K​∑i=1KRi<η\\frac\{1\}\{K\}\\sum\_\{i=1\}^\{K\}R\_\{i\}<\\etathen

6:Generate hindsight group

𝒢~\\tilde\{\\mathcal\{G\}\}from

𝒢\\mathcal\{G\}via

ℳψ\\mathcal\{M\}\_\{\\psi\};

ℬ~←ℬ~∪𝒢~\\tilde\{\\mathcal\{B\}\}\\leftarrow\\tilde\{\\mathcal\{B\}\}\\cup\\tilde\{\\mathcal\{G\}\}
7:endif

8:endfor

9:Update

θ\\thetausing

ℒGRPO​\(ℬ\)\+λ​ℒH​\-​GRPO​\(ℬ~\)\\mathcal\{L\}\_\{\\mathrm\{GRPO\}\}\(\\mathcal\{B\}\)\+\\lambda\\mathcal\{L\}\_\{\\mathrm\{H\\text\{\-\}GRPO\}\}\(\\tilde\{\\mathcal\{B\}\}\)
10:untilconvergence

## 5Experiments

We organize our experiments around the following questions: \(1\) Does LfH improve the sample efficiency of GRPO fine\-tuning under sparse rewards? \(Sections[5\.1](https://arxiv.org/html/2607.09042#S5.SS1)and[5\.2](https://arxiv.org/html/2607.09042#S5.SS2)\) \(2\) How does LfH improve sample efficiency? \(Section[5\.3](https://arxiv.org/html/2607.09042#S5.SS3)\) \(3\) Is LfH effective on physical robot experiments? \(Section[5\.4](https://arxiv.org/html/2607.09042#S5.SS4)\)\. Additional experimental results are present in Appendix[A\.1](https://arxiv.org/html/2607.09042#A1.SS1)and[A\.2](https://arxiv.org/html/2607.09042#A1.SS2)\.

General experiment setup\.Unless otherwise specified, all methods are initialized from the sameRLinf\-Pi05\-LIBERO\-SFTcheckpoint\[[7](https://arxiv.org/html/2607.09042#bib.bib13)\], which was trained with few\-shot demonstrations on the four LIBERO task suites\[[14](https://arxiv.org/html/2607.09042#bib.bib27)\]in simulation\. We implement LfH and the baselines in RLinf111[https://rlinf\.readthedocs\.io/](https://rlinf.readthedocs.io/)\. LfH usesQwen3\-VL\-235B\-A22B\-Thinking\-FP8for both instruction relabeling and reward relabeling \(Section[4\.1](https://arxiv.org/html/2607.09042#S4.SS1)\)\. Trajectories are represented as sequences of video frames from the global camera facing the robot\.

### 5\.1Illustrative Example

We use a task “close the microwave,” from LIBERO\-90 to illustrate how LfH improves sample efficiency and helps RL overcome a cold start from a weak initial policy\.

LfH makes better use of collected rollouts\.The initial policy has zero success on the target task\. Consequently, during standard GRPO training, almost all trajectory groups receive all\-zero rewards and are discarded, as shown in Figure[2\(a\)](https://arxiv.org/html/2607.09042#S5.F2.sf1)\(a\)\. In contrast, LfH retains nearly 80% of trajectory groups after hindsight relabeling\. Since GRPO filters out groups with no reward variation, this result shows that LfH recovers a large fraction of data that would otherwise provide no learning signal\. The remaining 20% of groups are still discarded because they correspond to uninformative failures for which the VLM relabeler does not assign a meaningful hindsight instruction and rewards\.

LfH improves sample efficiency under sparse rewards\.By converting otherwise zero\-reward failures into relabeled training examples, LfH enables GRPO to update from useful data even when the original task reward is absent\. As shown in Figure[2\(a\)](https://arxiv.org/html/2607.09042#S5.F2.sf1), GRPO\+LfH steadily improves success on the target task over training, whereas standard GRPO remains at zero success throughout\. This suggests that hindsight relabeling can provide the initial learning signal needed to escape cold\-start regimes where sparse task rewards alone are insufficient\.

What does LfH relabel?Figure[2\(b\)](https://arxiv.org/html/2607.09042#S5.F2.sf2)visualizes representative relabeled trajectories\. In this example, the initial policy repeatedly grasps and bumps the mug, behavior that is unrelated to the commanded task “close the microwave\.” LfH relabels these failures with instructions describing what the robot actually did\. Interestingly, although these hindsight instructions are not semantically related to the target task, the relabeled data still help RL make progress on the target policy\. One possible explanation is that relabeling provides a contrastive grounding signal: trajectories that do not complete the target task are explicitly associated with alternative instructions, helping the policy distinguish the target instruction from behaviors induced by spurious or irrelevant actions\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x2.png)\(a\)
“Tip over the white mug” ![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/illustrative/frames/traj_Tip_over_the_white_mug_3f.png) “Pick up the mug and move it left” ![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/illustrative/frames/traj_Pick_up_the_mug_and_move_it_le_3f.png)

\(b\)

Figure 2:\(a\)Fraction of groups kept in GRPO versus LfH during training\. LfH keeps more trajectory groups for training through relabeling, while GRPO discards most groups\. LfH also escapes the weak initial policy with zero success rate and reaches 60% success\.\(b\)Example hindsight relabeled trajectories with their respective VLM\-relabled instructions\. Although the relabeled instructions are not the target task of closing the microwave, they provide useful learning signal\.
### 5\.2Can LfH improve sample efficiency?

Setup\.We evaluate whether LfH can accelerate RL fine\-tuning when the initial VLA policy is weak\. To this end, we construct out\-of\-distribution \(OOD\) tasks from the LIBERO\-PRO dataset with the task perturbation\. These tasks preserve the original scenes, object instances, and environment dynamics, but change the task specification, such as the instructed object, target state, or desired action\. For example, a policy may have seen the same kitchen scene during SFT, but must now manipulate a new object or reach a different goal state\. This setting isolates the central challenge of sample\-efficient RL: improving from mostly failed rollouts\.

Implementation & Baselines\.We train each method for 40 steps, using the same number of collected rollouts for both GRPO and LfH, since GRPO plateaus\. For all experiments, we run with four random seeds and report the mean and standard error at each step\. We compare LfH with standard GRPO as well as GRPO trained with RoboMETER dense progress rewards\[[13](https://arxiv.org/html/2607.09042#bib.bib59)\]\. Standard GRPO uses the sparse environment success reward under the original instruction\. GRPO\+RoboMETER replaces this sparse signal with a dense progress reward toward this instruction\. This baseline tests whether providing denser rewards is sufficient to match the sample\-efficiency benefit of LfH\. All baselines are trained for the same number of steps and with the same number of collected environment rollouts\.

Metric\.During training, we periodically freeze the policy and evaluate it with 480 rollouts\. We report the normalized gain over the initial policy\. LetSR0\\mathrm\{SR\}\_\{0\}denote the success rate of the initial policy, and letSRt\\mathrm\{SR\}\_\{t\}denote the evaluation success rate after collectingtttraining rollouts\. We defineGain​\(t\)=SRtSR0−1\.\\mathrm\{Gain\}\(t\)=\\frac\{\\mathrm\{SR\}\_\{t\}\}\{\\mathrm\{SR\}\_\{0\}\}\-1\.This metric measures how quickly each method improves relative to the initial checkpoint as more rollouts are collected\. Additional implementation and evaluation details are provided in Appendix[B](https://arxiv.org/html/2607.09042#A2)\.

Results\.Figure[3\(a\)](https://arxiv.org/html/2607.09042#S5.F3.sf1)shows that LfH learns much faster than GRPO\. LfH matches GRPO’s final performance in about 5 training steps, while GRPO takes nearly 30 steps, yielding roughly a5×5\\timesgain in sample efficiency\. LfH also outperforms GRPO with RoboMETER rewards, indicating that dense progress feedback alone is not sufficient when the policy rarely makes progress toward the commanded instruction\.

Figure[3\(b\)](https://arxiv.org/html/2607.09042#S5.F3.sf2)helps explain this gain\. Standard GRPO and GRPO\+RoboMETER drop the groups of trajectories with zero reward variance \(Section[3](https://arxiv.org/html/2607.09042#S3)\) and keep only about 20–40% of trajectory groups for training, while LfH keeps roughly 70–80%\. Thus, LfH does not merely assign denser rewards; it changes how much of the collected experience becomes useful for learning\. By relabeling failed rollouts with hindsight instructions, LfH turns many otherwise discarded groups into positive training signal\.

Generality\.Finally, we test whether this benefit transfers beyond a single VLA backbone\. Using the same training protocol, we fine\-tune additional VLA models: GR00T\[[5](https://arxiv.org/html/2607.09042#bib.bib46)\]for 200 steps and OpenVLA\-OFT\[[11](https://arxiv.org/html/2607.09042#bib.bib32)\]for 60 steps on the LIBERO PRO Spatial, Goal, and Object suites \(again with task perturbation\)\. Figure[3\(c\)](https://arxiv.org/html/2607.09042#S5.F3.sf3)shows that LfH also consistently improves sample efficiency across these VLAs\.

Figure 3:\(a\)Gain during training for vanilla GRPO, Robometer, and LfH\. LfH improves sample efficiency over GRPO and RoboMETER, achieving higher gain with fewer training steps\.\(b\)The fraction of groups kept during training\. LfH keeps a larger fraction of trajectory groups usable for training, making better use of collected data\.\(c\)Final gain for other VLA models\. The benefit of LfH transfers beyondπ0\.5\\pi\_\{0\.5\}, improving the performance on both GR00T and OpenVLA\-OFT\.
### 5\.3Analysis: How does LfH improve sample efficiency?

Can LfH performance gain be attributed to instruction augmentation or reward relabeling alone?We isolate the two steps in LfH: hindsight instruction relabeling and reward relabeling \(Section[4](https://arxiv.org/html/2607.09042#S4)\)\. We compare against two ablations\.*Rephrase\-only*replaces the hindsight instructiong′g^\{\\prime\}with a paraphrase of the original instructionggwhile keeping the original rewards, testing whether language augmentation alone explains the gain\.*Reward\-only*keeps the instruction asggbut relabels rewards with the VLM, testing whether reward relabeling alone explains the gain\. Figure[4\(a\)](https://arxiv.org/html/2607.09042#S5.F4.sf1)shows that both ablations marginally improve over GRPO, remain substantially below the full LfH method throughout training\. This indicates that LfH is not merely instruction augmentation or reward relabeling; its benefit comes from coupling the two\.

Can LfH performance gain be attributed to perturbed rewards?We next test whether LfH works simply because it injects additional reward variance into otherwise discarded groups\. The random\-reward baseline creates nonzero advantages without using semantically meaningful hindsight labels\. The result in Figure[4\(a\)](https://arxiv.org/html/2607.09042#S5.F4.sf1)shows that the limited gains from using random rewards in the modified batch do not provide enough additional learning signal to account for the gains achieved by LfH\. Thus, the benefit of LfH is not merely due to avoiding zero\-variance groups\. Semantically grounded hindsight rewards provide useful credit assignment, whereas arbitrary reward perturbations produce noisy gradients that do not reliably improve the commanded tasks\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x7.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.09042v1/x8.png)\(b\)

Figure 4:\(a\)Comparison of full LfH with ablations that remove/modify parts of the hindsight relabeling mechanism\. LfH’s gains come from jointly relabeling instructions and rewards; instruction rephrasing or reward relabeling alone, or additional reward variation does not substantially improve performance\.\(b\)Real\-world physical\-robot fine\-tuning results\. LfH improves success rate over GRPO across rollout budgets\.
### 5\.4Real\-world results

Setup\.We evaluate LfH on a real Franka FR3 robot with manipulable objects, two drawers, and a two\-level rack\. We first SFTπ0\.5\\pi\_\{0\.5\}on 10 language\-conditioned manipulation tasks using 20 SpaceMouse demonstrations per task, spanning short pick\-and\-place and longer multi\-stage tasks\. We then fine\-tune on a held\-out task,*put the green container into the bowl*, where the initial SFT policy has zero success\. Both GRPO and GRPO\+LfH use the same binary human reward\. See Appendix[B\.2](https://arxiv.org/html/2607.09042#A2.SS2)for details\.

Results\.Figure[4\(b\)](https://arxiv.org/html/2607.09042#S5.F4.sf2)shows that LfH improves real\-world sample efficiency\. Both methods start from zero, but LfH improves faster\. At 128 rollouts, LfH achieves roughly twice the success rate of GRPO; at 160 rollouts, LfH reaches56%56\\%success, while GRPO plateaus at22%22\\%\.

Analysis\.The initial policy often grounds prompts to the wrong object, such as grasping the tape regardless of the commanded language prompt\. GRPO treats these rollouts only as failures\. LfH instead relabels them with prompts describing the executed behavior, such as “pick up the tape” or “move the tape to the bowl,” turning failed target\-task attempts into useful supervision for object grounding and manipulation\.

## 6Discussion and Conclusion

More bits per robot sample\.LfH is motivated by a simple question: how many useful bits do we extract from each costly robot rollout? Classic RL uses a rollout mainly to evaluate whether the intended outcome happened, often reducing a rich trajectory to one reward\. This is inefficient when physical interaction is expensive\. Following the broader view that learning systems should extract more information from each sample\[[12](https://arxiv.org/html/2607.09042#bib.bib2)\], LfH treats a trajectory as evidence for all language goals consistent with what happened\. In this sense, hindsight relabeling increases the bits per sample: one rollout can supervise multiple possible tasks, rather than only the task originally commanded\.

Practical implication\.This is especially natural for pretrained VLAs\. Because these models already generalize across related language instructions, training on what the robot actually did can improve what it was asked to do\. LfH therefore changes the role of robot data: failed attempts are no longer merely negative examples, but reusable experience that can be mined for nearby tasks\. This offers a practical path for VLA post\-training: instead of only collecting more rollouts or taking more optimization steps on the same labels, we can reinterpret each rollout through language and transfer learning across the tasks it supports\.

Limitations\.LfH is limited by the behaviors present in the collected data\. If the policy produces repetitive or uninformative trajectories, there are few meaningful hindsight goals to recover\. It also depends on relabeling quality: incorrect or overly generic prompts can introduce noisy supervision\. Better filtering, uncertainty estimation, and lightweight human verification are important directions for reliable deployment\.

## References

- \[1\]J\. Achiam, S\. Adler, S\. Agarwal, L\. Ahmad, I\. Akkaya, F\. L\. Aleman, D\. Almeida, J\. Altenschmidt, S\. Altman, S\. Anadkat,et al\.\(2023\)GPT\-4 technical report\.arXiv preprint arXiv:2303\.08774\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p1.1)\.
- \[2\]M\. Andrychowicz, F\. Wolski, A\. Ray, J\. Schneider, R\. Fong, P\. Welinder, B\. McGrew, J\. Tobin, P\. Abbeel, and W\. Zaremba\(2017\)Hindsight experience replay\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.
- \[3\]S\. Bai, K\. Chen, X\. Liu, J\. Wang, W\. Ge, S\. Song, K\. Dang, P\. Wang, S\. Wang, J\. Tang, H\. Zhong, Y\. Zhu, M\. Yang, Z\. Li, J\. Wan, P\. Wang, W\. Ding, Z\. Fu, Y\. Xu, J\. Ye, X\. Zhang, T\. Xie, Z\. Cheng, H\. Zhang, Z\. Yang, H\. Xu, and J\. Lin\(2025\)Qwen2\.5\-vl technical report\.arXiv preprint arXiv:2502\.13923\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1)\.
- \[4\]\(2016\)Unifying count\-based exploration and intrinsic motivation\.Advances in neural information processing systems29\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p2.1)\.
- \[5\]J\. Bjorck, F\. Castañeda, N\. Cherniadev, X\. Da, R\. Ding, L\. Fan, Y\. Fang, D\. Fox, F\. Hu, S\. Huang,et al\.\(2025\)Gr00t n1: an open foundation model for generalist humanoid robots\.arXiv preprint arXiv:2503\.14734\.Cited by:[§5\.2](https://arxiv.org/html/2607.09042#S5.SS2.p6.1)\.
- \[6\]K\. Black, N\. Brown, D\. Driess, A\. Esmail, M\. Equi, C\. Finn, N\. Fusai, L\. Groom, K\. Hausman, B\. Ichter, S\. Jakubczak, T\. Jones, L\. Ke, S\. Levine, A\. Li\-Bell, M\. Mothukuri, S\. Nair, K\. Pertsch, L\. X\. Schneider, A\. Stone, Q\. Tan, J\. Tanner, M\. Toussaint, Q\. Vuong, and A\. Zeng\(2024\)π0\\pi\_\{0\}: A vision\-language\-action flow model for general robot control\.arXiv preprint arXiv:2410\.24164\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p1.1),[§3](https://arxiv.org/html/2607.09042#S3.p1.12)\.
- \[7\]K\. Chen, Z\. Liu, T\. Zhang, Z\. Guo, S\. Xu, H\. Lin, H\. Zang, X\. Li, Q\. Zhang, Z\. Yu, G\. Fan, T\. Huang, Y\. Wang, and C\. Yu\(2025\)πRL\\pi\_\{\\texttt\{RL\}\}: online rl fine\-tuning for flow\-based vision\-language\-action models\.External Links:2510\.25889,[Document](https://dx.doi.org/10.48550/arXiv.2510.25889),[Link](https://arxiv.org/abs/2510.25889)Cited by:[§5](https://arxiv.org/html/2607.09042#S5.p2.1)\.
- \[8\]B\. Eysenbach, T\. Zhang, R\. Salakhutdinov, and S\. Levine\(2022\)Contrastive learning as goal\-conditioned reinforcement learning\.InAdvances in Neural Information Processing Systems,Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.
- \[9\]C\. Glossop, W\. Chen, A\. Bhorkar, D\. Shah, and S\. Levine\(2025\)Cast: counterfactual labels improve instruction following in vision\-language\-action models\.arXiv preprint arXiv:2508\.13446\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p3.1)\.
- \[10\]L\. P\. Kaelbling, M\. L\. Littman, and A\. W\. Moore\(1996\)Reinforcement learning: a survey\.Journal of Artificial Intelligence Research4,pp\. 237–285\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p1.1)\.
- \[11\]M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. Foster, G\. Lam, P\. Sanketi,et al\.\(2024\)Openvla: an open\-source vision\-language\-action model\.arXiv preprint arXiv:2406\.09246\.Cited by:[§3](https://arxiv.org/html/2607.09042#S3.p1.12),[§5\.2](https://arxiv.org/html/2607.09042#S5.SS2.p6.1)\.
- \[12\]Y\. LeCun\(2022\)A path towards autonomous machine intelligence\.OpenReview\.Cited by:[§6](https://arxiv.org/html/2607.09042#S6.p1.1)\.
- \[13\]A\. Liang, Y\. Korkmaz, J\. Zhang, M\. Hwang, A\. Anwar, S\. Kaushik, A\. Shah, A\. S\. Huang, L\. Zettlemoyer, D\. Fox,et al\.\(2026\)Robometer: scaling general\-purpose robotic reward models via trajectory comparisons\.arXiv preprint arXiv:2603\.02115\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§2](https://arxiv.org/html/2607.09042#S2.p2.1),[§5\.2](https://arxiv.org/html/2607.09042#S5.SS2.p2.1)\.
- \[14\]B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. Stone\(2023\)LIBERO: benchmarking knowledge transfer for lifelong robot learning\.InAdvances in Neural Information Processing Systems,Vol\.36\.Cited by:[§5](https://arxiv.org/html/2607.09042#S5.p2.1)\.
- \[15\]Y\. J\. Ma, J\. Hejna, C\. Fu, D\. Shah, J\. Liang, Z\. Xu, S\. Kirmani, P\. Xu, D\. Driess, T\. Xiao,et al\.\(2025\)Vision language models are in\-context value learners\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[16\]Y\. J\. Ma, V\. Kumar, A\. Zhang, O\. Bastani, and D\. Jayaraman\(2023\)Liv: language\-image representations and rewards for robotic control\.InInternational Conference on Machine Learning,pp\. 23301–23320\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[17\]Y\. J\. Ma, S\. Sodhani, D\. Jayaraman, O\. Bastani, V\. Kumar, and A\. Zhang\(2022\)Vip: towards universal visual reward and representation via value\-implicit pre\-training\.arXiv preprint arXiv:2210\.00030\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[18\]A\. Y\. Ng, D\. Harada, and S\. J\. Russell\(1999\)Policy invariance under reward transformations: theory and application to reward shaping\.InProceedings of the Sixteenth International Conference on Machine Learning,ICML ’99,San Francisco, CA, USA,pp\. 278–287\.External Links:ISBN 1558606122Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[19\]L\. Ouyang, J\. Wu, X\. Jiang, D\. Almeida, C\. L\. Wainwright, P\. Mishkin, C\. Zhang, S\. Agarwal, K\. Slama, A\. Ray, J\. Schulman, J\. Hilton, F\. Kelton, L\. Miller, M\. Simens, A\. Askell, P\. Welinder, P\. Christiano, J\. Leike, and R\. Lowe\(2022\)Training language models to follow instructions with human feedback\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 27730–27744\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p1.1)\.
- \[20\]D\. Pathak, P\. Agrawal, A\. A\. Efros, and T\. Darrell\(2017\)Curiosity\-driven exploration by self\-supervised prediction\.InInternational conference on machine learning,pp\. 2778–2787\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p2.1)\.
- \[21\]D\. Pathak, P\. Mahmoudieh, G\. Luo, P\. Agrawal, D\. Chen, Y\. Shentu, E\. Shelhamer, J\. Malik, A\. A\. Efros, and T\. Darrell\(2018\)Zero\-shot visual imitation\.InProceedings of the IEEE conference on computer vision and pattern recognition workshops,pp\. 2050–2053\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.
- \[22\]P\. Rauber, A\. Ummadisingu, F\. Mutz, and J\. Schmidhuber\(2017\)Hindsight policy gradients\.arXiv preprint arXiv:1711\.06006\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§4\.2](https://arxiv.org/html/2607.09042#S4.SS2.p1.5)\.
- \[23\]H\. Sahni, T\. Buckley, P\. Abbeel, and I\. Kuzovkin\(2019\)Visual hindsight experience replay\.Proc\. of NeurIPS\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p3.1),[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.
- \[24\]Z\. Shao, P\. Wang, Q\. Zhu, R\. Xu, J\. Song, X\. Bi, H\. Zhang, M\. Zhang, Y\. K\. Li, Y\. Wu, and D\. Guo\(2024\)DeepSeekMath: pushing the limits of mathematical reasoning in open language models\.arXiv preprint arXiv:2402\.03300\.Cited by:[§3](https://arxiv.org/html/2607.09042#S3.p2.5)\.
- \[25\]Y\. Wang, Z\. Sun, J\. Zhang, Z\. Xian, E\. Biyik, D\. Held, and Z\. Erickson\(2024\)Rl\-vlm\-f: reinforcement learning from vision language foundation model feedback\.arXiv preprint arXiv:2402\.03681\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[26\]Y\. Wu, A\. Li, T\. Hermans, F\. Ramos, A\. Bajcsy, and C\. PÊrez\-D’Arpino\(2026\)Do what you say: steering vision\-language\-action models via runtime reasoning\-action alignment verification\.2026 IEEE International Conference on Robotics & Automation \(ICRA\)\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p3.1)\.
- \[27\]T\. Xiao, H\. Chan, P\. Sermanet, A\. Wahid, A\. Brohan, K\. Hausman, S\. Levine, and J\. Tompson\(2023\)Robotic skill acquisition via instruction augmentation with vision\-language models\.InRobotics: Science and Systems,External Links:2211\.11736,[Link](https://arxiv.org/abs/2211.11736)Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p3.1)\.
- \[28\]S\. Yang, H\. Li, B\. Wang, Y\. Chen, Y\. Tian, T\. Wang, H\. Wang, F\. Zhao, Y\. Liao, and J\. Pang\(2025\)Instructvla: vision\-language\-action instruction tuning from understanding to manipulation\.arXiv preprint arXiv:2507\.17520\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p3.1)\.
- \[29\]S\. Zhai, Q\. Zhang, T\. Zhang, F\. Huang, H\. Zhang, M\. Zhou, S\. Zhang, L\. Liu, S\. Lin, and J\. Pang\(2025\)A vision\-language\-action\-critic model for robotic real\-world reinforcement learning\.arXiv preprint arXiv:2509\.15937\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p2.1)\.
- \[30\]J\. Zhang, K\. Pertsch, J\. Zhang, and J\. J\. Lim\(2024\)Sprint: scalable policy pre\-training via language instruction relabeling\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 9168–9175\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p3.1)\.
- \[31\]Z\. Zhang, Z\. Wang, Y\. Du, and F\. Fang\(2025\)Learning instruction\-following policies through open\-ended instruction relabeling with large language models\.arXiv preprint arXiv:2506\.20061\.Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.
- \[32\]X\. Zhou, Y\. Xu, G\. Tie, Y\. Chen, G\. Zhang, D\. Chu, P\. Zhou, and L\. Sun\(2025\)LIBERO\-pro: towards robust and fair evaluation of vision\-language\-action models beyond memorization\.\[arXiv preprint arXiv:2510\.03827\]\.Cited by:[§1](https://arxiv.org/html/2607.09042#S1.p5.1)\.
- \[33\]Z\. Zhou, P\. Atreya, A\. Lee, H\. Walke, O\. Mees, and S\. Levine\(2024\)Autonomous improvement of instruction following skills via foundation models\.External Links:2407\.20635,[Link](https://arxiv.org/abs/2407.20635)Cited by:[§2](https://arxiv.org/html/2607.09042#S2.p1.1)\.

## Appendix AAdditional experimental results

### A\.1Additional ablation studies

What is the best way to select groups for hindsight relabeling?We ablate which rollout groups should be selected for hindsight relabeling\. Relabeling every trajectory is not necessarily beneficial: VLM relabeling adds compute, and applying hindsight updates to already successful or near\-saturated groups can introduce redundant or noisy supervision\. Therefore, our default strategy relabels only groups whose mean original\-task reward is below a threshold, focusing LfH on groups with enough failed trajectories that hindsight relabeling can meaningfully transform them into successful examples under alternative instructions\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/group_selection_goal_selection_60.png)\(a\)
![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/group_selection_goal_threshold_60.png)\(b\)

Figure 5:\(a\)Selecting groups with mean reward below 0\.8 outperforms random selection and relabeling all groups, showing that targeted relabeling is important\.\(b\)Among the tested thresholds, 0\.8 performs best, suggesting that LfH benefits from relabeling a broad set of non\-saturated groups while avoiding indiscriminate relabeling of all groups\.As shown in Figure[5\(a\)](https://arxiv.org/html/2607.09042#A1.F5.sf1), threshold\-based group selection is important\. We compare three group\-selection strategies\. LfH selects groups whose mean reward is below 0\.8\. Random selection keeps 70% of groups uniformly at random, matching the fraction of groups retained by our LfH filter\. The all\-groups variant applies hindsight relabeling to every group\. Relabeling groups with mean reward below 0\.8 performs substantially better than random group selection and relabeling all groups\. Random selection uses the relabeling budget inefficiently because it often spends hindsight queries on groups where relabeling is not useful \(e\.g\. all successful groups\)\. Relabeling all groups is also worse than threshold\-based selection, suggesting that indiscriminate relabeling can dilute the learning signal or introduce unnecessary hindsight updates on trajectories that already provide useful original\-task supervision\.

We also ablate the threshold used for filtering groups\. As shown in Figure[5\(b\)](https://arxiv.org/html/2607.09042#A1.F5.sf2), a very low threshold, such as 0\.1, relabels only all\-failed groups and therefore misses many trajectories that fail the commanded task but still contain meaningful, reusable object interactions\. A more moderate threshold, such as 0\.5, improves over this stricter setting but still underperforms the default threshold of 0\.8\. The threshold of 0\.8 provides the best performance, indicating that LfH benefits most when relabeling a broad set of non\-saturated groups\.

Does increasing updates per rollout match LfH?A natural question is whether LfH’s gains could be matched by simply increasing the update\-to\-data ratio\. To test this, we compare LfH against GRPO and PPO with different update epochs, where a larger update epoch means taking more gradient steps on the same collected rollout batch before collecting new data\. Figure[6](https://arxiv.org/html/2607.09042#A1.F6)shows that increasing update epochs does not consistently improve performance and can even hurt, indicating that there is a narrow sweet spot for data reuse\. More importantly, even the best\-tuned GRPO and PPO variants remain below LfH\. This suggests that the bottleneck is not merely under\-optimization, but the usefulness of the supervision contained in each rollout: repeatedly updating on failed trajectories under the original prompt has limited value, whereas LfH relabels those trajectories with prompts they can support, making each interaction more informative\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x9.png)\(a\)LfH vs GRPO
![Refer to caption](https://arxiv.org/html/2607.09042v1/x10.png)\(b\)LfH vs PPO

Figure 6:\(a\)Increasing the number of GRPO update epochs does not match LfH, showing that the main bottleneck is not insufficient optimization on the same rollout data\.\(b\)Increasing PPO update epochs also fails to consistently match LfH, further indicating that LfH improves sample efficiency by making each collected trajectory more informative rather than by simply reusing data more aggressively\.How does data diversity affect the performance of LfH?We vary the Pi\-0\.5 rollout sampling noise during data collection to control the diversity of collected trajectories\. Figure[7](https://arxiv.org/html/2607.09042#A1.F7)shows that both GRPO and LfH exhibit an optimal noise level: too little noise limits exploration, while too much noise makes rollouts less coherent and less useful for learning\. However, LfH appears to benefit more from this optimal noise than standard GRPO\. This suggests that data diversity is especially important for LfH\. While GRPO can only use diversity when it leads to higher commanded\-task reward, LfH can additionally relabel diverse failures with hindsight instructions and rewards\. In this way, exploration produces alternative object interactions that LfH can convert into useful supervision by leveraging the VLA’s pretrained knowledge about language\.

Limitations of LfH\.LfH depends highly on the quality of the failures it receives\. When the policy produces diverse, interpretable failures, hindsight relabeling takes advantage of these useful alternative goals\. However, when it collapses to repetitive or uninformative behavior, there may be little meaningful signal to relabel\. Also, LfH depends on the VLM relabeler\. Incorrect object names, overly generic hindsight instructions, or hallucinated achievements can turn failed trajectories into noisy positive examples, while reward relabeling can introduce false positives or false negatives\. We are able to partially mitigate these issues with the two mechanism shown in Figure[8](https://arxiv.org/html/2607.09042#A1.F8):Nothingfilters out rollouts with no confirmed object contact \(such as just hovering or erratically moving around\) andunsuregives the option assigns reward0\.50\.5instead of assigning a false negative or positive reward\. However, this does not fully mitigate these limitations, and for certain tasks where the failures are not meaningful, LfH may be less suitable\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x11.png)Figure 7:Ablation on the noise level used for rollout sampling\. As the rollout sampling noise level increases, trajectory diversity increases and LfH benefits more than GRPO, showing that LfH actively benefits from and encourages a more diverse rollout distribution during training\.Impact of the VLM prompt on performance\.We ablate which rollout groups should be selected for hindsight relabeling on the LIBERO\-PRO Goal suite with task perturbation using Pi\-0\.5\. The full LfH prompt \([B\.1](https://arxiv.org/html/2607.09042#A2.SS1.SSS0.Px2)\) includes three key components: \(1\) a list of scene objects provided to the VLM, \(2\) a “Nothing” option that allows the relabeler to identify uninteresting trajectories, which are then filtered out during training, and \(3\) an “unsure” option that allows the reward evaluator to abstain when task completion cannot be visually verified, resulting in reward 0\.5\. We compare this prompt against variants that remove each of these components\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/prompt_ablation.png)Figure 8:VLM prompt ablation\. Removing the scene object list, theNothingoption for filtering uninteresting trajectories, or theunsureoption for reward evaluation each reduces the sample\-efficiency gains of LfH\.As shown in Figure[8](https://arxiv.org/html/2607.09042#A1.F8), all three components contribute to the sample\-efficiency gains of LfH\. Removing the object list makes relabeling more open\-ended: the VLM must infer object identities from visual appearance alone, which increases the likelihood of inconsistent, overly specific, or ungrounded labels\. Removing the “Nothing” option forces the VLM to assign an instruction even to idle, contact\-free, or otherwise uninformative trajectories, turning useless rollouts into noisy positive hindsight examples\. Removing the “unsure” option from reward evaluation forces ambiguous videos to be classified as either success or failure, which can introduce incorrect binary rewards when the final goal state is not clearly visible\.

These results suggest that effective hindsight relabeling requires not only a capable VLM, but also a prompt that constrains the VLM to produce meaningful relabelings\.

### A\.2Additional analysis of LfH

When does LfH help the most?LfH is most effective when failures are diverse and semantically interpretable\. Figure[9](https://arxiv.org/html/2607.09042#A1.F9)shows sample rollouts at training step 40 on the microwave task\. Standard GRPO produces highly similar trajectories: the robot repeatedly approaches the same mug with similar gripper poses, leading to an undiverse failure mode\. In contrast, LfH rollouts show more varied object interactions and spatial configurations around the microwave and mugs\. Since these samples are taken after training has progressed, this difference suggests that LfH may do more than exploit diversity already present in the initial policy; it may also help maintain a more diverse behavioral distribution during RL fine\-tuning\. One possible explanation is that hindsight relabeling gives credit to multiple meaningful alternative behaviors, whereas GRPO treats them all as failures under the commanded task\. As a result, GRPO can collapse toward repeated unsuccessful behaviors, while LfH continues to reinforce a broader set of grounded object interactions\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/ablation/grpo_rollout_frames_microwave.png)\(a\)Repeated GRPO failures
![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/ablation/lfh_rollout_frames_microwave.png)\(b\)Diverse LfH rollouts

Figure 9:Random mid\-training rollouts on the microwave task for\(a\)GRPO and\(b\)LfH\. Standard GRPO collapses toward repeated, low\-diversity failure modes, while LfH maintains more diverse and semantically meaningful behavior, indicating that hindsight relabeling not only benefits from, but also helps preserve useful behavioral diversity during training\.Can LfH improve sample efficiency on other OOD perturbations?We next evaluate a less obvious setting for language\-based hindsight: position perturbations\. Here, the commanded language prompt is unchanged, but the initial object positions are shifted\. Since LfH relabels trajectories in language space, it is not immediately clear that it should help when only the environment configuration changes\. Surprisingly, the preliminary results in Figure[13](https://arxiv.org/html/2607.09042#A1.F13)suggest that LfH can still improve over GRPO on position\-perturbation tasks, both in sample efficiency and maximum success rate\. This points to a possible broader benefit of LfH: relabeled failures may help the policy reuse experience from shifted layouts and learn object interactions that can transfer back to the commanded task\.

### A\.3Full Training Curves

The main metric used is Gain as defined in Section[5\.2](https://arxiv.org/html/2607.09042#S5.SS2)averaged across suites\. However, below we also report the full training curves for each method and model per suite\. We report both the raw success rate and the maximum cumulative success\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/x12.png)Figure 10:Full training curves for Pi\-0\.5![Refer to caption](https://arxiv.org/html/2607.09042v1/x13.png)Figure 11:Full training curves for GR00T![Refer to caption](https://arxiv.org/html/2607.09042v1/x14.png)Figure 12:Full training curves for OpenVLA\-OFT![Refer to caption](https://arxiv.org/html/2607.09042v1/x15.png)Figure 13:Full training curves for Pi\-0\.5 on position perturbation

## Appendix BImplementation details

### B\.1LfH experimental details

#### Hyperparameters\.

Throughout the experiments, we use6464environments with11rollout collection per step\. We use group sizeN=8N=8, microbatch3232, global batch512512,KL​\_​beta=0\\mathrm\{KL\\\_beta\}=0\(no KL penalty\), two gradient update epochs, and VLM relabeling temperature0\.80\.8\.

#### Prompts\.

LfH uses the VLM to both relabel the prompt and also assign rewards\. The relabeling prompt is:

Relabeling PromptWatch this video of a robot arm performing a manipulation task\.The robot was originally instructed to:"\{original\_instruction\}"However, the robot may have FAILED or done something DIFFERENT from the original instruction\.Your job is to describe what the robot ACTUALLY DID based solely on visual evidence\.Objects present in the scene:\{scene\_objects\_text\}First, determine whether the robot did anything INTERESTING\.Interesting behavior means the robot PHYSICALLY CONTACTED an object \-\-\- the gripper fingers visibly touched or closed around it\.Uninteresting behavior means the robot merely hovered near an object, approached without making contact, or stayed idle\.CONTACT CHECK \-\-\- before claiming the robot touched anything, ask yourself:\- Did the gripper fingers visibly close around or press against the object? \- Did the object move, tilt, or shift in response to the gripper? \- Did the object’s shadow change?If NONE of these are true, the robot did NOT contact the object \-\-\- even if the gripper moved close to it\.If the behavior is UNINTERESTING \(no confirmed physical contact\), output exactly:<thought\>brief reasoning about why nothing interesting happened</thought\><final\>Nothing</final\>If the behavior IS interesting \(confirmed contact\), focus on:1\. Which object\(s\) did the gripper physically contact? 2\. Where did the object\(s\) move to? 3\. What is the final spatial arrangement?Rules:\- Write ONE imperative instruction \(e\.g\., "pick up the mug and place it in the microwave"\)\. \- Use SHORT, GENERIC names for objects: strip any numeric suffixes \(\_1, \_2\) and brand/material prefixes \(e\.g\. "akita\_black\_bowl\_1" → "bowl", "flat\_stove\_1" → "stove"\)\. \- Do NOT copy the original instruction \-\-\- describe what you SEE\. \- Under 20 words\.Output:<thought\>brief reasoning about what happened</thought\><final\>your instruction</final\>

The reward prompt is:

Reward Evaluation PromptWatch this video of a robot arm performing a manipulation task\.The robot was instructed to:"\{instruction\}"Determine whether the robot FULLY and SUCCESSFULLY completed the instruction\.Follow these steps:1\. FIRST FRAME: Describe the positions of all relevant objects at the start\. 2\. LAST FRAME: Describe the positions of all relevant objects at the end\. 3\. CHANGES: What specifically changed between the first and last frames? List each object that moved and where it moved to\. 4\. VERIFICATION: For each requirement in the instruction, state whether it is satisfied based on what you can clearly see in the last frame\. 5\. JUDGMENT: Are ALL requirements satisfied?Rules:\- Only describe what you can clearly see in the frames\. Do NOT infer or assume actions that are not visible \-\-\- narrating a plausible sequence of events is not evidence of completion\. \- The task is complete ONLY if the last frame shows the EXACT goal state: correct objects in the correct locations as specified in the instruction\. \- "In" a compartment/container means INSIDE the enclosed space \-\-\- not on top, not adjacent, not leaning against it\. \- Moving an object to the WRONG location is a failure \(e\.g\. wrong compartment, wrong side of the table\), even if the object was moved\. \- The robot arm may still be touching the object \-\-\- what matters is whether the object is at the target location\. \- If you cannot clearly verify the target location in the last frame, answer UNSURE\.Output format must be exactly:<thought\>your step\-by\-step analysis following steps 1\-5 above</thought\><answer\>yes</answer\> or <answer\>no</answer\> or <answer\>unsure</answer\>Use YES if the goal state is clearly achieved\.Use NO if it clearly is not achieved\.Use UNSURE if the evidence is ambiguous \(e\.g\. you cannot tell which compartment an object is in, or whether it is on top vs inside\)\.

#### LfH Group selection\.

Trajectories are grouped into sets ofN=8N\{=\}8\. Groups whose mean reward falls within\[θlow,θhigh\]=\[0\.8,1\.0\]\[\\theta\_\{\\text\{low\}\},\\theta\_\{\\text\{high\}\}\]=\[0\.8,1\.0\]are skipped; all other groups \(mean reward<0\.8<0\.8\) are selected for hindsight relabeling\. This focuses relabeling on groups with substantial failure rates where hindsight instructions provide the most value, while leaving near\-saturated groups untouched\.

#### Hindsight loss clipping\.

LfH can occasionally produce very large hindsight\-policy losses, especially early in training when hindsight rewards are noisy\. To prevent a small number of hindsight groups from dominating optimization, we clip the hindsight GRPO loss before applying the hindsight weight:

LLfH​\(θ\)=LGRPO​\(θ\)\+λ⋅min⁡\(LH​\-​GRPO​\(θ\),c\),L\_\{\\mathrm\{LfH\}\}\(\\theta\)=L\_\{\\mathrm\{GRPO\}\}\(\\theta\)\+\\lambda\\cdot\\min\\left\(L\_\{\\mathrm\{H\\text\{\-\}GRPO\}\}\(\\theta\),c\\right\),whereccis a clipping threshold andλ\\lambdacontrols the overall contribution of hindsight updates\. This bounds the maximum contribution of the hindsight term toλ​c\\lambda c, while leaving the standard GRPO objective unchanged\.

We keep this maximum hindsight contribution fixed across all experiments\. To set the clipping threshold, we estimate the empirical distribution of hindsight losses during the early stage of training and chooseccas approximately the 85th percentile\. We then chooseλ\\lambdasuch that the weighted LfH batch loss is roughly equal to the original batch policy loss\.

#### Same\-instruction skip\.

Before reward evaluation, any trajectory whose relabeled instruction matches the original task \(after case/punctuation normalization\) reverts to its original instruction and environment reward, avoiding redundant VLM queries\.

### B\.2Real\-world experimental details

Setup\.Our real\-world setup consists of a Franka FR3 arm, a set of manipulable objects, two drawers, and a two\-level rack\. We first supervised fine\-tune \(SFT\)π0\.5\\pi\_\{0\.5\}on 10 language\-conditioned tasks that span different difficulty levels and horizon lengths, from short\-horizon pick\-and\-place to multi\-stage manipulation\. Table[1](https://arxiv.org/html/2607.09042#A2.T1)lists all tasks used for collecting SFT demonstrations\. For each task, we collect2020demonstration rollouts using SpaceMouse teleoperation\. Each demonstration lasts between3030and6060seconds\. The SFT hyperparameters are reported in Table[3](https://arxiv.org/html/2607.09042#A2.T3)\. Figure[14](https://arxiv.org/html/2607.09042#A2.F14)shows the setup of the real world experiments\.

RL fine\-tuning\.We then compare GRPO and GRPO\+LfH on the task*put the green container into the bowl*, where the initial SFT policy achieves zero success\. Rewards are provided by a human operator as binary labels, with11indicating success and0indicating failure\. Each rollout has a time limit of3030seconds, corresponding to512512action steps; each step executes an action chunk of size1616predicted by the VLA model\. The GRPO hyperparameters are reported in Table[2](https://arxiv.org/html/2607.09042#A2.T2)\.

Results\.Figure[4\(b\)](https://arxiv.org/html/2607.09042#S5.F4.sf2)shows that LfH improves sample efficiency over GRPO throughout real\-world training\. The initial SFT policy often fails by approaching or grasping the wrong object; for example, we observe that it frequently grasps the tape regardless of the language prompt\. LfH turns these failures into useful supervision by relabeling them with prompts that describe the behavior actually executed, such as “pick up the tape”, “lift the tape”, or “move the tape to the bowl”\. These relabeled rollouts provide additional training signal for object grounding and related manipulation skills, such as identifying the bowl and moving objects toward it\. As a result, LfH can extract useful learning signal from trajectories that would otherwise be treated as failures under the original task prompt\.

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/realworld_scene.png)Figure 14:Realworld experiment setupTable 1:Tasks used for collecting SFT demonstrations\.Table 2:Real\-world GRPO hyperparameters\.GroupHyperparameterValueRolloutepisodes per iteration32max episode steps \(chunks\)64poll interval \(s\)60heartbeat interval \(s\)60recompute logprobsFalseEnvironmenttotal num envs \(train / eval\)32 / 32group size4max episode steps \(robot\)1024Algorithmloss aggregationtoken\-meanupdate epochs2rollout epoch1KL penalty typelow\_var\_klKL coefficientβ\\beta1×10−31\{\\times\}10^\{\-3\}entropy bonus0clip ratio \(low / high\)0\.2 / 0\.2clip ratiocc3\.0Modelaction dim10num action chunks16flow\-matching steps10noise level0\.1Optimizeractor LR2×10−62\{\\times\}10^\{\-6\}value LR1×10−41\{\\times\}10^\{\-4\}Adam\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)\(0\.9,0\.95\)\(0\.9,\\,0\.95\)Adamε\\varepsilon1×10−81\{\\times\}10^\{\-8\}weight decay0\.01grad clip1\.0Batching / FSDPmicro batch size32global batch size512sharding strategyno\_shard\(DDP\-style\)gradient checkpointingFalseTable 3:Realworld SFT hyperparameters\.GroupHyperparameterValueModelprecisionbf16action dim10num action chunks50num flow steps10num image inputs2 \(two cameras\)train expert onlyTrueBatchingmicro batch size1global batch size256Optimizeractor LR1×10−41\{\\times\}10^\{\-4\}value LR1\.55×10−41\.55\{\\times\}10^\{\-4\}Adam\(β1,β2\)\(\\beta\_\{1\},\\beta\_\{2\}\)\(0\.95,0\.99\)\(0\.95,\\,0\.99\)Adamε\\varepsilon1×10−81\{\\times\}10^\{\-8\}weight decay1×10−61\{\\times\}10^\{\-6\}grad clip1\.0FSDP / mixed prec\.strategyFSDPsharding strategyno\_sharduse orig paramsTruegradient checkpointingFalsewrap policydisabledAMPenabledTrueprecisionbf16grad scalerFalse

## Appendix CAdditional examples of relabeled trajectories

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/lfh_examples/lfh_example_1.jpg)

Original:put the plate on the stove Relabeled:move the bowl to the left side of the table

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/lfh_examples/lfh_example_2.jpg)

Original:put the wine bottle on the plate Relabeled:place the wine bottle on the cabinet

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/lfh_examples/lfh_example_3.jpg)

Original:put the wine bottle on the plate Relabeled:nothing \(uninteresting\)

![Refer to caption](https://arxiv.org/html/2607.09042v1/analysis/appendix/lfh_examples/lfh_example_4.jpg)

Original:put the wine bottle in the bowl Relabeled:pour the wine bottle’s contents into the bowl

Figure 15:Qualitative examples of LfH relabeling\.For each rollout, we show the original instruction and the VLM\-generated hindsight instruction describing what the robot actually did\. Meaningful failures are relabeled as alternative successful behaviors, uninformative rollouts are filtered with a “nothing” label, and rare VLM hallucinations remain possible\.