OGPO：面向可恢复视觉-语言-动作模型的离线目标条件策略优化

# OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models
Source: [https://arxiv.org/html/2610.12914](https://arxiv.org/html/2610.12914)
Xule GaoAffiliation:National Key Laboratory of Space Integrated Information SystemAffiliation:Institute of Software, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesXi WangAffiliation:National Key Laboratory of Space Integrated Information SystemAffiliation:Institute of Software, Chinese Academy of SciencesAffiliation:University of Chinese Academy of SciencesRui WangAffiliation:National Key Laboratory of Space Integrated Information SystemAffiliation:Institute of Software, Chinese Academy of SciencesChangwen ZhengAffiliation:National Key Laboratory of Space Integrated Information SystemAffiliation:Institute of Software, Chinese Academy of SciencesChuxiong SunEmail:[gaoxule24@mails\.ucas\.ac\.cn†Corresponding author:chuxiong2016@iscas\.ac\.cn](mailto:%20Corresponding%20author:%20)Affiliation:National Key Laboratory of Space Integrated Information SystemAffiliation:Institute of Software, Chinese Academy of Sciences

###### Abstract

Vision\-language\-action \(VLA\) models have shown strong promise in robotic manipulation, yet their post\-training still predominantly relies on trajectory\-level supervised imitation over robot demonstrations\. While effective for learning feasible actions, this paradigm supervises policies to reproduce expert actions at each step, biasing them toward demonstration\-specific action paths and providing limited guidance for recovering toward task\-progress states once execution deviates from demonstrated trajectories\. To address this limitation, we proposeOGPO, anOfflineGoal\-conditionedPolicyOptimization framework for recoverable VLA post\-training\. Instead of treating offline demonstrations merely as step\-wise action labels, OGPO adaptively relabels trajectories with semantic key states and constructs goal\-conditioned process rewards to optimize the policy for reaching these task\-progress states\. In this way, OGPO shifts VLA post\-training from trajectory\-level action imitation to key\-state reachability learning, enabling policies to acquire a more robust ability to complete goal states from feasible off\-demonstration states without requiring additional online interaction\. Experiments on LIBERO and MetaWorld show that OGPO consistently improves VLA policy performance over standard supervised fine\-tuning\. Moreover, evaluations on LIBERO\-Plus and our proposed LIBERO\-ReAct, a mid\-execution perturbation benchmark, further demonstrate that OGPO achieves stronger robustness and goal\-directed recoverability under distribution shifts and recoverable off\-demonstration states\.

## 1Introduction

Vision\-language\-action \(VLA\) policies have shown strong potential for language\-conditioned robotic manipulation by scaling supervised action prediction on large\-scale robot datasets\[[3](https://arxiv.org/html/2610.12914#bib.bib25),[40](https://arxiv.org/html/2610.12914#bib.bib8),[17](https://arxiv.org/html/2610.12914#bib.bib9)\]\. Most existing VLA policies are adapted to downstream tasks through supervised fine\-tuning \(SFT\), where the model is optimized to predict expert actions from visual observations and language instructions on demonstration trajectories\[[16](https://arxiv.org/html/2610.12914#bib.bib19),[30](https://arxiv.org/html/2610.12914#bib.bib32),[2](https://arxiv.org/html/2610.12914#bib.bib6),[1](https://arxiv.org/html/2610.12914#bib.bib5)\]\. Although this paradigm is simple and stable, it treats demonstrations primarily as step\-wise action labels for trajectory\-level imitation\. As a result, policies trained purely with SFT tend to reproduce demonstration\-specific action paths and remain inherently bounded by the coverage and quality of demonstration data\. As shown in Figure[1](https://arxiv.org/html/2610.12914#S1.F1), when execution deviates from demonstrated trajectories, such policies may suffer from compounding errors in long\-horizon tasks and lack explicit guidance for recovering toward task\-progress states\[[28](https://arxiv.org/html/2610.12914#bib.bib27)\]\.

Recent studies have explored reinforcement learning \(RL\) to further improve pretrained VLA policies beyond supervised imitation, including iterative online RL fine\-tuning\[[11](https://arxiv.org/html/2610.12914#bib.bib29)\], PPO/GRPO\-style policy\-gradient optimization\[[24](https://arxiv.org/html/2610.12914#bib.bib21),[19](https://arxiv.org/html/2610.12914#bib.bib22),[5](https://arxiv.org/html/2610.12914#bib.bib23)\], residual policy learning\[[34](https://arxiv.org/html/2610.12914#bib.bib30)\], and RL\-based policy distillation\[[15](https://arxiv.org/html/2610.12914#bib.bib31)\]\. By optimizing task\-level rewards, RL can in principle encourage goal\-directed behaviors that are not directly captured by step\-wise action imitation\. However, existing RL\-based VLA post\-training methods typically require costly environment rollouts and suffer from limited sample efficiency, making them difficult to scale for real\-world robotic post\-training\.

In this work, we study whether VLA policies can acquire goal\-directed recoverability purely from offline robot data, without requiring additional online interaction\. We define this ability as recovering toward task\-progress key states when execution deviates from demonstrated trajectories\. Our key insight is that offline demonstrations should not be used merely as action sequences to imitate; they also contain implicit evidence about which task\-progress states are reachable and which behaviors move the policy toward them\. Based on this insight, we proposeOGPO, an offline goal\-conditioned policy optimization framework for recoverable VLA post\-training\. Specifically, we retain SFT as a lightweight action prior to preserve low\-level action feasibility and language\-action alignment, while introducing an offline goal\-conditioned RL objective to learn key\-state reachability from existing trajectories\. To turn offline data into goal\-reaching supervision,OGPOadaptively relabels trajectories with task\-progress key states and constructs goal\-conditioned process rewards to guide both policy optimization and value estimation\. Instead of supervising the policy to imitate every action along the demonstration path, this objective trains the policy to reach task\-progress states under goal\-conditioned rewards\. In this way,OGPOshifts VLA post\-training from trajectory\-level action imitation to key\-state reachability learning, enabling policies to improve goal\-directed recoverability under feasible off\-demonstration deviations while avoiding the high interaction cost of online RL\.

![Refer to caption](https://arxiv.org/html/2610.12914v1/case-gripper2.png)\(a\)Put the wine bottle on top of the cabinet\.
![Refer to caption](https://arxiv.org/html/2610.12914v1/case-position2.png)\(b\)Open the top of drawer and put the bowl inside\.

Figure 1:We design two representative perturbation cases to evaluate the robustness and goal\-directed recoverability of offline\-trained VLA policies under deviations from demonstration trajectories\. In case \(a\), the gripper is forcibly opened after grasping the object, while position perturbation is applied when the end effector approaches the drawer\. The results show thatπ0\.5\\pi\_\{0\.5\}\-SFT fails to complete the task, whereasπ0\.5\\pi\_\{0\.5\}\-OGPO successfully recovers toward the goal state and finishes the task, demonstrating stronger recoverability under execution\-time deviations\.We evaluate OGPO on LIBERO\[[21](https://arxiv.org/html/2610.12914#bib.bib1)\]and MetaWorld\[[35](https://arxiv.org/html/2610.12914#bib.bib2)\], and further test robustness and recoverability on LIBERO\-Plus\[[9](https://arxiv.org/html/2610.12914#bib.bib3)\]and our proposed LIBERO\-ReAct benchmark with mid\-execution perturbations\. The results show that OGPO consistently improves over SFT baselines, especially on long\-horizon and difficult tasks, while achieving stronger robustness and goal\-directed recoverability under distribution shifts and execution\-time deviations\. Our contributions are threefold\.

- •First, we formulate VLA post\-training from the perspective of goal\-directed recoverability and identify the limitation of trajectory\-level SFT in recovering toward task\-progress states under execution deviations\.
- •Second, we propose OGPO, an offline goal\-conditioned policy optimization framework that adaptively relabels fixed robot trajectories with task\-progress key states and constructs goal\-conditioned process rewards, allowing the policy to learn and optimize goal\-directed behaviors beyond step\-wise action imitation\.
- •Third, we conduct extensive experiments on standard manipulation, distribution\-shift, and mid\-execution perturbation benchmarks, demonstrating that OGPO consistently improves task performance, robustness, and goal\-directed recoverability without requiring additional online interaction\.

## 2Related Works

VLA post\-training aims to adapt pretrained vision\-language\-action policies to downstream robotic manipulation tasks while preserving their general action priors and improving task\-specific performance\. Existing approaches mainly follow two directions: SFT and RL\.

### 2\.1Supervised Fine\-tuning for VLA Policies

SFT has become a dominant post\-training strategy for adapting pretrained VLA policies to downstream robotic tasks\. In this paradigm, the model is fine\-tuned on expert demonstration trajectories by minimizing a supervised action prediction loss conditioned on visual observations and language instructions\. Existing VLA models instantiate this paradigm with different action representations and prediction objectives\. OpenVLA\[[17](https://arxiv.org/html/2610.12914#bib.bib9)\]discretizes continuous robot actions into action tokens and optimizes the policy with a next\-token prediction objective using cross\-entropy loss\. Octo\[[30](https://arxiv.org/html/2610.12914#bib.bib32)\]predicts continuous action chunks through a diffusion\-based action head trained with a diffusion loss\.π0\\pi\_\{0\}\[[2](https://arxiv.org/html/2610.12914#bib.bib6)\]andπ0\.5\\pi\_\{0\.5\}\[[1](https://arxiv.org/html/2610.12914#bib.bib5)\]further adopt flow\-matching objectives for continuous action\-chunk prediction\. Recent studies such as OpenVLA\-OFT\[[16](https://arxiv.org/html/2610.12914#bib.bib19)\]improve SFT\-based adaptation by introducing continuous action representations, action chunking, parallel decoding, and regression\-based imitation objectives\. Despite these advances, SFT remains inherently constrained by the coverage and quality of demonstration data: it primarily learns to reproduce expert actions along demonstrated trajectories, and thus provides limited support for generalizing to unseen states, handling execution perturbations, and mitigating long\-horizon compounding errors\[[28](https://arxiv.org/html/2610.12914#bib.bib27)\]\.

### 2\.2Reinforcement Learning for VLA Post\-training

Recent works have explored RL as an additional post\-training stage after SFT\. By optimizing task\-level rewards through online rollouts, RL allows VLA policies to move beyond the behavior distribution of expert demonstrations and potentially acquire more robust behaviors under distribution shift\[[22](https://arxiv.org/html/2610.12914#bib.bib28)\]\. For example, iRe\-VLA\[[11](https://arxiv.org/html/2610.12914#bib.bib29)\]alternates between online RL and supervised learning to improve policy performance while maintaining training stability\. Other methods apply PPO\- or GRPO\-style policy\-gradient optimization to VLA models\[[24](https://arxiv.org/html/2610.12914#bib.bib21),[19](https://arxiv.org/html/2610.12914#bib.bib22),[5](https://arxiv.org/html/2610.12914#bib.bib23)\], aiming to enhance long\-horizon decision\-making and task success through outcome\-driven rewards\. In addition, residual RL methods improve pretrained VLA policies by learning lightweight corrective policies around frozen generalist models\[[34](https://arxiv.org/html/2610.12914#bib.bib30)\], while RL\-based policy distillation methods leverage VLA teachers or RL experts to obtain more specialized policies\[[15](https://arxiv.org/html/2610.12914#bib.bib31)\]\. More recent studies further explore simulator\- or world\-model\-based reinforcement fine\-tuning to reduce the cost of direct real\-world interaction\[[33](https://arxiv.org/html/2610.12914#bib.bib34),[20](https://arxiv.org/html/2610.12914#bib.bib33),[38](https://arxiv.org/html/2610.12914#bib.bib35),[14](https://arxiv.org/html/2610.12914#bib.bib36)\]\. Although these methods demonstrate the promise of RL for improving VLA policies beyond SFT, they typically rely on online rollouts, real\-world interaction, or simulator\-based policy evaluation, making them costly and sample\-inefficient for long\-horizon robotic tasks\.

#### Our distinction\.

In this work, we combine the benefits of SFT and RL in a fully offline VLA post\-training framework\. Different from prior methods that rely mainly on action imitation, online rollouts, or simulator\-based policy evaluation, OGPO extracts goal\-reaching and failure\-aware supervision directly from offline robot data\. This enables VLA policies to improve goal\-directed recoverability without additional environment interaction\. To the best of our knowledge, OGPO is the first framework that integrates SFT regularization with offline goal\-conditioned reinforcement learning for recoverable VLA post\-training\.

## 3Method

In this section, we present OGPO, an offline goal\-conditioned policy optimization framework for recoverable VLA models\. OGPO operates in three stages\. First, it relabels offline trajectories into key\-state\-centered goal\-reaching segments, thereby exposing the task\-progress structure hidden in demonstrations\. Second, it converts segment\-level completion labels into goal\-conditioned process rewards, which provide dense learning signals for value estimation\. Third, it performs offline advantage\-weighted policy optimization with SFT regularization, enabling the policy to improve toward goal\-reaching behaviors while preserving feasible action priors\.

![Refer to caption](https://arxiv.org/html/2610.12914v1/framework0507.png)Figure 2:Offline Goal\-conditioned Policy Optimization\(OGPO\) framework\.### 3\.1Goal\-Relabeled Trajectory Segmentation

To discover the task\-progress structure embedded in offline trajectories, OGPO first segments each trajectory according to gripper state change\. Given an expert demonstration trajectoryτ\\tau:

τ=\{\{\(ot,at\)\}t=1T,l\}\\tau=\\left\\\{\\\{\(o\_\{t\},a\_\{t\}\)\\\}\_\{t=1\}^\{T\},l\\right\\\}\(1\)whereoto\_\{t\}andata\_\{t\}denote the observation and action at timesteptt, andllis a natural language instruction describing the task carried out in the demonstration\. We assume that each trajectoryτ\\tauis composed of a sequence of task\-progress stages\. The objective of trajectory segmentation is to split the trajectory into non\-overlapping temporal segments\{τ1,τ2,…,τM\}\\\{\\tau^\{1\},\\tau^\{2\},\\ldots,\\tau^\{M\}\\\}, each corresponding to a task\-progress stage\. Following Chen et al\.\[[39](https://arxiv.org/html/2610.12914#bib.bib37)\]and Zhu et al\.\[[7](https://arxiv.org/html/2610.12914#bib.bib38)\], we realize such segmentation based on keyframes containing the change of gripper state\. Despite its simplicity, gripper state change is a robust indicator of critical physical interaction, which can be viewed as a signal of the end of one stage and the beginning of the next\.

After segmentation, we perform goal relabeling for each segment\. Specifically, we first assign each segmentτm\\tau^\{m\}a semantic local goalgmg^\{m\}based on the task instruction, gripper transitions, and observed state changes, as detailed in Appendix[B\.2](https://arxiv.org/html/2610.12914#A2.SS2)\. We then assign each segment a binary completion labelym=𝕀⁡\[τm​achieves​gm\]y^\{m\}=\\mathbb\{I\}\[\\tau^\{m\}\\text\{ achieves \}g^\{m\}\], using different labeling procedures for successful and failed trajectories:

- •For a successful trajectory, each valid segment is treated as completing its assigned local goal and is therefore labeled withym=1y^\{m\}=1\. The segment\-end observation is regarded as the corresponding achieved task\-progress key state\.
- •For a failed trajectory, the task\-level outcome does not directly indicate which local goals have been completed, as meaningful intermediate progress may occur before the eventual failure\. We therefore introduce an offline VLM\-based annotator to evaluate each segment and assign a binary completion labelymy^\{m\}\. Details of the completion labeling are provided in Appendix[B\.3](https://arxiv.org/html/2610.12914#A2.SS3)and[B\.4](https://arxiv.org/html/2610.12914#A2.SS4)\.

After relabeling, we obtain a goal\-relabeled offline dataset:

𝒟~=\{\(ot,l,at,gt,yt\)\}\.\\widetilde\{\\mathcal\{D\}\}=\\\{\(o\_\{t\},l,a\_\{t\},g\_\{t\},y\_\{t\}\)\\\}\.\(2\)Through this process, we obtain a diverse goal\-relabeled dataset that contains both achieved and unfulfilled task\-progress key states, providing a solid foundation for subsequent VLA policy optimization\.

### 3\.2Designing Goal\-Conditioned Process Rewards for Value Learning

Directly constructing a goal\-conditioned policy on top of a pretrained VLA model would require architectural modifications and may interfere with the pretrained vision\-language representations\. To avoid this, we use goal\-conditioned process rewards as training\-time signals for offline policy optimization\. This design preserves the original VLA policy structure and instruction\-conditioned action interface, while still enabling the policy to acquire goal\-directed behaviors by optimizing toward task\-progress key states\. Concretely, we construct goal\-conditioned process rewards from the relabeled key states and completion labels\. The relabeled goalgtg\_\{t\}is used only to annotate rewards and define segment\-level goal progress during training; it is not introduced as an additional input to either the policy or the value function\.

For each relabeled segmentτm=\{\(ot,l,at,gt,yt\)\}t=smem\\tau^\{m\}=\\\{\(o\_\{t\},l,a\_\{t\},g\_\{t\},y\_\{t\}\)\\\}\_\{t=s\_\{m\}\}^\{e\_\{m\}\}, wheresms\_\{m\}andeme\_\{m\}denote the start and end time steps of the segment, we define a normalized progress variable:

ρtm=yt⋅t−smem\+1−sm,t∈\[sm,em\+1\]\.\\rho\_\{t\}^\{m\}=y\_\{t\}\\cdot\\frac\{t\-s\_\{m\}\}\{e\_\{m\}\+1\-s\_\{m\}\},\\qquad t\\in\[s\_\{m\},e\_\{m\}\+1\]\.\(3\)Whenyt=1y\_\{t\}=1,ρtm\\rho\_\{t\}^\{m\}increases from00to11along the segment, providing dense progress reward for a completed goal\. Whenyt=0y\_\{t\}=0,ρtm=0\\rho\_\{t\}^\{m\}=0for the entire segment, so failed segments do not receive progressive positive rewards\.

For each transitiont∈\[sm,em\]t\\in\[s\_\{m\},e\_\{m\}\], we use a segment\-end indicatorδtm=𝕀\[t=em\]\\delta\_\{t\}^\{m\}=\\mathbb\{I\}\[t=e\_\{m\}\], which equals 1 only at the final transition of the segment and 0 otherwise\. The reward is then annotated as:

rt=−λstep\+Rprog​\(ρt\+1m−ρtm\)\+δtm​\(Rsuccessgoal​yt−Rfailgoal​\(1−yt\)\)\.r\_\{t\}=\-\\lambda\_\{\\mathrm\{step\}\}\+R^\{\\mathrm\{prog\}\}\\left\(\\rho\_\{t\+1\}^\{m\}\-\\rho\_\{t\}^\{m\}\\right\)\+\\delta\_\{t\}^\{m\}\\left\(R^\{\\mathrm\{goal\}\}\_\{\\mathrm\{success\}\}y\_\{t\}\-R^\{\\mathrm\{goal\}\}\_\{\\mathrm\{fail\}\}\(1\-y\_\{t\}\)\\right\)\.\(4\)The first term is a small step penalty that discourages redundant actions\. The second term assigns dense positive credit to transitions inside achieved key\-state segments\. The last term provides a terminal bonus when the assigned key state is achieved and a terminal penalty when the key state remains unfulfilled\. The reward\-augmented offline dataset is written as:

𝒟~=\{\(ot,l,at,gt,yt,rt\)\}\.\\widetilde\{\\mathcal\{D\}\}=\\\{\(o\_\{t\},l,a\_\{t\},g\_\{t\},y\_\{t\},r\_\{t\}\)\\\}\.\(5\)With the annotated rewards, we train a value functionVψ​\(ot,l\)V\_\{\\psi\}\(o\_\{t\},l\)to estimate the expected discounted return induced by the key\-state relabeled rewards\. For value learning, we use annn\-step Bellman backup to construct the temporal\-difference target:

V^t=∑i=0n−1γi​rt\+i\+γn​Vψ¯​\(ot\+n,l\),\\hat\{V\}\_\{t\}=\\sum\_\{i=0\}^\{n\-1\}\\gamma^\{i\}r\_\{t\+i\}\+\\gamma^\{n\}V\_\{\\bar\{\\psi\}\}\(o\_\{t\+n\},l\),\(6\)whereVψ¯V\_\{\\bar\{\\psi\}\}denotes a target value network or a stop\-gradient copy of the value model\. In implementation,VψV\_\{\\psi\}is obtained by attaching a lightweight value head to the intermediate representation of the VLA backbone\. The value function is optimized by minimizing the temporal\-difference error:

ℒV​\(ψ\)=𝔼\(ot,l\)∼𝒟~​\[\(Vψ​\(ot,l\)−V^t\)2\]\.\\mathcal\{L\}\_\{V\}\(\\psi\)=\\mathbb\{E\}\_\{\(o\_\{t\},l\)\\sim\\widetilde\{\\mathcal\{D\}\}\}\\left\[\\left\(V\_\{\\psi\}\(o\_\{t\},l\)\-\\hat\{V\}\_\{t\}\\right\)^\{2\}\\right\]\.\(7\)Through this objective, states that are followed by achieved key\-state progress are assigned higher values, while states in failed or incomplete segments receive lower values due to the absence of progress rewards and the terminal failure penalty\.

### 3\.3Offline Goal\-conditioned Policy Optimization

In this work, we seek a lightweight policy improvement objective that can be seamlessly integrated into the existing action modeling loss of a pretrained VLA policy\. Concretely, we adopt an AWAC\-style actor update\[[25](https://arxiv.org/html/2610.12914#bib.bib4)\], which casts policy improvement as advantage\-weighted behavior regression\.

We calculate the advantage value for each actionata\_\{t\}using annn\-step return estimate induced by the goal\-conditioned rewards:

At=\(∑i=0n−1γi​rt\+i\+γn​Vψ​\(ot\+n,l\)\)−Vψ​\(ot,l\)\.A\_\{t\}=\\left\(\\sum\_\{i=0\}^\{n\-1\}\\gamma^\{i\}r\_\{t\+i\}\+\\gamma^\{n\}V\_\{\\psi\}\(o\_\{t\+n\},l\)\\right\)\-V\_\{\\psi\}\(o\_\{t\},l\)\.\(8\)A positive advantage indicates that the observed action leads to a higher return than expected from the current state, while a small or negative advantage suggests lower contribution to future progress\. Following AWAC\[[25](https://arxiv.org/html/2610.12914#bib.bib4)\], we convert the advantage into an exponential policy improvement weight:

ωt=sg⁡\[min⁡\(exp⁡\(Atβ\),ωmax\)\],\\omega\_\{t\}=\\operatorname\{sg\}\\left\[\\min\\left\(\\exp\\left\(\\frac\{A\_\{t\}\}\{\\beta\}\\right\),\\omega\_\{\\max\}\\right\)\\right\],\(9\)whereβ\\betacontrols the sharpness of advantage weighting,ωmax\\omega\_\{\\max\}prevents a small number of high\-advantage samples from dominating the update, andsg⁡\(⋅\)\\operatorname\{sg\}\(\\cdot\)denotes the stop\-gradient operation\.

We then optimize the policy with an AWAC\-style offline RL loss over the entire relabeled dataset𝒟~\\widetilde\{\\mathcal\{D\}\}:

ℒRL​\(θ\)=−𝔼x~t∼𝒟~​\[ωt​log⁡πθ​\(at∣ot,l\)\]\.\\mathcal\{L\}\_\{\\mathrm\{RL\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{\\tilde\{x\}\_\{t\}\\sim\\widetilde\{\\mathcal\{D\}\}\}\\left\[\\omega\_\{t\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},l\)\\right\]\.\(10\)This objective biases policy improvement toward high\-advantage actions in the relabeled dataset: actions that contribute to achieved key states receive larger weights, while actions with low progress return have limited influence on the policy update\. As a result, the policy is encouraged not merely to imitate demonstrated actions, but to improve its ability to recover from intermediate states and continue progressing toward the intended goal states\.

Furthermore, inspired by recent post\-training paradigms in large language models \(LLMs\) that unify supervised and reinforcement objectives within a single optimization stage\[[10](https://arxiv.org/html/2610.12914#bib.bib7),[23](https://arxiv.org/html/2610.12914#bib.bib39),[36](https://arxiv.org/html/2610.12914#bib.bib40)\], we further retain SFT in the policy objective\. Specifically, SFT is applied to successful demonstrations as a lightweight action prior, preserving low\-level action feasibility and maintaining the language\-action alignment inherited from the pretrained VLA model\.

ℒSFT​\(θ\)=−𝔼xt∼𝒟demo​\[log⁡πθ​\(at∣ot,l\)\]\.\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)=\-\\mathbb\{E\}\_\{x\_\{t\}\\sim\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\}\\left\[\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},l\)\\right\]\.\(11\)The final policy optimization objective is:

ℒtotal​\(θ\)=λRL​ℒRL​\(θ\)\+λSFT​ℒSFT​\(θ\)\\mathcal\{L\}\_\{\\mathrm\{total\}\}\(\\theta\)=\\lambda\_\{\\mathrm\{RL\}\}\\mathcal\{L\}\_\{\\mathrm\{RL\}\}\(\\theta\)\+\\lambda\_\{\\mathrm\{SFT\}\}\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)\(12\)
SFT and offline RL play complementary roles in this objective\. SFT provides a coarse\-grained behavioral prior, anchoring the policy to the feasible action distribution of expert demonstrations and stabilizing optimization\. In contrast, offline RL provides fine\-grained goal\-directed optimization signals through goal\-conditioned process rewards and advantage\-weighted policy improvement, enabling the policy to deviate from demonstration\-specific action paths when necessary and favor behaviors that better advance toward task\-progress key states\.

For notational simplicity, we represent the action\-matching term in the above objectives using the likelihood\-style formlog⁡πθ​\(at∣ot,l\)\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid o\_\{t\},l\)\. This term serves as a general action\-matching objective that encourages the policy output to align with the demonstrated actionata\_\{t\}\. In practice, it is instantiated according to the action head of the underlying VLA model, such as action\-token likelihood\[[17](https://arxiv.org/html/2610.12914#bib.bib9)\], continuous action regression\[[16](https://arxiv.org/html/2610.12914#bib.bib19)\], denoising prediction\[[8](https://arxiv.org/html/2610.12914#bib.bib24),[30](https://arxiv.org/html/2610.12914#bib.bib32)\], or flow\-matching training objectives\[[2](https://arxiv.org/html/2610.12914#bib.bib6),[1](https://arxiv.org/html/2610.12914#bib.bib5)\]\.

## 4Experiments

### 4\.1Experimental Setup

#### Benchmarks\.

To evaluate the effectiveness of our method, we conduct comprehensive experiments on two widely used robotic manipulation benchmarks, LIBERO\[[21](https://arxiv.org/html/2610.12914#bib.bib1)\]and MetaWorld\[[35](https://arxiv.org/html/2610.12914#bib.bib2)\]\. To further assess robustness and performance under off\-demonstration conditions, we design two additional evaluation settings\. First, we use LIBERO\-Plus\[[9](https://arxiv.org/html/2610.12914#bib.bib3)\]to test distribution shifts in camera viewpoint, robot initial state, visual noise, and object layout\. Second, to evaluate dynamic recovery under execution\-time disturbances, we introduceLIBERO\-ReAct, a mid\-execution perturbation benchmark built on LIBERO\[[21](https://arxiv.org/html/2610.12914#bib.bib1)\]\. LIBERO\-ReAct includes action noise, gripper\-state perturbations, and end\-effector position perturbations, and is designed to evaluate goal\-directed recoverability during policy execution\.

#### Backbones and Baselines\.

OGPO is built on theπ0\\pi\_\{0\}andπ0\.5\\pi\_\{0\.5\}backbones\. We compare with their SFT\-based few\-shot and full\-shot post\-training variants to evaluate improvements over standard supervised imitation\. We further report representative VLA models, including OpenVLA\[[17](https://arxiv.org/html/2610.12914#bib.bib9)\], SmolVLA\[[29](https://arxiv.org/html/2610.12914#bib.bib11)\], UniVLA\[[31](https://arxiv.org/html/2610.12914#bib.bib15)\], and OpenVLA\-OFT\[[16](https://arxiv.org/html/2610.12914#bib.bib19)\], as broader references, and include PPO\-based online RL baselines from RLinf\[[5](https://arxiv.org/html/2610.12914#bib.bib23)\]for comparison with online RL post\-training\. Overall, this baseline set provides a comprehensive and rigorous evaluation for assessing the effectiveness of our OGPO\.

### 4\.2Main Results

Table 1:Performance comparison on the LIBERO benchmark\. Policy input notation: T \(third\-view image\), I \(instruction\), W \(wrist image\), P \(proprioception\)\. OGPO is built upon the few\-shot SFT policy, with↑\\uparrowindicating performance gains over the few\-shot baseline\.Our evaluation begins with the overall performance on widely used manipulation benchmarks\. Tables[1](https://arxiv.org/html/2610.12914#S4.T1)and[2](https://arxiv.org/html/2610.12914#S4.T2)present the results on LIBERO and MetaWorld\-MT50\. On LIBERO, OGPO and PPO are initialized from the sameπ0\.5\\pi\_\{0\.5\}few\-shot policy, obtained with limited downstream demonstrations\. Starting from this backbone, OGPO improves the average success rate from 77\.1% to 97\.1%, outperforming theπ0\.5\\pi\_\{0\.5\}full\-SFT baseline of 96\.85% and achieving performance comparable toπ0\.5\\pi\_\{0\.5\}\+PPO, an online RL method requiring costly online rollout\. The improvement is particularly significant on the long\-horizonLIBERO\-Longsuite, where our method raises the success rate from 43\.9% to 93\.2%, also surpassing the full\-SFT result of 92\.4%, indicating its enhanced goal\-reaching capability in long\-horizon manipulation tasks\. On MetaWorld\-MT50, our method consistently outperforms SFT across different backbones, improvingπ0\\pi\_\{0\}from 50\.8% to 81\.0% and approaching the online PPO\-based result of 85\.8%\. The gains are especially large on more challenging tasks; for example, on theπ0\\pi\_\{0\}backbone, our method improves the success rate by 28\.4% on Hard tasks and 53\.6% onVery Hardtasks\. Overall, these results validate that OGPO can effectively extract goal\-reaching supervision from offline trajectories, enabling substantial policy improvement without online rollouts, particularly on long\-horizon and difficult manipulation tasks\.

Table 2:Performance comparison on the MetaWorld\-MT50 benchmark\. Policy input notation: T \(third\-view image\), P \(proprioception\), C \(task conditioning\)\. C denotes the task information used to distinguish MetaWorld tasks, and may be instantiated as a language instruction, task ID, or task embedding depending on the model\.Furthermore, to evaluate the robustness of OGPO and its ability to handle off\-demonstration deviations, Table[3](https://arxiv.org/html/2610.12914#S4.T3)reports performance under selected LIBERO\-Plus perturbations and our proposedLIBERO\-ReAct, a mid\-execution perturbation benchmark built on LIBERO\[[21](https://arxiv.org/html/2610.12914#bib.bib1)\]\. Among the perturbations provided by LIBERO\-Plus, we select four representative types: camera viewpoint changes, robot initial\-state variations, visual noise, and object layout changes, covering scene\-, observation\-, and initial\-state distribution shifts\. To further test dynamic recovery during policy execution, LIBERO\-ReAct introduces three mid\-episode action\-space perturbations across the four LIBERO task suites, including Spatial, Object, Goal, and Long\. For each task, we randomly select a 20\-step interval during the episode and apply the perturbation after the policy predicts an action and before the action is executed in the environment:

- •Action noise: we add uniform noise to the executed action command, with a range of\[−0\.1,0\.1\]\[\-0\.1,0\.1\]for arm control and\[−0\.5,0\.5\]\[\-0\.5,0\.5\]for gripper control\.
- •Gripper\-state perturbation: we flip the gripper command to the opposite state, forcing an open command to close and a close command to open\.
- •End\-effector position perturbation: we add uniform noise within\[−0\.2,0\.2\]\[\-0\.2,0\.2\]to the predicted position motion command, simulating spatial deviations of the end effector\.

Compared with the full\-shot SFT baseline, our offline fine\-tunedπ0\.5\\pi\_\{0\.5\}improves the overall robustness score from 76\.7% to 81\.8%\. It achieves clear gains on LIBERO\-Plus Robot\-Init and Noise perturbations, and shows even larger improvements under LIBERO\-ReAct disturbances\. In particular, the success rate under gripper\-state perturbation increases from 76\.8% to 87\.7%, suggesting that our method better recovers from erroneous grasping or releasing actions\. These results indicate that our method improves both robustness to static distribution shifts and recovery from execution\-time disturbances\.

Table 3:Robustness Evaluation on the LIBERO\-Plus Benchmark and Our Extended Perturbation Settings\. The red\-highlighted columns indicate the additional mid\-episode perturbations introduced by us\. All results are evaluated without perturbation\-specific fine\-tuning\.At last, to evaluate the scalability of OGPO, we conduct an additional set of experiments usingπ0\\pi\_\{0\}as the backbone\. As shown in Fig\.[3](https://arxiv.org/html/2610.12914#S4.F3), OGPO also achieves substantial improvements over standard SFT on LIBERO\-Spatial and LIBERO\-Goal under the same 3,000 training steps\. Specifically, our method reaches 93\.6% on LIBERO\-Spatial and 92\.3% on LIBERO\-Goal, outperforming SFT by 13\.6 and 11\.1 points, respectively\. These results further validate the effectiveness of OGPO, suggesting that the proposed offline objective provides more effective learning signals than pure supervised imitation and leads to more efficient policy improvement\.

\(a\)LIBERO\-Spatial\(b\)LIBERO\-Goal
Figure 3:Training efficiency comparison between Ours and SFT using theπ0\\pi\_\{0\}backbone\.
### 4\.3Ablation Study

To better understand the contribution of each component, we conduct ablation studies from two perspectives\. First, we examine the reward design, which provides the basis for offline advantage estimation\. Second, we analyze the policy optimization objective by removing each training signal, including offline RL and SFT\.

#### Ablation on reward designs\.

As shown in Fig\.[4](https://arxiv.org/html/2610.12914#S4.F4)\(a\), the proposed reward design achieves the best performance on both LIBERO\-Goal and LIBERO\-Spatial\. Among all variants, removingRgoalR^\{\\mathrm\{goal\}\}leads to the largest performance drop\. This indicates that subgoal\-level completion signals are crucial for offline value learning: without trajectory segmentation and subgoal relabeling, the value function cannot effectively distinguish achieved progress from failed segments\.

#### Ablation on policy fine\-tuning objectives\.

As shown in Fig\.[4](https://arxiv.org/html/2610.12914#S4.F4)\(b\), the full objective performs best across both tasks\. Removing the offline RL term clearly degrades performance, highlighting the importance of AWAC\-style advantage\-weighted policy improvement for learning higher\-return behaviors\. Meanwhile, the decline observed after removing SFT confirms its role as positive behavior regularization, which preserves feasible successful behaviors and prevents the AWAC\-style update from drifting away from the pretrained action prior\.

\(a\)Ablation on reward designs\.\(b\)Ablation on policy fine\-tuning objectives\.
Figure 4:Ablation studies using theπ0\\pi\_\{0\}backbone trained for 3,000 steps on LIBERO\-Goal and LIBERO\-Spatial\.

## 5Conclusion and Limitations

In this work, we investigate how to leverage offline data to train a robust VLA model capable of recovering toward goal states after deviating from demonstration trajectories\. Specifically, we propose OGPO, an offline goal\-conditioned policy optimization framework for recoverable VLA post\-training\. Instead of treating demonstrations as step\-wise action supervision, OGPO relabels offline trajectories into key\-state\-centered goal\-reaching segments and transforms segment completion signals into goal\-conditioned process rewards\. These rewards facilitate value learning and advantage\-weighted policy optimization\. In addition, OGPO retains SFT to provide a coarse\-grained behavioral prior that anchors the policy to the feasible action distribution induced by expert demonstrations\. Overall, OGPO shifts VLA post\-training from trajectory\-level action imitation toward key\-state reachability learning, thereby improving goal\-directed recoverability without requiring online interaction\. Extensive experiments demonstrate the effectiveness, robustness, and recovery capability of OGPO under off\-demonstration scenarios\. Despite the promising results, this work still has several limitations\. First, the current gripper\-state\-based segmentation strategy may not generalize well to tasks whose progress cannot be naturally characterized by such events\. Second, as a purely offline framework, OGPO is limited by the quality of the offline dataset, which may restrict its ability to recover from severe out\-of\-distribution deviations beyond the demonstrated trajectories\. In future work, we plan to investigate more general and robust goal\-state relabeling strategies, along with data generation and augmentation methods that can improve the diversity of offline data for recoverable policy learning\.

## References

- \[1\]K\. Black, N\. Brown, J\. Darpinian, K\. Dhabalia, D\. Driess, A\. Esmail, M\. R\. Equi, C\. Finn, N\. Fusai, M\. Y\. Galliker, D\. Ghosh, L\. Groom, K\. Hausman, b\. ichter, S\. Jakubczak, T\. Jones, L\. Ke, D\. LeBlanc, S\. Levine, A\. Li\-Bell, M\. Mothukuri, S\. Nair, K\. Pertsch, A\. Z\. Ren, L\. X\. Shi, L\. Smith, J\. T\. Springenberg, K\. Stachowicz, J\. Tanner, Q\. Vuong, H\. Walke, A\. Walling, H\. Wang, L\. Yu, and U\. Zhilinsky\(2025\)π0\.5\\pi\_\{0\.5\}: A vision\-language\-action model with open\-world generalization\.InProceedings of The 9th Conference on Robot Learning,J\. Lim, S\. Song, and H\. Park \(Eds\.\),Proceedings of Machine Learning Research, Vol\.305,pp\. 17–40\.External Links:[Link](https://proceedings.mlr.press/v305/black25a.html)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.15.1.1),[Table 3](https://arxiv.org/html/2610.12914#S4.T3.5.1.7.1)\.
- \[2\]K\. Black, N\. Brown, D\. Driess, A\. Esmail, M\. Equi, C\. Finn, N\. Fusai, L\. Groom, K\. Hausman, B\. Ichter, S\. Jakubczak, T\. Jones, L\. Ke, S\. Levine, A\. Li\-Bell, M\. Mothukuri, S\. Nair, K\. Pertsch, L\. X\. Shi, J\. Tanner, Q\. Vuong, A\. Walling, H\. Wang, and U\. Zhilinsky\(2026\)π0\\pi\_\{0\}: A vision\-language\-action flow model for general robot control\.External Links:2410\.24164,[Link](https://arxiv.org/abs/2410.24164)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.5.1),[Table 3](https://arxiv.org/html/2610.12914#S4.T3.5.1.3.1)\.
- \[3\]A\. Brohan, N\. Brown, J\. Carbajal, Y\. Chebotar, J\. Dabis, C\. Finn, K\. Gopalakrishnan, K\. Hausman, A\. Herzog, J\. Hsu, J\. Ibarz, B\. Ichter, A\. Irpan, T\. Jackson, S\. Jesmonth, N\. J\. Joshi, R\. Julian, D\. Kalashnikov, Y\. Kuang, I\. Leal, K\. Lee, S\. Levine, Y\. Lu, U\. Malla, D\. Manjunath, I\. Mordatch, O\. Nachum, C\. Parada, J\. Peralta, E\. Perez, K\. Pertsch, J\. Quiambao, K\. Rao, M\. Ryoo, G\. Salazar, P\. Sanketi, K\. Sayed, J\. Singh, S\. Sontakke, A\. Stone, C\. Tan, H\. Tran, V\. Vanhoucke, S\. Vega, Q\. Vuong, F\. Xia, T\. Xiao, P\. Xu, S\. Xu, T\. Yu, and B\. Zitkovich\(2023\)RT\-1: robotics transformer for real\-world control at scale\.External Links:2212\.06817,[Link](https://arxiv.org/abs/2212.06817)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1)\.
- \[4\]J\. Cen, C\. Yu, H\. Yuan, Y\. Jiang, S\. Huang, J\. Guo, X\. Li, Y\. Song, H\. Luo, F\. Wang, D\. Zhao, and H\. Chen\(2025\)WorldVLA: towards autoregressive action world model\.External Links:2506\.21539,[Link](https://arxiv.org/abs/2506.21539)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.7.1.1)\.
- \[5\]K\. Chen, Z\. Liu, T\. Zhang, Z\. Guo, S\. Xu, H\. Lin, H\. Zang, X\. Li, Q\. Zhang, Z\. Yu, G\. Fan, T\. Huang, Y\. Wang, and C\. Yu\(2026\)πRL\\pi\_\{\\texttt\{RL\}\}: Online rl fine\-tuning for flow\-based vision\-language\-action models\.External Links:2510\.25889,[Link](https://arxiv.org/abs/2510.25889)Cited by:[Appendix E](https://arxiv.org/html/2610.12914#A5.p1.1),[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.19.1.1),[Table 2](https://arxiv.org/html/2610.12914#S4.T2.5.6.1.1)\.
- \[6\]Z\. Chen, R\. Niu, H\. Kong, and Q\. Wang\(2025\)TGRPO :fine\-tuning vision\-language\-action model via trajectory\-wise group relative policy optimization\.ArXivabs/2506\.08440\.External Links:[Link](https://api.semanticscholar.org/CorpusID:279260912)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.16.1.1)\.
- \[7\]Z\. Chen, J\. Yin, Y\. Chen, J\. Huo, P\. Tian, J\. Shi, Y\. Hou, Y\. Li, and Y\. Gao\(2026\)DeCo: task decomposition and skill composition for zero\-shot generalization in long\-horizon 3d manipulation\.External Links:2505\.00527,[Link](https://arxiv.org/abs/2505.00527)Cited by:[§3\.1](https://arxiv.org/html/2610.12914#S3.SS1.p1.2)\.
- \[8\]C\. Chi, Z\. Xu, S\. Feng, E\. Cousineau, Y\. Du, B\. Burchfiel, R\. Tedrake, and S\. Song\(2025\)Diffusion policy: visuomotor policy learning via action diffusion\.The International Journal of Robotics Research44\(10\-11\),pp\. 1684–1704\.Cited by:[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1),[Table 2](https://arxiv.org/html/2610.12914#S4.T2.5.3.1.1)\.
- \[9\]S\. Fei, S\. Wang, J\. Shi, Z\. Dai, J\. Cai, P\. Qian, L\. Ji, X\. He, S\. Zhang, Z\. Fei, J\. Fu, J\. Gong, and X\. Qiu\(2025\)LIBERO\-plus: in\-depth robustness analysis of vision\-language\-action models\.External Links:2510\.13626,[Link](https://arxiv.org/abs/2510.13626)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px1.p1.1)\.
- \[10\]Y\. Fu, T\. Chen, J\. Chai, X\. Wang, S\. Tu, G\. Yin, W\. Lin, Q\. Zhang, Y\. Zhu, and D\. Zhao\(2026\)SRFT: a single\-stage method with supervised and reinforcement fine\-tuning for reasoning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=n6E0r6kQWQ)Cited by:[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p4.1)\.
- \[11\]Y\. Guo, J\. Zhang, X\. Chen, X\. Ji, Y\. Wang, Y\. Hu, and J\. Chen\(2025\)Improving vision\-language\-action model with online reinforcement learning\.In2025 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 15665–15672\.Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[12\]C\. Huang, Y\. Wu, M\. Chen, Y\. F\. Wang, and F\. Yang\(2025\)ThinkAct: vision\-language\-action reasoning via reinforced visual latent planning\.External Links:2507\.16815,[Link](https://arxiv.org/abs/2507.16815)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.12.1.1)\.
- \[13\]C\. Hung, Q\. Sun, P\. Hong, A\. Zadeh, C\. Li, U\. Tan, N\. Majumder, and S\. Poria\(2025\)NORA: a small open\-sourced generalist vision language action model for embodied tasks\.External Links:2504\.19854,[Link](https://arxiv.org/abs/2504.19854)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.8.1.1)\.
- \[14\]Z\. Jiang, S\. Zhou, Y\. Jiang, Z\. Huang, M\. Wei, Y\. Chen, T\. Zhou, Z\. Guo, H\. Lin, Q\. Zhang, Y\. Wang, H\. Li, C\. Yu, and D\. Zhao\(2026\)WoVR: world models as reliable simulators for post\-training vla policies with rl\.External Links:2602\.13977,[Link](https://arxiv.org/abs/2602.13977)Cited by:[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[15\]T\. Jülg, W\. Burgard, and F\. Walter\(2025\)Refined policy distillation: from vla generalists to rl experts\.2025 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 11677–11684\.External Links:[Link](https://api.semanticscholar.org/CorpusID:276903591)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[16\]M\. J\. Kim, C\. Finn, and P\. Liang\(2025\)Fine\-tuning vision\-language\-action models: optimizing speed and success\.External Links:2502\.19645,[Link](https://arxiv.org/abs/2502.19645)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.14.1.1),[Table 3](https://arxiv.org/html/2610.12914#S4.T3.5.1.6.1)\.
- \[17\]M\. J\. Kim, K\. Pertsch, S\. Karamcheti, T\. Xiao, A\. Balakrishna, S\. Nair, R\. Rafailov, E\. P\. Foster, P\. R\. Sanketi, Q\. Vuong, T\. Kollar, B\. Burchfiel, R\. Tedrake, D\. Sadigh, S\. Levine, P\. Liang, and C\. Finn\(2025\)OpenVLA: an open\-source vision\-language\-action model\.InProceedings of The 8th Conference on Robot Learning,P\. Agrawal, O\. Kroemer, and W\. Burgard \(Eds\.\),Proceedings of Machine Learning Research, Vol\.270,pp\. 2679–2713\.External Links:[Link](https://proceedings.mlr.press/v270/kim25c.html)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.3.1.1),[Table 3](https://arxiv.org/html/2610.12914#S4.T3.5.1.5.1)\.
- \[18\]J\. Lee, J\. Duan, H\. Fang, Y\. Deng, S\. Liu, B\. Li, B\. Fang, J\. Zhang, Y\. R\. Wang, S\. Lee, W\. Han, W\. Pumacay, A\. Wu, R\. Hendrix, K\. Farley, E\. VanderBilt, A\. Farhadi, D\. Fox, and R\. Krishna\(2025\)MolmoAct: action reasoning models that can reason in space\.External Links:2508\.07917,[Link](https://arxiv.org/abs/2508.07917)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.11.1.1)\.
- \[19\]H\. Li, Y\. Zuo, J\. Yu, Y\. Zhang, Y\. Zhaohui, K\. Zhang, X\. Zhu, Y\. Zhang, T\. Chen, G\. Cui, D\. Wang, D\. Luo, Y\. Fan, Y\. Sun, J\. Zeng, J\. Pang, S\. Zhang, Y\. Wang, Y\. Mu, B\. Zhou, and N\. Ding\(2026\)SimpleVLA\-RL: scaling VLA training via reinforcement learning\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=TQhSodCM4r)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.18.1.1)\.
- \[20\]H\. Li, P\. Ding, R\. Suo, Z\. Ge, D\. Zang, K\. Yu, M\. Sun, H\. Zhang, D\. Wang, and W\. Su\(2026\)VLA\-RFT: vision\-language\-action reinforcement fine\-tuning with verified rewards in world simulators\.External Links:[Link](https://openreview.net/forum?id=Jaut99EHeu)Cited by:[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[21\]B\. Liu, Y\. Zhu, C\. Gao, Y\. Feng, Q\. Liu, Y\. Zhu, and P\. Stone\(2023\)Libero: benchmarking knowledge transfer for lifelong robot learning\.Advances in Neural Information Processing Systems36,pp\. 44776–44791\.Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px1.p1.1),[§4\.2](https://arxiv.org/html/2610.12914#S4.SS2.p2.1)\.
- \[22\]J\. Liu, F\. Gao, B\. Wei, X\. Chen, Q\. Liao, Y\. Wu, C\. Yu, and Y\. Wang\(2026\)What can RL bring to VLA generalization? an empirical study\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=qmBMPInbZC)Cited by:[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[23\]M\. Liu, G\. Farina, and A\. E\. Ozdaglar\(2026\)UFT: unifying supervised and reinforcement fine\-tuning\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,External Links:[Link](https://openreview.net/forum?id=usOkGv1S7M)Cited by:[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p4.1)\.
- \[24\]G\. Lu, W\. Guo, C\. Zhang, Y\. Zhou, H\. Jiang, Z\. Gao, Y\. Tang, and Z\. Wang\(2025\)VLA\-rl: towards masterful and general robotic manipulation with scalable reinforcement learning\.External Links:2505\.18719,[Link](https://arxiv.org/abs/2505.18719)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.17.1.1)\.
- \[25\]A\. Nair, A\. Gupta, M\. Dalal, and S\. Levine\(2020\)Awac: accelerating online reinforcement learning with offline datasets\.arXiv preprint arXiv:2006\.09359\.Cited by:[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p2.2)\.
- \[26\]NVIDIA, :, J\. Bjorck, F\. Castañeda, N\. Cherniadev, X\. Da, R\. Ding, L\. "\. Fan, Y\. Fang, D\. Fox, F\. Hu, S\. Huang, J\. Jang, Z\. Jiang, J\. Kautz, K\. Kundalia, L\. Lao, Z\. Li, Z\. Lin, K\. Lin, G\. Liu, E\. Llontop, L\. Magne, A\. Mandlekar, A\. Narayan, S\. Nasiriany, S\. Reed, Y\. L\. Tan, G\. Wang, Z\. Wang, J\. Wang, Q\. Wang, J\. Xiang, Y\. Xie, Y\. Xu, Z\. Xu, S\. Ye, Z\. Yu, A\. Zhang, H\. Zhang, Y\. Zhao, R\. Zheng, and Y\. Zhu\(2025\)GR00T n1: an open foundation model for generalist humanoid robots\.External Links:2503\.14734,[Link](https://arxiv.org/abs/2503.14734)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.13.1.1)\.
- \[27\]K\. Pertsch, K\. Stachowicz, B\. Ichter, D\. Driess, S\. Nair, Q\. Vuong, O\. Mees, C\. Finn, and S\. Levine\(2025\)FAST: efficient action tokenization for vision\-language\-action models\.External Links:2501\.09747,[Link](https://arxiv.org/abs/2501.09747)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.4.1.1),[Table 3](https://arxiv.org/html/2610.12914#S4.T3.5.1.4.1)\.
- \[28\]S\. Ross, G\. Gordon, and D\. Bagnell\(2011\)A reduction of imitation learning and structured prediction to no\-regret online learning\.InProceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics,G\. Gordon, D\. Dunson, and M\. Dudík \(Eds\.\),Proceedings of Machine Learning Research, Vol\.15,Fort Lauderdale, FL, USA,pp\. 627–635\.External Links:[Link](https://proceedings.mlr.press/v15/ross11a.html)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1)\.
- \[29\]M\. Shukor, D\. Aubakirova, F\. Capuano, P\. Kooijmans, S\. Palma, A\. Zouitine, M\. Aractingi, C\. Pascal, M\. Russi, A\. Marafioti, S\. Alibert, M\. Cord, T\. Wolf, and R\. Cadène\(2025\)SmolVLA: a vision\-language\-action model for affordable and efficient robotics\.ArXivabs/2506\.01844\.External Links:[Link](https://api.semanticscholar.org/CorpusID:279119427)Cited by:[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.6.1.1),[Table 2](https://arxiv.org/html/2610.12914#S4.T2.5.5.1.1)\.
- \[30\]O\. M\. Team, D\. Ghosh, H\. Walke, K\. Pertsch, K\. Black, O\. Mees, S\. Dasari, J\. Hejna, T\. Kreiman, C\. Xu,et al\.\(2024\)Octo: an open\-source generalist robot policy\.arXiv preprint arXiv:2405\.12213\.Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1),[§2\.1](https://arxiv.org/html/2610.12914#S2.SS1.p1.1),[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p6.1)\.
- \[31\]Y\. Wang, X\. Li, W\. Wang, J\. Zhang, Y\. Li, Y\. Chen, X\. Wang, and Z\. Zhang\(2026\)Unified vision\-language\-action model\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=PklMD8PwUy)Cited by:[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px2.p1.1),[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.10.1.1)\.
- \[32\]J\. Wen, Y\. Zhu, J\. Li, M\. Zhu, Z\. Tang, K\. Wu, Z\. Xu, N\. Liu, R\. Cheng, C\. Shen, Y\. Peng, F\. Feng, and J\. Tang\(2024\)TinyVLA: toward fast, data\-efficient vision\-language\-action models for robotic manipulation\.IEEE Robotics and Automation Letters10,pp\. 3988–3995\.External Links:[Link](https://api.semanticscholar.org/CorpusID:272753287)Cited by:[Table 2](https://arxiv.org/html/2610.12914#S4.T2.5.4.1.1)\.
- \[33\]J\. Xiao, Y\. Yang, X\. Chang, R\. Chen, F\. Xiong, M\. Xu, W\. Zheng, and Q\. Zhang\(2025\)World\-env: leveraging world model as a virtual environment for VLA post\-training\.External Links:[Link](https://openreview.net/forum?id=otrng1WHxG)Cited by:[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[34\]W\. Xiao, H\. Lin, A\. Peng, H\. Xue, T\. He, Z\. Luo, Y\. Xie, F\. Hu, L\. Fan, G\. Shi, and Y\. Zhu\(2026\)Self\-improving vision\-language\-action models with data generation via residual RL\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=eUGoqrZ6Ea)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p2.1),[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[35\]T\. Yu, D\. Quillen, Z\. He, R\. Julian, K\. Hausman, C\. Finn, and S\. Levine\(2020\)Meta\-world: a benchmark and evaluation for multi\-task and meta reinforcement learning\.InConference on robot learning,pp\. 1094–1100\.Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p4.1),[§4\.1](https://arxiv.org/html/2610.12914#S4.SS1.SSS0.Px1.p1.1)\.
- \[36\]W\. Zhang, Y\. Xie, Y\. Sun, Y\. Chen, G\. Wang, Y\. Li, B\. Ding, and J\. Zhou\(2026\)On\-policy RL meets off\-policy experts: harmonizing supervised fine\-tuning and reinforcement learning via dynamic weighting\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dCm9bBrk5d)Cited by:[§3\.3](https://arxiv.org/html/2610.12914#S3.SS3.p4.1)\.
- \[37\]Q\. Zhao, Y\. Lu, M\. J\. Kim, Z\. Fu, Z\. Zhang, Y\. Wu, Z\. Li, Q\. Ma, S\. Han, C\. Finn, A\. Handa, M\. Liu, D\. Xiang, G\. Wetzstein, and T\. Lin\(2025\)CoT\-vla: visual chain\-of\-thought reasoning for vision\-language\-action models\.External Links:2503\.22020,[Link](https://arxiv.org/abs/2503.22020)Cited by:[Table 1](https://arxiv.org/html/2610.12914#S4.T1.5.9.1.1)\.
- \[38\]F\. Zhu, Y\. Zhengyang, Z\. Hong, Q\. Shou, X\. Ma, and S\. Guo\(2026\)WMPO: world model\-based policy optimization for vision\-language\-action models\.InThe Fourteenth International Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=qE2FyvRvuF)Cited by:[§2\.2](https://arxiv.org/html/2610.12914#S2.SS2.p1.1)\.
- \[39\]Y\. Zhu, W\. Wang, S\. Wu, Y\. Shi, and J\. Wang\(2025\)Learning semantic atomic skills for multi\-task robotic manipulation\.arXiv preprint arXiv:2512\.18368\.Cited by:[§3\.1](https://arxiv.org/html/2610.12914#S3.SS1.p1.2)\.
- \[40\]B\. Zitkovich, T\. Yu, S\. Xu, P\. Xu, T\. Xiao, F\. Xia, J\. Wu, P\. Wohlhart, S\. Welker, A\. Wahid, Q\. Vuong, V\. Vanhoucke, H\. Tran, R\. Soricut, A\. Singh, J\. Singh, P\. Sermanet, P\. R\. Sanketi, G\. Salazar, M\. S\. Ryoo, K\. Reymann, K\. Rao, K\. Pertsch, I\. Mordatch, H\. Michalewski, Y\. Lu, S\. Levine, L\. Lee, T\. E\. Lee, I\. Leal, Y\. Kuang, D\. Kalashnikov, R\. Julian, N\. J\. Joshi, A\. Irpan, B\. Ichter, J\. Hsu, A\. Herzog, K\. Hausman, K\. Gopalakrishnan, C\. Fu, P\. Florence, C\. Finn, K\. A\. Dubey, D\. Driess, T\. Ding, K\. M\. Choromanski, X\. Chen, Y\. Chebotar, J\. Carbajal, N\. Brown, A\. Brohan, M\. G\. Arenas, and K\. Han\(2023\)RT\-2: vision\-language\-action models transfer web knowledge to robotic control\.InProceedings of The 7th Conference on Robot Learning,J\. Tan, M\. Toussaint, and K\. Darvish \(Eds\.\),Proceedings of Machine Learning Research, Vol\.229,pp\. 2165–2183\.External Links:[Link](https://proceedings.mlr.press/v229/zitkovich23a.html)Cited by:[§1](https://arxiv.org/html/2610.12914#S1.p1.1)\.

## Appendix APseudo\-code of OGPO

Algorithm[1](https://arxiv.org/html/2610.12914#alg1)summarizes the training procedure ofOGPO\. The relabeled goals are only used to construct process rewards during training, and are not introduced as additional inputs to the policy or value function\.

Algorithm 1OGPO: Offline Goal\-conditioned Policy Optimization1:Successful demonstrations

𝒟demo\\mathcal\{D\}\_\{\\mathrm\{demo\}\}, failed trajectories

𝒟fail\\mathcal\{D\}\_\{\\mathrm\{fail\}\}, pretrained VLA policy

πθ\\pi\_\{\\theta\}, value function

VψV\_\{\\psi\}, VLM annotator

2:Optimized policy

πθ\\pi\_\{\\theta\}
3:Initialize relabeled dataset

𝒟~←∅\\widetilde\{\\mathcal\{D\}\}\\leftarrow\\emptyset
4:foreach trajectory

τ∈𝒟demo∪𝒟fail\\tau\\in\\mathcal\{D\}\_\{\\mathrm\{demo\}\}\\cup\\mathcal\{D\}\_\{\\mathrm\{fail\}\}do

5:Segment

τ\\tauinto stages

\{τ1,…,τM\}\\\{\\tau^\{1\},\\ldots,\\tau^\{M\}\\\}using gripper\-state changes

6:foreach segment

τm\\tau^\{m\}do

7:if

τ∈𝒟demo\\tau\\in\\mathcal\{D\}\_\{\\mathrm\{demo\}\}then

8:Set the segment terminal state as goal

gmg^\{m\}and label

ym←1y^\{m\}\\leftarrow 1
9:else

10:Use the VLM annotator to infer goal

gmg^\{m\}and completion label

ymy^\{m\}
11:endif

12:Add goal labels to transitions in

τm\\tau^\{m\}
13:endfor

14:endfor

15:foreach relabeled segment

τm\\tau^\{m\}do

16:Compute progress variable

ρtm\\rho\_\{t\}^\{m\}using Eq\.[3](https://arxiv.org/html/2610.12914#S3.E3)

17:Annotate process rewards

rtr\_\{t\}using Eq\.[4](https://arxiv.org/html/2610.12914#S3.E4)

18:Add reward\-augmented transitions to

𝒟~\\widetilde\{\\mathcal\{D\}\}
19:endfor

20:Value learning:Train

VψV\_\{\\psi\}on

𝒟~\\widetilde\{\\mathcal\{D\}\}with the

nn\-step target in Eq\.[6](https://arxiv.org/html/2610.12914#S3.E6)by minimizing

ℒV\\mathcal\{L\}\_\{V\}in Eq\.[7](https://arxiv.org/html/2610.12914#S3.E7)

21:Policy optimization:

22:whilenot convergeddo

23:Sample a relabeled minibatch from

𝒟~\\widetilde\{\\mathcal\{D\}\}
24:Compute advantages

AtA\_\{t\}and AWAC weights

ωt\\omega\_\{t\}using the trained value function

VψV\_\{\\psi\}
25:Compute

ℒRL\\mathcal\{L\}\_\{\\mathrm\{RL\}\}in Eq\.[10](https://arxiv.org/html/2610.12914#S3.E10)and

ℒSFT\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}in Eq\.[11](https://arxiv.org/html/2610.12914#S3.E11)

26:Update

πθ\\pi\_\{\\theta\}by minimizing

ℒtotal\\mathcal\{L\}\_\{\\mathrm\{total\}\}in Eq\.[12](https://arxiv.org/html/2610.12914#S3.E12)

27:endwhile

28:return

πθ\\pi\_\{\\theta\}

## Appendix BDetailed Trajectory Segmentation and Goal Relabeling

This section provides the implementation details of our trajectory segmentation and goal relabeling procedure\. The key idea is to exploit the gripper open\-close pattern in manipulation trajectories\. In many tabletop manipulation tasks, gripper state changes naturally correspond to important semantic transitions, such as approaching an object, grasping it, transporting it, and releasing it at the target location\. Therefore, we use gripper state changes as temporal anchors to divide a long trajectory into several goal\-level segments\. After segmentation, each segment is assigned a semantic goal label and a binary completion label\.

### B\.1Gripper\-Based Trajectory Segmentation

Given an offline trajectory

τ=\{\(ot,l,at\)\}t=1T,\\tau=\\\{\(o\_\{t\},l,a\_\{t\}\)\\\}\_\{t=1\}^\{T\},whereoto\_\{t\}is the observation,llis the language instruction, andata\_\{t\}is the action, we extract the gripper command from the action sequence:

ct∈\{open,close\}\.c\_\{t\}\\in\\\{\\mathrm\{open\},\\mathrm\{close\}\\\}\.We then identify gripper switching points as candidate segment boundaries:

ℬ=\{t∣ct≠ct−1\}\.\\mathcal\{B\}=\\\{t\\mid c\_\{t\}\\neq c\_\{t\-1\}\\\}\.To avoid noisy boundaries caused by short\-term command oscillations, we only keep a switching point if the new gripper state is maintained for a short temporal window\. The final boundary set is formed by adding the beginning and the end of the trajectory:

ℬ¯=\{1\}∪ℬ∪\{T\}\.\\bar\{\\mathcal\{B\}\}=\\\{1\\\}\\cup\\mathcal\{B\}\\cup\\\{T\\\}\.The trajectory is then divided into a sequence of segments:

τ=\{τ1,τ2,…,τM\},τm=\{\(ot,l,at\)\}t=smem,\\tau=\\\{\\tau^\{1\},\\tau^\{2\},\\ldots,\\tau^\{M\}\\\},\\qquad\\tau^\{m\}=\\\{\(o\_\{t\},l,a\_\{t\}\)\\\}\_\{t=s\_\{m\}\}^\{e\_\{m\}\},wheresms\_\{m\}andeme\_\{m\}are two adjacent boundaries inℬ¯\\bar\{\\mathcal\{B\}\}\.

This gripper\-based segmentation is simple but effective for manipulation tasks because most meaningful object interactions are synchronized with gripper events\. For example, an open\-to\-close transition near an object usually indicates a grasping attempt, while a close\-to\-open transition near a target region usually indicates a placing or releasing attempt\. The segment between these two events often corresponds to a transportation or alignment goal\.

### B\.2Goal Relabel

After obtaining the temporal segments, we assign a semantic goalgmg^\{m\}to each segmentτm\\tau^\{m\}\. The goal label is determined by the task instruction, the gripper transition type, and the visual state change within the segment\. For example, for a task instruction such as “put the black bowl in the bottom drawer of the cabinet and close it”, the trajectory can be decomposed into the following goals:

reach→grasp→move→lay→close\.\\text\{reach \}\\rightarrow\\text\{grasp\}\\rightarrow\\text\{move\}\\rightarrow\\text\{lay\}\\rightarrow\\text\{close\}\.
In practice, we use the following rules to assign goal labels:

- •If the segment ends with an open\-to\-close gripper transition near a target object, it is labeled as a grasping\-related goal\.
- •If the gripper remains closed and the object pose changes with the end\-effector, the segment is labeled as a moving or transporting goal\.
- •If the segment ends with a close\-to\-open gripper transition near the target region, it is labeled as a placing or releasing goal\.
- •For articulated\-object tasks, if the gripper remains closed while the joint state or object part pose changes, the segment is labeled as an opening, pulling, pushing, or closing goal\.

This procedure converts each original trajectory into a sequence of goal\-conditioned segments:

τ→\{\(τ1,g1\),\(τ2,g2\),…,\(τM,gM\)\}\.\\tau\\rightarrow\\\{\(\\tau^\{1\},g^\{1\}\),\(\\tau^\{2\},g^\{2\}\),\\ldots,\(\\tau^\{M\},g^\{M\}\)\\\}\.

### B\.3Completion Labeling

For each segmentτm\\tau^\{m\}, we further assign a binary completion label:

ym∈\{0,1\},y^\{m\}\\in\\\{0,1\\\},whereym=1y^\{m\}=1indicates that the segment successfully completes its assigned goalgmg^\{m\}, andym=0y^\{m\}=0indicates that the segment fails to complete the goal\.

#### Successful trajectories\.

For a trajectory that successfully completes the full task, we label all its valid goal segments as successful:

ym=1,∀m\.y^\{m\}=1,\\qquad\\forall m\.

#### Failed trajectories\.

To better localize the failure stage and provide more accurate supervision for value learning, we use a VLM\-based offline annotator to determine whether each segment in a failed trajectory completes its assigned goal\. The VLM is asked to judge whether the visual state at the end of the segment satisfies the assigned goal\. The resulting relabeled dataset is:

𝒟~=\{\(ot,l,at,gt,yt\)\},\\widetilde\{\\mathcal\{D\}\}=\\\{\(o\_\{t\},l,a\_\{t\},g\_\{t\},y\_\{t\}\)\\\},wheregtg\_\{t\}andyty\_\{t\}are inherited from the segment containing time steptt\.

### B\.4VLM Annotation Prompt for Failed Segments

For failed trajectories, we use Qwen2\.5\-VL\-72B\-Instruct as the VLM annotator to evaluate whether each segment completes its candidate goal\. As shown in Figure[5](https://arxiv.org/html/2610.12914#A2.F5), the prompt provides the task instruction, candidate goal, and representative segment observations, and asks the annotator to output a completion label with a short explanation\.

VLM Annotation PromptYour task is to determine whether a trajectory segment completes a given candidate subgoal\.Task instruction:\{instruction\}Candidate goal:\{goal\}Segment observation:The following images show the first, middle, and last frames of a trajectory segment\.Question:Does this segment successfully complete the candidate goal?Please answer with one of the following labels:SUCCESS: the final frame satisfies the candidate goal\.FAILURE: the final frame does not satisfy the candidate goal\.Also provide a short reason\.Figure 5:Prompt template for VLM\-based annotation of failed trajectory segments\. We use Qwen2\.5\-VL\-72B\-Instruct as the VLM annotator\.
### B\.5Annotated Cases

Figure[6](https://arxiv.org/html/2610.12914#A2.F6)and Figure[7](https://arxiv.org/html/2610.12914#A2.F7)show two annotated examples\. For visualization clarity, we only show frames sampled every 10 time steps as a coarse illustration\. The first case is a successful trajectory\. After gripper\-based segmentation, all goal segments are labeled as achieved\. The second case is a failed trajectory\. Although the full trajectory fails, the early segments still complete meaningful goals, while the later segment fails to complete the assigned goal\. Our VLM\-based annotation preserves the achieved segments and marks the failed segment as negative experience\.

![Refer to caption](https://arxiv.org/html/2610.12914v1/draw.png)Figure 6:Annotated example of a successful trajectory\.![Refer to caption](https://arxiv.org/html/2610.12914v1/failure.png)Figure 7:Annotated example of a failed trajectory\.

## Appendix CReward Design and Value Analysis

This section provides additional details on the reward annotation procedure and analyzes the learned value estimates\. Our reward design converts sparse goal completion labels into dense goal\-conditioned learning signals\. For a segmentτm\\tau^\{m\}, we use the normalized progress variableρtm\\rho\_\{t\}^\{m\}to assign dense progress rewards to completed segments, while failed segments do not receive progressive positive rewards\. The reward consists of three parts: a step penalty, a dense progress reward, and a segment\-end success or failure term\.

#### Reward annotation\.

For each transition within a segment, the reward is computed according to Eq\.[4](https://arxiv.org/html/2610.12914#S3.E4)\. The step penalty discourages unnecessarily long or redundant action sequences\. The progress reward provides smooth credit assignment for actions that move the system toward a completed goal\. The terminal success bonus encourages completion of the assigned goal, while the terminal failure penalty prevents failed segments from being treated as useful demonstrations\. In all experiments, we setλstep=0\.01\\lambda\_\{\\mathrm\{step\}\}=0\.01,Rprog=1\.0R^\{\\mathrm\{prog\}\}=1\.0,Rsuccessgoal=1\.0R^\{\\mathrm\{goal\}\}\_\{\\mathrm\{success\}\}=1\.0, andRfailgoal=2\.0R^\{\\mathrm\{goal\}\}\_\{\\mathrm\{fail\}\}=2\.0, unless otherwise specified\.

#### Value curve analysis\.

To verify whether the value function captures meaningful goal progress, we visualize the learned value estimates along representative trajectories\. As shown in Fig\.[8](https://arxiv.org/html/2610.12914#A3.F8), the learned value curves exhibit clear local high\-value regions along the execution trajectory\. Interestingly, these high\-value turning points are well aligned with semantically meaningful task\-progress key states, such as approaching the target object, establishing a stable grasp, opening the drawer, or placing the object at the goal location\. This suggests that the learned value function does not merely estimate final task success, but also captures intermediate progress toward the goal\. By assigning higher values to states that are closer to reachable task\-progress milestones, our method provides a progress\-aware optimization signal for offline policy improvement, supporting key\-state reachability learning from offline demonstrations\.

![Refer to caption](https://arxiv.org/html/2610.12914v1/value_1.png)\(a\)Put the bowl on the plate\.
![Refer to caption](https://arxiv.org/html/2610.12914v1/value_2.png)\(b\)Open the drawer and put the bowl inside\.
![Refer to caption](https://arxiv.org/html/2610.12914v1/value_3.png)\(c\)Push the plate to the front of the stove\.
![Refer to caption](https://arxiv.org/html/2610.12914v1/value_4.png)\(d\)Put the bowl on the stove\.

Figure 8:Visualization of the learned goal\-conditioned value along execution trajectories\. Local high\-value frames are aligned with semantically meaningful task\-progress key states, such as placing the bowl on the plate, opening the drawer, or moving the plate toward the stove\. This qualitative evidence suggests that the learned value provides dense progress signals beyond final task success\.

## Appendix DHyperparameter Analysis

In this section, we analyze the sensitivity of our method to the objective weights in Eq\.[12](https://arxiv.org/html/2610.12914#S3.E12)\. The final policy objective contains three terms: supervised fine\-tuning on achieved goal segments, advantage\-guided offline reinforcement fine\-tuning, and negative suppression on failed goal segments\. Their relative importance is controlled byλSFT\\lambda\_\{\\mathrm\{SFT\}\},λRL\\lambda\_\{\\mathrm\{RL\}\}respectively:

ℒtotal​\(θ\)=λSFT​ℒSFT​\(θ\)\+λRL​ℒRL​\(θ\)\.\\mathcal\{L\}\_\{\\mathrm\{total\}\}\(\\theta\)=\\lambda\_\{\\mathrm\{SFT\}\}\\mathcal\{L\}\_\{\\mathrm\{SFT\}\}\(\\theta\)\+\\lambda\_\{\\mathrm\{RL\}\}\\mathcal\{L\}\_\{\\mathrm\{RL\}\}\(\\theta\)\.Unless otherwise specified, we useλSFT=0\.4,λRL=0\.6\.\\lambda\_\{\\mathrm\{SFT\}\}=0\.4,\\lambda\_\{\\mathrm\{RL\}\}=0\.6\.

Table 4:Sensitivity analysis of the objective weights\. The default setting is highlighted in gray\.#### Discussion\.

The tow terms play complementary roles\. The SFT term provides a stable imitation anchor from achieved goal segments\. The offline RL term further distinguishes actions according to their estimated long\-term contribution to goal completion\. Empirically, we find that using all three components yields more stable and effective policy improvement than relying on any single term alone\.

## Appendix EImplementation Details

We implement our offline fine\-tuning method within the RLinf\[[5](https://arxiv.org/html/2610.12914#bib.bib23)\]framework\. For fair comparison, our implementation follows the same preprocessing, action normalization, backbone initialization, and optimization setup as the corresponding SFT baseline\. The value model is trained before policy fine\-tuning and is only used during offline training to compute advantage weights\.

Table 5:Optimization hyperparameters\.Table 6:Value model training settings\.Table 7:Offline objective hyperparameters\.
## Appendix FComputational Cost Analysis

We compare the computational cost of our offline fine\-tuning method with online RL\-based post\-training\. Unlike online RL methods, our method does not require policy rollout collection during training\. All reward annotation, value learning, and policy optimization are performed on the fixed offline dataset\. The comparison shows that our offline fine\-tuning method is substantially more computationally efficient than online RL\-based training\. More importantly, our method avoids online environment interaction, making it easier to scale to multiple task suites and safer for robotic manipulation settings where online rollout collection can be expensive or risky\.

Table 8:Computational cost comparison on LIBERO\. GRPO denotes the online RL training recipe in RLinf\. The GRPO cost is reported for one LIBERO task suite, while our method is reported for all four LIBERO suites\.