From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

Hugging Face Daily Papers Papers
reinforcement-learning robotics manipulation long-horizon real-world-rl fine-tuning minimal-intervention

Summary

The paper presents PARTS, a real-world subtask reinforcement learning framework that improves long-horizon manipulation tasks by focusing on bottleneck subtasks with minimal human intervention, achieving higher success rates in experiments.

A pretrained robot foundation policy may execute most of a long-horizon task yet repeatedly fail at a few critical subtasks. Collecting additional full-task demonstrations for supervised fine-tuning (SFT) requires operators to repeat behaviors the policy already performs well. Reinforcement learning (RL) fine-tuning offers a promising path to bridge this gap, but existing approaches struggle to solve long-horizon tasks using only sparse rewards. We present PARTS (Policy Adaptation with RL on Targeted Subtasks), a real-world subtask RL framework that concentrates practice at these bottlenecks while allowing training rollouts to proceed with minimal human intervention. The frozen pretrained policy supplies nominal actions throughout execution, while agent-generated selectors and success verifiers activate residual corrections and provide local outcome rewards. These rewards support learning from successful subtasks even when complete-task successes are scarce. Training combines online RL with success-reweighted retraining, and each retrained residual policy is redeployed to collect further experience. Humans identify bottlenecks during setup and perform physical resets when needed. On bimanual YAM and single-arm Franka tasks, PARTS improves complete-task success from 32% to 61% and from 50% to 95%, respectively, using tens of minutes of real-world RL rollouts per task on average. Compared with existing real-world RL fine-tuning methods, PARTS raises full-task success by more than 25% under the same robot-rollout budget while requiring less human involvement.
Original Article
View Cached Full Text

Cached at: 09/21/26, 07:22 PM

Paper page - From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Intervention

Source: https://huggingface.co/papers/2609.21788

Abstract

Apretrainedrobotfoundationpolicymayexecutemostofalong-horizontaskyetrepeatedlyfailatafewcriticalsubtasks.Collectingadditionalfull-taskdemonstrationsforsupervisedfine-tuning(SFT)requiresoperatorstorepeatbehaviorsthepolicyalreadyperformswell.Reinforcementlearning(RL)fine-tuningoffersapromisingpathtobridgethisgap,butexistingapproachesstruggletosolvelong-horizontasksusingonlysparserewards.WepresentPARTS(PolicyAdaptationwithRLonTargetedSubtasks),areal-worldsubtaskRLframeworkthatconcentratespracticeatthesebottleneckswhileallowingtrainingrolloutstoproceedwithminimalhumanintervention.Thefrozenpretrainedpolicysuppliesnominalactionsthroughoutexecution,whileagent-generatedselectorsandsuccessverifiersactivateresidualcorrectionsandprovidelocaloutcomerewards.Theserewardssupportlearningfromsuccessfulsubtasksevenwhencomplete-tasksuccessesarescarce.TrainingcombinesonlineRLwithsuccess-reweightedretraining,andeachretrainedresidualpolicyisredeployedtocollectfurtherexperience.Humansidentifybottlenecksduringsetupandperformphysicalresetswhenneeded.OnbimanualYAMandsingle-armFrankatasks,PARTSimprovescomplete-tasksuccessfrom32%to61%andfrom50%to95%,respectively,usingtensofminutesofreal-worldRLrolloutspertaskonaverage.Comparedwithexistingreal-worldRLfine-tuningmethods,PARTSraisesfull-tasksuccessbymorethan25%underthesamerobot-rolloutbudgetwhilerequiringlesshumaninvolvement.

View arXiv pageView PDFProject pageAdd to collection

Get this paper in your agent:

hf papers read 2609\.21788

Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash

Models citing this paper0

No model linking this paper

Cite arxiv.org/abs/2609.21788 in a model README.md to link it from this page.

Datasets citing this paper0

No dataset linking this paper

Cite arxiv.org/abs/2609.21788 in a dataset README.md to link it from this page.

Spaces citing this paper0

No Space linking this paper

Cite arxiv.org/abs/2609.21788 in a Space README.md to link it from this page.

Collections including this paper0

No Collection including this paper

Add this paper to acollectionto link it from this page.

Similar Articles

Efficient Reinforcement Learning for Long-Horizon Tool-Use Agentic Tasks

arXiv cs.LG

This paper introduces SinkFlex-RL, a modular training system for memory-feasible reinforcement learning in long-horizon tool-use agentic tasks. It combines a Gymnasium-compatible environment wrapper, GRPO-based policy optimization, and a sink-aware FlexAttention path, reducing peak VRAM by 19.7% at 4096 tokens and enabling 8192-token runs where eager attention runs out of memory.