ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

Hugging Face Daily Papers Papers
robotics reinforcement-learning continual-learning robot-foundation-models off-policy-rl real-world-robotics

Summary

ACCRUE proposes a continual off-policy reinforcement learning method for robot foundation models. By freezing the foundation model and only updating lightweight actor–critic networks while inheriting experience across tasks, robots can learn new skills within 4–12 minutes of real-world interaction while retaining 98.3% of success rates on previously acquired tasks.

Reinforcement learning (RL) post-training improves robot foundation models (RFMs) beyond their demonstrations, but real-world deployment demands more: learning new tasks efficiently while retaining previously acquired behaviors. Existing continual RL methods on top of VLAs mitigate forgetting but rely on on-policy updates with limited forward transfer, whereas off-policy methods improve single-task sample efficiency without addressing learner persistence across tasks. We propose ACCRUE, a continual off-policy RL framework that carries lightweight actor and critic networks across tasks, allowing each stage to reuse behavioral and value knowledge acquired through prior RL. The inherited critic adapts only on current-task transitions and guides current-task actor learning, while prior behavior is preserved by distilling the actor toward cached targets on a small number of retained episodes. Fixed visual features keep the inherited learner and its replay compatible with base-policy updates. Across contact-rich assembly sequences in simulation and on real-robot hardware, ACCRUE accelerates subsequent-task learning with near-zero forgetting and significantly reduces real-robot training time by 31.0\% to 68.5\%.
Original Article
View Cached Full Text

Cached at: 10/14/26, 06:53 PM

Paper page — ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models

Source: https://huggingface.co/papers/2610.18596

🤖 Real-robot RL. Few demonstrations. New skills in minutes.

Introducing ACCRUE: Continual Off-Policy Reinforcement Learning for Robot Foundation Models.

We use RL to improve robot foundation model behavior and carry that experience across tasks, helping robots learn faster while retaining earlier skills.

🦾 Learning directly in the real world For our real-world policies, all RL interaction comes from the physical robot. Each task starts with a small set of demonstrations used to fine-tune the foundation model.

💻 Low additional GPU training overhead During RL, the foundation model stays frozen. Only a compact actor–critic is updated to refine its reference actions.

🧠 Make the experience continual We inherit that actor–critic into the next task and distill earlier behavior using a small fraction of prior-task replay.

⏱️ New skills in minutes of robot interaction Using π0.5 + FastTD3, subsequent real-world tasks reach the 95% success stopping criterion in about 4–12 minutes of robot interaction.

Compared with single-task RL:

  • AutoMate assembly: 31.0–68.5% less robot interaction
  • NIST assembly: 38.7–42.9% less robot interaction

Both methods use the same stopping rule over the latest 20 episodes. These are robot interaction times, including online warmup and excluding demonstration collection and foundation-model fine-tuning.

🛡️ Earlier skills remain After each full sequence, the final policies achieve 118/120 successes across six prior tasks (98.3% overall). Each task is evaluated separately for 20 trials across the two independently trained suites.

We also evaluate ACCRUE separately in simulation using GR00T + FastSAC.

🎬 Watch the attached 60-second overview, then explore more videos and interactive results on the project page.

📄 Paper: https://arxiv.org/abs/2610.18596 🌐 Project: https://accrue-rl.github.io/

Similar Articles

Player-1: Learning to Play from Synthetic Experience

Hugging Face Daily Papers

Tencent ARC 的论文提出 Player-1 一套面向游戏控制的后训练框架，通过扩展动作空间、在 6,044 条游戏轨迹上进行 SFT 初始化，并利用 16 个可程序化验证的合成环境进行 PPO 多环境强化学习，显著超越监督基线与现有最强 baseline。

Embodied Turing Machines: Stateful Code for Robot Recursive Self-Improvement

Hugging Face Daily Papers

本文提出 Code-Only-As-Policy (COAP) 范式，将机器人本体视为图灵机，用显式代码替代 VLM/VLA 进行决策与状态跟踪，并借助编码智能体实现闭环递归自我改进；在 RoboDojo 的 42 项双臂任务上达到 70.24% 成功率，且测试时无需模型在环。

OGPO: Offline Goal-conditioned Policy Optimization for Recoverable Vision-Language-Action Models

arXiv cs.AI

OGPO 框架通过语义关键状态重标注和目标条件过程奖励，将视觉-语言-动作模型的后训练从轨迹级模仿转向关键状态可达性学习，从而提升机器人操作策略在偏离演示轨迹时的恢复能力。实验在 LIBERO 和 MetaWorld 上超越标准监督微调，并在 LIBERO-Plus 及新提出的 LIBERO-ReAct 基准中展现出更强的分布偏移鲁棒性。

Recurrent Harness for Agentic Continual Learning

arXiv cs.LG

提出 ReHarness 框架，通过在严格在线流式协议下结合前向读写与后向修订（利用轨迹反馈修订检索到的反思），实现 LLM 智能体的持续学习，在三个基准和三种模型骨干上均优于基线方法。