WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
Summary
Proposes WarpSAC, a regime-aware off-policy reinforcement learning algorithm that adapts stabilizers to data availability, achieving significant performance improvements over FlashSAC in CPU-scale and GPU-parallel environments.
View Cached Full Text
Cached at: 08/27/26, 07:20 AM
Paper page - WarpSAC: Towards the Pinnacle of Scalable Off-policy RL by Rethinking Exploration and Exploitation
Source: https://huggingface.co/papers/2608.24479 Authors:
,
,
,
,
,
,
,
,
,
,
Abstract
Off-policy reinforcement learning stabilizers vary with data availability, motivating regime-aware algorithms that adapt normalization and Q-function clipping to improve efficiency across CPU and GPU-parallel training.
Massively parallel simulation changes the data regime in whichoff-policy reinforcement learning(RL) is trained, challenging stabilizers designed for data-limitedreplay. Through controlled experiments across eight benchmark families, we show that these stabilizers are data-regime-dependent:parameter normalizationhelps with narrowreplaycoverage but restricts value fitting when data are abundant, whileclipped double-Qcan be relaxed in high-throughput manipulation.Age-biased replay weightingimproves learning efficiency across regimes, especially with limited network capacity. Based on these findings, we proposeWarpSAC, a regime-aware family of off-policy RL algorithms.WarpSACusesSample Weight Decayfor efficient exploitation and provides two variants:WarpSAC-L (Norm ON,clipped double-Q) for data-limited CPU-scale training, andWarpSAC-A (Norm OFF, single-Q) for data-abundant GPU-parallel training.WarpSACimproves normalized score--step AUC over FlashSAC by 4.5% across nine CPU-scale environments and 23.1% across fourteen GPU-parallel environments. It increases UnitreeG1TransportBox-v1 success rate from 19.8% to 96.4%, improves mean normalized wall-time AUC on MuJoCo Playground by 19.1%, and achieves 36.4% fastersim-to-realdeployment on Unitree G1 than FlashSAC. These results show that scalable off-policy RL should adapt its stabilizers to the available data regime.
View arXiv pageView PDFProject pageGitHub1Add to collection
Get this paper in your agent:
hf papers read 2608\.24479
Don’t have the latest CLI?curl \-LsSf https://hf\.co/cli/install\.sh \| bash
Models citing this paper0
No model linking this paper
Cite arxiv.org/abs/2608.24479 in a model README.md to link it from this page.
Datasets citing this paper0
No dataset linking this paper
Cite arxiv.org/abs/2608.24479 in a dataset README.md to link it from this page.
Spaces citing this paper0
No Space linking this paper
Cite arxiv.org/abs/2608.24479 in a Space README.md to link it from this page.
Collections including this paper0
No Collection including this paper
Add this paper to acollectionto link it from this page.
Similar Articles
Dreamer-SAC: Off-Policy Learning in Latent World Models for Sample-Efficient Autonomous Driving
This paper proposes Dreamer-SAC, a model-based reinforcement learning framework that integrates a recurrent state-space world model with soft actor-critic in latent space for sample-efficient autonomous driving. It outperforms DreamerV3, SAC, and PPO baselines while requiring fewer real environment interactions.
SAPO: Single-Rollout Autoregressive Policy Optimization for Agentic Reinforcement Learning
SAPO is a low-memory and compute-efficient framework for agentic reinforcement learning that shares policy and value functions in a single autoregressive backbone, outperforming PPO and GRPO in experiments on ALFWorld and WebShop.
Warp RL: Reshaping Base Policy Distributions for Dynamics Adaptation
Warp RL replaces additive residual corrections in reinforcement learning with an invertible, state-conditioned transformation of the base policy's action distribution using monotonic rational-quadratic spline flows, enabling adaptation of distribution shape, scale, and geometry under dynamics shifts. It matches or outperforms residual correction in ManiSkill3 manipulation tasks and achieves 30% faster task completion in a real robot peg-insertion task.
Soft Adaptive Policy Optimization
SAPO introduces a smooth, temperature-controlled gate to adaptively attenuate off-policy updates in reinforcement learning for large language models, enhancing training stability and performance compared to methods with hard clipping.
Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
This paper proposes an exploration-aware reinforcement learning framework that enables LLM agents to adaptively explore only when uncertainty is high, improving performance on text-based and GUI-based benchmarks.