RBGA：面向灵巧操作中离线到在线安全强化学习的剩余预算引导仲裁

# RBGA: Residual-Budget Guide Arbitration for Offline-to-Online Safe Reinforcement Learning in Dexterous Manipulation
Source: [https://arxiv.org/html/2610.33494](https://arxiv.org/html/2610.33494)
Zongyou YuAo ZhouXiongbo ShiJiayi GuanShihang DuXudong ZhangShangding GuJunqiao ZhaoHanli WangGuang Chen Tongji UniversityShanghai Innovation InstituteXiaomi EVShanghai Jiao Tong UniversityUniversity of California, Berkeley††thanks:Corresponding author: Guang Chen \(guangchen@tongji\.edu\.cn\)\.

###### Abstract

Dexterous manipulation requires robots to leverage existing datasets for rapid skill acquisition while maintaining safety during real\-world deployment to prevent damage to the robot or its surroundings\. Offline\-to\-online safe reinforcement learning \(O2O safe RL\) provides a promising solution for this problem by leveraging prior experience for online improvement under safety constraints, thereby mitigating the sample inefficiency and exploration risk of learning safe policies from scratch\. However, existing O2O safe RL approaches do not explicitly account for the safety budget already consumed when making decisions within an episode\. Excessive cost incurred early can therefore leave little room for subsequent exploration and policy improvement\. To address this limitation, we proposeResidual\-BudgetGuideArbitration \(RBGA\), an algorithm that explicitly tracks the remaining episode safety budget and uses it to adaptively select between offline guidance and the evolving online policy\. At each step, RBGA compares the cost\-to\-go estimates of the guide and online actions against the remaining budget, enabling step\-wise action arbitration throughout the episode\. We provide theoretical analysis characterizing the safety implications of residual\-budget\-based selection and its effect on guide\-assisted exploration\. Extensive experiments demonstrate that RBGA enables effective online policy improvement while maintaining strong safety performance\. Our code is available at this[link](https://github.com/sii-pilab)\.

## 1Introduction

Dexterous manipulation enables multi\-fingered robotic hands to perform a wide range of contact\-rich tasks, including in\-hand manipulation, grasping, and tool use\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.33494#bib.bib16);[OpenAI et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib20);[Yuan et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib52)\)\. Despite substantial progress, improving the safety of dexterous manipulation remains a pressing challenge\([García and Fernández, 2015](https://arxiv.org/html/2610.33494#bib.bib41);[Jain et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib27);[Geng et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib15)\)\. Safe reinforcement learning \(RL\) addresses this challenge by optimizing task performance subject to explicit cost constraints\([Chow et al\., 2018](https://arxiv.org/html/2610.33494#bib.bib38);[Tessler et al\., 2019](https://arxiv.org/html/2610.33494#bib.bib48);[Stooke et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib3);[Satija et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib47)\)\. However, learning a safe dexterous policy from scratch is often prohibitive, as it requires extensive environment interactions that are both costly and risky\([Ray et al\., 2019](https://arxiv.org/html/2610.33494#bib.bib5);[Thananjeyan et al\., 2021](https://arxiv.org/html/2610.33494#bib.bib49);[Dulac\-Arnold et al\., 2021](https://arxiv.org/html/2610.33494#bib.bib39)\)\. Offline safe RL reduces this burden by learning policies from a fixed dataset without additional online interaction\([Xu et al\., 2022](https://arxiv.org/html/2610.33494#bib.bib51);[Lee et al\., 2022a](https://arxiv.org/html/2610.33494#bib.bib44);[Liu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib45)\)\. Yet its performance is constrained by the coverage of the offline data\([Levine et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib6);[Liu et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib46)\), making learned policies vulnerable to distribution shift and out\-of\-distribution \(OOD\) actions when encountering unseen states\([Fujimoto et al\., 2019](https://arxiv.org/html/2610.33494#bib.bib40);[Kumar et al\., 2019](https://arxiv.org/html/2610.33494#bib.bib42);[Kumar et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib43)\)\. Offline\-to\-online \(O2O\) safe RL therefore offers an appealing alternative by leveraging policies learned offline to accelerate online adaptation, substantially reducing the extensive environment interactions required to learn policies from scratch\.

Compared with O2O RL, O2O safe RL remains relatively underexplored\([Li et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib12);[Cao et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib13);[Chen et al\., 2026](https://arxiv.org/html/2610.33494#bib.bib14)\)\. Existing policy\-finetuning methods continue online learning from an offline\-pretrained policy, although this transfer can suffer from mismatched value estimates and Lagrange multipliers between offline and online learning\([Chen et al\., 2026](https://arxiv.org/html/2610.33494#bib.bib14)\)\. To facilitate safer and more effective adaptation, some works instead use an offline policy to guide the learning of a separate online policy\([Li et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib12)\), while others preserve offline knowledge through model\-based fine\-tuning and safe policy expansion\([Cao et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib13)\)\. Despite these advances, existing methods do not explicitly account for the cumulative cost already incurred in the current episode when making online decisions\. This limitation is particularly important in dexterous manipulation, where contact\-rich interactions can produce abrupt safety violations and nonuniform cost accumulation over an episode\. Therefore, developing a dedicated O2O safe RL algorithm that explicitly accounts for within\-episode cost accumulation is particularly important for dexterous manipulation\.

In this work, to address the lack of trajectory\-level budget awareness in O2O safe RL, we propose Residual\-Budget Guide Arbitration \(RBGA\), an algorithm that explicitly incorporates the remaining episode cost budget into online policy learning\. First, RBGA tracks the remaining cost budget along each trajectory to characterize the current safety margin throughout online interaction\. Then, at each interaction step, it evaluates the actions proposed by the offline guide and the online policy based on their estimated cost\-to\-go, and adaptively selects between them according to the remaining budget\. This step\-wise arbitration allows the online policy to take control when the guide action becomes unfavorable under the current budget, while retaining useful offline guidance when it remains admissible\. Finally, extensive experiments on Redman\([Geng et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib15)\)demonstrate the effectiveness of RBGA across a range of dexterous manipulation tasks\. The main contributions of this work are summarized as follows:

- •To the best of our knowledge, we are the first to explicitly incorporate trajectory\-level episode cost budget information into O2O safe RL for dexterous manipulation\.
- •We propose RBGA, a step\-wise action arbitration algorithm that adaptively balances offline guidance and the evolving online policy based on the remaining episode budget and estimated cost\-to\-go\.
- •Extensive experiments across diverse dexterous manipulation tasks demonstrate the effectiveness and advantages of RBGA, showing strong policy performance while maintaining favorable safety behavior during online adaptation\.

## 2Related Work

RL for Dexterous Manipulation\.RL has enabled multi\-fingered robotic hands to acquire complex manipulation skills from interaction, with early studies demonstrating effective learning from demonstrations and large\-scale simulation\([Rajeswaran et al\., 2018](https://arxiv.org/html/2610.33494#bib.bib16);[Zhu et al\., 2019](https://arxiv.org/html/2610.33494#bib.bib17);[OpenAI et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib20)\)\. Subsequent work has improved the applicability of learned policies through model\-based planning and more efficient policy optimization\([Nagabandi et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib21);[Charlesworth and Montana, 2021](https://arxiv.org/html/2610.33494#bib.bib22)\)\. Recent studies further extend dexterous RL to unseen objects and real\-world manipulation\([Chen et al\., 2022a](https://arxiv.org/html/2610.33494#bib.bib18);[Qin et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib19);[Qi et al\., 2023a](https://arxiv.org/html/2610.33494#bib.bib24);[Qi et al\., 2023b](https://arxiv.org/html/2610.33494#bib.bib25)\), as well as more challenging bimanual tasks\([Chen et al\., 2022b](https://arxiv.org/html/2610.33494#bib.bib23);[Huang et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib26)\)\. Despite these advances, most prior work considers dexterous manipulation without explicit safety constraints\. Safety is nevertheless critical for dexterous manipulation, as unsafe actions during contact\-rich interaction may damage the robotic hand or its surroundings\. Research on safe dexterous manipulation remains limited\. A few studies incorporate explicit constraints into dexterous policy learning or extend safe RL to bimanual manipulation\([Jain et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib27);[Zhan and Chin, 2024](https://arxiv.org/html/2610.33494#bib.bib28)\)\. ReDMan further establishes a dedicated benchmark for evaluating safe RL algorithms on dexterous manipulation tasks\([Geng et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib15)\)\. These works mainly learn or evaluate safe policies through online interaction, leaving how offline experience can facilitate subsequent safe adaptation largely unexplored\.

Offline\-to\-Online RL\.Offline\-to\-online \(O2O\) RL leverages offline pretraining to accelerate online policy improvement\. A natural strategy is to initialize online learning with an offline\-trained policy and continue fine\-tuning it using collected experience\([Nair et al\., 2020](https://arxiv.org/html/2610.33494#bib.bib7)\)\. To improve the stability and efficiency of this transition, some methods balance offline and online samples during replay\([Lee et al\., 2022b](https://arxiv.org/html/2610.33494#bib.bib8);[Ball et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib29)\), while others correct value estimates inherited from offline training before or during online fine\-tuning\([Yu and Zhang, 2023](https://arxiv.org/html/2610.33494#bib.bib30);[Nakamoto et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib9);[Zhang et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib31);[Luo et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib32)\)\. Instead of directly fine\-tuning the offline policy, JSRL uses a fixed guide policy to facilitate exploration through progressively shortened roll\-ins\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\), whereas PEX retains separate offline and online policies and selects between their proposed actions according to estimated values\([Zhang et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib11)\)\. Despite substantial progress, most O2O RL methods are developed for unconstrained reward maximization and do not address safety constraints during online adaptation\. Recent O2O safe RL studies begin to address this problem by incorporating safety into offline\-to\-online transfer\. GOLD uses an offline sequence policy to guide the learning of an online safe policy\([Li et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib12)\)\. However, GOLD relies on a Decision Transformer as the offline guide and performs guided online distillation, differing from the matched offline\-policy setting considered in this work\. FOSP combines offline model learning with safe policy expansion for online fine\-tuning\([Cao et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib13)\)\. However, FOSP adopts a model\-based safe O2O setting with a learned world model, which is not directly comparable to the model\-free setting considered in this work\.

Despite these advances, existing O2O safe RL studies have not addressed dexterous manipulation, where unsafe exploratory actions can damage the robotic hand or its surroundings\. Learning a safe dexterous policy from scratch requires costly and risky online interaction, while offline safe RL remains limited by the coverage of the fixed dataset and degrades under distribution shift\. O2O safe RL is therefore particularly appealing for dexterous manipulation, as it reuses offline knowledge to accelerate online adaptation under safety constraints\. To the best of our knowledge, we are the first to study O2O safe RL for dexterous manipulation\.

## 3Preliminaries

This section introduces the constrained RL formulation and the O2O safe RL setting used throughout the paper\. We first describe finite\-horizon Markov decision processes and their extension to explicit cost constraints\. We then specify the offline and online policies considered by RBGA\.

### 3\.1Constrained Markov Decision Processes

Standard RL is formulated as a finite\-horizon Markov decision process \(MDP\)ℳMDP=\(𝒮,𝒜,P,r,ρ0,γ,H\)\\mathcal\{M\}\_\{\\mathrm\{MDP\}\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\rho\_\{0\},\\gamma,H\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}are the state and action spaces,P\(⋅∣s,a\)P\(\\cdot\\mid s,a\)is the transition kernel,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}is a bounded reward function,ρ0\\rho\_\{0\}is the initial\-state distribution,γ∈\(0,1\]\\gamma\\in\(0,1\]is the discount factor, andHHis the finite episode horizon\. A policyπ\\piinduces a trajectory throughs0∼ρ0s\_\{0\}\\sim\\rho\_\{0\},at∼π\(⋅∣st\)a\_\{t\}\\sim\\pi\(\\cdot\\mid s\_\{t\}\), andst\+1∼P\(⋅∣st,at\)s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\. For policies that depend on the episode step, we omit the time index for simplicity\. Standard RL maximizes the expected discounted reward\.

Safe RL augments the MDP with a bounded non\-negative cost functionc:𝒮×𝒜→ℝc:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}and a cost limitdd, giving the constrained MDPℳ=\(𝒮,𝒜,P,r,c,ρ0,γ,H,d\)\\mathcal\{M\}=\(\\mathcal\{S\},\\mathcal\{A\},P,r,c,\\rho\_\{0\},\\gamma,H,d\)\([Altman, 1999](https://arxiv.org/html/2610.33494#bib.bib1);[Achiam et al\., 2017](https://arxiv.org/html/2610.33494#bib.bib2)\)\. For any policyπ\\pi, the expected reward and cost are

JR​\(π\)=𝔼τ∼π​\[∑t=0H−1γt​r​\(st,at\)\],JC​\(π\)=𝔼τ∼π​\[∑t=0H−1γt​c​\(st,at\)\],J\_\{R\}\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\],\\qquad J\_\{C\}\(\\pi\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}c\(s\_\{t\},a\_\{t\}\)\\right\],\(1\)whereτ\\taudenotes a trajectory induced by policyπ\\pi, andRRandCCdenote reward and cost, respectively\.

The common objective of constrained RL is to maximize the cumulative reward while satisfying the cumulative cost constraint\. Accordingly, the optimal feasible policyπ⋆\\pi^\{\\star\}is defined as

π⋆∈arg​maxπ⁡JR​\(π\)s\.t\.JC​\(π\)≤d\.\\pi^\{\\star\}\\in\\operatorname\*\{arg\\,max\}\_\{\\pi\}J\_\{R\}\(\\pi\)\\quad\\text\{s\.t\.\}\\quad J\_\{C\}\(\\pi\)\\leq d\.\(2\)The objective maximizes task return while constraining the expected cumulative cost\. In dexterous manipulation, the state represents the hand–object configuration, the action specifies a control command for the robotic hand, and the reward measures progress toward the manipulation objective, while the cost function penalizes unsafe behavior during interaction\.

### 3\.2Offline\-to\-Online Safe Reinforcement Learning

Offline\-to\-online \(O2O\) RL leverages offline data or a previously learned policy to support subsequent online policy learning\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10);[Zhang et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib11)\), while O2O safe RL further requires this online improvement to satisfy the cost constraint in Eq\. \([2](https://arxiv.org/html/2610.33494#S3.E2)\)\([Li et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib12);[Cao et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib13);[Chen et al\., 2026](https://arxiv.org/html/2610.33494#bib.bib14)\)\. In O2O safe RL, an offline policyπg\\pi\_\{g\}is first trained on a fixed offline dataset𝒟off\\mathcal\{D\}\_\{\\mathrm\{off\}\}of transitions\(s,a,r,c,s′,δ\)\(s,a,r,c,s^\{\\prime\},\\delta\), wherer=r⁡\(s,a\)r=r\(s,a\)andc=c⁡\(s,a\)c=c\(s,a\)are the observed reward and cost,s′s^\{\\prime\}is the next state, andδ∈\{0,1\}\\delta\\in\\\{0,1\\\}indicates episode termination\. The learned offline policy is then used to support subsequent online policy learning\. We denote the online policy byπo\\pi\_\{o\}and update it from newly collected transitions using a constrained RL algorithm\.

## 4Methodology

In this section, we first introduce Residual\-Budget Guide Arbitration \(RBGA\) for O2O safe RL in dexterous manipulation\. We first describe how guide horizon scheduling and residual\-budget\-based action selection combine offline guidance with online learning\. We then provide theoretical analysis of the cost of the composite policy and the reward performance of the learned online policy\.

### 4\.1Residual\-Budget Guide Arbitration

Guide horizon scheduling\.Inspired by JSRL\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\), RBGA combines a fixed offline guide policyπg\\pi\_\{g\}and a trainable online policyπo\\pi\_\{o\}into a composite policyπ\\pifor online interaction\. Consider a dexterous manipulation episode of at mostHHcontrol steps and a total ofNNonline training iterations\. To limit dependence on the offline guide, we specify a guidance cutoffNg≤NN\_\{g\}\\leq N\. During the firstNgN\_\{g\}training iterations, the offline guide participates in action selection, while the online policy acts alone thereafter\. The online policy is initialized independently and updated from collected experience\. At iterationii, the maximum guide horizonHi∈\{0,…,H\}H\_\{i\}\\in\\\{0,\\ldots,H\\\}controls the number of initial episode steps during which guide actions will be considered\. DecreasingHiH\_\{i\}allows the online policy to take control earlier as training progresses\. Specifically, in JSRL, the guide horizon is directly reduced across curriculum stages, introducing abrupt changes in the state distribution encountered by the online policy\. To provide a smoother transition from offline guidance to online control, we instead gradually decrease the maximum guide horizon as training progresses and consider two scheduling strategies,LinearandStep\. For the linear schedule, we set

Hi=round⁡\(H0​\[1−iNg\]\+\),Ng\>0,H\_\{i\}=\\operatorname\{round\}\\\!\\left\(H\_\{0\}\\left\[1\-\\frac\{i\}\{N\_\{g\}\}\\right\]\_\{\+\}\\right\),\\qquad N\_\{g\}\>0,\(3\)wherei∈\{0,…,N\}i\\in\\\{0,\\ldots,N\\\}counts completed online training iterations,H0≤HH\_\{0\}\\leq His the initial maximum guide horizon,\[⋅\]\+\[\\cdot\]\_\{\+\}clips negative values to zero, andround⁡\(⋅\)\\operatorname\{round\}\(\\cdot\)rounds to the nearest integer\. Thus,Hi=0H\_\{i\}=0fori≥Ngi\\geq N\_\{g\}\. A step schedule is described in Appendix[A\.5](https://arxiv.org/html/2610.33494#A1.SS5)\.

To allow the online policy to take over at different stages of an episode, we samplehhuniformly from\{0,…,Hi\}\\\{0,\\ldots,H\_\{i\}\\\}at the beginning of each episode, independently of the initial state\. The sampled horizonhhremains fixed within that episode, and guide actions are considered only whent<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}\. Here,ttcounts control steps within the current manipulation attempt, whileiicounts training iterations\. If an episode spans several updates, the currentHiH\_\{i\}continues to apply, so a decrease inHiH\_\{i\}can end guidance before the sampled horizonhhis reached\.

Residual budget\.A limited guide horizon does not by itself prevent unsafe actions\. The offline guide may produce actions that substantially violate safety constraints and potentially damage the robotic hand or its surroundings due to distribution shift between offline training and online interaction\. We therefore account for the costs already incurred when selecting actions within the guide horizon\. Before executing the action at steptt, we define the residual budget as

bt=d−∑j=0t−1γj​cj,b0=d,b\_\{t\}=d\-\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\},\\qquad b\_\{0\}=d,\(4\)whereddis the cost limit for a complete manipulation episode,cj=c⁡\(sj,aj\)c\_\{j\}=c\(s\_\{j\},a\_\{j\}\)measures the safety cost incurred by the hand at stepjj, andγ\\gammais the discount factor\. In dexterous tasks, the cost penalizes unsafe behaviors of the robotic hand during manipulation\. Thus,btb\_\{t\}is the budget left for the rest of the manipulation, andbt<0b\_\{t\}<0means that the accumulated discounted cost has already exceededdd\.

To select an action under the residual budget, we first obtain candidate actions from the two policies\. Given the current statests\_\{t\}, we sample an online candidateato∼πo\(⋅∣st\)a\_\{t\}^\{o\}\\sim\\pi\_\{o\}\(\\cdot\\mid s\_\{t\}\)and, whent<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}, a guide candidateatg∼πg\(⋅∣st\)a\_\{t\}^\{g\}\\sim\\pi\_\{g\}\(\\cdot\\mid s\_\{t\}\)\. To compare the candidates against the residual budget, we evaluate the expected discounted cost of executing each candidate and following the current online policy thereafter\. For a fixedπo\\pi\_\{o\}, this cost\-to\-go is defined as:

QCπo\(st,at\)=𝔼πo\[∑j=tH−1γj−tcj\|st,at\],Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)=\\mathbb\{E\}\_\{\\pi\_\{o\}\}\\left\[\\sum\_\{j=t\}^\{H\-1\}\\gamma^\{j\-t\}c\_\{j\}\\,\\middle\|\\,s\_\{t\},a\_\{t\}\\right\],\(5\)whereata\_\{t\}is the candidate control command being evaluated and subsequent actions followπo\\pi\_\{o\}\. For notational simplicity, we omit the time index on finite\-horizon value and advantage functions when the corresponding episode step is clear from context\. Since this expected cost is not directly available, we learn an auxiliary cost estimatorQ^Cπo​\(s,a\)\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s,a\)from online experience to approximate Eq\. \([5](https://arxiv.org/html/2610.33494#S4.E5)\) and use it to evaluate both candidates, as detailed in Appendix[A\.4](https://arxiv.org/html/2610.33494#A1.SS4)\. For comparison withbtb\_\{t\}, we defineq^tg=γt​Q^Cπo​\(st,atg\)\\widehat\{q\}\_\{t\}^\{g\}=\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)andq^to=γt​Q^Cπo​\(st,ato\)\\widehat\{q\}\_\{t\}^\{o\}=\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\. These estimates express the candidates’ remaining costs on the same discount scale as the episode budget\.

Step\-wise arbitration\.RBGA combines the guide horizon, residual budget, and estimated cost\-to\-go to select the executed action\. To retain useful guidance while limiting its additional cost, we introduce a comparison marginζ≥0\\zeta\\geq 0, which bounds the additional estimated cost\-to\-go of the guide candidate relative to the online candidate\. The executed action is then selected as

at=\{ato,t≥min⁡\{h,Hi\},atg,t<min⁡\{h,Hi\}​and​q^tg≤min⁡\{bt,q^to\+γt​ζ\},ato,t<min\{h,Hi\},q^to≤btandq^tg\>min\{bt,q^to\+γtζ\},arg​mina∈\{atg,ato\}⁡Q^Cπo​\(st,a\),otherwise\.a\_\{t\}=\\begin\{cases\}a\_\{t\}^\{o\},&t\\geq\\min\\\{h,H\_\{i\}\\\},\\\\ a\_\{t\}^\{g\},&t<\\min\\\{h,H\_\{i\}\\\}\\ \\text\{and\}\\ \\widehat\{q\}\_\{t\}^\{g\}\\leq\\min\\\{b\_\{t\},\\widehat\{q\}\_\{t\}^\{o\}\+\\gamma^\{t\}\\zeta\\\},\\\\ a\_\{t\}^\{o\},&t<\\min\\\{h,H\_\{i\}\\\},\\ \\widehat\{q\}\_\{t\}^\{o\}\\leq b\_\{t\}\\ \\text\{and\}\\ \\widehat\{q\}\_\{t\}^\{g\}\>\\min\\\{b\_\{t\},\\widehat\{q\}\_\{t\}^\{o\}\+\\gamma^\{t\}\\zeta\\\},\\\\ \\operatorname\*\{arg\\,min\}\_\{a\\in\\\{a\_\{t\}^\{g\},a\_\{t\}^\{o\}\\\}\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\),&\\text\{otherwise\}\.\\end\{cases\}\(6\)Whent≥min⁡\{h,Hi\}t\\geq\\min\\\{h,H\_\{i\}\\\}, the guide is no longer eligible, and RBGA executes the online candidateatoa\_\{t\}^\{o\}\. Whent<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}andq^tg≤min⁡\{bt,q^to\+γt​ζ\}\\widehat\{q\}\_\{t\}^\{g\}\\leq\\min\\\{b\_\{t\},\\widehat\{q\}\_\{t\}^\{o\}\+\\gamma^\{t\}\\zeta\\\}, the guide remains eligible, its estimated cost\-to\-go satisfies the remaining budget, and its additional estimated cost relative to the online candidate is within the comparison margin; RBGA therefore selectsatga\_\{t\}^\{g\}\. Whent<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\},q^to≤bt\\widehat\{q\}\_\{t\}^\{o\}\\leq b\_\{t\}, andq^tg\>min⁡\{bt,q^to\+γt​ζ\}\\widehat\{q\}\_\{t\}^\{g\}\>\\min\\\{b\_\{t\},\\widehat\{q\}\_\{t\}^\{o\}\+\\gamma^\{t\}\\zeta\\\}, the online candidate satisfies the remaining budget while the guide fails either the budget or comparison\-margin condition, so RBGA selectsatoa\_\{t\}^\{o\}\. Otherwise, both candidates exceed the remaining budget, and RBGA selects the candidate with the smaller estimated cost\-to\-go\. We provide a detailed description of RBGA in Algorithm[1](https://arxiv.org/html/2610.33494#alg1)\.

Algorithm 1Residual\-Budget Guide Arbitration \(RBGA\)1:Input:fixed guide policy

πg\\pi\_\{g\}, task horizon

HH, total training iterations

NN, guide horizon schedule with cutoff

NgN\_\{g\}, cost limit

dd, discount factor

γ\\gamma, and comparison margin

ζ\\zeta\.

2:Initialize the online learner with policy

πo\\pi\_\{o\}and the auxiliary estimator

Q^Cπo\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}independently of

πg\\pi\_\{g\}; set

ℬ←∅\\mathcal\{B\}\\leftarrow\\varnothingand

i←0i\\leftarrow 0\.

3:Reset the environment; set

t←0t\\leftarrow 0,

bt←db\_\{t\}\\leftarrow d, and sample

h∼Unif⁡\{0,…,Hi\}h\\sim\\operatorname\{Unif\}\\\{0,\\ldots,H\_\{i\}\\\}\.

4:while

i<Ni<Ndo

5:foreach environment stepdo

6:Select

ata\_\{t\}using Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) after warm\-up; use the warm\-up procedure otherwise\.

7:Execute

ata\_\{t\}; observe

rt,ct,st\+1,δtr\_\{t\},c\_\{t\},s\_\{t\+1\},\\delta\_\{t\}; append the transition to

ℬ\\mathcal\{B\}\.

8:Set

bt\+1←bt−γt​ctb\_\{t\+1\}\\leftarrow b\_\{t\}\-\\gamma^\{t\}c\_\{t\}and

t←t\+1t\\leftarrow t\+1\.

9:ifthe episode has endedthen

10:Reset the environment; set

t←0t\\leftarrow 0,

bt←db\_\{t\}\\leftarrow d, and sample a new

h∼Unif⁡\{0,…,Hi\}h\\sim\\operatorname\{Unif\}\\\{0,\\ldots,H\_\{i\}\\\}\.

11:endif

12:endfor

13:

πo,Q^Cπo,i←TrainPolicy​\(πo,Q^Cπo,ℬ,i,N\)\\pi\_\{o\},\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\},i\\leftarrow\\textsc\{TrainPolicy\}\(\\pi\_\{o\},\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\},\\mathcal\{B\},i,N\); obtain

HiH\_\{i\}from the schedule\.

14:endwhile

15:Return:the learned online policy

πo\\pi\_\{o\}\.

### 4\.2Theoretical Analysis

In this section, we analyze the cost of RBGA action selection and the reward performance of the learned policy\. We first derive local cost bounds from the remaining budget and cost estimates, and then obtain an expected cost bound for the composite policy\. Subsequently, we analyze the reward gap relative to the optimal feasible policy under state coverage and policy advantage conditions\. To account for the accuracy of the cost estimates in Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\), we introduce the following assumption\.

###### Assumption 4\.1\(Cost estimation error\)\.

For a fixed online policyπo\\pi\_\{o\}and cost estimatorQ^Cπo\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}, there exists a finite constantεC≥0\\varepsilon\_\{C\}\\geq 0such that, at each time stept<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}, the cost estimates of both candidate actions in Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) satisfy:

\|Q^Cπo​\(st,a\)−QCπo​\(st,a\)\|≤εC,∀a∈\{atg,ato\},\\left\|\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\)\\right\|\\leq\\varepsilon\_\{C\},\\qquad\\forall a\\in\\\{a\_\{t\}^\{g\},a\_\{t\}^\{o\}\\\},\(7\)whereεC\\varepsilon\_\{C\}is a uniform bound on their estimation errors\.

Based on Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1)and the remaining budget in Eq\. \([4](https://arxiv.org/html/2610.33494#S4.E4)\), we derive the cost bounds for a selected action, as shown in Proposition[4\.2](https://arxiv.org/html/2610.33494#S4.Thmtheorem2)\. Proofs and discussions are in Appendix[C\.2](https://arxiv.org/html/2610.33494#A3.SS2)\.

###### Proposition 4\.2\(Local cost bounds\)\.

Under Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), consider a time stept<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}\. For a selected actionata\_\{t\}satisfyingγt​Q^Cπo​\(st,at\)≤bt\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq b\_\{t\}, the accumulated discounted cost plus the expected discounted cost of executingata\_\{t\}and followingπo\\pi\_\{o\}thereafter satisfies:

∑j=0t−1γj​cj\+γt​QCπo​\(st,at\)≤d\+γt​εC≤d\+εC\.\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\}\+\\gamma^\{t\}Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq d\+\\gamma^\{t\}\\varepsilon\_\{C\}\\leq d\+\\varepsilon\_\{C\}\.\(8\)For every guide action selected by Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\), the difference between its cost\-to\-go and that of the online candidate satisfies:

QCπo​\(st,atg\)−QCπo​\(st,ato\)≤ζ\+2​εC\.Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\leq\\zeta\+2\\varepsilon\_\{C\}\.\(9\)

Proposition[4\.2](https://arxiv.org/html/2610.33494#S4.Thmtheorem2)bounds the accumulated discounted cost together with the expected continuation cost for actions satisfying the budget condition, and the additional cost\-to\-go of a guide action relative to the online candidate\. By combining Eq\. \([9](https://arxiv.org/html/2610.33494#S4.E9)\) with the finite\-horizon cost form of the performance difference lemma\([Kakade and Langford, 2002](https://arxiv.org/html/2610.33494#bib.bib33)\), we obtain the episode cost bound in Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3)\.

###### Proposition 4\.3\(Cost bound for the composite policy\)\.

Under Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), fixπo\\pi\_\{o\},Q^Cπo\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}, andHiH\_\{i\}throughout an episode, and sampleh∼Unif⁡\{0,…,Hi\}h\\sim\\operatorname\{Unif\}\\\{0,\\ldots,H\_\{i\}\\\}independently of the initial state\. The composite policyπ\\pidefined by Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) satisfies:

\[JC​\(π\)−d\]\+≤\[JC​\(πo\)−d\]\+\+\(ζ\+2​εC\)​∑t=0Hi−1γt​Hi−tHi\+1,\[J\_\{C\}\(\\pi\)\-d\]\_\{\+\}\\leq\[J\_\{C\}\(\\pi\_\{o\}\)\-d\]\_\{\+\}\+\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\}\-t\}\{H\_\{i\}\+1\},\(10\)where\[u\]\+=max⁡\{u,0\}\[u\]\_\{\+\}=\\max\\\{u,0\\\},JC​\(π\)J\_\{C\}\(\\pi\)is the expected discounted episode cost under the composite policy, with expectation over the initial state, sampled guide horizon, candidate actions, and state transitions, and\(Hi−t\)/\(Hi\+1\)=Pr⁡\(h\>t\)\(H\_\{i\}\-t\)/\(H\_\{i\}\+1\)=\\Pr\(h\>t\)is the probability that the sampled guide horizon extends beyond stepttunder uniform sampling, for0≤t<Hi0\\leq t<H\_\{i\}\.

Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3)bounds the additional violation of the expected cost constraint in terms of the comparison margin, estimation error, and maximum guide horizon\. The final term does not increase as the maximum guide horizon is reduced and equals zero when the guide is removed\. Proposition[C\.1](https://arxiv.org/html/2610.33494#A3.Thmtheorem1)in Appendix[C\.3](https://arxiv.org/html/2610.33494#A3.SS3)extends the analysis to policy updates within an episode\.

To analyze reward performance under RBGA roll\-ins, we consider the state distributions used for policy learning, following the state coverage analysis of guide\-based policy search\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\)\. For a learned online policyπo\\pi\_\{o\}, the reward advantage at timettis defined as:

ARπo\(s,a\)=QRπo\(s,a\)−𝔼a′∼πo\(⋅∣s\)\[QRπo\(s,a′\)\],A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)=Q\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\-\\mathbb\{E\}\_\{a^\{\\prime\}\\sim\\pi\_\{o\}\(\\cdot\\mid s\)\}\[Q\_\{R\}^\{\\pi\_\{o\}\}\(s,a^\{\\prime\}\)\],\(11\)whereQRπo​\(s,a\)Q\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)is the expected discounted reward∑j=tH−1γj−t​rj\\sum\_\{j=t\}^\{H\-1\}\\gamma^\{j\-t\}r\_\{j\}obtained by executingaainssand followingπo\\pi\_\{o\}thereafter, andrj=r⁡\(sj,aj\)r\_\{j\}=r\(s\_\{j\},a\_\{j\}\)\. Based on this advantage function, we introduce conditions on state coverage and the remaining advantage of the optimal feasible policy\.

###### Assumption 4\.4\(State coverage and policy advantage\)\.

For the learned online policyπo\\pi\_\{o\}, there exist a finite coverage coefficientD∞≥1D\_\{\\infty\}\\geq 1and finite reward advantage boundsεR,t≥0\\varepsilon\_\{R,t\}\\geq 0such that, at every time steptt, the state distributions satisfy:

ρtπ⋆≤D∞​μt,\\rho\_\{t\}^\{\\pi^\{\\star\}\}\\leq D\_\{\\infty\}\\mu\_\{t\},\(12\)whereμt\\mu\_\{t\}is the marginal state distribution at timettinduced by RBGA roll\-ins,ρtπ⋆\\rho\_\{t\}^\{\\pi^\{\\star\}\}is the corresponding distribution underπ⋆\\pi^\{\\star\}, andD∞D\_\{\\infty\}bounds their relative probabilities\. Under the roll\-in distribution, the expected positive advantage of the optimal feasible policy satisfies:

𝔼s∼μt\[\[𝔼a∼π⋆\(⋅∣s\)ARπo\(s,a\)\]\+\]≤εR,t,\\mathbb\{E\}\_\{s\\sim\\mu\_\{t\}\}\\left\[\\left\[\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\\right\]\_\{\+\}\\right\]\\leq\\varepsilon\_\{R,t\},\(13\)whereεR,t\\varepsilon\_\{R,t\}bounds the remaining positive reward advantage ofπ⋆\\pi^\{\\star\}under the roll\-in distribution\.

By applying the state coverage condition to the positive advantage in Eq\. \([13](https://arxiv.org/html/2610.33494#S4.E13)\), we obtain the reward performance bound shown in Proposition[4\.5](https://arxiv.org/html/2610.33494#S4.Thmtheorem5)\. Proofs and discussions are in Appendix[C\.4](https://arxiv.org/html/2610.33494#A3.SS4)and[C\.5](https://arxiv.org/html/2610.33494#A3.SS5)\.

###### Proposition 4\.5\(Reward performance\)\.

Under Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4), the reward gap between the learned online policyπo\\pi\_\{o\}and the optimal feasible policyπ⋆\\pi^\{\\star\}satisfies:

JR​\(π⋆\)−JR​\(πo\)≤D∞​∑t=0H−1γt​εR,t\.J\_\{R\}\(\\pi^\{\\star\}\)\-J\_\{R\}\(\\pi\_\{o\}\)\\leq D\_\{\\infty\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\varepsilon\_\{R,t\}\.\(14\)

Proposition[4\.5](https://arxiv.org/html/2610.33494#S4.Thmtheorem5)bounds the reward gap between the learned online policyπo\\pi\_\{o\}and the optimal feasible policyπ⋆\\pi^\{\\star\}\. OnceHi=0H\_\{i\}=0, Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) givesπ=πo\\pi=\\pi\_\{o\}\. The same bound therefore applies to the composite policy after the guide is removed, provided that the learned online policy satisfies Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4)\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/main_results_iclr.png)Figure 1:The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms in the comparison experiment\. The shaded areas represent variance over three random seeds\. The dashed line indicates the episode cost limitd=30d=30\.

## 5Experiment

In this section, we conduct comprehensive comparative experiments between RBGA and prior O2O safe RL methods in dexterous manipulation tasks\. Moreover, we conduct ablation studies to evaluate the effectiveness of guide horizon scheduling and the contribution of residual\-budget arbitration\.

### 5\.1Task and Baseline

Task\.We evaluate RBGA on four dexterous manipulation tasks from the ReDMan benchmark\([Geng et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib15)\), including*Joint*,*Finger*,*ArmJoint*, and*ArmFinger*\. These tasks combine two bimanual object\-transfer settings with constraints on either a single joint or multiple finger joints; task descriptions are provided in Appendix[A\.1](https://arxiv.org/html/2610.33494#A1.SS1)\. The original ReDMan environments use binary cost signals to indicate safety violations, resulting in abrupt changes between safe and unsafe states\. To provide a more informative safety signal for policy learning, we replace the binary cost with a continuous cost while preserving the original safety definition of each task\. The resulting continuous cost signalct=c⁡\(st,at\)c\_\{t\}=c\(s\_\{t\},a\_\{t\}\)is used to compute the cumulative cost over an episode\. We set the episode cost limit tod=30d=30for all tasks\.

Baselines\.We compare RBGA with the following six baselines:\(i\)JSRL\-Random\-Lag\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\): the random\-switching variant of JSRL, which uniformly samples the number of initial guide steps for each episode and then switches from the offline guide to the online policy;\(ii\)JSRL\-Curr\-Lag\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\): the curriculum variant of JSRL, which progressively reduces the number of initial guide steps during training, allowing the online policy to take control earlier in each episode;\(iii\)PEX\-Lag\([Zhang et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib11)\): adaptively selects between actions proposed by a fixed offline policy and a trainable online policy using reward value estimates with a Lagrangian cost penalty;\(iv\)SO2\-Lag\([Zhang et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib31)\): fine\-tunes an offline\-pretrained policy using a mixture of offline and online replay data, perturbed target actions, and more frequent critic updates;\(v\)Direct: initializes the online actor and the reward and cost critics from an offline\-pretrained checkpoint, and then directly fine\-tunes them using SAC\-Lag;\(vi\)Scratch: trains SAC\-Lag from scratch using online interactions, without offline pretraining\. Since JSRL, PEX, and SO2 were originally developed for unconstrained O2O RL, we adapt them to O2O safe RL by using cost critics and adaptive Lagrange multipliers to incorporate episode cost constraints into online policy learning, following the Lagrangian formulation used in RBGA\. We additionally reportOffline, the performance of the offline\-pretrained policy without online fine\-tuning, as a reference\.

Implementation Details\.We use IQL as the offline policy and SAC\-Lag as the online policy\. See the Appendix[A](https://arxiv.org/html/2610.33494#A1)for implementation details\. Unless otherwise stated, RBGA uses the Linear schedule withNg=40,000N\_\{g\}=40\{,\}000iterations\. In addition, we evaluate the compatibility of RBGA with DDPG\-Lag\([Lillicrap et al\., 2016](https://arxiv.org/html/2610.33494#bib.bib37);[Ji et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib35)\)as an alternative online learner, with detailed results provided in the Appendix[B\.2](https://arxiv.org/html/2610.33494#A2.SS2)\.

### 5\.2Performance Comparison Experiment

To evaluate the performance of RBGA across various dexterous manipulation tasks, we conduct comprehensive comparative experiments\. The results involve comparing RBGA with the baseline algorithms on four tasks:*Finger*,*Joint*,*ArmFinger*, and*ArmJoint*\. Figure[1](https://arxiv.org/html/2610.33494#S4.F1)illustrates the reward, cost, and success rate curves of RBGA and the baseline algorithms on the four tasks\. From the illustrated results, it can be observed that, compared with the baseline algorithms, RBGA maintains the episode cost close to the predefined cost limit while achieving higher rewards and success rates\. Furthermore, RBGA demonstrates more stable performance compared with the other baseline algorithms during online training\. Notably, RBGA achieves the best performance across the four tasks while maintaining favorable safety performance under the cost constraint\. The above results indicate that RBGA consistently enables effective online policy improvement across various safe dexterous manipulation tasks while maintaining the episode cost around the predefined cost limit\. We also evaluate RBGA on other manipulation tasks in Appendix[B\.3](https://arxiv.org/html/2610.33494#A2.SS3), and describe its deployment for real\-world cube grasping in Appendix[B\.4](https://arxiv.org/html/2610.33494#A2.SS4)\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/ablation_a_all_tasks_reward_cost_success.png)Figure 2:The figure depicts the reward, cost, and success rate curves under different guide horizon scheduling strategies\. The shaded areas represent variance over three random seeds\. The labels 40k, 60k, and 80k indicate the iterations at which guidance ends, while Random and Fixed retain guidance throughout training\. The dashed line indicates the episode cost limitd=30d=30\.
### 5\.3Ablation Study

To evaluate the effect of different guide horizon scheduling strategies on online learning, we conduct ablation experiments across the four dexterous manipulation tasks\. Specifically, we compare the proposed Linear and Step schedules with JSRL\-Random and JSRL\-Curriculum\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10)\), as well as a Fixed schedule\. For Linear, Step, and JSRL\-Curriculum, guidance ends at 40k, 60k, or 80k online training iterations, whereas JSRL\-Random and Fixed retain guide participation throughout training\. Figure[2](https://arxiv.org/html/2610.33494#S5.F2)illustrates the reward, cost, and success rate curves of the different scheduling strategies across the four tasks\. From the illustrated results, it can be observed that the Linear and Step schedules generally provide stable policy improvement and achieve competitive reward and success rate\. Notably, Linear\-40k achieves consistently strong performance across the four tasks while maintaining favorable cost control\. In contrast, several JSRL\-Curriculum configurations exhibit large performance fluctuations or sharp degradation during online training, while JSRL\-Random and Fixed also do not consistently match the performance of the decreasing\-horizon schedules\. These results indicate that sampling the guide horizon within a gradually decreasing maximum horizon provides more stable online learning than using a deterministic stage\-wise guide horizon\. This is because sampling the guide horizon within the current maximum horizon as it gradually decreases over training allows the online policy to take over at different episode steps, leading to smoother changes in the state distribution and reducing abrupt shifts between curriculum stages\. We additionally evaluate the contribution of residual\-budget arbitration in Appendix[B\.1](https://arxiv.org/html/2610.33494#A2.SS1)\.

## 6Conclusion

In this work, we study O2O safe RL for dexterous manipulation and propose Residual\-Budget Guide Arbitration \(RBGA\)\. Specifically, we first use a decreasing guide horizon to control the participation of a fixed offline guide during online learning\. Based on this, we introduce a residual budget to track the episode cost already incurred and select between the guide and online actions according to the remaining budget and their estimated cost\-to\-go\. Furthermore, we provide theoretical analyses of the cost of the composite policy and the reward performance of the learned online policy\. Finally, extensive comparison and ablation experiments on dexterous manipulation tasks demonstrate that RBGA achieves strong task performance while maintaining favorable safety performance\.

## AI Assistance Disclosure

In this work, we used generative AI tools to aid and polish writing, retrieve and discover relevant literature, support research ideation and execution, draft and revise portions of the manuscript, and assist in formulating and checking mathematical claims and proofs\. The authors reviewed and verified all AI\-assisted content, including literature references, technical descriptions, mathematical derivations, and proofs\. We take responsibility for the final content of this work, including text, claims, results, and mathematical statements produced with the aid of generative AI\.

## References

- J\. Achiam, D\. Held, A\. Tamar, and P\. AbbeelConstrained policy optimization\.InProceedings of the 34th International Conference on Machine Learning,D\. Precup and Y\. W\. Teh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.70,pp\. 22–31\.External Links:[Link](https://proceedings.mlr.press/v70/achiam17a.html)Cited by:[§3\.1](https://arxiv.org/html/2610.33494#S3.SS1.p2.1)\.
- Altman \(1999\)E\. AltmanConstrained Markov decision processes\.Chapman and Hall/CRC,Boca Raton, Florida\.Cited by:[§3\.1](https://arxiv.org/html/2610.33494#S3.SS1.p2.1)\.
- Ballet al\.\(2023\)P\. J\. Ball, L\. Smith, I\. Kostrikov, and S\. LevineEfficient online reinforcement learning with offline data\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 1577–1594\.External Links:[Link](https://proceedings.mlr.press/v202/ball23a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- Caoet al\.\(2025\)C\. Cao, Y\. Xin, S\. Wu, L\. He, Z\. Yan, J\. Tan, and X\. WangFOSP: fine\-tuning offline safe policy through world models\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=dbuFJg7eaw)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p2.1),[§2](https://arxiv.org/html/2610.33494#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.33494#S3.SS2.p1.1)\.
- Charlesworth and Montana \(2021\)H\. J\. Charlesworth and G\. MontanaSolving challenging dexterous manipulation tasks with trajectory optimisation and reinforcement learning\.InProceedings of the 38th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.139,pp\. 1496–1506\.External Links:[Link](https://proceedings.mlr.press/v139/charlesworth21a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Chenet al\.\(2026\)K\. Chen, H\. Wei, Z\. Deng, and S\. LinTowards fast safe online reinforcement learning via policy finetuning\.Transactions on Machine Learning Research\.External Links:[Link](https://openreview.net/forum?id=1SO7vmLFUq)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p2.1),[§3\.2](https://arxiv.org/html/2610.33494#S3.SS2.p1.1)\.
- Chenet al\.\(2022a\)T\. Chen, J\. Xu, and P\. AgrawalA system for general in\-hand object re\-orientation\.InProceedings of the 5th Conference on Robot Learning,A\. Faust, D\. Hsu, and G\. Neumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.164,pp\. 297–307\.External Links:[Link](https://proceedings.mlr.press/v164/chen22a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Chenet al\.\(2022b\)Y\. Chen, T\. Wu, S\. Wang, X\. Feng, J\. Jiang, Z\. Lu, S\. McAleer, H\. Dong, S\. Zhu, and Y\. YangTowards human\-level bimanual dexterous manipulation with reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.35,pp\. 5150–5163\.External Links:[Document](https://dx.doi.org/10.52202/068431-0372),[Link](https://proceedings.neurips.cc/paper_files/paper/2022/hash/217a2a387f52c30755c37b0a73430291-Abstract-Datasets_and_Benchmarks.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Chowet al\.\(2018\)Y\. Chow, O\. Nachum, E\. Duenez\-Guzman, and M\. GhavamzadehA Lyapunov\-based approach to safe reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.31\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2018/file/4fe5149039b52765bde64beb9f674940-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Dulac\-Arnoldet al\.\(2021\)G\. Dulac\-Arnold, N\. Levine, D\. J\. Mankowitz, J\. Li, C\. Paduraru, S\. Gowal, and T\. HesterChallenges of real\-world reinforcement learning: definitions, benchmarks and analysis\.Machine Learning110\(9\),pp\. 2419–2468\.External Links:[Document](https://dx.doi.org/10.1007/s10994-021-05961-4),[Link](https://link.springer.com/article/10.1007/s10994-021-05961-4)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Fujimotoet al\.\(2019\)S\. Fujimoto, D\. Meger, and D\. PrecupOff\-policy deep reinforcement learning without exploration\.InProceedings of the 36th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.97,pp\. 2052–2062\.External Links:[Link](https://proceedings.mlr.press/v97/fujimoto19a.html)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- García and Fernández \(2015\)J\. García and F\. FernándezA comprehensive survey on safe reinforcement learning\.Journal of Machine Learning Research16\(42\),pp\. 1437–1480\.External Links:[Link](https://jmlr.org/papers/v16/garcia15a.html)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Genget al\.\(2025\)Y\. Geng, J\. Ji, Y\. Chen, H\. Geng, F\. Zhong, and Y\. YangReDMan: reliable dexterous manipulation with safe reinforcement learning\.Machine Learning114\(8\),pp\. 189\.External Links:[Document](https://dx.doi.org/10.1007/s10994-025-06825-x)Cited by:[§A\.1](https://arxiv.org/html/2610.33494#A1.SS1.p1.1),[§1](https://arxiv.org/html/2610.33494#S1.p1.1),[§1](https://arxiv.org/html/2610.33494#S1.p3.1),[§2](https://arxiv.org/html/2610.33494#S2.p1.1),[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p1.1)\.
- Haarnojaet al\.\(2018\)T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. LevineSoft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.External Links:[Link](https://proceedings.mlr.press/v80/haarnoja18b.html)Cited by:[§A\.2](https://arxiv.org/html/2610.33494#A1.SS2.p1.1)\.
- Huanget al\.\(2023\)B\. Huang, Y\. Chen, T\. Wang, Y\. Qin, Y\. Yang, N\. Atanasov, and X\. WangDynamic handover: throw and catch with bimanual hands\.InProceedings of the 7th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.229,pp\. 1887–1902\.External Links:[Link](https://proceedings.mlr.press/v229/huang23d.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Jainet al\.\(2023\)A\. Jain, J\. Kolb, and H\. RavichandarConstrained reinforcement learning for dexterous manipulation\.External Links:2301\.09766,[Link](https://arxiv.org/abs/2301.09766)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1),[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Jiet al\.\(2023\)J\. Ji, B\. Zhang, J\. Zhou, X\. Pan, W\. Huang, R\. Sun, Y\. Geng, Y\. Zhong, J\. Dai, and Y\. YangSafety\-Gymnasium: a unified safe reinforcement learning benchmark\.InAdvances in Neural Information Processing Systems,External Links:[Link](https://arxiv.org/abs/2310.12567)Cited by:[§B\.3](https://arxiv.org/html/2610.33494#A2.SS3.p1.1)\.
- Jiet al\.\(2024\)J\. Ji, J\. Zhou, B\. Zhang, J\. Dai, X\. Pan, R\. Sun, W\. Huang, Y\. Geng, M\. Liu, and Y\. YangOmniSafe: an infrastructure for accelerating safe reinforcement learning research\.Journal of Machine Learning Research25\(285\),pp\. 1–6\.External Links:[Link](https://jmlr.org/papers/v25/23-0681.html)Cited by:[§A\.2](https://arxiv.org/html/2610.33494#A1.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p3.1)\.
- Kakade and Langford \(2002\)S\. Kakade and J\. LangfordApproximately optimal approximate reinforcement learning\.InProceedings of the 19th International Conference on Machine Learning,pp\. 267–274\.External Links:[Link](https://homes.cs.washington.edu/~sham/papers/rl/aoarl.pdf)Cited by:[§C\.3](https://arxiv.org/html/2610.33494#A3.SS3.p3.1.1),[§C\.5](https://arxiv.org/html/2610.33494#A3.SS5.p3.1.1),[§4\.2](https://arxiv.org/html/2610.33494#S4.SS2.p3.1)\.
- Kostrikovet al\.\(2022\)I\. Kostrikov, A\. Nair, and S\. LevineOffline reinforcement learning with implicit Q\-Learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=68n2s9ZJWF8)Cited by:[§A\.2](https://arxiv.org/html/2610.33494#A1.SS2.p1.1)\.
- Kumaret al\.\(2019\)A\. Kumar, J\. Fu, M\. Soh, G\. Tucker, and S\. LevineStabilizing off\-policy Q\-Learning via bootstrapping error reduction\.InAdvances in Neural Information Processing Systems,Vol\.32\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2019/file/c2073ffa77b5357a498057413bb09d3a-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Kumaret al\.\(2020\)A\. Kumar, A\. Zhou, G\. Tucker, and S\. LevineConservative Q\-Learning for offline reinforcement learning\.InAdvances in Neural Information Processing Systems,Vol\.33,pp\. 1179–1191\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Leeet al\.\(2022a\)J\. Lee, C\. Paduraru, D\. J\. Mankowitz, N\. Heess, D\. Precup, K\. Kim, and A\. GuezCOptiDICE: offline constrained reinforcement learning via stationary distribution correction estimation\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=FLA55mBee6Q)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Leeet al\.\(2022b\)S\. Lee, Y\. Seo, K\. Lee, P\. Abbeel, and J\. ShinOffline\-to\-online reinforcement learning via balanced replay and pessimistic Q\-Ensemble\.InProceedings of the 5th Conference on Robot Learning,A\. Faust, D\. Hsu, and G\. Neumann \(Eds\.\),Proceedings of Machine Learning Research, Vol\.164,pp\. 1702–1712\.External Links:[Link](https://proceedings.mlr.press/v164/lee22d.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- Levineet al\.\(2020\)S\. Levine, A\. Kumar, G\. Tucker, and J\. FuOffline reinforcement learning: tutorial, review, and perspectives on open problems\.External Links:2005\.01643,[Link](https://arxiv.org/abs/2005.01643)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Liet al\.\(2024\)J\. Li, X\. Liu, B\. Zhu, J\. Jiao, M\. Tomizuka, C\. Tang, and W\. ZhanGuided online distillation: promoting safe reinforcement learning by offline demonstration\.In2024 IEEE International Conference on Robotics and Automation \(ICRA\),pp\. 7447–7454\.External Links:[Document](https://dx.doi.org/10.1109/ICRA57147.2024.10611123)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p2.1),[§2](https://arxiv.org/html/2610.33494#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.33494#S3.SS2.p1.1)\.
- Lillicrapet al\.\(2016\)T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. WierstraContinuous control with deep reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://arxiv.org/abs/1509.02971)Cited by:[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p3.1)\.
- Liuet al\.\(2024\)Z\. Liu, Z\. Guo, H\. Lin, Y\. Yao, J\. Zhu, Z\. Cen, H\. Hu, W\. Yu, T\. Zhang, J\. Tan, and D\. ZhaoDatasets and benchmarks for offline safe reinforcement learning\.Journal of Data\-centric Machine Learning Research\.External Links:[Link](https://data.mlr.press/assets/pdf/v01-12.pdf)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Liuet al\.\(2023\)Z\. Liu, Z\. Guo, Y\. Yao, Z\. Cen, W\. Yu, T\. Zhang, and D\. ZhaoConstrained decision transformer for offline safe reinforcement learning\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 21611–21630\.External Links:[Link](https://proceedings.mlr.press/v202/liu23m.html)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Luoet al\.\(2024\)Q\. Luo, M\. Xie, Y\. Wang, and S\. HuangOptimistic critic reconstruction and constrained fine\-tuning for general offline\-to\-online RL\.InAdvances in Neural Information Processing Systems,Vol\.37,pp\. 108167–108207\.External Links:[Document](https://dx.doi.org/10.52202/079017-3435),[Link](https://proceedings.neurips.cc/paper_files/paper/2024/hash/c3b3c5f9297ad0012cfe7188c34cea0e-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- Nagabandiet al\.\(2020\)A\. Nagabandi, K\. Konolige, S\. Levine, and V\. KumarDeep dynamics models for learning dexterous manipulation\.InProceedings of the Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.100,pp\. 1101–1112\.External Links:[Link](https://proceedings.mlr.press/v100/nagabandi20a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Nairet al\.\(2020\)A\. Nair, A\. Gupta, M\. Dalal, and S\. LevineAWAC: accelerating online reinforcement learning with offline datasets\.External Links:2006\.09359,[Link](https://arxiv.org/abs/2006.09359)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- Nakamotoet al\.\(2023\)M\. Nakamoto, S\. Zhai, A\. Singh, M\. Sobol Mark, Y\. Ma, C\. Finn, A\. Kumar, and S\. LevineCal\-QL: calibrated offline RL pre\-training for efficient online fine\-tuning\.InAdvances in Neural Information Processing Systems,Vol\.36,pp\. 62244–62269\.External Links:[Link](https://proceedings.neurips.cc/paper_files/paper/2023/hash/c44a04289beaf0a7d968a94066a1d696-Abstract-Conference.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- OpenAIet al\.\(2020\)OpenAI, M\. Andrychowicz, B\. Baker, M\. Chociej, R\. Józefowicz, B\. McGrew, J\. Pachocki, A\. Petron, M\. Plappert, G\. Powell, A\. Ray, J\. Schneider, S\. Sidor, J\. Tobin, P\. Welinder, L\. Weng, and W\. ZarembaLearning dexterous in\-hand manipulation\.The International Journal of Robotics Research39\(1\),pp\. 3–20\.External Links:[Document](https://dx.doi.org/10.1177/0278364919887447),[Link](https://journals.sagepub.com/doi/10.1177/0278364919887447)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1),[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Qiet al\.\(2023a\)H\. Qi, A\. Kumar, R\. Calandra, Y\. Ma, and J\. MalikIn\-hand object rotation via rapid motor adaptation\.InProceedings of the 6th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.205,pp\. 1722–1732\.External Links:[Link](https://proceedings.mlr.press/v205/qi23a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Qiet al\.\(2023b\)H\. Qi, B\. Yi, S\. Suresh, M\. Lambeta, Y\. Ma, R\. Calandra, and J\. MalikGeneral in\-hand object rotation with vision and touch\.InProceedings of the 7th Conference on Robot Learning,Proceedings of Machine Learning Research, Vol\.229,pp\. 2549–2564\.External Links:[Link](https://proceedings.mlr.press/v229/qi23a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Qinet al\.\(2023\)Y\. Qin, B\. Huang, Z\. Yin, H\. Su, and X\. WangDexPoint: generalizable point cloud reinforcement learning for sim\-to\-real dexterous manipulation\.InProceedings of the 6th Conference on Robot Learning,K\. Liu, D\. Kulic, and J\. Ichnowski \(Eds\.\),Proceedings of Machine Learning Research, Vol\.205,pp\. 594–605\.External Links:[Link](https://proceedings.mlr.press/v205/qin23a.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Rajeswaranet al\.\(2018\)A\. Rajeswaran, V\. Kumar, A\. Gupta, G\. Vezzani, J\. Schulman, E\. Todorov, and S\. LevineLearning complex dexterous manipulation with deep reinforcement learning and demonstrations\.InProceedings of Robotics: Science and Systems,External Links:[Document](https://dx.doi.org/10.15607/RSS.2018.XIV.049)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1),[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Rayet al\.\(2019\)A\. Ray, J\. Achiam, and D\. AmodeiBenchmarking safe exploration in deep reinforcement learning\.Technical reportOpenAI\.External Links:[Link](https://cdn.openai.com/safexp-short.pdf)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Satijaet al\.\(2020\)H\. Satija, P\. Amortila, and J\. PineauConstrained Markov decision processes via backward value functions\.InProceedings of the 37th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.119,pp\. 8502–8511\.External Links:[Link](https://proceedings.mlr.press/v119/satija20a.html)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Stookeet al\.\(2020\)A\. Stooke, J\. Achiam, and P\. AbbeelResponsive safety in reinforcement learning by PID lagrangian methods\.InProceedings of the 37th International Conference on Machine Learning,H\. Daumé III and A\. Singh \(Eds\.\),Proceedings of Machine Learning Research, Vol\.119,pp\. 9133–9143\.External Links:[Link](https://proceedings.mlr.press/v119/stooke20a.html)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Tessleret al\.\(2019\)C\. Tessler, D\. J\. Mankowitz, and S\. MannorReward constrained policy optimization\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=SkfrvsA9FX)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Thananjeyanet al\.\(2021\)B\. Thananjeyan, A\. Balakrishna, S\. Nair, M\. Luo, K\. Srinivasan, M\. Hwang, J\. E\. Gonzalez, J\. Ibarz, C\. Finn, and K\. GoldbergRecovery RL: safe reinforcement learning with learned recovery zones\.IEEE Robotics and Automation Letters6\(3\),pp\. 4915–4922\.External Links:[Document](https://dx.doi.org/10.1109/LRA.2021.3070252),[Link](https://ieeexplore.ieee.org/document/9392290/)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Uchenduet al\.\(2023\)I\. Uchendu, T\. Xiao, Y\. Lu, B\. Zhu, M\. Yan, J\. Simon, M\. Bennice, C\. Fu, C\. Ma, J\. Jiao, S\. Levine, and K\. HausmanJump\-start reinforcement learning\.InProceedings of the 40th International Conference on Machine Learning,A\. Krause, E\. Brunskill, K\. Cho, B\. Engelhardt, S\. Sabato, and J\. Scarlett \(Eds\.\),Proceedings of Machine Learning Research, Vol\.202,pp\. 34556–34583\.External Links:[Link](https://proceedings.mlr.press/v202/uchendu23a.html)Cited by:[§C\.4](https://arxiv.org/html/2610.33494#A3.SS4.p1.1),[§2](https://arxiv.org/html/2610.33494#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.33494#S3.SS2.p1.1),[§4\.1](https://arxiv.org/html/2610.33494#S4.SS1.p1.1),[§4\.2](https://arxiv.org/html/2610.33494#S4.SS2.p5.1),[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p2.1),[§5\.3](https://arxiv.org/html/2610.33494#S5.SS3.p1.1)\.
- Wenet al\.\(2024\)B\. Wen, W\. Yang, J\. Kautz, and S\. BirchfieldFoundationPose: unified 6D pose estimation and tracking of novel objects\.InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pp\. 17868–17879\.External Links:[Link](https://openaccess.thecvf.com/content/CVPR2024/html/Wen_FoundationPose_Unified_6D_Pose_Estimation_and_Tracking_of_Novel_Objects_CVPR_2024_paper.html)Cited by:[§B\.4](https://arxiv.org/html/2610.33494#A2.SS4.p1.1)\.
- Xuet al\.\(2022\)H\. Xu, X\. Zhan, and X\. ZhuConstraints penalized Q\-learning for safe offline reinforcement learning\.Proceedings of the AAAI Conference on Artificial Intelligence36\(8\),pp\. 8753–8760\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v36i8.20855),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/20855)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Yu and Zhang \(2023\)Z\. Yu and X\. ZhangActor\-critic alignment for offline\-to\-online reinforcement learning\.InProceedings of the 40th International Conference on Machine Learning,Proceedings of Machine Learning Research, Vol\.202,pp\. 40452–40474\.External Links:[Link](https://proceedings.mlr.press/v202/yu23k.html)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1)\.
- Yuanet al\.\(2025\)H\. Yuan, B\. Zhou, Y\. Fu, and Z\. LuCross\-embodiment dexterous grasping with reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=twIPSx9qHn)Cited by:[§1](https://arxiv.org/html/2610.33494#S1.p1.1)\.
- Zhan and Chin \(2024\)W\. Zhan and P\. ChinSafe multi\-agent reinforcement learning for bimanual dexterous manipulation\.In2024 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 12420–12427\.External Links:[Document](https://dx.doi.org/10.1109/IROS58592.2024.10801490),[Link](https://ieeexplore.ieee.org/document/10801490)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.
- Zhanget al\.\(2023\)H\. Zhang, W\. Xu, and H\. YuPolicy expansion for bridging offline\-to\-online reinforcement learning\.InInternational Conference on Learning Representations,External Links:[Link](https://openreview.net/forum?id=-Y34L45JR6z)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1),[§3\.2](https://arxiv.org/html/2610.33494#S3.SS2.p1.1),[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p2.1)\.
- Zhanget al\.\(2024\)Y\. Zhang, J\. Liu, C\. Li, Y\. Niu, Y\. Yang, Y\. Liu, and W\. OuyangA perspective of Q\-Value estimation on offline\-to\-online reinforcement learning\.Proceedings of the AAAI Conference on Artificial Intelligence38\(15\),pp\. 16908–16916\.External Links:[Document](https://dx.doi.org/10.1609/aaai.v38i15.29633),[Link](https://ojs.aaai.org/index.php/AAAI/article/view/29633)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p2.1),[§5\.1](https://arxiv.org/html/2610.33494#S5.SS1.p2.1)\.
- Zhuet al\.\(2019\)H\. Zhu, A\. Gupta, A\. Rajeswaran, S\. Levine, and V\. KumarDexterous manipulation with deep reinforcement learning: efficient, general, and low\-cost\.In2019 International Conference on Robotics and Automation \(ICRA\),pp\. 3651–3657\.External Links:[Document](https://dx.doi.org/10.1109/ICRA.2019.8794102)Cited by:[§2](https://arxiv.org/html/2610.33494#S2.p1.1)\.

## Appendix AAdditional Experimental Details

### A\.1Task Descriptions

To evaluate the performance of RBGA in various safe dexterous manipulation tasks with different manipulation settings and joint\-range constraints, we select four tasks including*Finger*,*Joint*,*ArmFinger*, and*ArmJoint*from ReDMan\([Geng et al\., 2025](https://arxiv.org/html/2610.33494#bib.bib15)\)as the experimental tasks in this work\.

Finger\.Two fixed\-base dexterous hands cooperate to transfer a ball to a target position\. The safety constraint restricts the ranges of motion of three index\-finger joints\.

Joint\.The manipulation objective and hand configuration are the same as in Finger, but the safety constraint applies to a single index\-finger joint\.

ArmFinger\.The object is thrown from a vertically oriented hand and caught by a second hand with its palm facing upward\. Multiple index\-finger joints are subject to range\-of\-motion constraints\.

ArmJoint\.This task uses the same throwing\-and\-catching setting as ArmFinger, with the safety constraint imposed on a single index\-finger joint\.

### A\.2Offline Guide and Online Learning

Unless otherwise stated, RBGA uses implicit Q\-learning \(IQL\)\([Kostrikov et al\., 2022](https://arxiv.org/html/2610.33494#bib.bib34)\)to train the offline guide and SAC\-Lagrangian \(SAC\-Lag\)\([Haarnoja et al\., 2018](https://arxiv.org/html/2610.33494#bib.bib4);[Ji et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib35)\)for online learning\. We train the offline policyπg\\pi\_\{g\}on a fixed dataset𝒟off\\mathcal\{D\}\_\{\\mathrm\{off\}\}containing 1,000,000 transitions per task and keep it fixed during online learning\. The critics and the auxiliary cost estimator associated with the online policyπo\\pi\_\{o\}are initialized independently of the guide policy\.

The composite policyπ\\picollects transitions into the online replay bufferℬ\\mathcal\{B\}\. The guide policy remains fixed throughout online learning, whereas the online policy is continually updated using transitions collected in the online replay buffer\. Each collection comprises eight steps in each of 512 parallel environments\. Learning begins after the buffer contains 500,000 transitions; each subsequent collection is followed by ten training iterations, up to the total ofNN\. Each iteration updates the SAC\-Lag learner and then the auxiliary estimator using a fresh minibatch\. Table[1](https://arxiv.org/html/2610.33494#A1.T1)summarizes the architecture and training settings\.

Table 1:RBGA hyperparameters used in the main experiments\. The online SAC\-Lag cost critic and the auxiliary cost estimator are separate networks\.
### A\.3Online SAC\-Lag Updates

Critic updates\.For a transition sampled fromℬ\\mathcal\{B\}, the reward and cost TD targets are

yR\\displaystyle y\_\{R\}=r\+γ⁡\(1−δ\)​minj∈\{1,2\}​Qϕ¯R,j​\(s′,a′\),\\displaystyle=r\+\\gamma\(1\-\\delta\)\\min\_\{j\\in\\\{1,2\\\}\}Q\_\{\\bar\{\\phi\}\_\{R,j\}\}\(s^\{\\prime\},a^\{\\prime\}\),\(15\)yC\\displaystyle y\_\{C\}=c\+γ⁡\(1−δ\)​maxj∈\{1,2\}​Qϕ¯C,j​\(s′,a′\)\.\\displaystyle=c\+\\gamma\(1\-\\delta\)\\max\_\{j\\in\\\{1,2\\\}\}Q\_\{\\bar\{\\phi\}\_\{C,j\}\}\(s^\{\\prime\},a^\{\\prime\}\)\.where\(s,a,r,c,s′,δ\)\(s,a,r,c,s^\{\\prime\},\\delta\)is a sampled transition,a′∼πo\(⋅∣s′\)a^\{\\prime\}\\sim\\pi\_\{o\}\(\\cdot\\mid s^\{\\prime\}\), andδ\\deltaindicates episode termination\. The functionsQϕR,jQ\_\{\\phi\_\{R,j\}\}andQϕC,jQ\_\{\\phi\_\{C,j\}\},j∈\{1,2\}j\\in\\\{1,2\\\}, denote the reward and cost critics, respectively, and an overbar denotes the corresponding target\-network parameters\.

Policy and temperature updates\.The online policy minimizes

ℒπo=11\+λ​𝔼s∼ℬa∼πo\(⋅∣s\)​\[α​log⁡πo​\(a∣s\)−minj∈\{1,2\}⁡QϕR,j​\(s,a\)\+λ​maxj∈\{1,2\}​QϕC,j​\(s,a\)\]\.\\mathcal\{L\}\_\{\\pi\_\{o\}\}=\\frac\{1\}\{1\+\\lambda\}\\mathbb\{E\}\_\{\\begin\{subarray\}\{c\}s\\sim\\mathcal\{B\}\\\\ a\\sim\\pi\_\{o\}\(\\cdot\\mid s\)\\end\{subarray\}\}\\left\[\\alpha\\log\\pi\_\{o\}\(a\\mid s\)\-\\min\_\{j\\in\\\{1,2\\\}\}Q\_\{\\phi\_\{R,j\}\}\(s,a\)\+\\lambda\\max\_\{j\\in\\\{1,2\\\}\}Q\_\{\\phi\_\{C,j\}\}\(s,a\)\\right\]\.\(16\)wheressis sampled from replay,aais a newly sampled online action, andλ≥0\\lambda\\geq 0is the Lagrange multiplier\. Critic parameters,α\\alpha, andλ\\lambdaare held fixed during this update\. We adjustα\\alphaby optimizinglog⁡α\\log\\alphawith a target entropy equal to the negative action dimension\.

Multiplier and target\-network updates\.Once learning has started, each collection with completed episodes is followed by one Adam step minimizing

ℒλ=−λ⁡\(C¯ep−d\),\\mathcal\{L\}\_\{\\lambda\}=\-\\lambda\\bigl\(\\bar\{C\}\_\{\\mathrm\{ep\}\}\-d\\bigr\),\(17\)whereC¯ep\\bar\{C\}\_\{\\mathrm\{ep\}\}is the mean cumulative cost of the episodes completed in that collection\. We clipλ\\lambdato\[0,5\]\[0,5\]after this update\. Each training iteration updates the reward critics, cost critics, online policy, and entropy temperature, then soft\-updates the target parameters with weight0\.0050\.005on the current parameters and0\.9950\.995on their previous target values\.

### A\.4Auxiliary Cost\-to\-Go Estimator

Candidate evaluation\.We train two auxiliary cost estimators to estimate the cost\-to\-go in Eq\. \([5](https://arxiv.org/html/2610.33494#S4.E5)\)\. Both take the state and action as input\. Arbitration uses the non\-negative mean of their target predictions:

Q^Cπo​\(s,a\)=\[12​∑j=12Qψ¯j​\(s,a\)\]\+,\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s,a\)=\\left\[\\frac\{1\}\{2\}\\sum\_\{j=1\}^\{2\}Q\_\{\\bar\{\\psi\}\_\{j\}\}\(s,a\)\\right\]\_\{\+\},\(18\)whereQψ¯jQ\_\{\\bar\{\\psi\}\_\{j\}\}denotes the target network of thejj\-th auxiliary cost critic, and\[⋅\]\+\[\\cdot\]\_\{\+\}clips negative predictions to zero\.

Estimator update\.The auxiliary cost estimators are trained using the TD target

yaux=c\+γ\(1−δ\)Q^Cπo\(s′,a′\),a′∼πo\(⋅∣s′\),y^\{\\mathrm\{aux\}\}=c\+\\gamma\(1\-\\delta\)\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s^\{\\prime\},a^\{\\prime\}\),\\qquad a^\{\\prime\}\\sim\\pi\_\{o\}\(\\cdot\\mid s^\{\\prime\}\),\(19\)where\(s,a,c,s′,δ\)\(s,a,c,s^\{\\prime\},\\delta\)is a transition sampled fromℬ\\mathcal\{B\}, andγ\\gammais the discount factor used in Eq\. \([5](https://arxiv.org/html/2610.33494#S4.E5)\)\. The target estimates the cost of executing the stored action and following the current online policy thereafter\. The auxiliary estimators are optimized using the Huber loss, with the remaining training details provided in Table[1](https://arxiv.org/html/2610.33494#A1.T1)\.

Warm\-up\.The auxiliary estimator is trained during the first 2,000 online training iterations, while actions are chosen using the guide horizon without budget arbitration\. Once both the online learner and auxiliary estimator have completed 2,000 updates, action selection follows Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\)\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/ablation_b_all_tasks_reward_cost_success.png)Figure 3:The figure depicts the reward, cost, and success rate curves of RBGA and RBGA without residual\-budget arbitration on four tasks\. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds\. The dashed line indicates the episode cost limitd=30d=30\.
### A\.5Guide Horizon Schedules

The linear schedule is defined in Eq\. \([3](https://arxiv.org/html/2610.33494#S4.E3)\) in Section[4](https://arxiv.org/html/2610.33494#S4)\. For the step schedule, we predefine decreasing horizon values fromH0H\_\{0\}to zero and assign them to approximately evenly spaced training stages over the firstNgN\_\{g\}iterations\. The value ofHiH\_\{i\}is constant within each stage\. By default, the horizon values are obtained by evenly spacing values betweenH0H\_\{0\}and zero\. Both schedules control guide participation without changing the task horizonHH\.

## Appendix BAdditional Experimental Results

### B\.1Effect of Residual\-Budget Arbitration

To evaluate the contribution of residual\-budget arbitration, we disable the residual\-budget mechanism while keeping the remaining settings unchanged\. As shown in Figure[3](https://arxiv.org/html/2610.33494#A1.F3), RBGA satisfies the cost constraint earlier without degrading reward\. These results indicate that residual\-budget arbitration achieves a more favorable safety–performance trade\-off during the early stage of online training\. This suggests that incorporating the remaining episode budget helps RBGA select actions with lower predicted cost\-to\-go while preserving task performance\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/ddpg_results_iclr.png)Figure 4:The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms using DDPG\-Lag as the online learner\. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds\. The dashed line indicates the episode cost limitd=30d=30\.
### B\.2Results with a DDPG\-Lag Online Learner

To evaluate the performance of RBGA with a different online RL algorithm, we conduct comparative experiments using DDPG\-Lag as the online learner\. Specifically, we replace SAC\-Lag with DDPG\-Lag for RBGA and all corresponding baselines while retaining their original offline\-to\-online mechanisms\. The remaining data collection, guide scheduling, and arbitration settings are kept unchanged\.

Figure[4](https://arxiv.org/html/2610.33494#A2.F4)illustrates the reward, cost, and success rate curves of RBGA and the baseline algorithms on the four tasks\. From the illustrated results, it can be observed that RBGA maintains the episode cost close to the predefined cost limit while achieving competitive reward and success rate across the four tasks\. Furthermore, RBGA demonstrates stable policy improvement during online training, whereas several baseline algorithms exhibit higher costs or lower task performance\. Although the reward of RBGA temporarily decreases during the early stage of training on*ArmJoint*, it subsequently recovers while maintaining a relatively low episode cost and a high success rate\. The above results indicate that RBGA can be effectively combined with different constrained online RL algorithms\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/appendix_main_results_iclr.png)Figure 5:The figure depicts the reward, cost, and success rate curves of RBGA and the baseline algorithms on the*Close Drawer*and*Pick and Place*tasks\. The shaded areas on the curves represent the variance obtained from online testing conducted with three random seeds\. The dashed line indicates the episode cost limitd=30d=30\.
### B\.3Results on Additional Manipulation Tasks

We further evaluate RBGA on two additional manipulation tasks including*Close Drawer*and*Pick and Place*, two Safety\-Gymnasium manipulation tasks from Safe Policy Optimization\([Ji et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib36)\)\. Both use a mobile base equipped with a Franka arm and a gripper\.*Close Drawer*requires the robot to close a drawer while keeping the base outside a restricted region near the cabinet, whereas*Pick and Place*requires the robot to move an object from a lower platform to a higher location while avoiding the restricted region between the platforms\. RBGA and all six baselines use SAC\-Lag for online learning\.

Figure[5](https://arxiv.org/html/2610.33494#A2.F5)illustrates the reward, cost, and success rate curves of RBGA and the baseline algorithms on the two tasks\. From the illustrated results, it can be observed that RBGA consistently improves task performance during online training while maintaining the episode cost within or close to the predefined cost limit\. On*Close Drawer*, RBGA achieves high reward and success rate while reducing the episode cost toward the cost limit, whereas several baseline algorithms exhibit higher costs or lower success rates during training\. On*Pick and Place*, RBGA continues to improve beyond the offline policy and achieves higher reward and success rate than the compared methods, while several baselines show limited improvement or performance degradation\. The above results indicate that RBGA can effectively leverage offline guidance for safe online policy improvement across different manipulation tasks\.

### B\.4Real\-world Deployment

To further evaluate the practical applicability of RBGA, we deploy the policy trained in simulation environment directly on a real\-world robotic platform consisting of a RealMan RM65 robotic arm and a Linker Hand L20 dexterous hand\. We consider aCube Graspingtask, where the same arm–hand platform is modeled in simulation for policy training\. For real\-world execution, an external Intel RealSense D435 RGB\-D camera captures the scene, and FoundationPose\([Wen et al\., 2024](https://arxiv.org/html/2610.33494#bib.bib50)\)estimates the 6D pose of the target object\. The estimated object pose is transformed into the robot reference frame and combined with the robot proprioception to construct the same policy observation used during IsaacGym training\. The resulting observation is directly fed into the learned policy to generate control commands for the robotic arm and dexterous hand\.

As shown in Figure[6](https://arxiv.org/html/2610.33494#A2.F6), we present the corresponding grasping behaviors in simulation and on the physical robot\. The policy learned with RBGA can be transferred to the physical platform and successfully execute the cube grasping behavior while maintaining stable and safety\-aware manipulation\. This result demonstrates the potential of RBGA for transferring safe dexterous manipulation policies from simulation to real robotic systems\.

![Refer to caption](https://arxiv.org/html/2610.33494v1/sim2real_comparison.png)Figure 6:Simulation and real\-world deployment of RBGA on theCube Graspingtask\. Top: simulation\. Bottom: the physical robot\. The platform consists of a RealMan RM65 robotic arm equipped with a Linker Hand L20 dexterous hand\.

## Appendix CProofs and Discussions

We provide the proofs of the cost and reward bounds in Section[4\.2](https://arxiv.org/html/2610.33494#S4.SS2)\. We first derive the Bellman recursion used in Section[4\.1](https://arxiv.org/html/2610.33494#S4.SS1), and then prove the local and episode cost bounds\. Subsequently, we discuss the state coverage and policy advantage conditions and prove the reward performance bound\.

### C\.1Proof of the Cost Bellman Recursion

For a fixed online policyπo\\pi\_\{o\}, the cost state values att=0,…,H−1t=0,\\ldots,H\-1and the terminal value are given as:

VCπo\(st\)=𝔼ato∼πo\(⋅∣st\)\[QCπo\(st,ato\)\],VCπo\(sH\)=0,V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\}\)=\\mathbb\{E\}\_\{a\_\{t\}^\{o\}\\sim\\pi\_\{o\}\(\\cdot\\mid s\_\{t\}\)\}\[Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\],\\qquad V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{H\}\)=0,\(20\)whereVCπo​\(st\)V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\}\)denotes the expected discounted cost\-to\-go from statests\_\{t\}at timettunderπo\\pi\_\{o\}\. The cost\-to\-go in Eq\. \([5](https://arxiv.org/html/2610.33494#S4.E5)\) satisfies the following Bellman recursion:

QCπo\(st,at\)=c\(st,at\)\+γ𝔼st\+1∼P\(⋅∣st,at\)\[VCπo\(st\+1\)\]\.Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)=c\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\[V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\+1\}\)\]\.\(21\)
###### Proof\.

By separating the immediate cost from the remaining discounted cost in Eq\. \([5](https://arxiv.org/html/2610.33494#S4.E5)\), we obtain:

QCπo​\(st,at\)\\displaystyle Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)=c\(st,at\)\+γ𝔼πo\[∑j=t\+1H−1γj−\(t\+1\)cj\|st,at\]\\displaystyle=c\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{\\pi\_\{o\}\}\\left\[\\sum\_\{j=t\+1\}^\{H\-1\}\\gamma^\{j\-\(t\+1\)\}c\_\{j\}\\,\\middle\|\\,s\_\{t\},a\_\{t\}\\right\]\(22\)=c\(st,at\)\+γ𝔼st\+1∼P\(⋅∣st,at\)\[VCπo\(st\+1\)\],\\displaystyle=c\(s\_\{t\},a\_\{t\}\)\+\\gamma\\mathbb\{E\}\_\{s\_\{t\+1\}\\sim P\(\\cdot\\mid s\_\{t\},a\_\{t\}\)\}\\left\[V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\+1\}\)\\right\],The first equality usesct=c⁡\(st,at\)c\_\{t\}=c\(s\_\{t\},a\_\{t\}\), and the second equality follows by conditioning on the next statest\+1s\_\{t\+1\}and using Eq\. \([20](https://arxiv.org/html/2610.33494#A3.E20)\)\. Att=H−1t=H\-1, the remaining sum is empty andVCπo​\(sH\)=0V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{H\}\)=0, so the same equality holds\. Therefore, the cost\-to\-go satisfies Eq\. \([21](https://arxiv.org/html/2610.33494#A3.E21)\)\. ∎

### C\.2Proof and Discussion of Proposition[4\.2](https://arxiv.org/html/2610.33494#S4.Thmtheorem2)

Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1)\.For a fixed online policyπo\\pi\_\{o\}and cost estimatorQ^Cπo\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}, there exists a finite constantεC≥0\\varepsilon\_\{C\}\\geq 0such that, at each time stept<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}, the cost estimates of both candidate actions in Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) satisfy:

\|Q^Cπo​\(st,a\)−QCπo​\(st,a\)\|≤εC,∀a∈\{atg,ato\},\\left\|\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\)\\right\|\\leq\\varepsilon\_\{C\},\\qquad\\forall a\\in\\\{a\_\{t\}^\{g\},a\_\{t\}^\{o\}\\\},\(23\)whereaaranges over the guide and online candidates, andεC\\varepsilon\_\{C\}is a uniform bound on their estimation errors\.

Proposition[4\.2](https://arxiv.org/html/2610.33494#S4.Thmtheorem2)\.Under Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), consider a time stept<min⁡\{h,Hi\}t<\\min\\\{h,H\_\{i\}\\\}\. For a selected action satisfyingγt​Q^Cπo​\(st,at\)≤bt\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq b\_\{t\}, the accumulated discounted cost and expected continuation cost satisfy:

∑j=0t−1γj​cj\+γt​QCπo​\(st,at\)≤d\+γt​εC≤d\+εC\.\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\}\+\\gamma^\{t\}Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq d\+\\gamma^\{t\}\\varepsilon\_\{C\}\\leq d\+\\varepsilon\_\{C\}\.\(24\)For every guide action selected by Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\), the difference in cost\-to\-go satisfies:

QCπo​\(st,atg\)−QCπo​\(st,ato\)≤ζ\+2​εC\.Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\leq\\zeta\+2\\varepsilon\_\{C\}\.\(25\)
###### Proof\.

By applying the estimation error bound to the selected action and substituting the budget condition, we obtain:

∑j=0t−1γj​cj\+γt​QCπo​\(st,at\)\\displaystyle\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\}\+\\gamma^\{t\}Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)≤∑j=0t−1γj​cj\+γt​Q^Cπo​\(st,at\)\+γt​εC\\displaystyle\\leq\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\}\+\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\+\\gamma^\{t\}\\varepsilon\_\{C\}\(26\)≤∑j=0t−1γj​cj\+bt\+γt​εC\\displaystyle\\leq\\sum\_\{j=0\}^\{t\-1\}\\gamma^\{j\}c\_\{j\}\+b\_\{t\}\+\\gamma^\{t\}\\varepsilon\_\{C\}=d\+γt​εC≤d\+εC,\\displaystyle=d\+\\gamma^\{t\}\\varepsilon\_\{C\}\\leq d\+\\varepsilon\_\{C\},where the first inequality follows fromQCπo​\(st,at\)≤Q^Cπo​\(st,at\)\+εCQ\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\+\\varepsilon\_\{C\}in Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), the second usesγt​Q^Cπo​\(st,at\)≤bt\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\\leq b\_\{t\}, the equality follows from Eq\. \([4](https://arxiv.org/html/2610.33494#S4.E4)\), and the final inequality usesγt≤1\\gamma^\{t\}\\leq 1\. Therefore, Eq\. \([8](https://arxiv.org/html/2610.33494#S4.E8)\) holds\.

To derive the guide comparison bound, we consider the branches of Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) that select a guide action\. If the selected guide action passes the budget condition, the second branch of the selection rule gives:

q^tg−q^to≤min⁡\{γt​ζ,bt−q^to\}\.\\widehat\{q\}\_\{t\}^\{g\}\-\\widehat\{q\}\_\{t\}^\{o\}\\leq\\min\\\{\\gamma^\{t\}\\zeta,b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\\\}\.\(27\)This inequality covers the cases in which both candidates pass the budget condition and in which only the guide candidate passes\. If the selected guide action fails the budget condition, the online candidate also fails, and the minimum\-cost fallback givesq^tg−q^to≤0\\widehat\{q\}\_\{t\}^\{g\}\-\\widehat\{q\}\_\{t\}^\{o\}\\leq 0\. Sinceγt​ζ≥0\\gamma^\{t\}\\zeta\\geq 0, the estimate difference in both cases satisfies:

q^tg−q^to≤min⁡\{γt​ζ,\[bt−q^to\]\+\}\.\\widehat\{q\}\_\{t\}^\{g\}\-\\widehat\{q\}\_\{t\}^\{o\}\\leq\\min\\\{\\gamma^\{t\}\\zeta,\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}\\\}\.\(28\)where\[bt−q^to\]\+\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}is the non\-negative budget remaining after accounting for the online candidate’s estimated discounted cost\.

By Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), the guide candidate satisfiesQCπo​\(st,atg\)≤Q^Cπo​\(st,atg\)\+εCQ\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\\leq\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\+\\varepsilon\_\{C\}, while the online candidate satisfiesQCπo​\(st,ato\)≥Q^Cπo​\(st,ato\)−εCQ\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\geq\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\-\\varepsilon\_\{C\}\. Applying these two bounds to the cost\-to\-go difference gives:

γt​\[QCπo​\(st,atg\)−QCπo​\(st,ato\)\]\\displaystyle\\gamma^\{t\}\\left\[Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\right\]\(29\)≤γt​\[Q^Cπo​\(st,atg\)\+εC−\(Q^Cπo​\(st,ato\)−εC\)\]\\displaystyle\\leq\\gamma^\{t\}\\left\[\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\+\\varepsilon\_\{C\}\-\\left\(\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\-\\varepsilon\_\{C\}\\right\)\\right\]=γt​\[Q^Cπo​\(st,atg\)−Q^Cπo​\(st,ato\)\]\+2​γt​εC\\displaystyle=\\gamma^\{t\}\\left\[\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{g\}\)\-\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\right\]\+2\\gamma^\{t\}\\varepsilon\_\{C\}=q^tg−q^to\+2​γt​εC\\displaystyle=\\widehat\{q\}\_\{t\}^\{g\}\-\\widehat\{q\}\_\{t\}^\{o\}\+2\\gamma^\{t\}\\varepsilon\_\{C\}≤2​γt​εC\+min⁡\{γt​ζ,\[bt−q^to\]\+\}\\displaystyle\\leq 2\\gamma^\{t\}\\varepsilon\_\{C\}\+\\min\\\{\\gamma^\{t\}\\zeta,\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}\\\}≤γt​\(ζ\+2​εC\)\.\\displaystyle\\leq\\gamma^\{t\}\(\\zeta\+2\\varepsilon\_\{C\}\)\.The second equality uses the definitions ofq^tg\\widehat\{q\}\_\{t\}^\{g\}andq^to\\widehat\{q\}\_\{t\}^\{o\}\. The second inequality follows from the preceding bound on their difference, and the final inequality usesmin⁡\{γt​ζ,\[bt−q^to\]\+\}≤γt​ζ\\min\\\{\\gamma^\{t\}\\zeta,\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}\\\}\\leq\\gamma^\{t\}\\zeta\. Dividing both sides byγt\>0\\gamma^\{t\}\>0yields Eq\. \([9](https://arxiv.org/html/2610.33494#S4.E9)\)\.

∎

### C\.3Proof and Discussion of Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3)

Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3)\.Under Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1), fixπo\\pi\_\{o\},Q^Cπo\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}\}, andHiH\_\{i\}throughout an episode, and sampleh∼Unif⁡\{0,…,Hi\}h\\sim\\operatorname\{Unif\}\\\{0,\\ldots,H\_\{i\}\\\}independently of the initial state\. The composite policyπ\\pisatisfies:

\[JC​\(π\)−d\]\+≤\[JC​\(πo\)−d\]\+\+\(ζ\+2​εC\)​∑t=0Hi−1γt​Hi−tHi\+1\.\[J\_\{C\}\(\\pi\)\-d\]\_\{\+\}\\leq\[J\_\{C\}\(\\pi\_\{o\}\)\-d\]\_\{\+\}\+\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\}\-t\}\{H\_\{i\}\+1\}\.\(30\)
###### Proof\.

The composite policyπ\\pimay depend on the interaction history throughbtb\_\{t\}and on the sampled horizonhh, while the online candidate is sampled fromπo\(⋅∣st\)\\pi\_\{o\}\(\\cdot\\mid s\_\{t\}\)\.

To compare the expected discounted costs ofπ\\piandπo\\pi\_\{o\}, we use the finite\-horizon cost form of the performance difference lemma\([Kakade and Langford, 2002](https://arxiv.org/html/2610.33494#bib.bib33)\)\. With both policies starting froms0∼ρ0s\_\{0\}\\sim\\rho\_\{0\}, we obtain:

JC​\(π\)−JC​\(πo\)=∑t=0H−1γt​𝔼π​\[QCπo​\(st,at\)−VCπo​\(st\)\],J\_\{C\}\(\\pi\)\-J\_\{C\}\(\\pi\_\{o\}\)=\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\_\{\\pi\}\\left\[Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\-V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\}\)\\right\],\(31\)where the expectation is over trajectories generated byπ\\pi, including the sampled guide horizon and candidate actions\. Taking expectations in Eq\. \([21](https://arxiv.org/html/2610.33494#A3.E21)\) and summing the discounted differences cancels the intermediate value terms\. UsingVCπo​\(sH\)=0V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{H\}\)=0and𝔼s0∼ρ0​\[VCπo​\(s0\)\]=JC​\(πo\)\\mathbb\{E\}\_\{s\_\{0\}\\sim\\rho\_\{0\}\}\[V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{0\}\)\]=J\_\{C\}\(\\pi\_\{o\}\)then gives Eq\. \([31](https://arxiv.org/html/2610.33494#A3.E31)\)\.

For RBGA, the online candidate is sampled asato∼πo\(⋅∣st\)a\_\{t\}^\{o\}\\sim\\pi\_\{o\}\(\\cdot\\mid s\_\{t\}\)\. Conditional on the history before candidate sampling, its expected cost\-to\-go equalsVCπo​\(st\)V\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\}\)by Eq\. \([20](https://arxiv.org/html/2610.33494#A3.E20)\)\. Substituting this relation into Eq\. \([31](https://arxiv.org/html/2610.33494#A3.E31)\) gives:

JC​\(π\)−JC​\(πo\)=∑t=0H−1γt​𝔼π​\[QCπo​\(st,at\)−QCπo​\(st,ato\)\]\.J\_\{C\}\(\\pi\)\-J\_\{C\}\(\\pi\_\{o\}\)=\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\_\{\\pi\}\\left\[Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}\)\-Q\_\{C\}^\{\\pi\_\{o\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\right\]\.\(32\)
When the online candidate is selected, the action\-value difference is zero\. When the guide candidate is selected, Eq\. \([9](https://arxiv.org/html/2610.33494#S4.E9)\) bounds this difference byζ\+2​εC\\zeta\+2\\varepsilon\_\{C\}\. By substituting these bounds into Eq\. \([32](https://arxiv.org/html/2610.33494#A3.E32)\), we obtain:

JC​\(π\)−JC​\(πo\)≤\(ζ\+2​εC\)​∑t=0H−1γt​Prπ⁡\(Itg=1\),J\_\{C\}\(\\pi\)\-J\_\{C\}\(\\pi\_\{o\}\)\\leq\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\Pr\_\{\\pi\}\(I\_\{t\}^\{g\}=1\),\(33\)whereItg∈\{0,1\}I\_\{t\}^\{g\}\\in\\\{0,1\\\}equals one when the guide candidate is selected under Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) before episode termination, including selection by the fallback, and zero otherwise\. The probabilityPrπ⁡\(Itg=1\)\\Pr\_\{\\pi\}\(I\_\{t\}^\{g\}=1\)includes the randomness of the trajectory, candidate actions, and sampled horizon\. Since guide selection requirest<ht<handhhis uniform on\{0,…,Hi\}\\\{0,\\ldots,H\_\{i\}\\\}, the discounted sum of these probabilities satisfies:

∑t=0H−1γt​Prπ⁡\(Itg=1\)≤∑t=0H−1γt​Pr⁡\(h\>t\)=∑t=0Hi−1γt​Hi−tHi\+1\.\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\Pr\_\{\\pi\}\(I\_\{t\}^\{g\}=1\)\\leq\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\Pr\(h\>t\)=\\sum\_\{t=0\}^\{H\_\{i\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\}\-t\}\{H\_\{i\}\+1\}\.\(34\)
Combining this inequality with Eq\. \([33](https://arxiv.org/html/2610.33494#A3.E33)\), taking positive parts, and using\[u\+v\]\+≤\[u\]\+\+v\[u\+v\]\_\{\+\}\\leq\[u\]\_\{\+\}\+vforv≥0v\\geq 0completes the proof:

\+\\displaystyle\{\}\_\{\+\}≤\[JC​\(πo\)−d\+\(ζ\+2​εC\)​∑t=0Hi−1γt​Hi−tHi\+1\]\+\\displaystyle\\leq\\left\[J\_\{C\}\(\\pi\_\{o\}\)\-d\+\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\}\-t\}\{H\_\{i\}\+1\}\\right\]\_\{\+\}\(35\)≤\[JC​\(πo\)−d\]\+\+\(ζ\+2​εC\)​∑t=0Hi−1γt​Hi−tHi\+1\.\\displaystyle\\leq\[J\_\{C\}\(\\pi\_\{o\}\)\-d\]\_\{\+\}\+\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\}\-t\}\{H\_\{i\}\+1\}\.∎

Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3)establishes the episode cost bound with the online policy, cost estimator, and maximum guide horizon fixed throughout the episode\. In online learning, an episode may instead span multiple policy updates\. These updates can change the expected remaining cost, so their effect must be included when bounding the cost of the resulting interaction\. Proposition[C\.1](https://arxiv.org/html/2610.33494#A3.Thmtheorem1)extends the analysis to this setting by adding a term for increases in the online policy’s cost value\. Under the fixed\-policy conditions of Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3), this term vanishes and the original bound is recovered\.

###### Proposition C\.1\(Cost bound with policy updates\)\.

Letiti\_\{t\}be the current training iteration before action selection at episode steptt, and letπo\(t\)\\pi\_\{o\}^\{\(t\)\}denote the online policy available at that step\. The superscript\(t\)\(t\)distinguishes policy versions only in this extension\. Fix the initial learner state, includingi0i\_\{0\}andπo\(0\)\\pi\_\{o\}^\{\(0\)\}, before samplings0∼ρ0s\_\{0\}\\sim\\rho\_\{0\}andh∼Unif⁡\{0,…,Hi0\}h\\sim\\operatorname\{Unif\}\\\{0,\\ldots,H\_\{i\_\{0\}\}\\\}independently\. Keephhfixed throughout the episode\. At each step, the current online policy and cost estimator are determined before candidate sampling\. Assume Eq\. \([7](https://arxiv.org/html/2610.33494#S4.E7)\) holds with the sameεC\\varepsilon\_\{C\}for both candidates at every step satisfyingt<min⁡\{h,Hit\}t<\\min\\\{h,H\_\{i\_\{t\}\}\\\}\. If Eq\. \([6](https://arxiv.org/html/2610.33494#S4.E6)\) is used throughout the episode with the current policy, estimator, and maximum guide horizon, then:

𝔼⁡\[∑t=0H−1γt​ct\]\\displaystyle\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}c\_\{t\}\\right\]≤JC​\(πo\(0\)\)\\displaystyle\\leq J\_\{C\}\(\\pi\_\{o\}^\{\(0\)\}\)\(36\)\+𝔼⁡\[∑t=0H−1Itg​\(2​γt​εC\+min⁡\{γt​ζ,\[bt−q^to\]\+\}\)\]\\displaystyle\+\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}I\_\{t\}^\{g\}\\left\(2\\gamma^\{t\}\\varepsilon\_\{C\}\+\\min\\\{\\gamma^\{t\}\\zeta,\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}\\\}\\right\)\\right\]\+∑t=1H−1γt𝔼\[\[VCπo\(t\)\(st\)−VCπo\(t−1\)\(st\)\]\+\],\\displaystyle\+\\sum\_\{t=1\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\\left\[\\left\[V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\-V\_\{C\}^\{\\pi\_\{o\}^\{\(t\-1\)\}\}\(s\_\{t\}\)\\right\]\_\{\+\}\\right\],whereq^to=γt​Q^Cπo\(t\)​\(st,ato\)\\widehat\{q\}\_\{t\}^\{o\}=\\gamma^\{t\}\\widehat\{Q\}\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)uses the current cost estimator, andItgI\_\{t\}^\{g\}indicates guide selection\. The expectations include randomness in interaction and policy updates, conditional on the initial learner state\. Both value functions in the last term are evaluated at the same statests\_\{t\}and episode steptt\. For episodes that terminate beforeHH, remaining costs and terminal\-state values are set to zero\.

###### Proof\.

Applying the Bellman recursion in Eq\. \([21](https://arxiv.org/html/2610.33494#A3.E21)\) to the current policy, held fixed for each value evaluation, and taking expectations gives:

𝔼⁡\[QCπo\(t\)​\(st,at\)−VCπo\(t\)​\(st\)\]\\displaystyle\\mathbb\{E\}\\left\[Q\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\},a\_\{t\}\)\-V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\\right\]\(37\)=𝔼⁡\[ct\+γ​VCπo\(t\)​\(st\+1\)−VCπo\(t\)​\(st\)\]\.\\displaystyle=\\mathbb\{E\}\\left\[c\_\{t\}\+\\gamma V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\+1\}\)\-V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\\right\]\.Multiplying byγt\\gamma^\{t\}and summing overttleaves the value differences between successive policies at the intermediate states\. Moreover, conditional on the history before candidate sampling,ato∼πo\(t\)\(⋅∣st\)a\_\{t\}^\{o\}\\sim\\pi\_\{o\}^\{\(t\)\}\(\\cdot\\mid s\_\{t\}\)implies that its expected action value equalsVCπo\(t\)​\(st\)V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\. Using the common initial distribution and zero terminal values therefore yields:

𝔼⁡\[∑t=0H−1γt​ct\]−JC​\(πo\(0\)\)\\displaystyle\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}c\_\{t\}\\right\]\-J\_\{C\}\(\\pi\_\{o\}^\{\(0\)\}\)\(38\)=∑t=0H−1γt​𝔼​\[QCπo\(t\)​\(st,at\)−QCπo\(t\)​\(st,ato\)\]\\displaystyle=\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\\left\[Q\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\},a\_\{t\}\)\-Q\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\},a\_\{t\}^\{o\}\)\\right\]\+∑t=1H−1γt𝔼\[VCπo\(t\)\(st\)−VCπo\(t−1\)\(st\)\]\.\\displaystyle\+\\sum\_\{t=1\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\\left\[V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\-V\_\{C\}^\{\\pi\_\{o\}^\{\(t\-1\)\}\}\(s\_\{t\}\)\\right\]\.The action\-value difference is zero when the online candidate is selected\. For a selected guide candidate, Eq\. \([29](https://arxiv.org/html/2610.33494#A3.E29)\) applies to the current policy and estimator under the stated error condition\. Applying this bound to the first sum and replacing each policy value difference in the second sum by its positive part proves Eq\. \([36](https://arxiv.org/html/2610.33494#A3.E36)\)\. ∎

Even when policies and guide limits change within an episode, guide selection requirest<ht<h\. The guide contribution can therefore be bounded using the horizon sampled at episode start:

𝔼⁡\[∑t=0H−1Itg​\(2​γt​εC\+min⁡\{γt​ζ,\[bt−q^to\]\+\}\)\]\\displaystyle\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}I\_\{t\}^\{g\}\\left\(2\\gamma^\{t\}\\varepsilon\_\{C\}\+\\min\\\{\\gamma^\{t\}\\zeta,\[b\_\{t\}\-\\widehat\{q\}\_\{t\}^\{o\}\]\_\{\+\}\\\}\\right\)\\right\]\(39\)≤\(ζ\+2​εC\)​∑t=0H−1γt​Pr⁡\(h\>t\)\\displaystyle\\leq\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\Pr\(h\>t\)=\(ζ\+2​εC\)​∑t=0Hi0−1γt​Hi0−tHi0\+1\.\\displaystyle=\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\_\{0\}\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\_\{0\}\}\-t\}\{H\_\{i\_\{0\}\}\+1\}\.Taking positive parts in Eq\. \([36](https://arxiv.org/html/2610.33494#A3.E36)\) then gives:

\[𝔼⁡\[∑t=0H−1γt​ct\]−d\]\+\\displaystyle\\left\[\\mathbb\{E\}\\left\[\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}c\_\{t\}\\right\]\-d\\right\]\_\{\+\}\(40\)≤\[JC​\(πo\(0\)\)−d\]\+\+\(ζ\+2​εC\)​∑t=0Hi0−1γt​Hi0−tHi0\+1\\displaystyle\\leq\[J\_\{C\}\(\\pi\_\{o\}^\{\(0\)\}\)\-d\]\_\{\+\}\+\(\\zeta\+2\\varepsilon\_\{C\}\)\\sum\_\{t=0\}^\{H\_\{i\_\{0\}\}\-1\}\\gamma^\{t\}\\frac\{H\_\{i\_\{0\}\}\-t\}\{H\_\{i\_\{0\}\}\+1\}\+∑t=1H−1γt𝔼\[\[VCπo\(t\)\(st\)−VCπo\(t−1\)\(st\)\]\+\]\.\\displaystyle\+\\sum\_\{t=1\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\\left\[\\left\[V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\-V\_\{C\}^\{\\pi\_\{o\}^\{\(t\-1\)\}\}\(s\_\{t\}\)\\right\]\_\{\+\}\\right\]\.
The final term accounts for increases in cost value caused by policy updates\. It vanishes when the policy is unchanged or each update satisfiesVCπo\(t\)​\(st\)≤VCπo\(t−1\)​\(st\)V\_\{C\}^\{\\pi\_\{o\}^\{\(t\)\}\}\(s\_\{t\}\)\\leq V\_\{C\}^\{\\pi\_\{o\}^\{\(t\-1\)\}\}\(s\_\{t\}\)almost surely at the visited state and episode step\. In particular, under the fixed\-policy conditions of Proposition[4\.3](https://arxiv.org/html/2610.33494#S4.Thmtheorem3), Eq\. \([40](https://arxiv.org/html/2610.33494#A3.E40)\) reduces to its bound\. For general updates, Eq\. \([36](https://arxiv.org/html/2610.33494#A3.E36)\) guarantees that the expected episode cost does not exceedddif its right\-hand side is at mostdd\. Assumption[4\.1](https://arxiv.org/html/2610.33494#S4.Thmtheorem1)alone does not control the changes in policy value\.

### C\.4Discussion of Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4)

Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4)gives conditions on the state distributions generated by RBGA roll\-ins and the reward advantage of the learned policy\. The distributionμt\\mu\_\{t\}is induced by RBGA roll\-ins, whose action selections may depend on the remaining budget, sampled horizon, and earlier observations\. The valueQRπoQ\_\{R\}^\{\\pi\_\{o\}\}evaluates the action at timettfollowed by the learned online policyπo\\pi\_\{o\}, as in guide\-based policy search\([Uchendu et al\., 2023](https://arxiv.org/html/2610.33494#bib.bib10), Appendix A\.5\)\.

The state coverage condition in Eq\. \([12](https://arxiv.org/html/2610.33494#S4.E12)\) compares probabilities of measurable sets\. In a finite state space, it givesρtπ⋆​\(s\)≤D∞​μt​\(s\)\\rho\_\{t\}^\{\\pi^\{\\star\}\}\(s\)\\leq D\_\{\\infty\}\\mu\_\{t\}\(s\)for every state\. Under this condition, the expectation of any non\-negative measurable function satisfies:

∫f⁡\(s\)​ρtπ⋆​\(𝑑s\)≤D∞​∫f⁡\(s\)​μt​\(𝑑s\),\\int f\(s\)\\,\\rho\_\{t\}^\{\\pi^\{\\star\}\}\(\\mathrm\{d\}s\)\\leq D\_\{\\infty\}\\int f\(s\)\\,\\mu\_\{t\}\(\\mathrm\{d\}s\),\(41\)whereffis a non\-negative measurable function of the state\. The constantD∞D\_\{\\infty\}bounds how much more frequently the optimal feasible policy visits a set of states than the roll\-in policy\. Since the remaining budget affects action selection, the coverage condition is imposed on the distributions generated by the complete RBGA selection rule\.

The advantage condition in Eq\. \([13](https://arxiv.org/html/2610.33494#S4.E13)\) takes the positive part after averaging over the comparator’s actions and before averaging over states\. This gives the non\-negative function to which the state coverage inequality applies\. For the optimal feasible policy itself, the expected advantage satisfies:

𝔼a∼π⋆\(⋅∣s\)ARπ⋆\(s,a\)=0\.\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi^\{\\star\}\}\(s,a\)=0\.\(42\)

### C\.5Proof and Discussion of Proposition[4\.5](https://arxiv.org/html/2610.33494#S4.Thmtheorem5)

Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4)\.For the learned online policyπo\\pi\_\{o\}, there exist a finite coverage coefficientD∞≥1D\_\{\\infty\}\\geq 1and finite reward advantage boundsεR,t≥0\\varepsilon\_\{R,t\}\\geq 0such that, at every time steptt, the state distributions satisfy:

ρtπ⋆≤D∞​μt,\\rho\_\{t\}^\{\\pi^\{\\star\}\}\\leq D\_\{\\infty\}\\mu\_\{t\},\(43\)whereμt\\mu\_\{t\}is the marginal state distribution at timettinduced by RBGA roll\-ins,ρtπ⋆\\rho\_\{t\}^\{\\pi^\{\\star\}\}is the corresponding distribution underπ⋆\\pi^\{\\star\}, andD∞D\_\{\\infty\}bounds their relative probabilities\. Under the roll\-in distribution, the expected positive advantage of the optimal feasible policy satisfies:

𝔼s∼μt\[\[𝔼a∼π⋆\(⋅∣s\)ARπo\(s,a\)\]\+\]≤εR,t,\\mathbb\{E\}\_\{s\\sim\\mu\_\{t\}\}\\left\[\\left\[\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\\right\]\_\{\+\}\\right\]\\leq\\varepsilon\_\{R,t\},\(44\)whereεR,t\\varepsilon\_\{R,t\}bounds the remaining positive reward advantage ofπ⋆\\pi^\{\\star\}under the roll\-in distribution\.

Proposition[4\.5](https://arxiv.org/html/2610.33494#S4.Thmtheorem5)\.Under Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4), the reward gap between the learned online policyπo\\pi\_\{o\}and the optimal feasible policyπ⋆\\pi^\{\\star\}satisfies:

JR​\(π⋆\)−JR​\(πo\)≤D∞​∑t=0H−1γt​εR,t\.J\_\{R\}\(\\pi^\{\\star\}\)\-J\_\{R\}\(\\pi\_\{o\}\)\\leq D\_\{\\infty\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\varepsilon\_\{R,t\}\.\(45\)
###### Proof\.

To compare the reward performance ofπ⋆\\pi^\{\\star\}andπo\\pi\_\{o\}, we use the finite\-horizon reward form of the performance difference lemma\([Kakade and Langford, 2002](https://arxiv.org/html/2610.33494#bib.bib33), Lemma 6\.1\)\. With both policies starting fromρ0\\rho\_\{0\}, we obtain:

JR\(π⋆\)−JR\(πo\)=∑t=0H−1γt𝔼s∼ρtπ⋆\[𝔼a∼π⋆\(⋅∣s\)ARπo\(s,a\)\]\.J\_\{R\}\(\\pi^\{\\star\}\)\-J\_\{R\}\(\\pi\_\{o\}\)=\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\_\{s\\sim\\rho\_\{t\}^\{\\pi^\{\\star\}\}\}\\left\[\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\\right\]\.\(46\)
By replacing each state’s expected advantage with its positive part and applying the two conditions in Assumption[4\.4](https://arxiv.org/html/2610.33494#S4.Thmtheorem4), we obtain:

JR​\(π⋆\)−JR​\(πo\)\\displaystyle J\_\{R\}\(\\pi^\{\\star\}\)\-J\_\{R\}\(\\pi\_\{o\}\)≤∑t=0H−1γt𝔼s∼ρtπ⋆\[\[𝔼a∼π⋆\(⋅∣s\)ARπo\(s,a\)\]\+\]\\displaystyle\\leq\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\_\{s\\sim\\rho\_\{t\}^\{\\pi^\{\\star\}\}\}\\left\[\\left\[\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\\right\]\_\{\+\}\\right\]\(47\)≤D∞∑t=0H−1γt𝔼s∼μt\[\[𝔼a∼π⋆\(⋅∣s\)ARπo\(s,a\)\]\+\]\\displaystyle\\leq D\_\{\\infty\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\mathbb\{E\}\_\{s\\sim\\mu\_\{t\}\}\\left\[\\left\[\\mathbb\{E\}\_\{a\\sim\\pi^\{\\star\}\(\\cdot\\mid s\)\}A\_\{R\}^\{\\pi\_\{o\}\}\(s,a\)\\right\]\_\{\+\}\\right\]≤D∞​∑t=0H−1γt​εR,t,\\displaystyle\\leq D\_\{\\infty\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\varepsilon\_\{R,t\},where the second inequality applies Eq\. \([43](https://arxiv.org/html/2610.33494#A3.E43)\) to a non\-negative function, and the last inequality follows from Eq\. \([44](https://arxiv.org/html/2610.33494#A3.E44)\)\. Therefore, the reward gap satisfies Eq\. \([14](https://arxiv.org/html/2610.33494#S4.E14)\)\. ∎

For the same learned policy satisfyingJC​\(πo\)≤dJ\_\{C\}\(\\pi\_\{o\}\)\\leq d, the optimality ofπ⋆\\pi^\{\\star\}gives:

0≤JR​\(π⋆\)−JR​\(πo\)≤D∞​∑t=0H−1γt​εR,t\.0\\leq J\_\{R\}\(\\pi^\{\\star\}\)\-J\_\{R\}\(\\pi\_\{o\}\)\\leq D\_\{\\infty\}\\sum\_\{t=0\}^\{H\-1\}\\gamma^\{t\}\\varepsilon\_\{R,t\}\.\(48\)Thus, for a sequence of feasible learned policies with a common finite coverage constant andεR,t\\varepsilon\_\{R,t\}tending to zero at every time step, the reward gap tends to zero\.