SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions

# SafeExplorer: An Unbiased Policy Gradient for Reinforcement Learning with Recovery Interventions
Source: [https://arxiv.org/html/2607.08925](https://arxiv.org/html/2607.08925)
Elham Daneshmandelham\.daneshmand@mail\.mcgill\.ca McGill University & MilaMajid Khadivmajid\.khadiv@tum\.de Technical University of MunichGlen Bersethglen\.berseth@umontreal\.ca Université de Montréal & MilaHsiu\-Chin Linhsiu\-chin\.lin@mcgill\.ca McGill University & Mila

###### Abstract

Training reinforcement\-learning agents directly on physical robots makes every fall costly, since a fall can damage the platform and cannot be undone like a simulator reset; the goal is therefore to minimize falls during training rather than trade them off against return, as constrainedMarkov decision process\(MDP\) formulations do\. A standard mitigation hands control to a separate*recovery policy*whenever the agent leaves a designer\-specified*safe region*\(a subset of state space it should stay within\), but the resulting mixed\-policy rollouts silently bias every on\-policy update, and the importance\-sampling correction that would remove this bias is ill\-defined whenever the recovery policy is deterministic\. We address this bias with a drop\-in modification ofproximal policy optimization\(PPO\)\. Its core is an unbiased policy\-gradient estimator that uses the score function only at safe timesteps and never evaluates the recovery policy’s density, so it stays valid even when the recovery policy is deterministic, exactly where importance sampling breaks, and it empirically dominates importance sampling even when the recovery policy is stochastic\. Because the recovery policy still makes credit assignment slow near the safe\-region boundary, two further components accelerate learning: a closed\-form value for recovery\-triggering states when dynamics and recovery are deterministic, and an imitation loss that copies recovery actions only when recovery*succeeds*\. On a three\-environment, five\-seed benchmark, the resulting algorithm reduces training\-time falls by factors of233×233\\times,48×48\\times, and26×26\\timeson HalfCheetah, Ant, and Unitree Go1 over standardPPO, while matching or exceedingPPO’s final reward, and on Ant, where the recovery policy is unreliable, it is the only method that reaches80%80\\%of the best final reward\.

## 1Introduction

\\Ac

RL policies often perform best when trained directly on the task they will be deployed on, but moving that training onto a physical robot is hard for a reason unrelated to asymptotic performance: learning requires failures, and on real hardware every failure has a price\. A*fall*, a loss of balance that ends the episode, can damage the platform and cannot be undone the way a simulator reset can\. Recent methods learn capable controllers by collecting hundreds of thousands of episodes in massively parallel simulation\(Lee et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib32); Rudin et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib43); Agarwal et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib2)\), where these failures are free, but on a real robot they are not\(Ibarz et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib22); Smith et al\.,[2023b](https://arxiv.org/html/2607.08925#bib.bib49)\)\. What gates real\-world training is therefore not the final reward but the number of falls spent reaching it, and the goal of SafeExplorer is to keep that number small\.

The common way to keep learning on real hardware is to pair the agent with a separate*recovery policy*, a controller that takes over whenever the agent leaves a designer\-specified*safe region*and steers the system back toward a nominal state\. Because the safe region is set conservatively, the agent leaves it routinely, so the recovery policy fires far more often than a fall would occur, converting most would\-be falls into cheap, reversible interventions\. The controller is easy to assemble from off\-the\-shelf parts, anmodel predictive control\(MPC\) solver\(Pua & Khadiv,[2024](https://arxiv.org/html/2607.08925#bib.bib40)\), asoft actor\-critic\(SAC\) expert\(Haarnoja et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib18)\), or a hand\-coded routine\(Lee et al\.,[2019](https://arxiv.org/html/2607.08925#bib.bib31)\)\. But it carries a hidden cost: the data the agent collects is no longer produced by the agent alone, but by a*mixed policy*, the main policy inside the safe region and the recovery outside it, so each rollout is a blend of two controllers rather than a sample from the one being improved\.

This blend breaks the on\-policy assumption\.PPO\(Schulman et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib46)\)and related methods estimate the gradient of the policy that produced the rollout, so they are valid only when the rollout distribution matches the policy being updated\. The moment the recovery policy intervenes, every update is biased toward the recovery policy’s behavior\. Most safe\-reinforcement learning\(RL\) work leaves this mismatch unaddressed, updating as if all data came from the main policy\(Srinivasan et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib53); Thananjeyan et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib58); Yang et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib65)\), or sidesteps it by shaping the reward to discourage entering recovery\(Tessler et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib57); Stooke et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib54)\)\. The standard remedy, an importance\-sampling correction\(Degris et al\.,[2012](https://arxiv.org/html/2607.08925#bib.bib12)\), does not solve the problem either: a deterministic recovery \(anMPCsolver or a greedySACactor\) produces a single action rather than a distribution, so the importance ratio has no density in its denominator and is undefined precisely where it is needed, the same obstruction that motivates deterministic policy gradients\(Silver et al\.,[2014](https://arxiv.org/html/2607.08925#bib.bib47)\)\. The mechanism that makes real\-world training feasible is thus the one that corrupts the learning signal, and a correct gradient must factor the recovery policy out rather than reweight through it\.

Contributions\.We close this gap with SafeExplorer, a practical algorithm built on a theoretical foundation for safeRLwith a recovery policy\. We make four contributions\.*First*, we prove an unbiased policy\-gradient theorem \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\) for any mixed policy that hands control to an external recovery policy on a subset of states\. The gradient uses the main\-policy score function only where the main policy acts and never evaluates the recovery policy’s action density, so it applies unchanged to deterministic and stochastic recoveries, and empirically matches or outperformsimportance sampling\(IS\) corrections in every measured regime, with orders\-of\-magnitude lower gradient variance where importance ratios explode \([Remark˜1](https://arxiv.org/html/2607.08925#Thmremark1)\)\. The theorem is general: safe\-region intervention is the instance we develop \([Corollary˜2](https://arxiv.org/html/2607.08925#Thmtheorem2)\), and Jump\-StartRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)\([Corollary˜8](https://arxiv.org/html/2607.08925#Thmtheorem8)\) and state\-triggered shieldedRL\([Corollary˜9](https://arxiv.org/html/2607.08925#Thmtheorem9)\) follow as corollaries of the same factorization\.*Second*, we bound the gap between the mixed\-policy return we optimize and the main\-policy return we deploy \([Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\); this gap shrinks as the safe region grows and vanishes in the idealized limit where the region covers all reachable states\.*Third*, when dynamics and recovery are deterministic, the value of a recovery\-triggering state admits a closed\-form expression \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\); used as the critic’s target there, it provides dense, accurate supervision at the safe\-region boundary where on\-policy signal is otherwise scarce\.*Fourth*, we add an outcome\-gated compatibility regularizer \([Definition˜7](https://arxiv.org/html/2607.08925#Thmtheorem7)\) that pulls the main policy toward actions from*successful*recovery segments only, a warm start from behavior the recovery policy has shown to work; without the gate it reduces to aDataset Aggregation\(DAgger\)\-style behavioral\-cloning loss\(Ross et al\.,[2011](https://arxiv.org/html/2607.08925#bib.bib42)\)\([Proposition˜10](https://arxiv.org/html/2607.08925#Thmtheorem10)\)\.

Together these four pieces turn the recovery policy from a source of bias into a source of signal\. On a three\-environment, five\-seed benchmark, they reduce training\-time falls by233×233\\times,48×48\\times, and26×26\\timesonHalfCheetah,Ant, andGo1over standardPPO, while matching or exceedingPPO’s final reward\. OnAnt, where the recovery policy is unreliable, SafeExplorer is the only method that reaches the success threshold of80%80\\%of the best reward\.

## 2Related Work

The setup we study, an agent paired with an external recovery policy that takes over outside a safe region, makes training rollouts follow a mixed policy that biases every on\-policy update\. We evaluate on legged locomotion, where end\-to\-endRLhas advanced rapidly\(Ha et al\.,[2025](https://arxiv.org/html/2607.08925#bib.bib17); Lee et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib32); Peng et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib38); Tan et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib56); Peng et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib39); Rudin et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib43); Agarwal et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib2); Kumar et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib30); Haarnoja et al\.,[2019](https://arxiv.org/html/2607.08925#bib.bib19); Bogdanovic et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib9)\)and real\-world fine\-tuning is now routine\(Smith et al\.,[2023b](https://arxiv.org/html/2607.08925#bib.bib49);[2024](https://arxiv.org/html/2607.08925#bib.bib50);[a](https://arxiv.org/html/2607.08925#bib.bib48); Liu et al\.,[2024](https://arxiv.org/html/2607.08925#bib.bib33)\)\. These methods secure safety through reward design and sim\-to\-real robustness, but none confronts the bias a recovery policy injects into the on\-policy gradient\. Because that bias comes from the mixed rollout rather than any property of legged robots, our correction plugs into a locomotion pipeline without being tied to locomotion or any one robot\.

Off\-policy RLCorrecting that bias looks like a standard off\-policy problem, but off\-policy policy\-gradient and evaluation methods\(Degris et al\.,[2012](https://arxiv.org/html/2607.08925#bib.bib12); Gu et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib16); Jiang & Li,[2016](https://arxiv.org/html/2607.08925#bib.bib24)\)rely on the importance ratio that[Section˜1](https://arxiv.org/html/2607.08925#S1)showed a deterministic recovery leaves undefined\. Truncated\-ISschemes such as V\-trace\(Espeholt et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib13)\)and Retrace\(Munos et al\.,[2016](https://arxiv.org/html/2607.08925#bib.bib34)\)cannot help, because the singularity sits upstream of the truncation\. Our[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)factors the recovery measure out instead of reweighting, so it applies uniformly to deterministic and stochastic recoveries,MPCcontrollers\(Chiu et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib10)\)included\.

Mixed\-policy and intervention dataIf reweighting is not the route, the alternative is to ask how prior work uses the same mixed\-policy data, and rollouts in which a second policy intervenes are in fact common: they appear inDAgger\(Ross et al\.,[2011](https://arxiv.org/html/2607.08925#bib.bib42); Kelly et al\.,[2019](https://arxiv.org/html/2607.08925#bib.bib29)\), in human\-in\-the\-loopRL\(Spencer et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib52); Saunders et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib44)\), and inJump\-Start RL\(JSRL\)\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)\. These either imitate the intervening policy or discard its transitions, so none extract a bias\-corrected on\-policy gradient from that data\. Our masked gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\) recovers that gradient directly, and our compatibility regularizer \([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\) reduces in theJSRLsetting to a state\-dependent imitation loss with an outcome\-based gate \([Proposition˜10](https://arxiv.org/html/2607.08925#Thmtheorem10)\)\.

Constraints and safety filtersDelegating safety to an external recovery policy is itself the minority choice: the dominant lines of safeRL\(García & Fernández,[2015](https://arxiv.org/html/2607.08925#bib.bib15)\)build safety into the policy rather than handing it to a separate controller\.\\AcpCMDP encode safety as constraints on long\-run cost\(Altman,[1999](https://arxiv.org/html/2607.08925#bib.bib5)\), optimized via primal\-dual schemes such asconstrained policy optimization\(CPO\)\(Achiam et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib1)\), reward\-constrained policy optimization\(Tessler et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib57)\), and PID\-Lagrangian variants\(Stooke et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib54)\)\. Because they trade cost against return, they suit soft\-constraint settings where some failures are tolerable, whereas we treat falls as something to minimize rather than to budget against return\. Closer to a hard guarantee,control barrier function\(CBF\) approaches\(Ames et al\.,[2019](https://arxiv.org/html/2607.08925#bib.bib6)\)and shields\(Junges et al\.,[2015](https://arxiv.org/html/2607.08925#bib.bib25); Alshiekh et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib4); Dalal et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib11); Srinivasan et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib53); Hasanbeig et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib20); Kang et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib27)\)project unsafe actions at execution time; this is exactly the case of our mechanism in which the intervention is a single\-step projection, so our setup subsumes it as one instance\. Closely related are the state\-wise constrainedMDPfamily\(Zhao et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib67)\)and almost\-surely safeRL\(Sootla et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib51)\)\.[Section˜3](https://arxiv.org/html/2607.08925#S3)places all four formalisms in the constraint\-formulation taxonomy ofWachi et al\. \([2024](https://arxiv.org/html/2607.08925#bib.bib63)\)\.

RL with a recovery policyThe work closest to ours commits fully to that delegation: it uses an explicit recovery policy as an alternative toconstrained\(CMDP\) costs and pointwise filters, facing the same mixed\-policy bias extended from the single\-step projections of our shielded\-RLcorollary to multi\-step recovery segments\.Thananjeyan et al\. \([2021](https://arxiv.org/html/2607.08925#bib.bib58)\)train a safety critic and switch to the recovery policy when constraint risk crosses a threshold; they deliberately relabel recovery transitions with the task policy’s proposed action, coherent for their off\-policy Q\-learning objective \([Section˜B\.7](https://arxiv.org/html/2607.08925#A2.SS7)\) but the source of the bias we correct once carried into an on\-policy update\.Yang et al\. \([2022](https://arxiv.org/html/2607.08925#bib.bib65)\)apply a similar template to legged locomotion with reward shaping, andLee et al\. \([2019](https://arxiv.org/html/2607.08925#bib.bib31)\)train a model\-free quadruped recovery controller, an example of the external recovery policy our masked gradient handles directly\. Closest,Wagener et al\. \([2021](https://arxiv.org/html/2607.08925#bib.bib64)\)intervene on an advantage criterion and prove the intervened process safe, but still optimize the policy on the mixed rollouts without correcting the bias we address\. Reverse\-curriculum methods\(Florensa et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib14)\)grow the region of starting states, and teacher\-scheduled interventions\(Turchetta et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib61)\)induce a safety curriculum, whereas safe\-region curricula like ours grow the region the agent is permitted to enter\. None of these treat the resulting distributional bias on the on\-policy gradient, which our masked\-gradient theorem \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\) addresses without requiring the recovery policy to have a density\.

## 3Problem Setting and Notation

We studyRLin the*hard\-safety regime*of the constraint\-formulation taxonomy ofWachi et al\. \([2024](https://arxiv.org/html/2607.08925#bib.bib63)\)\. On a real robot a fall can break hardware, damage property, or injure a bystander, so the goal is to keep falls as rare as possible rather than to budget them against return\.

The strictest formalism related to this regime is the state\-wise constrainedMDPfamily\(Zhao et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib67)\), which demands constraint satisfaction at every step\. We relax that requirement to*violation\-minimization*, the model\-free goal the survey identifies: without prior knowledge of the dynamics, hard state\-wise safety cannot be guaranteed during training, so the target is to incur as few constraint violations \(falls\) as possible\.

Among the ways to enforce safety in this regime \([Section˜2](https://arxiv.org/html/2607.08925#S2)\), ours is an external*multi\-step*recovery policy that takes over whenever the state leaves a designer\-specified safe region and returns the agent toward a nominal configuration, unlike single\-step action filters such as control\-barrier projections and shields\. We call this the*safe\-region intervention*mechanism\.

We instantiate the mechanism with a discountedMDP\(𝒮,𝒜,P,r,γ,T\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,\\gamma,T\)with continuous state space𝒮\\mathcal\{S\}and action space𝒜\\mathcal\{A\}, transition kernelPP, rewardr∈\[−rmax,rmax\]r\\in\[\-r\_\{\\max\},r\_\{\\max\}\], discountγ∈\(0,1\)\\gamma\\in\(0,1\), and episode horizonTT\. The analysis uses the infinite\-horizon discounted convention \(absorption at failure states\); experiments truncate atT=1000T=1000steps\. Letπθ​\(a∣s\)\\pi\_\{\\theta\}\(a\\mid s\)be the main policy, stochastic andθ\\theta\-parameterized\. The recovery policyπrec​\(a∣s\)\\pi^\{\\mathrm\{rec\}\}\(a\\mid s\)is an arbitrary state\-conditioned measure that may be deterministic \(MPC, greedySAC\) or stochastic, with noθ\\theta\-dependence \(its own fixed parameters,SACweights or anMPCcost, are never optimized, so we suppress them\)\. The safe region is the sublevel setℛ​\(d\):=\{s:𝒟​\(s\)≤d\}\\mathcal\{R\}\(d\):=\\\{s:\\mathcal\{D\}\(s\)\\leq d\\\}of a task\-space distance𝒟\\mathcal\{D\}to a nominal configuration, withd∈\[0,dmax\]d\\in\[0,d\_\{\\max\}\]grown over training by a curriculum schedulediter​\(⋅\)d\_\{\\mathrm\{iter\}\}\(\\cdot\)\(dmaxd\_\{\\max\}a per\-environment hyperparameter,[Section˜5](https://arxiv.org/html/2607.08925#S5)\)\. The*mixed policy*πθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}followsπθ\\pi\_\{\\theta\}insideℛ​\(d\)\\mathcal\{R\}\(d\)andπrec\\pi^\{\\mathrm\{rec\}\}outside it\. We give its general form and an unbiased gradient in[Section˜4\.1](https://arxiv.org/html/2607.08925#S4.SS1)\. The trajectory distributions underπθ\\pi\_\{\\theta\}alone and underπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}arepθ​\(τ\)=p​\(s1\)​∏tπθ​\(at∣st\)​P​\(st\+1∣st,at\)p\_\{\\theta\}\(\\tau\)=p\(s\_\{1\}\)\\prod\_\{t\}\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)P\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)andpθmix​\(τ\)=p​\(s1\)​∏tπθmix​\(at∣st\)​P​\(st\+1∣st,at\)p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)=p\(s\_\{1\}\)\\prod\_\{t\}\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)P\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)\. The*main\-policy return*isJ​\(θ\)=𝔼τ∼pθ​\(τ\)​\[r​\(τ\)\]J\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\theta\}\(\\tau\)\}\[r\(\\tau\)\]withr​\(τ\):=∑tγt​rtr\(\\tau\):=\\sum\_\{t\}\\gamma^\{t\}r\_\{t\}\. It is not directly observable during training, since rollouts are drawn frompθmix​\(τ\)p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\), notpθ​\(τ\)p\_\{\\theta\}\(\\tau\)\. The*mixed\-policy return*, the quantity training rollouts actually realize, isJmix​\(θ\)=𝔼τ∼pθmix​\(τ\)​\[r​\(τ\)\]J^\{\\mathrm\{mix\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)\}\[r\(\\tau\)\]\. Its state\-conditional form, the*mixed\-policy value*Vπθmix​\(s\):=𝔼τ∼pθmix​\(τ\)​\[r​\(τ\)∣s1=s\]V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\):=\\mathbb\{E\}\_\{\\tau\\sim p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)\}\[r\(\\tau\)\\mid s\_\{1\}=s\], is whatPPO’s learned criticVθV\_\{\\theta\}estimates\. Discounted state\-visitations underπθ\\pi\_\{\\theta\}andπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}areνπθ\\nu^\{\\pi\_\{\\theta\}\}andνπθmix\\nu^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}, respectively\. We use two rates, distinguished by their sampling distribution\. The*recovery rate*α​\(θ,d\):=Prs∼νπθmix⁡\[s∉ℛ​\(d\)\]\\alpha\(\\theta,d\):=\\Pr\_\{s\\sim\\nu^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\}\[s\\notin\\mathcal\{R\}\(d\)\]is sampled under the*mixed*distribution that training rollouts produce, hence*observable*as the fraction of rollout steps on which the recovery policy fired\. The*out\-of\-region rate*β​\(θ,d\):=Prs∼νπθ⁡\[s∉ℛ​\(d\)\]\\beta\(\\theta,d\):=\\Pr\_\{s\\sim\\nu^\{\\pi\_\{\\theta\}\}\}\[s\\notin\\mathcal\{R\}\(d\)\]is sampled under the*main\-only*distribution never deployed during training, hence*not directly observable*, yet it is the quantity that controls the gap in[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\. A consolidated notation table is in[Section˜B\.2](https://arxiv.org/html/2607.08925#A2.SS2)\.

Together, these objects give the setup four operational features: binary set\-membership safety, a possibly\-deterministic recovery policy, multi\-step recovery \(a genuine mixed\-policy distribution\), and the unmodified main\-policy objective, each addressed by one method component in[Section˜4](https://arxiv.org/html/2607.08925#S4)\([Section˜B\.3](https://arxiv.org/html/2607.08925#A2.SS3)\)\.

## 4Method

SafeExplorer builds onPPOwith four pieces, each removing a failure mode of recovery\-usingRL: \(i\) a*masked policy gradient*\([Section˜4\.1](https://arxiv.org/html/2607.08925#S4.SS1)\) that removes the bias from treating recovery transitions as main\-policy transitions; \(ii\) an*objective\-gap bound*\([Section˜4\.2](https://arxiv.org/html/2607.08925#S4.SS2)\) relating the training \(mixed\-policy\) return to the deployment \(main\-policy\) return and showing a safe\-region curriculum closes the gap; \(iii\) an*analytic recovery value*\([Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3)\) that replaces the learned critic at recovery\-triggering states with a closed form under deterministic dynamics; and \(iv\) an*outcome\-gated compatibility regularizer*\([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\) that imitates recovery only after successful segments\. We develop each in turn\.

### 4\.1Unbiased policy gradient via partition masking

To make the most of training data, we want every rollout to contribute to the task gradient, even those where the recovery policy took over for long stretches\. The obstacle is that treating recovery transitions as if the main policy generated them biases the on\-policy update, and theISfix breaks down whenever the recovery policy is deterministic\.πθ\\pi\_\{\\theta\}, a Gaussian in ourPPOimplementation, has a differentiable log\-density, whereas a deterministicπrec\\pi^\{\\mathrm\{rec\}\}is a Dirac measure with no density\. Because this obstacle arises whenever an external,θ\\theta\-independent policy takes over on a subset of states, not only in safety, we solve it in full generality first and then specialize to safe\-region intervention\.

Setup\.We generalize the mixed policy of[Section˜3](https://arxiv.org/html/2607.08925#S3)\. Letℳ⊆𝒮\\mathcal\{M\}\\subseteq\\mathcal\{S\}be a designer\-specified*main\-policy set*\(whereπθ\\pi\_\{\\theta\}acts\), and letμ\(⋅∣s\)\\mu\(\\cdot\\mid s\)be any state\-conditioned action measure that is*independent of*θ\\theta\(it may be deterministic or stochastic\)\. The resulting*partition policy*is

πθmix​\(a∣s\)=\{πθ​\(a∣s\)s∈ℳ,μ​\(a∣s\)s∉ℳ,\\pi\_\{\\theta\}^\{\\mathrm\{mix\}\}\(a\\mid s\)=\\begin\{cases\}\\pi\_\{\\theta\}\(a\\mid s\)&s\\in\\mathcal\{M\},\\\\ \\mu\(a\\mid s\)&s\\notin\\mathcal\{M\},\\end\{cases\}\(1\)and the trajectory returnJmix​\(θ\)=𝔼τ∼pθmix​\[r​\(τ\)\]J^\{\\mathrm\{mix\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\theta\}^\{\\mathrm\{mix\}\}\}\[r\(\\tau\)\]\.

###### Assumption 1\(Regularity\)\.

\(R1\)For every\(s,a\)\(s,a\)on the support ofpθmixp\_\{\\theta\}^\{\\mathrm\{mix\}\}withs∈ℳs\\in\\mathcal\{M\},πθ​\(a∣s\)\>0\\pi\_\{\\theta\}\(a\\mid s\)\>0andθ↦log⁡πθ​\(a∣s\)\\theta\\mapsto\\log\\pi\_\{\\theta\}\(a\\mid s\)is differentiable\.\(R2\)T<∞T<\\infty, or\|rt\|≤rmax\|r\_\{t\}\|\\leq r\_\{\\max\}uniformly\.\(R3\)Differentiation and integration commute in[Equation˜2](https://arxiv.org/html/2607.08925#S4.E2)\.

For diagonal\-Gaussianπθ\\pi\_\{\\theta\}with bounded mean and log\-std, and bounded reward, \(R1\) to \(R3\) all hold\.

###### Theorem 1\(Unbiased partition\-policy gradient\)\.

Under[˜1](https://arxiv.org/html/2607.08925#Thmassumption1),

∇θJmix​\(θ\)=𝔼τ∼pθmix​\[\(∑t=1T𝟙​\[st∈ℳ\]​∇θlog⁡πθ​\(at∣st\)\)​r​\(τ\)\]\.\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\theta\}^\{\\mathrm\{mix\}\}\}\\\!\\left\[\\left\(\\sum\_\{t=1\}^\{T\}\\mathbb\{1\}\[s\_\{t\}\\in\\mathcal\{M\}\]\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\right\)\\,r\(\\tau\)\\right\]\.\(2\)Here𝟙​\[⋅\]\\mathbb\{1\}\[\\cdot\]is the indicator function, so the sum runs only over timesteps withst∈ℳs\_\{t\}\\in\\mathcal\{M\}\.

We prove this in[Section˜A\.1](https://arxiv.org/html/2607.08925#A1.SS1): becauseμ\\muisθ\\theta\-independent, its factor in the trajectory density vanishes under differentiation, so the masked gradient never evaluatesμ\\mu’s density and holds for anyμ\\mu\. This is exactly the case the intra\-option policy\-gradient results\(Sutton et al\.,[1999](https://arxiv.org/html/2607.08925#bib.bib55); Bacon et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib7)\)assume away, extending the factorization to the density\-free external controllers they exclude, and it is why we never need theIScorrection\.

Specializations\.Different choices of\(ℳ,μ\)\(\\mathcal\{M\},\\mu\)specialize[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)to concrete settings\.

###### Corollary 2\(Safe\-region intervention\)\.

Takeℳ=ℛ\\mathcal\{M\}=\\mathcal\{R\}\(the safe region\) andμ=πrec\\mu=\\pi^\{\\mathrm\{rec\}\}\(the recovery policy,θ\\theta\-independent by setup\), recovering the mixed policyπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}of[Section˜3](https://arxiv.org/html/2607.08925#S3)with returnJmix​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\)\. Then[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)gives∇θJmix​\(θ\)=𝔼τ∼pθmix​\(τ\)​\[\(∑t𝟙​\[st∈ℛ\]​∇θlog⁡πθ​\(at∣st\)\)​r​\(τ\)\]\.\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)\}\\\!\\left\[\\left\(\\sum\_\{t\}\\mathbb\{1\}\[s\_\{t\}\\in\\mathcal\{R\}\]\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\right\)r\(\\tau\)\\right\]\.

The same factorization yields Jump\-StartRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)and state\-triggered shieldedRLas corollaries \([Corollaries˜8](https://arxiv.org/html/2607.08925#Thmtheorem8)and[9](https://arxiv.org/html/2607.08925#Thmtheorem9),[Section˜A\.1](https://arxiv.org/html/2607.08925#A1.SS1)\)\. The rest of the paper develops safe\-region intervention \([Corollary˜2](https://arxiv.org/html/2607.08925#Thmtheorem2)\)\.

###### Corollary 3\(PPOsurrogate with safe\-step masking\)\.

Specializing[Corollary˜2](https://arxiv.org/html/2607.08925#Thmtheorem2)to the clippedPPOsurrogate, the empirical masked\-gradient estimator is

∇Jmix^​\(θ\)=1\|Tℛ\|​∑t:st∈ℛ∇θlog⁡πθ​\(at∣st\)​A^tfull,\\widehat\{\\nabla J^\{\\mathrm\{mix\}\}\}\(\\theta\)=\\frac\{1\}\{\|T\_\{\\mathcal\{R\}\}\|\}\\sum\_\{t:s\_\{t\}\\in\\mathcal\{R\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\},\(3\)whereA^tfull\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\}is thegeneralized advantage estimation\(GAE\)\(Schulman et al\.,[2016](https://arxiv.org/html/2607.08925#bib.bib45)\)advantage over the full mixed\-policy trajectory \(recovery segments enter through the return exactly as in[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1), while the mask restricts only the score function\) and\|Tℛ\|=\|\{t:st∈ℛ\}\|\|T\_\{\\mathcal\{R\}\}\|=\|\\\{t:s\_\{t\}\\in\\mathcal\{R\}\\\}\|\. The clippedPPOsurrogate is

LsafePPO​\(θ\)=−1\|Tℛ\|​∑t:st∈ℛmin⁡\(ρt​\(θ\)​A^tfull,clip​\(ρt​\(θ\),1−ϵ,1\+ϵ\)​A^tfull\),L^\{\\mathrm\{PPO\}\}\_\{\\mathrm\{safe\}\}\(\\theta\)=\-\\frac\{1\}\{\|T\_\{\\mathcal\{R\}\}\|\}\\sum\_\{t:s\_\{t\}\\in\\mathcal\{R\}\}\\min\\\!\\left\(\\rho\_\{t\}\(\\theta\)\\,\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\},\\;\\mathrm\{clip\}\(\\rho\_\{t\}\(\\theta\),1\\\!\-\\\!\\epsilon,1\\\!\+\\\!\\epsilon\)\\,\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\}\\right\),\(4\)withρt​\(θ\)=πθ​\(at∣st\)/πθold​\(at∣st\)\\rho\_\{t\}\(\\theta\)=\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\pi\_\{\\theta\_\{\\mathrm\{old\}\}\}\(a\_\{t\}\\mid s\_\{t\}\), well\-defined since both are the main policy\.

The masked score\-function term is exactly unbiased for∇θJmix\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\)\. Relative to the standard clipped\-PPOsurrogate over the same rollouts,[Equation˜3](https://arxiv.org/html/2607.08925#S4.E3)changes only two things: the sums run over safe steps, and the normalizer is\|Tℛ\|\|T\_\{\\mathcal\{R\}\}\|, the number of such steps in a rollout and hence random, which the curriculum onddkeeps large in practice \([Section˜5](https://arxiv.org/html/2607.08925#S5),[Section˜B\.5](https://arxiv.org/html/2607.08925#A2.SS5)\)\.GAE, the clipping, and thusPPO’s stability heuristics are otherwise unchanged\.

### 4\.2Bounding the gap between mixed\-policy and main\-policy returns

[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)’s gradient targets the*mixed\-policy return*Jmix​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\), the quantity training rollouts realize, but the deployment\-relevant quantity is the*main\-policy return*J​\(θ\)J\(\\theta\), whatπθ\\pi\_\{\\theta\}earns alone, which training never samples directly\. Optimizing one while caring about the other is only safe if the two cannot drift far apart, so this section bounds\|Jmix−J\|\|J^\{\\mathrm\{mix\}\}\-J\|and shows it is exactly zero under reading \(R\-A\) below\.

The two returns differ only on the steps whereπθ\\pi\_\{\\theta\}acts butπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}would have called the recovery policy, that is, the steps whereπθ\\pi\_\{\\theta\}alone leaves the safe region\. The gap is therefore controlled by how reliablyπθ\\pi\_\{\\theta\}keeps itself insideℛ\\mathcal\{R\}, which we make precise with a one\-step invariance property\.

###### Assumption 2\(Approximateπθ\\pi\_\{\\theta\}\-invariance ofℛ\\mathcal\{R\}\)\.

There existsη≥0\\eta\\geq 0such that, starting from any reachables∈ℛs\\in\\mathcal\{R\},πθ\\pi\_\{\\theta\}stays inℛ\\mathcal\{R\}at the next step with probability at least1−η1\-\\etauniformly over time\.

This one\-step set\-invariance property holds withη=0\\eta=0exactly in the idealized limit \(R\-A\) below and approximately in our setting \(R\-B\)\.

Summed over a discounted trajectory, this one\-step slackη\\etacontrols the out\-of\-region rateβ​\(θ,d\)\\beta\(\\theta,d\)\([Section˜3](https://arxiv.org/html/2607.08925#S3)\), which inherits a boundβ​\(θ,d\)≤η​γ/\(1−γ\)≤η/\(1−γ\)\\beta\(\\theta,d\)\\leq\\eta\\gamma/\(1\-\\gamma\)\\leq\\eta/\(1\-\\gamma\)whenever episodes start inside the region, as ours do \([Section˜A\.2](https://arxiv.org/html/2607.08925#A1.SS2)\)\.

###### Theorem 4\(Objective\-gap bound\)\.

Under[˜1](https://arxiv.org/html/2607.08925#Thmassumption1),

\|J​\(θ\)−Jmix​\(θ\)\|≤2​rmax\(1−γ\)2​β​\(θ,d\)\.\|J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)\|\\leq\\frac\{2\\,r\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\,\\beta\(\\theta,d\)\.\(5\)

The proof \([Section˜A\.2](https://arxiv.org/html/2607.08925#A1.SS2)\) applies the Performance Difference Lemma\(Kakade & Langford,[2002](https://arxiv.org/html/2607.08925#bib.bib26)\)\. The advantage ofπθ\\pi\_\{\\theta\}againstπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}vanishes insideℛ\\mathcal\{R\}, where the two policies agree, and is bounded by2​rmax/\(1−γ\)2r\_\{\\max\}/\(1\-\\gamma\)outside it, so the gap scales with the main\-policy out\-of\-region rateβ​\(θ,d\)\\beta\(\\theta,d\)\. The bound is non\-vacuous only forβ​\(θ,d\)<1−γ\\beta\(\\theta,d\)<1\-\\gamma\(η⋆<\(1−γ\)2\\eta\_\{\\star\}<\(1\-\\gamma\)^\{2\}in[Corollary˜5](https://arxiv.org/html/2607.08925#Thmtheorem5)\); we read it structurally \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\)\.

How far[Equation˜5](https://arxiv.org/html/2607.08925#S4.E5)can be pushed as the curriculum radiusditerd\_\{\\mathrm\{iter\}\}grows todmaxd\_\{\\max\}depends on the largest safe region it reaches,ℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\), which admits two readings\.

###### Corollary 5\(Conditional fixed\-point\)\.

\(Exact, R\-A\.\)Ifℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)covers the reachable state space, then atditer=dmaxd\_\{\\mathrm\{iter\}\}=d\_\{\\max\},πθ≡πθmix\\pi\_\{\\theta\}\\equiv\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}on the reachable support ofpθ​\(τ\)p\_\{\\theta\}\(\\tau\),β​\(θ,dmax\)=0\\beta\(\\theta,d\_\{\\max\}\)=0, andJmix​\(θ\)=J​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\)=J\(\\theta\)exactly\.\(Approximate, R\-B\.\)Otherwise\|J​\(θ\)−Jmix​\(θ\)\|≤2​rmax​η⋆/\(1−γ\)3\|J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)\|\\leq 2r\_\{\\max\}\\eta\_\{\\star\}/\(1\-\\gamma\)^\{3\}, whereη⋆=η​\(θ,dmax\)≥0\\eta\_\{\\star\}=\\eta\(\\theta,d\_\{\\max\}\)\\geq 0is the env\-dependent invariance slack that \(R\-B\) does not force to zero\.

The proof specializes[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)to the two readings ofℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)\([Section˜A\.3](https://arxiv.org/html/2607.08925#A1.SS3)\)\. Our setting is \(R\-B\):ℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)is a tuned hyperparameter strictly inside the reachable set, so the bound is approximate, the theory no longer forcingη=0\\eta=0, though the curriculum can still drive it low in practice\.[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)then says any reduction inβ​\(θ,d\)\\beta\(\\theta,d\)tightens the gap proportionally, which the linear schedulediter:d0→dmaxd\_\{\\mathrm\{iter\}\}\\\!:\\\!d\_\{0\}\\to d\_\{\\max\}achieves, with the observableα​\(θ,d\)\\alpha\(\\theta,d\)as its diagnostic \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\)\. Falls are orthogonal todd, and those still possible under \(R\-B\) are what the compatibility regularizer \([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\) targets\.[Section˜A\.7](https://arxiv.org/html/2607.08925#A1.SS7)sketches whyβ​\(θ,d\)\\beta\(\\theta,d\)is hard to bound from*observable*quantities in continuous\-action settings\.

### 4\.3Analytic recovery value

The masked gradient fixes the policy gradient, but it leaves a second error untouched: the critic\. A learned critic trained over the recovery segments bootstraps its value targets*through*them, so even with a corrected gradient the value\-target error persists; empirically, the learned\-VVvariant under\-performs every other variant onHalfCheetah\. Replacing the learned critic at recovery\-triggering states with the analytic value derived below removes this error, and is the largest single\-ingredient reward gain onHalfCheetahat no cost in falls \(onAntandGo1the outcome gate contributes more; quantified in[Section˜8](https://arxiv.org/html/2607.08925#S8)\)\.

When dynamics and recovery are both deterministic, the practically common case \(MPC, greedySAC\), the value at a recovery\-triggering state has a closed form: the recovery segment unrolls along a single fixed path, so its contribution is the return along that path rather than something the critic must learn\.

###### Proposition 6\(Analytic recovery value\)\.

Suppose dynamicsPPand recoveryπrec\\pi^\{\\mathrm\{rec\}\}are both deterministic\. Letsts\_\{t\}be a recovery\-triggering state,k≥1k\\geq 1the number of recovery steps until either re\-entry intoℛ\\mathcal\{R\}att\+kt\+k\(*success*\) or termination att\+kt\+k\(*failure*\), andGt,k=∑j=0k−1γj​rt\+jG\_\{t,k\}=\\sum\_\{j=0\}^\{k\-1\}\\gamma^\{j\}r\_\{t\+j\}the realized segment return\. Then

Vπθmix​\(st\)=\{Gt,k\+γk​Vπθmix​\(st\+k\)success,Gt,kfailure\.V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)=\\begin\{cases\}G\_\{t,k\}\+\\gamma^\{k\}\\,V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\+k\}\)&\\text\{success\},\\\\ G\_\{t,k\}&\\text\{failure\}\.\\end\{cases\}\(6\)Under our convention that recovery accrues zero per\-step task reward and failure incurs a one\-time terminal failure rewardrtermr\_\{\\mathrm\{term\}\}, the segment returnGt,kG\_\{t,k\}is0on success andγk−1​rterm\\gamma^\{k\-1\}r\_\{\\mathrm\{term\}\}on failure\. In practice we bootstrap the re\-entry value with the learned criticVθV\_\{\\theta\}, writingγk​Vθ​\(st\+k\)\\gamma^\{k\}V\_\{\\theta\}\(s\_\{t\+k\}\)into the buffer on success \([Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\)\. This carries no Monte\-Carlo variance under deterministicPPandπrec\\pi^\{\\mathrm\{rec\}\}\.

The proof is direct \([Section˜A\.4](https://arxiv.org/html/2607.08925#A1.SS4)\): under deterministic dynamics and recovery, the post\-trigger trajectory is a single deterministic path of lengthkk, so its return is the realized return\.111[Theorems1](https://arxiv.org/html/2607.08925#Thmtheorem1)and[4](https://arxiv.org/html/2607.08925#Thmtheorem4)hold for any bounded reward, covering this training\-signal convention; the induced train\-to\-deploy return gap is controlled by the sameβ​\(θ,d\)\\beta\(\\theta,d\)\([AppendixA](https://arxiv.org/html/2607.08925#A1)\)\. Reported returns always use the unmodified task reward \([Section6](https://arxiv.org/html/2607.08925#S6)\)\.

In practice we write[Equation˜6](https://arxiv.org/html/2607.08925#S4.E6)into the rollout buffer beforeGAE, which then proceeds normally and feeds the masked gradient \([Corollary˜3](https://arxiv.org/html/2607.08925#Thmtheorem3)\) only at safe steps \([Appendices˜B](https://arxiv.org/html/2607.08925#A2)and[B\.5](https://arxiv.org/html/2607.08925#A2.SS5)\)\.

### 4\.4The compatibility regularizer

The masked gradient and analytic value together define a provably unbiased on\-policy update forJmix​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\), but unbiasedness alone does not make the agent self\-sufficient: neither component pullsπθ\\pi\_\{\\theta\}toward behaving like recovery at unsafe states\. This matters precisely when recovery is unreliable \([Section˜7\.2](https://arxiv.org/html/2607.08925#S7.SS2)\): an unbiased update over repeated failed recoveries teaches the policy nothing about avoiding those states unaided\. We close this gap with a lightweight imitation\-from\-recovery loss with an outcome\-conditioned weight\. As an imitation signal rather than an importance correction, it needs no recovery density and covers deterministic controllers\. The closest methods,DAgger\(Ross et al\.,[2011](https://arxiv.org/html/2607.08925#bib.bib42)\)andJSRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\), do not gate the imitation signal by segment outcome, and self\-imitation learning\(Oh et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib37)\), which does, imitates the agent’s own past actions rather than an external recovery policy, the distinction we now make precise\.

###### Definition 7\(Compatibility regularizer\)\.

Letatrec=πrec​\(st\)a\_\{t\}^\{\\mathrm\{rec\}\}=\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)be the recovery action executed at unsafe steptt\. The trajectory\-level compatibility score is

Cθ​\(τ\)=∏t:st∉ℛπθ​\(atrec∣st\),log⁡Cθ​\(τ\)=∑t:st∉ℛlog⁡πθ​\(atrec∣st\)\.C\_\{\\theta\}\(\\tau\)=\\prod\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\\mid s\_\{t\}\),\\quad\\log C\_\{\\theta\}\(\\tau\)=\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}\\log\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\\mid s\_\{t\}\)\.\(7\)

A highCθ​\(τ\)C\_\{\\theta\}\(\\tau\)measures exactly the agreement we want,πθ\\pi\_\{\\theta\}assigning high likelihood to the recovery actions at the states where it intervened\. Raising it indiscriminately is harmful: a failed recovery \(segment ends in termination\) is exactly whatπθ\\pi\_\{\\theta\}should*not*imitate\. We therefore gate the imitation signal by the realized outcome of each segment\. Letσk∈\{0,1\}\\sigma\_\{k\}\\in\\\{0,1\\\}indicate success of recovery segmentkk\(re\-entry intoℛ\\mathcal\{R\}\), andseg⁡\(t\)\\operatorname\{seg\}\(t\)the segment containing steptt\. Reading success off the realized rollout keeps the gate applicable whether recovery is deterministic or stochastic\. The*hard outcome gate*is

wthard=σseg⁡\(t\),t:st∉ℛ\.w\_\{t\}^\{\\mathrm\{hard\}\}=\\sigma\_\{\\operatorname\{seg\}\(t\)\},\\quad t:s\_\{t\}\\notin\\mathcal\{R\}\.\(8\)Two boundary cases follow the same reading: a segment cut off by rollout truncation counts as a tentative success with a bootstrapped value, and an immediate re\-exit opens a new segment\. A signed soft variant based on per\-step value change is ablated in[Section˜B\.4](https://arxiv.org/html/2607.08925#A2.SS4); the hard gate dominates empirically, sowt=wthardw\_\{t\}=w\_\{t\}^\{\\mathrm\{hard\}\}throughout unless noted\.

Putting the score and its gate together, the regularizer is the outcome\-gated form of−log⁡Cθ​\(τ\)\-\\log C\_\{\\theta\}\(\\tau\)from[Equation˜7](https://arxiv.org/html/2607.08925#S4.E7), added to thePPOloss with coefficientλcompat\\lambda\_\{\\mathrm\{compat\}\}:

Lcompat​\(θ\)=−λcompatNrec​∑t:st∉ℛwt​log⁡πθ​\(atrec∣st\),L​\(θ\)=LsafePPO\+cv​LV\+Lcompat\.L^\{\\mathrm\{compat\}\}\(\\theta\)=\-\\,\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}w\_\{t\}\\,\\log\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\\mid s\_\{t\}\),\\quad L\(\\theta\)=L^\{\\mathrm\{PPO\}\}\_\{\\mathrm\{safe\}\}\+c\_\{v\}\\,L^\{V\}\+L^\{\\mathrm\{compat\}\}\.\(9\)HereNrecN\_\{\\mathrm\{rec\}\}counts the recovery\-controlled minibatch steps \([Section˜B\.2](https://arxiv.org/html/2607.08925#A2.SS2); per\-segment normalization ablated in[Section˜E\.9](https://arxiv.org/html/2607.08925#A5.SS9)\), andLVL^\{V\}is the standardPPOcritic loss with coefficientcvc\_\{v\}\(entropy bonus retained,ce=0c\_\{e\}=0by default;[Appendix˜B](https://arxiv.org/html/2607.08925#A2)\)\.

LcompatL^\{\\mathrm\{compat\}\}is a supervised auxiliary term, deliberately outside the unbiasedJmixJ^\{\\mathrm\{mix\}\}gradient: it trades a tunable amount of bias, scaled byλcompat\\lambda\_\{\\mathrm\{compat\}\}, for a signal the unbiased components cannot supply, pullingπθ\\pi\_\{\\theta\}toward the actions of*successful*recovery segments\. That trade pays off exactly where it should: the term costs a little reward onHalfCheetahbut adds substantially onGo1andAnt\(\+1121\+1121and\+2683\+2683reward over the gate\-free variant,[Section˜8](https://arxiv.org/html/2607.08925#S8)\)\.

The gate is all that separates this term from standard imitation: always on \(σk≡1\\sigma\_\{k\}\\equiv 1\),LcompatL^\{\\mathrm\{compat\}\}reduces toDAgger\-style\(Ross et al\.,[2011](https://arxiv.org/html/2607.08925#bib.bib42)\)cloning on the recovery\-controlled states,πrec\\pi^\{\\mathrm\{rec\}\}as teacher \([Proposition˜10](https://arxiv.org/html/2607.08925#Thmtheorem10),[Section˜A\.6](https://arxiv.org/html/2607.08925#A1.SS6)\); the gate echoes the Q\-filter ofNair et al\. \([2018](https://arxiv.org/html/2607.08925#bib.bib35)\), reading a realized segment outcome instead of a learned value estimate\.

## 5Algorithm

SafeExplorer assembles the three algorithmic pieces developed in the previous sections, the masked policy gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1),[Corollary˜3](https://arxiv.org/html/2607.08925#Thmtheorem3)\), the analytic recovery value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\), and the hard outcome\-gated compatibility regularizer \([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\), into a singlePPOupdate \([Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\); the fourth piece, the objective\-gap bound \([Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\), needs no implementation beyond the curriculum below\. Because each piece intervenes only where the recovery policy acts, the resulting algorithm is a small edit to a standardPPOimplementation: the changes touch only the rollout’s recovery branch and the loss, everything else is unchanged\. That locality keeps the method cheap: the added cost is one recovery\-policy forward pass per triggered step plus segment bookkeeping \(runtimes in[Appendix˜C](https://arxiv.org/html/2607.08925#A3)\)\.

This leaves one moving part, the safe\-region radius that decides when recovery is triggered, which we anneal over training with a linear curriculum\(Bengio et al\.,[2009](https://arxiv.org/html/2607.08925#bib.bib8); Narvekar et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib36)\):diter​\(u\)=d0\+u−1N​dmaxd\_\{\\mathrm\{iter\}\}\(u\)=d\_\{0\}\+\\frac\{u\-1\}\{N\}\\,d\_\{\\max\}, growing the region from a tightd0d\_\{0\}to the per\-environmentdmaxd\_\{\\max\}over theNNupdates \(d0d\_\{0\},dmaxd\_\{\\max\},NNin[Appendix˜C](https://arxiv.org/html/2607.08925#A3)\)\.[Section˜E\.7](https://arxiv.org/html/2607.08925#A5.SS7)ablates logarithmic, step, and constant schedules onAnt, our least reliable recovery policy, where the linear schedule attains the highest reward\. This annealing is the practical mechanism by which the recovery rateα​\(θ,d\)\\alpha\(\\theta,d\)falls over training \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\), an empirical diagnostic for the gap\-tightening predicted by[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\.

## 6Experimental Setup

Our experiments evaluate three claims\. The first is the*safety\-reward trade\-off*of SafeExplorer against standardPPO, recovery\-using baselines, andCMDPbaselines, judged on both falls and reward\. The second is the*unreliable\-recovery regime*that most motivates the method, where the recovery policy itself often fails and the trade\-off is hardest to win\. The third asks*which ingredient*drives the gains: the unbiased masked gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\), the analytic recovery value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\), or the outcome\-gated regularizer \([Definition˜7](https://arxiv.org/html/2607.08925#Thmtheorem7)\)\. A single metric adjudicates all three: the number of training falls a method incurs before first reaching task success, where success is80%80\\%of the best final reward attained by any method in that environment\. OnAntandGo1that best method is SafeExplorer, so the reward winner partly sets its own bar;[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)therefore reports the underlying rewards and fall counts separately\.

Environments\.We evaluate on three continuous\-control locomotion environments:HalfCheetahandAnt\(custom MuJoCo\(Todorov et al\.,[2012](https://arxiv.org/html/2607.08925#bib.bib59)\)/Gymnasium\(Towers et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib60)\)variants with a configurable safe\-region indicator; full specification in[Appendix˜D](https://arxiv.org/html/2607.08925#A4)\), andGo1, a Unitree Go1 quadruped built on themujoco\_menageriemodel\(Zakka et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib66)\)with a velocity\-tracking reward adapted fromlegged\_gym\(Rudin et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib43)\)\. The mechanism is identical across all three: the distance𝒟\\mathcal\{D\}definingℛ​\(d\)\\mathcal\{R\}\(d\)is a\(z,tilt\)\(z,\\mathrm\{tilt\}\)task\-space distance to the nominal upright pose \(exact formulas in[Section˜D\.2](https://arxiv.org/html/2607.08925#A4.SS2)\), and each environment keeps its standard locomotion reward, zeroed only during recovery \([Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3)\)\. The environments differ in how hard they make safety:HalfCheetah\(planar, strong recovery\) sets the baseline,Ant\(3D, weaker recovery\) tests whether the compatibility regularizer compensates for an unreliable fall\-back, andGo1\(12\-degrees of freedom\(DoF\) quadruped\) scales the claim to a higher\-dimensional, multi\-contact platform\.

Recovery policies\.Each environment uses a separately pre\-trainedSAC\(Haarnoja et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib18)\)actor asπrec\\pi^\{\\mathrm\{rec\}\}: a stand\-and\-stabilize objective on its own environment, task terms disabled, no demonstrations, aggressively randomized resets \(recipes in[Section˜D\.3](https://arxiv.org/html/2607.08925#A4.SS3)\)\. At rollout time it takes theSACmean action, deterministic, so it exercises the no\-density case the masked gradient is built for \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\)\. TheAntrecovery policy is meaningfully weaker than the others, the most informative axis of variation: its strong control\-magnitude penalty caps how aggressively it can correct, and the harder 3D stabilization needs those corrections \([Appendix˜D](https://arxiv.org/html/2607.08925#A4)\)\.

The remaining choice is the radiusdmaxd\_\{\\max\}, the one quantity a skeptic might suspect was tuned\. Unbiasedness holds for anydmaxd\_\{\\max\}\([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\), so no radius can manufacture statistical validity, though it still shapes results through the gap bound \([Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\) and the curriculum\. We set eachdmaxd\_\{\\max\}\([Appendix˜C](https://arxiv.org/html/2607.08925#A3)\) by a qualitative criterion, that the curriculum lowers the observable intervention rate \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\), without a quantitative search\. The fixed\-ddsweep \([Table˜10](https://arxiv.org/html/2607.08925#A5.T10)\) shows sensitivity to a*constant*radius that the curriculum mostly recovers \([Section˜E\.7](https://arxiv.org/html/2607.08925#A5.SS7)\); no systematic sweep variesdmaxd\_\{\\max\}under the curriculum, a limitation \([Appendix˜E](https://arxiv.org/html/2607.08925#A5)\)\.

Variants compared\.SafeExplorer \([Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\) is compared against three external baselines spanning the established alternatives identified in[Section˜2](https://arxiv.org/html/2607.08925#S2)\. StandardPPOis the no\-recovery extreme\. Recovery RL\(Thananjeyan et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib58)\)and Safe Legged\(Yang et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib65)\)are*on\-policy ports*of the two published recovery\-using alternatives: both run inside the samePPOloop, sharing SafeExplorer’s safe\-region trigger and frozenSACrecovery policy, and each isolates its published method’s data\-handling rule at recovery steps, action relabeling for Recovery RL versus relabeling plus a fixed reward penalty of11for Safe Legged \(exact rules and deviations in[Section˜B\.7](https://arxiv.org/html/2607.08925#A2.SS7)\)\. A controlled ablation isolating each ingredient of SafeExplorer is reported in[Section˜8](https://arxiv.org/html/2607.08925#S8); additional soft\-gate variants are in[Section˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4)\.

CMDPbaselines\.To position SafeExplorer against the standard no\-recovery alternative, we compare to twoCMDPsolvers,CPO\(Achiam et al\.,[2017](https://arxiv.org/html/2607.08925#bib.bib1)\)andPPO\-Lagrangian\(Ray et al\.,[2019](https://arxiv.org/html/2607.08925#bib.bib41); Stooke et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib54)\), which act with a single policy and encode safety as a constraint on an indicator cost \(1 per fall\)\. We run both via OmniSafe\(Ji et al\.,[2024](https://arxiv.org/html/2607.08925#bib.bib23)\)at settings matched to SafeExplorer, except the solvers’ default observation and reward normalization, which SafeExplorer does not use, favoring the baselines \([Section˜E\.14](https://arxiv.org/html/2607.08925#A5.SS14)\)\.[Section˜7\.3](https://arxiv.org/html/2607.08925#S7.SS3)analyzes their constraint feasibility\.

Training and evaluation\.We train each variant for88M environment steps onHalfCheetahandAntand1616M onGo1\(its higher\-dimensional action space needs a longer horizon\), with five random seeds\. Final reward is the per\-episode mean of the last10%10\\%of training, computed from the unmodified task reward, not the recovery\-zeroed learning signal; falls are the cumulative count over the full training run, a fall being a per\-environment unhealthy termination \([Table˜6](https://arxiv.org/html/2607.08925#A4.T6)\), never a time\-limit truncation\. Tables report mean±\\pmstandard deviation over seeds; the headline and ablation plots aggregate with interquartile means and95%95\\%stratified\-bootstrap confidence intervals\(Agarwal et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib3)\)\. A method’s falls\-to\-success is the cumulative falls before its trailing\-mean reward first crosses the success bar; a method that never crosses it is counted as not reaching success, and falls\-to\-success aggregates use the succeeding seeds only \(thek/nk/nlabels in[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)\)\. SharedPPOhyperparameters follow CleanRL’s defaults\(Huang et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib21)\)and are listed in[Table˜4](https://arxiv.org/html/2607.08925#A3.T4)\.

## 7Main Results

### 7\.1SafeExplorer reaches task success with the fewest falls

Addressing our first claim, we compare all methods on the three environments:[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)reports falls\-to\-success with the per\-method solve countsk/nk/n,[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)the separate final\-reward and total\-training\-fall figures \(different quantities, since total falls span the whole run, not just the climb to success\), and[Figure˜2](https://arxiv.org/html/2607.08925#A5.F2)the per\-metric rliable interquartile\-mean view\. SafeExplorer reaches success with the fewest training falls in every environment, an interquartile mean of1212falls onHalfCheetah,1717onAnt, and118118onGo1\. The advantage is most stark onAnt, where it is the only method to reach success at all, on all five seeds, while every baseline fails to cross the bar on any seed \(quantified in[Section˜7\.2](https://arxiv.org/html/2607.08925#S7.SS2)\)\. The separation also widens with the difficulty of staying safe: onGo1the next\-cheapest method after SafeExplorer to reach success at all is Recovery RL on a single seed \(309309falls\), withPPO\-Lagrangian andPPOat9,6589\{,\}658and15,32015\{,\}320\. The same lead shows up in the table’s lifetime view: measured as total training falls \([Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\), SafeExplorer reduces falls overPPOby𝟐𝟑𝟑×\\mathbf\{233\\times\}onHalfCheetah,𝟒𝟖×\\mathbf\{48\\times\}onAnt, and𝟐𝟔×\\mathbf\{26\\times\}onGo1while matching or beatingPPO’s final reward\.

Table 1:Final reward \(mean over the last10%10\\%of training, on the same unmodified task reward for every method\) and total training\-time falls for SafeExplorer and all baselines\. For recovery\-using methods reward is the training\-time mixed\-policy return, including recovery actions \(recovery\-disabled return for SafeExplorer:[Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\)\.CPOandPPO\-Lagrangian areCMDPbaselines with a per\-fall cost \([Section˜6](https://arxiv.org/html/2607.08925#S6)\)\. Bold marks the best value per column; onHalfCheetahfalls, SafeExplorer,PPO\-Lagrangian, andCPOare within seed noise\. This is the canonical body\-protocol batch anchoring the headline fall\-reduction ratios; the appendix sensitivity tables use separate re\-run batches \(batch provenance in[Appendix˜E](https://arxiv.org/html/2607.08925#A5)\)\.HalfCheetahAntGo1MethodReward↑\\uparrowFalls↓\\downarrowReward↑\\uparrowFalls↓\\downarrowReward↑\\uparrowFalls↓\\downarrowPPO\(no recovery\)3246±5213246\\pm 5213311±9403311\\pm 940308±165308\\pm 16536467±1459836467\\pm 145983347±2153347\\pm 21518683±140318683\\pm 1403Recovery RL1951±14231951\\pm 1423947±1831947\\pm 183176±2576\\pm 2553535±761153535\\pm 76113136±5113136\\pm 5111689±13791689\\pm 1379Safe Legged𝟒𝟎𝟓𝟒±𝟔𝟎𝟒\\mathbf\{4054\\pm 604\}1552±10841552\\pm 108483±2083\\pm 2053548±1168253548\\pm 116822347±3532347\\pm 3533022±12253022\\pm 1225CPO2835±4032835\\pm 40323±3023\\pm 301047±1061047\\pm 106899±117899\\pm 117359±48359\\pm 4839613±249839613\\pm 2498PPO\-Lagrangian2939±4142939\\pm 41418±1918\\pm 191339±4021339\\pm 4025094±27745094\\pm 27743915±9533915\\pm 95312572±163612572\\pm 1636SafeExplorer \(ours\)3581±9713581\\pm 971𝟏𝟒±𝟖\\mathbf\{14\\pm 8\}𝟐𝟖𝟓𝟕±𝟏𝟐𝟗\\mathbf\{2857\\pm 129\}𝟕𝟓𝟒±𝟒𝟔𝟑\\mathbf\{754\\pm 463\}𝟓𝟓𝟗𝟒±𝟏𝟕𝟕\\mathbf\{5594\\pm 177\}𝟕𝟑𝟏±𝟏𝟕𝟕\\mathbf\{731\\pm 177\}

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/falls_to_success_80pct.png)Figure 1:Training falls to reach task success \(success=80%=80\\%of the best final reward in each environment\), interquartile mean with95%95\\%bootstrap confidence intervals\(Agarwal et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib3)\); log axis, fewer falls is better\. Labels show the seeds reaching success \(k/nk/n\); never\-reachers are marked at right\. Baselines are gray, SafeExplorer ablation variants \([Section˜8](https://arxiv.org/html/2607.08925#S8)\) blue, and SafeExplorer the red star, which reaches success with the fewest falls in every environment and is the only method to reach it onAnt\.Reading the environments one at a time, the same ranking holds but the reason shifts with how hard safety is\. OnHalfCheetah, where the recovery policy is reliable,PPOis competitive on reward but pays for it in falls, while SafeExplorer matches that reward at a fraction of the falls\. OnGo1, the larger action space and longer horizon makePPOhighly unsafe, while Recovery RL and Safe Legged control falls only at substantial reward cost and SafeExplorer attains both the highest reward and the lowest fall count\.Antis the informative exception: the ranking still favors SafeExplorer, but it behaves differently because the recovery policy itself is unreliable, the case we isolate next in[Section˜7\.2](https://arxiv.org/html/2607.08925#S7.SS2)\.

This dependence on recovery is exactly what[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)predicts:[Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)confirms on body\-matched 5\-seed runs that the gapJmix​\(θk\)−J​\(θk\)J^\{\\mathrm\{mix\}\}\(\\theta\_\{k\}\)\-J\(\\theta\_\{k\}\)tracks the recovery rateα​\(θ,d\)\\alpha\(\\theta,d\), closing to seed noise onHalfCheetahandAnt, where the policy becomes self\-stable, and settling at a proportional residual onGo1, where it does not\.

Per\-environment learning curves are deferred to[Figure˜3](https://arxiv.org/html/2607.08925#A5.F3)in[Appendix˜E](https://arxiv.org/html/2607.08925#A5)\.

### 7\.2The unreliable\-recovery regime:Ant

This subsection addresses our second claim, the unreliable\-recovery regime:[Section˜7\.1](https://arxiv.org/html/2607.08925#S7.SS1)cannot show*why*SafeExplorer remains effective where the recovery policy itself fails, so we isolate the one environment where that happens,Ant, whoseSACrecovery is markedly less reliable\. Replaying the recovery from the states a trained policy actually drives it into, it completes the segment only7070to91%91\\%of the time onAnt, against9797to100%100\\%onHalfCheetahandGo1\(one representative run per environment;[Table˜7](https://arxiv.org/html/2607.08925#A4.T7)\)\. The recovery\-based baselines reveal the latter:PPO, Recovery RL, and Safe Legged all fail catastrophically \(3636k to5454k falls, reward≤308\\leq 308\)\. The mechanism is a feedback cycle, because when the recovery policy is unreliable, any method that relies on it*reactively*keeps re\-entering unsafe states, recovery keeps failing, and the agent never learns to avoid them unaided\. SafeExplorer breaks that cycle and alone reaches the22862286success bar, at28572857reward and754754falls, an order of magnitude fewer falls than any other recovery\-using method; the strongest baselines,PPO\-Lagrangian andCPO, reach13391339and10471047\. The ablations pinpoint why: every variant that drops the hard outcome gate, including the soft\-gate variants, also fails onAnt\([Section˜8](https://arxiv.org/html/2607.08925#S8)\); the gate, not the masked gradient or the analytic value, supplies the fallback skill when recovery is unreliable\.

### 7\.3Comparison toCMDPbaselines

This subsection completes our first claim by comparing SafeExplorer against theCMDPalternative to a recovery policy, which casts safety as a constraint and solves the resultingCMDPrather than intervening on the trajectory\.CPOandPPO\-Lagrangian, the trust\-region and Lagrangian solvers, appear in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)and[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)with a cost of11per fall, at settings identical to SafeExplorer\. SafeExplorer attains the highest reward of the three in every environment and the fewest total training falls onAntandGo1, with all three methods low and within seed noise onHalfCheetah\([Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\)\. The more revealing comparison is constraint feasibility: whether each solver keeps the falls it promised to\.

The constrained solvers’ poor safety on the hard environments is not under\-tuning\. EachCMDPsolver is configured with a cost limit on falls, as its formulation requires, and meets it onHalfCheetahwhere staying safe is easy, but*violates its own limit*onAntandGo1by up to19×19\\times:CPOconverges to an episodic cost of0\.960\.96onGo1against the0\.050\.05budget it was set, and even incurs moreGo1falls than unconstrainedPPOat a tenth of the reward \([Section˜E\.14](https://arxiv.org/html/2607.08925#A5.SS14)\)\. Constraint satisfaction thus holds where safety is easy and fails exactly where the safe\-region intervention is needed\. The difference is one of mechanism: theCMDPbaselines act alone and only penalize the violation after the fact, whereas SafeExplorer acts on the safe region directly, which is why it keeps falls low where the constrained solvers cannot\.

## 8Ablation Analysis

This section addresses our third claim, which ingredient drives SafeExplorer’s gains over prior recovery\-using methods\. We isolate each of its three new ingredients with a ladder of controlled variants that share the recovery policy and curriculum and differ only at recovery steps\. The ladder starts from the biased update those methods use, which keeps recovery transitions as if the main policy had generated them \(a learned\-critic on\-policy gradient\), and turns on one ingredient at a time:*Unmasked PG*is that biased baseline;*masked, learnedVV*switches to the unbiased masked policy gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\);*masked, analyticVV*adds the analytic recovery value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\); and SafeExplorer adds the hard outcome gate \([Equation˜8](https://arxiv.org/html/2607.08925#S4.E8)\), withPPO\(no recovery\) anchoring the no\-intervention end\.[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)ranks these variants by falls\-to\-success and[Figure˜4](https://arxiv.org/html/2607.08925#A5.F4)\([Section˜E\.3](https://arxiv.org/html/2607.08925#A5.SS3)\) gives the complementary reward\-vs\-falls view\. Each ingredient contributes a distinct improvement, attributed rung by rung below\.

The first rung, the masked policy gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\), is the foundation\. Its biased alternative, Unmasked PG, reaches competitive or higher reward onHalfCheetahandGo1, but pays14×14\\times,7×7\\times, and4×4\\timesmore falls than SafeExplorer onHalfCheetah,Ant, andGo1\([Table˜8](https://arxiv.org/html/2607.08925#A5.T8)\), and is ill\-posed for deterministic recovery\. Masking removes that bias, the prerequisite for the next two ingredients\.

Removing the bias exposes a second problem: the unbiased gradient is sparse at the safe\-region boundary, so the masked, learned\-VVvariant alone regresses to19821982reward onHalfCheetah, the lowest of any rung \([Table˜8](https://arxiv.org/html/2607.08925#A5.T8)\)\. The analytic recovery value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\) supplies dense, correct targets exactly there, liftingHalfCheetahreward to38703870, the largest single\-ingredient gain on that environment \(\+1888\+1888\)\.

The top rung adds the hard outcome gate \(SafeExplorer\), which dominates on safety in every environment and supplies the largest single\-ingredient reward gains onGo1andAnt\(\+1121\+1121and\+2683\+2683;[Table˜8](https://arxiv.org/html/2607.08925#A5.T8)\), where the analytic value alone leaves reward flat; it is the only variant to reach theAntsuccess bar \(22862286; next best12531253\), the strongest evidence for the imitation reading ofCθ​\(τ\)C\_\{\\theta\}\(\\tau\)\([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\)\.

In theλcompat\\lambda\_\{\\mathrm\{compat\}\}sweep \([Table˜9](https://arxiv.org/html/2607.08925#A5.T9);HalfCheetahandAntonly, the default transferred toGo1untested\),HalfCheetahreward stays at47684768to57835783across\[10−3,1\]\[10^\{\-3\},1\], peaking at10−210^\{\-2\}, but falls are minimized at the10−310^\{\-3\}default \(4747\) and rise8×8\\timeswithin a decade \(374374at10−210^\{\-2\}\); we choose the default on falls, not reward\.[Appendix˜E](https://arxiv.org/html/2607.08925#A5)reports the rest: robustness to hyperparameters and noise \([Sections˜E\.5](https://arxiv.org/html/2607.08925#A5.SS5),[E\.6](https://arxiv.org/html/2607.08925#A5.SS6),[E\.7](https://arxiv.org/html/2607.08925#A5.SS7),[E\.9](https://arxiv.org/html/2607.08925#A5.SS9)and[E\.10](https://arxiv.org/html/2607.08925#A5.SS10)\), the soft\-gate variants \([Section˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4)\), compute\-scaling \([Section˜E\.8](https://arxiv.org/html/2607.08925#A5.SS8)\), the full ladder \([Section˜E\.3](https://arxiv.org/html/2607.08925#A5.SS3)\), and a failure\-mode catalog \([Section˜E\.11](https://arxiv.org/html/2607.08925#A5.SS11)\)\.

## 9Discussion and Limitations

When the method helps, and when it does not\.SafeExplorer’s advantage is smallest whenPPOis already safe and recovery is reliable, and largest in the opposite regimes: weak recovery \(Ant\), dangerous unaidedPPO\(Go1\)\. The hard outcome gate makes this work: a soft gate imitates failed recovery actions and degrades both metrics \([Section˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4)\)\. We treat the recovery policy as a permanent fallback, not temporary scaffolding: onGo1it stays mildly engaged \(α​\(θ,d\)≈0\.05\\alpha\(\\theta,d\)\\approx 0\.05\) even at1616M steps, and keeping it available is what bounds falls there \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\); our claim is reduced training falls, not their elimination\.

Assumptions behind the guarantees\.The analytic value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\) is exact only under deterministic dynamics and recovery\. Under noise it becomes a single\-sample estimate, unbiased up to critic error at re\-entry, favorable in practice sinceπrec\\pi^\{\\mathrm\{rec\}\}is near\-deterministic; a multi\-sample extension is in[Section˜A\.5](https://arxiv.org/html/2607.08925#A1.SS5)\. The objective\-gap bound \([Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\) is controlled by the unobservable out\-of\-region rateβ​\(θ,d\)\\beta\(\\theta,d\), tracked via the observableα​\(θ,d\)\\alpha\(\\theta,d\)as a diagnostic \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\)\. A certified observable bound remains open \(a deterministic recovery makes the natural simulation\-lemma bound vacuous; routes in[Section˜A\.7](https://arxiv.org/html/2607.08925#A1.SS7)\)\.

Generality of the compatibility regularizer\.The gate is also what makes the regularizer general: without itLcompatL^\{\\mathrm\{compat\}\}isDAgger\-style imitation \([Proposition˜10](https://arxiv.org/html/2607.08925#Thmtheorem10)\), so outcome\-gated imitation should transfer to other teacher\-intervention settings, human\-in\-the\-loopRL\(Spencer et al\.,[2020](https://arxiv.org/html/2607.08925#bib.bib52)\)orJSRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)\.

Beyond locomotion\.That transfer argument is theoretical; our evaluation is locomotion\-only, though the correction applies wherever a fixed recovery policy takes over at unsafe states \(a scripted reset in manipulation, an emergency\-stop in driving\); validating these is future work\. The method also presupposes a pre\-trained recovery policy and a hand\-designed distance𝒟\\mathcal\{D\}, whose acquisition cost is excluded from the reported budgets \([Appendices˜C](https://arxiv.org/html/2607.08925#A3)and[D](https://arxiv.org/html/2607.08925#A4)\)\.

## 10Conclusion

We built SafeExplorer, an on\-policyRLalgorithm for safety\-critical control that, as a singlePPOupdate, reduces training\-time falls by26×26\\timesto233×233\\timesacross three locomotion environments while matching or beating standardPPOon reward\.

These gains rest on a theoretical foundation forRLwith an external recovery policy: a masked policy\-gradient theorem unbiased for the mixed\-policy return under any recovery, an objective\-gap bound with an exact fixed point, an analytic recovery value, and an outcome\-gated compatibility regularizer, all instantiated for*safe\-region intervention*to minimize training falls\.

## References

- Achiam et al\. \(2017\)Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel\.Constrained policy optimization\.In*Proceedings of the 34th International Conference on Machine Learning \(ICML\)*, pp\. 22–31, 2017\.
- Agarwal et al\. \(2022\)Ananye Agarwal, Ashish Kumar, Jitendra Malik, and Deepak Pathak\.Legged locomotion in challenging terrains using egocentric vision\.In*Conference on Robot Learning*, 2022\.
- Agarwal et al\. \(2021\)Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C\. Courville, and Marc G\. Bellemare\.Deep reinforcement learning at the edge of the statistical precipice\.*Advances in Neural Information Processing Systems \(NeurIPS\)*, 34:29304–29320, 2021\.
- Alshiekh et al\. \(2018\)Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu\.Safe reinforcement learning via shielding\.In*Proceedings of the 32nd AAAI Conference on Artificial Intelligence*, 2018\.
- Altman \(1999\)Eitan Altman\.*Constrained Markov Decision Processes*\.Chapman & Hall/CRC, 1999\.
- Ames et al\. \(2019\)Aaron D\. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada\.Control barrier functions: Theory and applications\.In*Proceedings of the 2019 European Control Conference \(ECC\)*, pp\. 3420–3431, 2019\.
- Bacon et al\. \(2017\)Pierre\-Luc Bacon, Jean Harb, and Doina Precup\.The option\-critic architecture\.In*Proceedings of the 31st AAAI Conference on Artificial Intelligence \(AAAI\)*, pp\. 1726–1734, 2017\.
- Bengio et al\. \(2009\)Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston\.Curriculum learning\.In*Proceedings of the 26th Annual International Conference on Machine Learning*, ICML ’09, pp\. 41–48, New York, NY, USA, 2009\. Association for Computing Machinery\.ISBN 9781605585161\.doi:10\.1145/1553374\.1553380\.URL[https://doi\.org/10\.1145/1553374\.1553380](https://doi.org/10.1145/1553374.1553380)\.
- Bogdanovic et al\. \(2022\)Miroslav Bogdanovic, Majid Khadiv, and Ludovic Righetti\.Model\-free reinforcement learning for robust locomotion using demonstrations from trajectory optimization\.*Frontiers in Robotics and AI*, 9, 2022\.
- Chiu et al\. \(2022\)Jiawei Chiu, Jean\-Pierre Sleiman, Mayank Mittal, Farbod Farshidian, and Marco Hutter\.A collision\-free mpc for whole\-body dynamic locomotion and manipulation\.In*2022 International Conference on Robotics and Automation \(ICRA\)*, pp\. 4686–4693, 2022\.
- Dalal et al\. \(2018\)Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerík, Todd Hester, Cosmin Paduraru, and Yuval Tassa\.Safe exploration in continuous action spaces\.*CoRR*, abs/1801\.08757, 2018\.
- Degris et al\. \(2012\)Thomas Degris, Martha White, and Richard S\. Sutton\.Off\-policy actor\-critic\.In*Proceedings of the 29th International Conference on Machine Learning \(ICML\)*, 2012\.
- Espeholt et al\. \(2018\)Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu\.IMPALA: Scalable distributed deep\-RL with importance weighted actor\-learner architectures\.In*Proceedings of the 35th International Conference on Machine Learning \(ICML\)*, pp\. 1407–1416, 2018\.
- Florensa et al\. \(2017\)Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel\.Reverse curriculum generation for reinforcement learning\.In*Proceedings of the 1st Annual Conference on Robot Learning \(CoRL\)*, pp\. 482–495, 2017\.
- García & Fernández \(2015\)Javier García and Fernando Fernández\.A comprehensive survey on safe reinforcement learning\.*Journal of Machine Learning Research*, 16\(42\):1437–1480, 2015\.
- Gu et al\. \(2017\)Shixiang Gu, Tim Lillicrap, Zoubin Ghahramani, Richard E\. Turner, Bernhard Schölkopf, and Sergey Levine\.Interpolated policy gradient: Merging on\-policy and off\-policy gradient estimation for deep reinforcement learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2017\.
- Ha et al\. \(2025\)Sehoon Ha, Joonho Lee, Michiel van de Panne, Zhaoming Xie, Wenhao Yu, and Majid Khadiv\.Learning\-based legged locomotion: State of the art and future perspectives\.*The International Journal of Robotics Research*, 44\(8\):1396–1427, 2025\.
- Haarnoja et al\. \(2018\)Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine\.Soft actor\-critic: Off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.In Jennifer Dy and Andreas Krause \(eds\.\),*Proceedings of the 35th International Conference on Machine Learning*, volume 80 of*Proceedings of Machine Learning Research*, pp\. 1861–1870\. PMLR, 10–15 Jul 2018\.URL[https://proceedings\.mlr\.press/v80/haarnoja18b\.html](https://proceedings.mlr.press/v80/haarnoja18b.html)\.
- Haarnoja et al\. \(2019\)Tuomas Haarnoja, Aurick Zhou, Sehoon Ha, Jie Tan, G\. Tucker, and Sergey Levine\.Learning to walk via deep reinforcement learning\.In*Robotics: Science and Systems XV*, 2019\.doi:10\.15607/RSS\.2019\.XV\.011\.
- Hasanbeig et al\. \(2020\)Mohammadhosein Hasanbeig, Alessandro Abate, and Daniel Kroening\.Cautious reinforcement learning with logical constraints\.In Amal El Fallah Seghrouchni, Gita Sukthankar, Bo An, and Neil Yorke\-Smith \(eds\.\),*Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS ’20, Auckland, New Zealand, May 9\-13, 2020*, pp\. 483–491\. International Foundation for Autonomous Agents and Multiagent Systems, 2020\.doi:10\.5555/3398761\.3398821\.
- Huang et al\. \(2022\)Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and Jo ao G\. M\. Araújo\.Cleanrl: High\-quality single\-file implementations of deep reinforcement learning algorithms\.*Journal of Machine Learning Research*, 23\(274\):1–18, 2022\.URL[http://jmlr\.org/papers/v23/21\-1342\.html](http://jmlr.org/papers/v23/21-1342.html)\.
- Ibarz et al\. \(2021\)Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine\.How to train your robot with deep reinforcement learning: lessons we have learned\.*The International Journal of Robotics Research*, 40:698 – 721, 2021\.
- Ji et al\. \(2024\)Jiaming Ji, Jiayi Zhou, Borong Zhang, Juntao Dai, Xuehai Pan, Ruiyang Sun, Weidong Huang, Yiran Geng, Mickel Liu, and Yaodong Yang\.OmniSafe: An infrastructure for accelerating safe reinforcement learning research\.*Journal of Machine Learning Research*, 25\(285\):1–6, 2024\.
- Jiang & Li \(2016\)Nan Jiang and Lihong Li\.Doubly robust off\-policy value evaluation for reinforcement learning\.In*Proceedings of the 33rd International Conference on Machine Learning \(ICML\)*, pp\. 652–661, 2016\.
- Junges et al\. \(2015\)Sebastian Junges, N\. Jansen, Christian Dehnert, Ufuk Topcu, and Joost\-Pieter Katoen\.Safety\-constrained reinforcement learning for mdps\.In*International Conference on Tools and Algorithms for Construction and Analysis of Systems*, 2015\.
- Kakade & Langford \(2002\)Sham Kakade and John Langford\.Approximately optimal approximate reinforcement learning\.In*Proceedings of the 19th International Conference on Machine Learning \(ICML\)*, pp\. 267–274, 2002\.
- Kang et al\. \(2022\)Katie Kang, Paula Gradu, Jason J\. Choi, Michael Janner, Claire J\. Tomlin, and Sergey Levine\.Lyapunov density models: Constraining distribution shift in learning\-based control\.In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato \(eds\.\),*International Conference on Machine Learning, ICML 2022, 17\-23 July 2022, Baltimore, Maryland, USA*, volume 162 of*Proceedings of Machine Learning Research*, pp\. 10708–10733\. PMLR, 2022\.
- Kearns et al\. \(2002\)Michael Kearns, Yishay Mansour, and Andrew Y\. Ng\.A sparse sampling algorithm for near\-optimal planning in large Markov decision processes\.*Machine Learning*, 49\(2–3\):193–208, 2002\.
- Kelly et al\. \(2019\)Michael Kelly, Chelsea Sidrane, Katherine Driggs\-Campbell, and Mykel J\. Kochenderfer\.HG\-DAgger: Interactive imitation learning with human experts\.In*2019 International Conference on Robotics and Automation \(ICRA\)*, pp\. 8077–8083, 2019\.doi:10\.1109/ICRA\.2019\.8793698\.
- Kumar et al\. \(2021\)Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik\.RMA: rapid motor adaptation for legged robots\.In Dylan A\. Shell, Marc Toussaint, and M\. Ani Hsieh \(eds\.\),*Robotics: Science and Systems XVII, Virtual Event, July 12\-16, 2021*, 2021\.doi:10\.15607/RSS\.2021\.XVII\.011\.
- Lee et al\. \(2019\)Joonho Lee, Jemin Hwangbo, and Marco Hutter\.Robust recovery controller for a quadrupedal robot using deep reinforcement learning\.*CoRR*, abs/1901\.07517, 2019\.
- Lee et al\. \(2020\)Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter\.Learning quadrupedal locomotion over challenging terrain\.*Science Robotics*, 5, 2020\.
- Liu et al\. \(2024\)Minghuan Liu, Zixuan Chen, Xuxin Cheng, Yandong Ji, Ruihan Yang, and Xiaolong Wang\.Visual whole\-body control for legged loco\-manipulation\.In*Conference on Robot Learning \(CoRL\)*, 2024\.
- Munos et al\. \(2016\)Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc G\. Bellemare\.Safe and efficient off\-policy reinforcement learning\.In*Advances in Neural Information Processing Systems \(NeurIPS\)*, 2016\.
- Nair et al\. \(2018\)Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel\.Overcoming exploration in reinforcement learning with demonstrations\.In*2018 IEEE International Conference on Robotics and Automation \(ICRA\)*, pp\. 6292–6299, 2018\.doi:10\.1109/ICRA\.2018\.8463162\.
- Narvekar et al\. \(2020\)Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E\. Taylor, and Peter Stone\.Curriculum learning for reinforcement learning domains: A framework and survey\.*Journal of Machine Learning Research*, 21\(181\):1–50, 2020\.URL[http://jmlr\.org/papers/v21/20\-212\.html](http://jmlr.org/papers/v21/20-212.html)\.
- Oh et al\. \(2018\)Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee\.Self\-imitation learning\.In*Proceedings of the 35th International Conference on Machine Learning \(ICML\)*, volume 80 of*Proceedings of Machine Learning Research*, pp\. 3878–3887\. PMLR, 2018\.
- Peng et al\. \(2018\)Xue Bin Peng, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel\.Sim\-to\-real transfer of robotic control with dynamics randomization\.In*2018 IEEE International Conference on Robotics and Automation \(ICRA\)*, pp\. 3803–3810, 2018\.doi:10\.1109/ICRA\.2018\.8460528\.
- Peng et al\. \(2020\)Xue Bin Peng, Erwin Coumans, Tingnan Zhang, Tsang\-Wei Edward Lee, Jie Tan, and Sergey Levine\.Learning agile robotic locomotion skills by imitating animals\.In Marc Toussaint, Antonio Bicchi, and Tucker Hermans \(eds\.\),*Robotics: Science and Systems XVI, Virtual Event / Corvalis, Oregon, USA, July 12\-16, 2020*, 2020\.doi:10\.15607/RSS\.2020\.XVI\.064\.
- Pua & Khadiv \(2024\)Xun Pua and Majid Khadiv\.Safe learning of locomotion skills from mpc\.In*2024 IEEE\-RAS 23rd International Conference on Humanoid Robots \(Humanoids\)*, pp\. 459–466, 2024\.doi:10\.1109/Humanoids58906\.2024\.10769799\.
- Ray et al\. \(2019\)Alex Ray, Joshua Achiam, and Dario Amodei\.Benchmarking safe exploration in deep reinforcement learning\.Technical report, OpenAI, 2019\.URL[https://cdn\.openai\.com/safexp\-short\.pdf](https://cdn.openai.com/safexp-short.pdf)\.
- Ross et al\. \(2011\)Stéphane Ross, Geoffrey J\. Gordon, and J\. Andrew Bagnell\.A reduction of imitation learning and structured prediction to no\-regret online learning\.In*Proceedings of the 14th International Conference on Artificial Intelligence and Statistics \(AISTATS\)*, pp\. 627–635, 2011\.
- Rudin et al\. \(2021\)Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter\.Learning to walk in minutes using massively parallel deep reinforcement learning\.In Aleksandra Faust, David Hsu, and Gerhard Neumann \(eds\.\),*Conference on Robot Learning, 8\-11 November 2021, London, UK*, volume 164 of*Proceedings of Machine Learning Research*, pp\. 91–100\. PMLR, 2021\.
- Saunders et al\. \(2018\)William Saunders, Girish Sastry, Andreas Stuhlmüller, and Owain Evans\.Trial without error: Towards safe reinforcement learning via human intervention\.In*Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems*, AAMAS ’18, pp\. 2067–2069, Richland, SC, 2018\. International Foundation for Autonomous Agents and Multiagent Systems\.
- Schulman et al\. \(2016\)John Schulman, Philipp Moritz, Sergey Levine, Michael I\. Jordan, and Pieter Abbeel\.High\-dimensional continuous control using generalized advantage estimation\.In*International Conference on Learning Representations \(ICLR\)*, 2016\.
- Schulman et al\. \(2017\)John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov\.Proximal policy optimization algorithms\.*CoRR*, abs/1707\.06347, 2017\.
- Silver et al\. \(2014\)David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller\.Deterministic policy gradient algorithms\.In*Proceedings of the 31st International Conference on Machine Learning \(ICML\)*, 2014\.
- Smith et al\. \(2023a\)Laura M\. Smith, J\. Chase Kew, Tianyu Li, Linda Luu, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine\.Learning and adapting agile locomotion skills by transferring experience\.In Kostas E\. Bekris, Kris Hauser, Sylvia L\. Herbert, and Jingjin Yu \(eds\.\),*Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10\-14, 2023*, 2023a\.doi:10\.15607/RSS\.2023\.XIX\.051\.
- Smith et al\. \(2023b\)Laura M\. Smith, Ilya Kostrikov, and Sergey Levine\.A walk in the park: Learning to walk in 20 minutes with model\-free reinforcement learning\.In*2023 IEEE International Conference on Robotics and Automation \(ICRA\)*, 2023b\.
- Smith et al\. \(2024\)Laura M\. Smith, Yunhao Cao, and Sergey Levine\.Grow your limits: Continuous improvement with real\-world rl for robotic locomotion\.In*2024 IEEE International Conference on Robotics and Automation \(ICRA\)*, pp\. 10829–10836, 2024\.
- Sootla et al\. \(2022\)Aivar Sootla, Alexander I Cowen\-Rivers, Taher Jafferjee, Ziyan Wang, David H Mguni, Jun Wang, and Haitham Ammar\.Saute RL: Almost surely safe reinforcement learning using state augmentation\.In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato \(eds\.\),*Proceedings of the 39th International Conference on Machine Learning*, volume 162 of*Proceedings of Machine Learning Research*, pp\. 20423–20443\. PMLR, 17–23 Jul 2022\.URL[https://proceedings\.mlr\.press/v162/sootla22a\.html](https://proceedings.mlr.press/v162/sootla22a.html)\.
- Spencer et al\. \(2020\)Jonathan Spencer, Sanjiban Choudhury, Matthew Barnes, Matthew Schmittle, Mung Chiang, Peter Ramadge, and Sidd Srinivasa\.Learning from interventions: Human\-robot interaction as both explicit and implicit feedback\.In*Proceedings of Robotics: Science and Systems \(RSS\)*, 2020\.
- Srinivasan et al\. \(2020\)Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn\.Learning to be safe: Deep RL with a safety critic\.*CoRR*, abs/2010\.14603, 2020\.
- Stooke et al\. \(2020\)Adam Stooke, Joshua Achiam, and Pieter Abbeel\.Responsive safety in reinforcement learning by PID Lagrangian methods\.In*Proceedings of the 37th International Conference on Machine Learning \(ICML\)*, 2020\.
- Sutton et al\. \(1999\)Richard S\. Sutton, Doina Precup, and Satinder Singh\.Between MDPs and semi\-MDPs: A framework for temporal abstraction in reinforcement learning\.*Artificial Intelligence*, 112\(1\-2\):181–211, 1999\.
- Tan et al\. \(2018\)Jie Tan, Tingnan Zhang, Erwin Coumans, Atil Iscen, Yunfei Bai, Danijar Hafner, Steven Bohez, and Vincent Vanhoucke\.Sim\-to\-real: Learning agile locomotion for quadruped robots\.In Hadas Kress\-Gazit, Siddhartha S\. Srinivasa, Tom Howard, and Nikolay Atanasov \(eds\.\),*Robotics: Science and Systems XIV, Carnegie Mellon University, Pittsburgh, Pennsylvania, USA, June 26\-30, 2018*, 2018\.doi:10\.15607/RSS\.2018\.XIV\.010\.
- Tessler et al\. \(2018\)Chen Tessler, Daniel J\. Mankowitz, and Shie Mannor\.Reward constrained policy optimization\.*CoRR*, abs/1805\.11074, 2018\.
- Thananjeyan et al\. \(2021\)Brijen Thananjeyan, Ashwin Balakrishna, Suraj Nair, Michael Luo, Krishnan Srinivasan, Minho Hwang, Joseph E\. Gonzalez, Julian Ibarz, Chelsea Finn, and Ken Goldberg\.Recovery RL: Safe reinforcement learning with learned recovery zones\.*IEEE Robotics and Automation Letters*, 6\(3\):4915–4922, 2021\.doi:10\.1109/LRA\.2021\.3070252\.
- Todorov et al\. \(2012\)Emanuel Todorov, Tom Erez, and Yuval Tassa\.Mujoco: A physics engine for model\-based control\.In*2012 IEEE/RSJ International Conference on Intelligent Robots and Systems*, pp\. 5026–5033\. IEEE, 2012\.doi:10\.1109/IROS\.2012\.6386109\.
- Towers et al\. \(2023\)Mark Towers, Jordan K\. Terry, Ariel Kwiatkowski, John U\. Balis, Gianluca de Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Arjun KG, Markus Krimmel, Rodrigo Perez\-Vicente, Andrea Pierré, Sander Schulhoff, Jun Jet Tai, Andrew Tan Jin Shen, and Omar G\. Younis\.Gymnasium, March 2023\.
- Turchetta et al\. \(2020\)Matteo Turchetta, Andrey Kolobov, Shital Shah, Andreas Krause, and Alekh Agarwal\.Safe reinforcement learning via curriculum induction\.In*Advances in Neural Information Processing Systems 33 \(NeurIPS\)*, 2020\.
- Uchendu et al\. \(2023\)Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman\.Jump\-start reinforcement learning\.In*Proceedings of the 40th International Conference on Machine Learning \(ICML\)*, 2023\.
- Wachi et al\. \(2024\)Akifumi Wachi, Xun Shen, and Yanan Sui\.A survey of constraint formulations in safe reinforcement learning\.In Kate Larson \(ed\.\),*Proceedings of the Thirty\-Third International Joint Conference on Artificial Intelligence, IJCAI\-24*, pp\. 8262–8271\. International Joint Conferences on Artificial Intelligence Organization, 8 2024\.doi:10\.24963/ijcai\.2024/913\.URL[https://doi\.org/10\.24963/ijcai\.2024/913](https://doi.org/10.24963/ijcai.2024/913)\.Survey Track\.
- Wagener et al\. \(2021\)Nolan C\. Wagener, Byron Boots, and Ching\-An Cheng\.Safe reinforcement learning using advantage\-based intervention\.In*Proceedings of the 38th International Conference on Machine Learning \(ICML\)*, volume 139 of*Proceedings of Machine Learning Research*, pp\. 10630–10640\. PMLR, 2021\.
- Yang et al\. \(2022\)Tsung\-Yen Yang, Tingnan Zhang, Linda Luu, Sehoon Ha, Jie Tan, and Wenhao Yu\.Safe reinforcement learning for legged locomotion\.In*2022 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\)*, pp\. 2454–2461, 2022\.doi:10\.1109/IROS47612\.2022\.9982038\.
- Zakka et al\. \(2022\)Kevin Zakka, Yuval Tassa, and MuJoCo Menagerie Contributors\.MuJoCo Menagerie: A collection of high\-quality models for the MuJoCo physics engine, 2022\.URL[http://github\.com/google\-deepmind/mujoco\_menagerie](http://github.com/google-deepmind/mujoco_menagerie)\.
- Zhao et al\. \(2023\)Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu\.State\-wise safe reinforcement learning: A survey\.In Edith Elkind \(ed\.\),*Proceedings of the Thirty\-Second International Joint Conference on Artificial Intelligence, IJCAI\-23*, pp\. 6814–6822\. International Joint Conferences on Artificial Intelligence Organization, 8 2023\.doi:10\.24963/ijcai\.2023/763\.URL[https://doi\.org/10\.24963/ijcai\.2023/763](https://doi.org/10.24963/ijcai.2023/763)\.Survey Track\.

## Appendix AFull Proofs

### A\.1Proof of[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\(unbiased gradient ofJmix​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\)\)

###### Proof\.

Starting from the definition,Jmix​\(θ\)=∫pθmix​r​\(τ\)​𝑑τ\.J^\{\\mathrm\{mix\}\}\(\\theta\)=\\int p\_\{\\theta\}^\{\\mathrm\{mix\}\}\\,r\(\\tau\)\\,d\\tau\.Under[˜1](https://arxiv.org/html/2607.08925#Thmassumption1), differentiation and integration commute:∇θJmix​\(θ\)=∫∇θpθmix​r​\(τ\)​𝑑τ\.\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\(\\theta\)=\\int\\nabla\_\{\\theta\}p\_\{\\theta\}^\{\\mathrm\{mix\}\}\\,r\(\\tau\)\\,d\\tau\.Factorpθmixp\_\{\\theta\}^\{\\mathrm\{mix\}\}explicitly by partition membership of the visited states:

pθmix=p​\(s1\)⏟no​θ⋅∏t:st∈ℳπθ​\(at∣st\)⏟onlyθ\-dependent factor⋅∏t:st∉ℳμ​\(at∣st\)⏟no​θ⋅∏t=1TP​\(st\+1∣st,at\)⏟no​θ,p\_\{\\theta\}^\{\\mathrm\{mix\}\}=\\underbrace\{p\(s\_\{1\}\)\}\_\{\\text\{no \}\\theta\}\\cdot\\underbrace\{\\prod\_\{t:s\_\{t\}\\in\\mathcal\{M\}\}\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\}\_\{\\text\{only $\\theta$\-dependent factor\}\}\\cdot\\underbrace\{\\prod\_\{t:s\_\{t\}\\notin\\mathcal\{M\}\}\\mu\(a\_\{t\}\\mid s\_\{t\}\)\}\_\{\\text\{no \}\\theta\}\\cdot\\underbrace\{\\prod\_\{t=1\}^\{T\}P\(s\_\{t\+1\}\\mid s\_\{t\},a\_\{t\}\)\}\_\{\\text\{no \}\\theta\},\(10\)whereμ\(⋅\|s\)\\mu\(\\cdot\|s\)denotes the \(possibly Dirac\)θ\\theta\-independent action measure ats∉ℳs\\notin\\mathcal\{M\}\. Equation \([10](https://arxiv.org/html/2607.08925#A1.E10)\) is an equality of measures on trajectory space and does*not*requireμ\\muto have a Lebesgue density\. Deterministic, stochastic, point\-mass at a hand\-coded action, and pre\-trained\-policy measures are all covered\. Only theℳ\\mathcal\{M\}\-step factor depends onθ\\theta\. By the product rule and the score\-function identity applied factor\-wise,

∇θpθmix=pθmix​∑t:st∈ℳ∇θlog⁡πθ​\(at∣st\)\.\\nabla\_\{\\theta\}p\_\{\\theta\}^\{\\mathrm\{mix\}\}=p\_\{\\theta\}^\{\\mathrm\{mix\}\}\\,\\sum\_\{t:s\_\{t\}\\in\\mathcal\{M\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\.\(11\)No gradient ofμ\\muappears, becauseμ\\mucarries noθ\\theta\-dependence; at no point do we evaluatelog\\logof a Dirac measure\. Substituting back,

∇θJmix​\(θ\)=𝔼τ∼pθmix​\[\(∑t:st∈ℳ∇θlog⁡πθ​\(at∣st\)\)​r​\(τ\)\]\.∎\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\(\\theta\)=\\mathbb\{E\}\_\{\\tau\\sim p\_\{\\theta\}^\{\\mathrm\{mix\}\}\}\\\!\\left\[\\Big\(\\sum\_\{t:s\_\{t\}\\in\\mathcal\{M\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\Big\)\\,r\(\\tau\)\\right\]\.\\qed

Corollaries \([Corollaries˜2](https://arxiv.org/html/2607.08925#Thmtheorem2),[8](https://arxiv.org/html/2607.08925#Thmtheorem8)and[9](https://arxiv.org/html/2607.08925#Thmtheorem9)\)\.The proof never used the specific form ofμ\\mu, only thatμ\\mucarries noθ\\theta\-dependence, so each corollary follows by simply instantiating the pair\(ℳ,μ\)\(\\mathcal\{M\},\\mu\)\. The safe\-region intervention case \(ℳ=ℛ\\mathcal\{M\}=\\mathcal\{R\},μ=πrec\\mu=\\pi^\{\\mathrm\{rec\}\}\) is the body of this paper\. The Jump\-StartRLcase \(ℳ=\{\(s,t\):t≥h\}\\mathcal\{M\}=\\\{\(s,t\):t\\geq h\\\}for a per\-trajectory handoff stephh,μ=πteach\\mu=\\pi^\{\\mathrm\{teach\}\}for a fixed teacher\) inherits the same factorization: the teacher isθ\\theta\-independent, the handoff step is data\-dependent but notθ\\theta\-dependent, and the score function evaluates only on the student\-controlled tail of each trajectory\. The state\-triggered shieldedRLcase \(ℳ=\{s:Σ​does not engage at​s\}\\mathcal\{M\}=\\\{s:\\Sigma\\text\{ does not engage at \}s\\\},μ​\(a∣s\)=δ​\(a−σ​\(s\)\)\\mu\(a\\mid s\)=\\delta\(a\-\\sigma\(s\)\)for the shield’s corrective actionσ\\sigma\) inherits the same factorization withμ\\mua Dirac measure that the proof’s factor\-level differentiation handles directly\. The common payoff is the same in all three: theISfix is ill\-defined whenμ\\muhas no density at the relevant states, whereas the factor\-level proof never forms the ratio and so bypasses the need for a density onμ\\mu\. We state the two non\-safe\-region specializations formally for reference\.

###### Corollary 8\(Jump\-StartRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)\)\.

Letπteach\\pi^\{\\mathrm\{teach\}\}be a fixed teacher policy andhha per\-trajectory handoff step\. Takeℳ=\{\(s,t\):t≥h\}\\mathcal\{M\}=\\\{\(s,t\):t\\geq h\\\}\(states reached after handoff; the timestepttis absorbed into the state, standard for finite\-horizon MDPs, soℳ\\mathcal\{M\}is a subset of the augmented state space\) andμ=πteach\\mu=\\pi^\{\\mathrm\{teach\}\}\. Then[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)gives an unbiased gradient of the student’s contribution to the mixed return, where the score function is evaluated only on the student\-controlled tail of each trajectory\.

###### Corollary 9\(State\-triggered shieldedRL\)\.

LetΣ\\Sigmabe a*state\-triggered*shield, whose engagement is decided by the state alone and which, at unsafe states, replaces the proposed action with aθ\\theta\-independent corrective actionσ​\(s\)\\sigma\(s\)\. Takeℳ=\{s:Σ​does not engage at​s\}\\mathcal\{M\}=\\\{s:\\Sigma\\text\{ does not engage at \}s\\\}andμ​\(a∣s\)=δ​\(a−σ​\(s\)\)\\mu\(a\\mid s\)=\\delta\(a\-\\sigma\(s\)\)\. Then[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)gives an unbiased gradient evaluated only at unshielded states, with no density required on the shield’s correction\.

Action\-triggered shields, as inAlshiekh et al\. \([2018](https://arxiv.org/html/2607.08925#bib.bib4)\), decide the override from the proposed actiona∼πθ\(⋅∣s\)a\\sim\\pi\_\{\\theta\}\(\\cdot\\mid s\): at states where only some actions are unsafe, the executed\-action distribution mixesπθ\\pi\_\{\\theta\}restricted to safe actions with the correction weighted by theθ\\theta\-dependent probability of proposing an unsafe action, so it depends onθ\\thetaand falls outside[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)’s hypothesis of aθ\\theta\-independentμ\\mu\.

Truncated importance sampling\.One might hope that the singularity is only a problem for the naiveISestimator and that a more careful clipped estimator escapes it\. It does not\. Truncated\-ISestimators such as V\-trace\(Espeholt et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib13)\)and Retrace\(Munos et al\.,[2016](https://arxiv.org/html/2607.08925#bib.bib34)\)reach the mixed\-policy gradient by clipping the importance ratiopθ/pθmixp\_\{\\theta\}/p\_\{\\theta\}^\{\\mathrm\{mix\}\}, but that ratio is still undefined at the deterministic\-recovery steps whereμ\\muis a point mass\. Clipping bounds variance downstream of the singularity rather than removing it\. The factor\-level proof above avoids the ratio entirely, which is exactly why it extends to the deterministic recovery policy our method uses\.

### A\.2Proof of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\(objective\-gap bound\)

This subsection and the remainder of the appendix work in the infinite\-horizon discounted convention fixed in[Section˜3](https://arxiv.org/html/2607.08925#S3), under whichVπθmixV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\},QπθmixQ^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\},AπθmixA^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}, and the normalized discounted visitationsνπθ\\nu^\{\\pi\_\{\\theta\}\},νπθmix\\nu^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}are stationary\.

###### Proof\.

[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)shows we optimizeJmixJ^\{\\mathrm\{mix\}\}rather than the deployment objectiveJJ, so the question is how far apart the two can be\. We bound the gap by routing it through the states where the two policies actually differ\. By the Performance Difference Lemma\(Kakade & Langford,[2002](https://arxiv.org/html/2607.08925#bib.bib26)\),

J​\(θ\)−Jmix​\(θ\)=11−γ​𝔼s∼νπθ,a∼πθ\(⋅\|s\)​\[Aπθmix​\(s,a\)\],J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)=\\frac\{1\}\{1\-\\gamma\}\\,\\mathbb\{E\}\_\{s\\sim\\nu^\{\\pi\_\{\\theta\}\},\\,a\\sim\\pi\_\{\\theta\}\(\\cdot\|s\)\}\\\!\\left\[A^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)\\right\],\(12\)whereAπθmix​\(s,a\)=Qπθmix​\(s,a\)−Vπθmix​\(s\)A^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)=Q^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)\-V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\)\. Split the expectation by the safe region:

J​\(θ\)−Jmix​\(θ\)=11−γ​\(𝔼s∈ℛ​\[⋅\]⏟=0\+𝔼s∉ℛ​\[⋅\]\)\.J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)=\\frac\{1\}\{1\-\\gamma\}\\\!\\left\(\\underbrace\{\\mathbb\{E\}\_\{s\\in\\mathcal\{R\}\}\[\\cdot\]\}\_\{=\\,0\}\+\\mathbb\{E\}\_\{s\\notin\\mathcal\{R\}\}\[\\cdot\]\\right\)\.Ons∈ℛs\\in\\mathcal\{R\}, bothπθ\\pi\_\{\\theta\}andπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}sample fromπθ\\pi\_\{\\theta\}, so the inner expectation𝔼a∼πθ\(⋅\|s\)​\[Aπθmix​\(s,a\)\]=0\\mathbb\{E\}\_\{a\\sim\\pi\_\{\\theta\}\(\\cdot\|s\)\}\[A^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)\]=0by definition ofVπθmixV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\. Ons∉ℛs\\notin\\mathcal\{R\},\|Aπθmix​\(s,a\)\|≤\|Qπθmix​\(s,a\)\|\+\|Vπθmix​\(s\)\|≤2​rmax/\(1−γ\)\|A^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)\|\\leq\|Q^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s,a\)\|\+\|V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\)\|\\leq 2r\_\{\\max\}/\(1\-\\gamma\)since both\|Q\|\|Q\|and\|V\|\|V\|are bounded byrmax/\(1−γ\)r\_\{\\max\}/\(1\-\\gamma\)\. Therefore

\|J​\(θ\)−Jmix​\(θ\)\|≤11−γ⋅2​rmax1−γ⋅Prs∼νπθ⁡\[s∉ℛ\]=2​rmax\(1−γ\)2​β​\(θ,d\)\.∎\|J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)\|\\leq\\frac\{1\}\{1\-\\gamma\}\\cdot\\frac\{2r\_\{\\max\}\}\{1\-\\gamma\}\\cdot\\Pr\_\{s\\sim\\nu^\{\\pi\_\{\\theta\}\}\}\[s\\notin\\mathcal\{R\}\]=\\frac\{2r\_\{\\max\}\}\{\(1\-\\gamma\)^\{2\}\}\\,\\beta\(\\theta,d\)\.\\qed

Bound on the out\-of\-region rateβ​\(θ,d\)\\beta\(\\theta,d\)\([Section˜4\.2](https://arxiv.org/html/2607.08925#S4.SS2)\)\.The gap above scales withβ​\(θ,d\)\\beta\(\\theta,d\), so the bound is only useful onceβ​\(θ,d\)\\beta\(\\theta,d\)itself is controlled\.[Section˜4\.2](https://arxiv.org/html/2607.08925#S4.SS2)usesβ​\(θ,d\)≤η​γ/\(1−γ\)≤η/\(1−γ\)\\beta\(\\theta,d\)\\leq\\eta\\gamma/\(1\-\\gamma\)\\leq\\eta/\(1\-\\gamma\), which we now derive from the per\-step invariance slack, under the additional hypothesis that episodes start inside the safe region,supp⁡p​\(s1\)⊆ℛ​\(d\)\\operatorname\{supp\}\\,p\(s\_\{1\}\)\\subseteq\\mathcal\{R\}\(d\)\(otherwise the initial step alone contributes\(1−γ\)​Pr⁡\[s1∉ℛ\]\(1\-\\gamma\)\\Pr\[s\_\{1\}\\notin\\mathcal\{R\}\]toβ​\(θ,d\)\\beta\(\\theta,d\)even withη=0\\eta=0\)\. All three of our environments satisfy the hypothesis: episodes reset to an upright pose inside the region\. Starting from a reachable initial states0∈ℛs\_\{0\}\\in\\mathcal\{R\}\(this derivation counts steps from zero, sos0s\_\{0\}here is thes1s\_\{1\}of the trajectory notation\),[˜2](https://arxiv.org/html/2607.08925#Thmassumption2)givesPr⁡\[s0,…,st∈ℛ\]≥\(1−η\)t\\Pr\[s\_\{0\},\\dots,s\_\{t\}\\in\\mathcal\{R\}\]\\geq\(1\-\\eta\)^\{t\}\. Since\{st∈ℛ\}⊇\{s0,…,st∈ℛ\}\\\{s\_\{t\}\\in\\mathcal\{R\}\\\}\\supseteq\\\{s\_\{0\},\\dots,s\_\{t\}\\in\\mathcal\{R\}\\\}, the marginal in\-region probability dominates the all\-stay probability,Pr⁡\[st∈ℛ\]≥\(1−η\)t\\Pr\[s\_\{t\}\\in\\mathcal\{R\}\]\\geq\(1\-\\eta\)^\{t\}, hencePr⁡\[st∉ℛ\]≤1−\(1−η\)t\\Pr\[s\_\{t\}\\notin\\mathcal\{R\}\]\\leq 1\-\(1\-\\eta\)^\{t\}\. With the discounted state\-visitationβ​\(θ,d\)=\(1−γ\)​∑t≥0γt​Pr⁡\[st∉ℛ\]\\beta\(\\theta,d\)=\(1\-\\gamma\)\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\Pr\[s\_\{t\}\\notin\\mathcal\{R\}\],

β​\(θ,d\)≤\(1−γ\)​∑t≥0γt​\(1−\(1−η\)t\)=η​γ1−γ​\(1−η\)≤η​γ1−γ≤η1−γ,\\beta\(\\theta,d\)\\leq\(1\-\\gamma\)\\sum\_\{t\\geq 0\}\\gamma^\{t\}\\big\(1\-\(1\-\\eta\)^\{t\}\\big\)=\\frac\{\\eta\\gamma\}\{1\-\\gamma\(1\-\\eta\)\}\\leq\\frac\{\\eta\\gamma\}\{1\-\\gamma\}\\leq\\frac\{\\eta\}\{1\-\\gamma\},the last step using1−γ​\(1−η\)≥1−γ1\-\\gamma\(1\-\\eta\)\\geq 1\-\\gammaandγ≤1\\gamma\\leq 1\.

Train\-to\-deploy reward gap \(footnote in[Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3)\)\.The training\-signal convention of[Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\(zero per\-step reward during recovery, one\-time terminal rewardrtermr\_\{\\mathrm\{term\}\}\) and the unmodified task reward differ only at states outsideℛ\\mathcal\{R\}\(and through the one\-timertermr\_\{\\mathrm\{term\}\}\)\. Bounding the per\-step discrepancy byrmax\+\|rterm\|r\_\{\\max\}\+\|r\_\{\\mathrm\{term\}\}\|on those steps and summing the discounted visitation underπθ\\pi\_\{\\theta\}, the returns the two conventions induce underπθ\\pi\_\{\\theta\}differ by at most\(rmax\+\|rterm\|\)​β​\(θ,d\)/\(1−γ\)\(r\_\{\\max\}\+\|r\_\{\\mathrm\{term\}\}\|\)\\,\\beta\(\\theta,d\)/\(1\-\\gamma\), so the train\-to\-deploy chain is controlled by the sameβ​\(θ,d\)\\beta\(\\theta,d\)as[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\.

### A\.3Proof of[Corollary˜5](https://arxiv.org/html/2607.08925#Thmtheorem5)\(conditional fixed point\)

###### Proof\.

The gap bound of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)leavesβ​\(θ,d\)\\beta\(\\theta,d\)free, so the natural endpoint of the analysis is the case that drivesβ​\(θ,d\)\\beta\(\\theta,d\)to zero\. The corollary splits into two readings of when that happens\.

Reading \(R\-A\):Ifℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)covers the reachable state space, thenPr⁡\[s∉ℛ​\(dmax\)\]=0\\Pr\[s\\notin\\mathcal\{R\}\(d\_\{\\max\}\)\]=0under*any*action distribution, includingπθ\\pi\_\{\\theta\}alone\. Henceβ​\(θ,dmax\)=0\\beta\(\\theta,d\_\{\\max\}\)=0, and[Equation˜5](https://arxiv.org/html/2607.08925#S4.E5)givesJmix​\(θ\)=J​\(θ\)J^\{\\mathrm\{mix\}\}\(\\theta\)=J\(\\theta\)exactly\. Furthermore,πθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}is identical toπθ\\pi\_\{\\theta\}on the reachable support ofpθ​\(τ\)p\_\{\\theta\}\(\\tau\)\(recovery is never engaged\)\.

Reading \(R\-B\):Ifℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)is strictly contained in the reachable state space, the theory no longer forcesπθ\\pi\_\{\\theta\}to remain inℛ\\mathcal\{R\}, so its invariance slackη⋆:=η​\(θ,dmax\)≥0\\eta\_\{\\star\}:=\\eta\(\\theta,d\_\{\\max\}\)\\geq 0\([˜2](https://arxiv.org/html/2607.08925#Thmassumption2)\) need not be zero\. Substitutingβ≤η⋆/\(1−γ\)\\beta\\leq\\eta\_\{\\star\}/\(1\-\\gamma\)\(derived above under the initial\-state hypothesissupp⁡p​\(s1\)⊆ℛ\\operatorname\{supp\}\\,p\(s\_\{1\}\)\\subseteq\\mathcal\{R\}\) into[Equation˜5](https://arxiv.org/html/2607.08925#S4.E5)gives the upper bound\|J​\(θ\)−Jmix​\(θ\)\|≤2​rmax​η⋆/\(1−γ\)3\|J\(\\theta\)\-J^\{\\mathrm\{mix\}\}\(\\theta\)\|\\leq 2r\_\{\\max\}\\eta\_\{\\star\}/\(1\-\\gamma\)^\{3\}, which, unlike \(R\-A\), is not pinned to zero\. Whetherη⋆\\eta\_\{\\star\}\(and hence the gap\) is small is environment\-dependent:[Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)findsη⋆→0\\eta\_\{\\star\}\\to 0onHalfCheetahandAnt, where the gap closes to seed noise, andη⋆\>0\\eta\_\{\\star\}\>0onGo1, where a proportional residual remains\. ∎

### A\.4Proof of[Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\(analytic recovery value\)

###### Proof\.

The gradient and gap results above treatVπθmixV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}as given; what remains is to compute it cheaply on the recovery segments themselves, where the trajectory is no longer under the agent’s control\. By the definition ofVπθmixV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\},Vπθmix​\(st\)=𝔼τ∼pθmix​\(τ\)\|st​\[∑j=0∞γj​rt\+j\]\.V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)=\\mathbb\{E\}\_\{\\tau\\sim p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)\|s\_\{t\}\}\\\!\\left\[\\sum\_\{j=0\}^\{\\infty\}\\gamma^\{j\}r\_\{t\+j\}\\right\]\.Under deterministic dynamics and deterministic recovery, the trajectory starting from a recovery\-triggering statests\_\{t\}is fully determined for the duration of the recovery segment\[t,t\+k\]\[t,t\+k\]\. The expectation collapses\. Two cases:

Recovery success\(st\+k∈ℛs\_\{t\+k\}\\in\\mathcal\{R\}\): the realized return over the segment isGt,k=∑j=0k−1γj​rt\+jG\_\{t,k\}=\\sum\_\{j=0\}^\{k\-1\}\\gamma^\{j\}r\_\{t\+j\}\. Fromst\+ks\_\{t\+k\}the agent resumes underπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}with valueVπθmix​\(st\+k\)V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\+k\}\)\. HenceVπθmix​\(st\)=Gt,k\+γk​Vπθmix​\(st\+k\)V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)=G\_\{t,k\}\+\\gamma^\{k\}V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\+k\}\)\.

Recovery failure\(terminal\): the realized return isGt,kG\_\{t,k\}over thekkdeterministic steps, the episode ends, no further reward accrues\. HenceVπθmix​\(st\)=Gt,kV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)=G\_\{t,k\}\.

Combining gives[Equation˜6](https://arxiv.org/html/2607.08925#S4.E6)\. Our implementation instantiates this general statement under a chosen reward: zero per\-step reward during recovery and a one\-time terminal rewardrtermr\_\{\\mathrm\{term\}\}accrued at the final segment step \(t\+k−1t\{\+\}k\{\-\}1\) on failure\. The terminal penalty is thus the only nonzero contribution to the segment returnGt,k=∑j=0k−1γj​rt\+jG\_\{t,k\}=\\sum\_\{j=0\}^\{k\-1\}\\gamma^\{j\}r\_\{t\+j\}, which is therefore0on success andγk−1​rterm\\gamma^\{k\-1\}r\_\{\\mathrm\{term\}\}on failure, matching[Equation˜6](https://arxiv.org/html/2607.08925#S4.E6)\. ∎

### A\.5Analytic value under stochastic dynamics

The closed form just derived rests on the determinism assumption, so it is worth asking what survives when that assumption is relaxed\.[Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)assumes deterministicPPandπrec\\pi^\{\\mathrm\{rec\}\}\. Under process noise or stochastic recovery, the realizedγk​Vθ​\(st\+k\)\\gamma^\{k\}V\_\{\\theta\}\(s\_\{t\+k\}\)is no longer equal toVπθmix​\(st\)V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)\. It becomes a single Monte\-Carlo realization drawn from the distribution over post\-trigger trajectories, whose variance grows with the noise ofPPandπrec\\pi^\{\\mathrm\{rec\}\}accumulated over the segment\. Two natural extensions preserve the spirit of bypassing critic bootstrapping across recovery\. \(i\) A multi\-sample MC target: re\-roll the recoveryMMtimes fromsts\_\{t\}\(where the simulator supports state\-resets\) and averageGt,km\+γkm​Vθ​\(st\+km\)G\_\{t,k\_\{m\}\}\+\\gamma^\{k\_\{m\}\}V\_\{\\theta\}\(s\_\{t\+k\_\{m\}\}\)across theMMrealizations\. This is an unbiased estimator ofVπθmix​\(st\)V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)whose variance falls as1/M1/M\. \(ii\) The single\-sample plug\-in we already use is an unbiased one\-sample MC estimate ofVπθmix​\(st\)V^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\(s\_\{t\}\)\. Its bias is zero in expectation, but its single\-realization noise reaches downstreamGAE\. The bias\-variance trade\-off is favorable in our setting becauseπrec\\pi^\{\\mathrm\{rec\}\}is approximately deterministic in practice \(MPC, greedySAC\), so the per\-segment variance is small\. A formal analysis under controlled stochasticity, and the regime where \(i\) is worth the extra simulator calls, is left to follow\-up\.

### A\.6Proof of[Proposition˜10](https://arxiv.org/html/2607.08925#Thmtheorem10)

The recovery segments that supplied the analytic value also supply state\-action pairs for the compatibility regularizer, and identifying its limiting cases shows that this loss is not an ad\-hoc addition but a familiar imitation objective in disguise\.

###### Proposition 10\(Special cases of the compatibility regularizer\)\.

Letπrec\\pi^\{\\mathrm\{rec\}\}be deterministic, soatrec=πrec​\(st\)a\_\{t\}^\{\\mathrm\{rec\}\}=\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)at recovery\-controlled states, and consider the gate\-set choiceσk≡1\\sigma\_\{k\}\\equiv 1for all segments\.

1. \(a\)Behavioral cloning on the recovery dataset\.Underσk≡1\\sigma\_\{k\}\\equiv 1, Lσ≡1compat​\(θ\)=−λcompatNrec​∑t:st∉ℛlog⁡πθ​\(πrec​\(st\)∣st\),L^\{\\mathrm\{compat\}\}\_\{\\sigma\\equiv 1\}\(\\theta\)=\-\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}\\log\\pi\_\{\\theta\}\\\!\\left\(\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\\mid s\_\{t\}\\right\),\(13\)the \(scaled\) negative log\-likelihood ofπθ\\pi\_\{\\theta\}on the datasetℬ=\{\(st,πrec​\(st\)\):st∉ℛ\}\\mathcal\{B\}=\\\{\(s\_\{t\},\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\):s\_\{t\}\\notin\\mathcal\{R\}\\\}of recovery\-controlled state\-action pairs collected from rollouts ofπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\. Equivalently,Lσ≡1compatL^\{\\mathrm\{compat\}\}\_\{\\sigma\\equiv 1\}is the per\-step behavioral\-cloning objective applied toπrec\\pi^\{\\mathrm\{rec\}\}’s actions at the statesπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}visits during recovery\.
2. \(b\)A region\-restrictedDAggeraggregation step\.The datasetℬ\\mathcal\{B\}is collected as in one iteration ofDAgger\(Ross et al\.,[2011](https://arxiv.org/html/2607.08925#bib.bib42)\)with the teacher queried only on the intervention set: states are visited under the mixed roll\-out policyπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\(predominantly student\-induced up to the safe\-region boundary\), and labels are the controllerπrec\\pi^\{\\mathrm\{rec\}\}’s actions at the recovery\-controlled states only, whereasDAggerlabels every visited state\. MinimizingLσ≡1compatL^\{\\mathrm\{compat\}\}\_\{\\sigma\\equiv 1\}as the per\-iteration training step then reproduces a region\-restricted analogue ofDAgger’s update withπrec\\pi^\{\\mathrm\{rec\}\}in the teacher role andπθ\\pi\_\{\\theta\}as the student\.
3. \(c\)Outcome gate as a designer filter\.The general gateσk∈\{0,1\}\\sigma\_\{k\}\\in\\\{0,1\\\}restrictsℬ\\mathcal\{B\}to recovery segments that re\-enteredℛ\\mathcal\{R\}, discarding the rest\.JSRL\-style mixed\-policy data collection\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)produces the same mixed\-policy roll\-out distribution but does not include an imitation loss\.LcompatL^\{\\mathrm\{compat\}\}withσk≡1\\sigma\_\{k\}\\equiv 1supplies the analogous student\-imitation step that theJSRLalgorithm itself omits\. What is new in the caseσk≢1\\sigma\_\{k\}\\not\\equiv 1is gating imitation of an*external recovery controller*by the*realized success of its multi\-step segment*, as opposed to gating by value estimates of one’s own past actions\(Oh et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib37)\)or of demonstrations\(Nair et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib35)\)\.

###### Proof\.

\(a\)From[Equation˜9](https://arxiv.org/html/2607.08925#S4.E9)withwt=σseg⁡\(t\)w\_\{t\}=\\sigma\_\{\\operatorname\{seg\}\(t\)\}, whereseg⁡\(t\)\\operatorname\{seg\}\(t\)is the index of the recovery segment containingtt\([Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)\), and the assumptionσk≡1\\sigma\_\{k\}\\equiv 1,

Lσ≡1compat​\(θ\)=−λcompatNrec​∑t:st∉ℛ1⋅log⁡πθ​\(atrec∣st\)=−λcompatNrec​∑t:st∉ℛlog⁡πθ​\(πrec​\(st\)∣st\),L^\{\\mathrm\{compat\}\}\_\{\\sigma\\equiv 1\}\(\\theta\)=\-\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}1\\cdot\\log\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\\mid s\_\{t\}\)=\-\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\}\}\\log\\pi\_\{\\theta\}\\\!\\left\(\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\\mid s\_\{t\}\\right\),usingatrec=πrec​\(st\)a\_\{t\}^\{\\mathrm\{rec\}\}=\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)for deterministicπrec\\pi^\{\\mathrm\{rec\}\}\. This is, up to the constant scaleλcompat/Nrec\\lambda\_\{\\mathrm\{compat\}\}/N\_\{\\mathrm\{rec\}\}, the empirical negative log\-likelihood ofπθ\\pi\_\{\\theta\}on the datasetℬ=\{\(st,πrec​\(st\)\):st∉ℛ\}\\mathcal\{B\}=\\\{\(s\_\{t\},\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\):s\_\{t\}\\notin\\mathcal\{R\}\\\}, the \(population\-level\) maximum\-likelihood / behavioral\-cloning objective forπθ\\pi\_\{\\theta\}onℬ\\mathcal\{B\}\.

\(b\)The datasetℬ\\mathcal\{B\}is collected by rolling outπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}, retaining the state\-action pairs at the unsafe\-region timesteps\. Underπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\},πθ\\pi\_\{\\theta\}controls the rollout up to the first exit fromℛ\\mathcal\{R\}\(so the visited boundary statests\_\{t\}hasπθ\\pi\_\{\\theta\}\-induced distribution\), andπrec\\pi^\{\\mathrm\{rec\}\}provides actions thereafter\. This matches the data\-collection rule of a singleDAggeriteration withπrec\\pi^\{\\mathrm\{rec\}\}as the teacher andπθ\\pi\_\{\\theta\}as the student, states visited under the mixed roll\-out policy and labels from the teacher, except that the teacher is queried only at the recovery\-controlled states rather than at every visited state as inRoss et al\. \([2011](https://arxiv.org/html/2607.08925#bib.bib42)\)\. MinimizingLσ≡1compatL^\{\\mathrm\{compat\}\}\_\{\\sigma\\equiv 1\}as the per\-iteration update therefore reproduces a region\-restricted analogue of theDAggeraggregation step\.

\(c\)For generalσk∈\{0,1\}\\sigma\_\{k\}\\in\\\{0,1\\\}, the regularizer is

Lcompat​\(θ\)=−λcompatNrec​∑kσk​∑t:st∉ℛ,seg⁡\(t\)=klog⁡πθ​\(πrec​\(st\)∣st\),L^\{\\mathrm\{compat\}\}\(\\theta\)=\-\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{k\}\\sigma\_\{k\}\\sum\_\{t:s\_\{t\}\\notin\\mathcal\{R\},\\,\\operatorname\{seg\}\(t\)=k\}\\log\\pi\_\{\\theta\}\\\!\\left\(\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\\mid s\_\{t\}\\right\),i\.e\. the behavioral\-cloning objective restricted to the sub\-datasetℬsucc=\{\(st,πrec​\(st\)\):st∉ℛ,σseg⁡\(t\)=1\}\\mathcal\{B\}\_\{\\mathrm\{succ\}\}=\\\{\(s\_\{t\},\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\)\):s\_\{t\}\\notin\\mathcal\{R\},\\,\\sigma\_\{\\operatorname\{seg\}\(t\)\}=1\\\}collected from segments that re\-enteredℛ\\mathcal\{R\}\.JSRL\(Uchendu et al\.,[2023](https://arxiv.org/html/2607.08925#bib.bib62)\)likewise rolls out a mixed teacher\-then\-student policy but trains the student only by reinforcement on the student\-controlled tail\. It does not include a teacher\-imitation step\. Theσk≡1\\sigma\_\{k\}\\equiv 1specialization ofLcompatL^\{\\mathrm\{compat\}\}is the analogous imitation lossJSRLomits\. In theσk≢1\\sigma\_\{k\}\\not\\equiv 1case, the new element is the gating criterion and target: imitation of an external recovery controller gated by the realized success of its multi\-step segment, rather than by value estimates of the agent’s own past actions\(Oh et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib37)\)or of demonstrations\(Nair et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib35)\)\. ∎

### A\.7Sketch: why an observable upper bound onβ\\betais hard in continuous actions

The bound onβ​\(θ,d\)\\beta\(\\theta,d\)above is stated in terms of the per\-step invariance slackη\\eta, which is not directly measurable; a bound in terms of an observable quantity would be far more useful, and we close by explaining why that is hard\. A natural attempt is to boundβ\\betaby the observable recovery rateα\\alphavia a simulation\-lemma argument:β=Prνπθ⁡\[s∉ℛ\]≤α\+TV​\(νπθ,νπθmix\)\.\\beta=\\Pr\_\{\\nu^\{\\pi\_\{\\theta\}\}\}\[s\\notin\\mathcal\{R\}\]\\leq\\alpha\+\\mathrm\{TV\}\(\\nu^\{\\pi\_\{\\theta\}\},\\nu^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}\)\.HereTV​\(⋅,⋅\)\\mathrm\{TV\}\(\\cdot,\\cdot\)denotes total variation \(TV\) distance\. The TV term can in turn be bounded by the expected per\-step TV betweenπθ\\pi\_\{\\theta\}andπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}via the standard simulation lemma\(Kearns et al\.,[2002](https://arxiv.org/html/2607.08925#bib.bib28)\)\. But on unsafe statesπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}collapses to a Dirac \(deterministic recovery\) whileπθ\\pi\_\{\\theta\}remains absolutely continuous\. The per\-step TV at unsafe states is11pointwise, so the simulation\-lemma term contributes a full1/\(1−γ\)\>11/\(1\-\\gamma\)\>1, and the boundβ≤α\+1/\(1−γ\)\\beta\\leq\\alpha\+1/\(1\-\\gamma\)exceeds11and is therefore vacuous \(sinceβ≤1\\beta\\leq 1trivially\)\. Routes that may produce a non\-vacuous bound include \(i\) replacing the deterministic recovery with a smoothed \(LSE / Gaussian\-mixture\) variant whose TV withπθ\\pi\_\{\\theta\}is finite, \(ii\) Wasserstein\-action\-gap bounds under known dynamics Lipschitz constants, and \(iii\) periodic offline evaluation ofπθ\\pi\_\{\\theta\}alone\. We sketch \(i\) below and leave a complete treatment to follow\-up work\.

Sketch via smoothed recovery\.Replaceπrec\(⋅\|s\)=δ\(a−a¯\(s\)\)\\pi^\{\\mathrm\{rec\}\}\(\\cdot\|s\)=\\delta\(a\-\\bar\{a\}\(s\)\)withπ~rec\(⋅\|s\)=𝒩\(a¯\(s\),σrec2I\)\\tilde\{\\pi\}\_\{\\mathrm\{rec\}\}\(\\cdot\|s\)=\\mathcal\{N\}\(\\bar\{a\}\(s\),\\sigma\_\{\\mathrm\{rec\}\}^\{2\}I\)for smallσrec\\sigma\_\{\\mathrm\{rec\}\}\. Then bothπθ\(⋅\|s\)\\pi\_\{\\theta\}\(\\cdot\|s\)andπ~rec\(⋅\|s\)\\tilde\{\\pi\}\_\{\\mathrm\{rec\}\}\(\\cdot\|s\)are absolutely continuous and, when the scales are matched \(σrec≈σθ​\(s\)\\sigma\_\{\\mathrm\{rec\}\}\\approx\\sigma\_\{\\theta\}\(s\), withσθ\\sigma\_\{\\theta\}the main policy’s standard deviation\), the Gaussian TV bound givesTV​\(πθ,π~rec\)≤const⋅‖μθ​\(s\)−a¯​\(s\)‖/σrec\\mathrm\{TV\}\(\\pi\_\{\\theta\},\\tilde\{\\pi\}\_\{\\mathrm\{rec\}\}\)\\leq\\mathrm\{const\}\\cdot\\\|\\mu\_\{\\theta\}\(s\)\-\\bar\{a\}\(s\)\\\|/\\sigma\_\{\\mathrm\{rec\}\}at unsafe states up to a scale\-mismatch term \(with mismatched covariances the TV is bounded away from zero even at equal means\), so the per\-step TV is integrable under these stated conditions\. The constant, however, grows with the action dimension, so even this smoothed bound loosens in high dimensions, the same mechanism by which the importance weights collapse on the 12\-dimensionalGo1action space \([Section˜E\.13](https://arxiv.org/html/2607.08925#A5.SS13)\)\. Moreover, the TV to a fixed\-scaleπθ\\pi\_\{\\theta\}approaches one asσrec→0\\sigma\_\{\\mathrm\{rec\}\}\\to 0, so the smoothing scale trades fidelity to the deployed deterministic recovery against tightness of the bound; the sketch therefore yields an observable upper\-bound proxy forβ\\betaonly under the matched\-scale conditions above, not a general bound\.

## Appendix BAlgorithm Details

### B\.1Full pseudocode

[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)gives the complete SafeExplorer update\. The per\-environment curriculum grows the safe\-region radius linearly,diter​\(u\)=d0\+u−1N​dmaxd\_\{\\mathrm\{iter\}\}\(u\)=d\_\{0\}\+\\frac\{u\-1\}\{N\}\\,d\_\{\\max\}at updateuuofNN, withd0=0\.01d\_\{0\}=0\.01onHalfCheetahandAntandd0=0\.05d\_\{0\}=0\.05onGo1, up to the per\-environmentdmaxd\_\{\\max\}values in[Table˜4](https://arxiv.org/html/2607.08925#A3.T4)\([Appendix˜C](https://arxiv.org/html/2607.08925#A3)\)\.

Algorithm 1SafeExplorer:PPOwith safe\-step masking, analytic recovery V, and hardCθ​\(τ\)C\_\{\\theta\}\(\\tau\)\.1:Main policy

πθ\\pi\_\{\\theta\}, recovery policy

πrec\\pi^\{\\mathrm\{rec\}\}, value function

VθV\_\{\\theta\}, safe\-region radius schedule

diterd\_\{\\mathrm\{iter\}\}, total updates

NN, rollout length

TroT\_\{\\mathrm\{ro\}\}, num envs

EE, compatibility coef

λcompat\\lambda\_\{\\mathrm\{compat\}\}, learning rate

ηlr\\eta\_\{\\mathrm\{lr\}\}, terminal reward convention

rtermr\_\{\\mathrm\{term\}\}\([Section˜C\.2](https://arxiv.org/html/2607.08925#A3.SS2)\)\.

2:for

update=1,…,N\\mathrm\{update\}=1,\\ldots,Ndo

3:

d←diter​\(update\)d\\leftarrow d\_\{\\mathrm\{iter\}\}\(\\mathrm\{update\}\)
4:Rollout phase:for

TroT\_\{\\mathrm\{ro\}\}steps in each of

EEparallel envs,

5:if

st∈ℛ​\(d\)s\_\{t\}\\in\\mathcal\{R\}\(d\):

at∼πθ\(⋅\|st\)a\_\{t\}\\sim\\pi\_\{\\theta\}\(\\cdot\|s\_\{t\}\), store

\(st,at,rt,log⁡πθ​\(at\|st\),Vθ​\(st\)\)\(s\_\{t\},a\_\{t\},r\_\{t\},\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\),V\_\{\\theta\}\(s\_\{t\}\)\)
6:if

st∉ℛ​\(d\)s\_\{t\}\\notin\\mathcal\{R\}\(d\):

at=πrec​\(st\)a\_\{t\}=\\pi^\{\\mathrm\{rec\}\}\(s\_\{t\}\),

𝚒𝚜​\_​𝚛𝚎𝚌t←1\\,\\mathtt\{is\\\_rec\}\_\{t\}\\leftarrow 1
7:store

\(st,at,rt,log⁡πθ​\(at\|st\),Vθ​\(st\)\)\(s\_\{t\},a\_\{t\},r\_\{t\},\\log\\pi\_\{\\theta\}\(a\_\{t\}\|s\_\{t\}\),V\_\{\\theta\}\(s\_\{t\}\)\)⊳\\trianglerightlog⁡πθ​\(atrec\|st\)\\log\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\|s\_\{t\}\)feedsLcompatL^\{\\mathrm\{compat\}\}only; never enters thePPOratio \(safe\-step mask\)\.

8:Analytic\-V overwrite \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\):for each recovery segment

\[tkstart,tkend\]\[t\_\{k\}^\{\\mathrm\{start\}\},t\_\{k\}^\{\\mathrm\{end\}\}\]of length

ℓk\\ell\_\{k\},

9:if segment ended in re\-entry:

Vθ​\(stkstart\)←γℓk​Vθ​\(stkend\)V\_\{\\theta\}\(s\_\{t\_\{k\}^\{\\mathrm\{start\}\}\}\)\\leftarrow\\gamma^\{\\ell\_\{k\}\}V\_\{\\theta\}\(s\_\{t\_\{k\}^\{\\mathrm\{end\}\}\}\)⊳\\trianglerightthe segment returnGt,k=0G\_\{t,k\}\{=\}0under the zeroed\-recovery\-reward convention,[Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)

10:if segment ended in termination:

Vθ​\(stkstart\)←γℓk−1​rtermV\_\{\\theta\}\(s\_\{t\_\{k\}^\{\\mathrm\{start\}\}\}\)\\leftarrow\\gamma^\{\\ell\_\{k\}\-1\}r\_\{\\mathrm\{term\}\}⊳\\trianglerightthe terminal reward; recovery steps are zeroed, so this isGt,kG\_\{t,k\}on failure

11:if segment cut off by rollout truncation: treat as re\-entry at the truncation boundary⊳\\trianglerighttentative success with a bootstrapped value,[Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4)

12:Advantage estimation:compute

A^full\\hat\{A\}^\{\\mathrm\{full\}\}viaGAEon the overwritten value sequence; the value target is

R^t=A^tfull\+Vθ​\(st\)\\hat\{R\}\_\{t\}=\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\}\+V\_\{\\theta\}\(s\_\{t\}\)\.

13:Outcome gates:for each recovery segment

kk, set

σk←𝟙​\[segment ended in re\-entry\]\\sigma\_\{k\}\\leftarrow\\mathbb\{1\}\[\\text\{segment ended in re\-entry\}\]\.

14:forepoch

=1,…,K=1,\\ldots,Kdo

15:forminibatch

⊂\\subsetrolloutdo

16:

Tℛ←\{t:𝚒𝚜​\_​𝚛𝚎𝚌t=0\}T\_\{\\mathcal\{R\}\}\\leftarrow\\\{t:\\mathtt\{is\\\_rec\}\_\{t\}=0\\\}\(safe\-step mask\)

17:

LsafePPO←−1\|Tℛ\|​∑t∈Tℛmin⁡\(ρt​A^tfull,clip​\(ρt\)​A^tfull\)L^\{\\mathrm\{PPO\}\}\_\{\\mathrm\{safe\}\}\\leftarrow\-\\frac\{1\}\{\|T\_\{\\mathcal\{R\}\}\|\}\\sum\_\{t\\in T\_\{\\mathcal\{R\}\}\}\\min\(\\rho\_\{t\}\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\},\\,\\mathrm\{clip\}\(\\rho\_\{t\}\)\\hat\{A\}\_\{t\}^\{\\mathrm\{full\}\}\)
18:

LV←1Tro​∑t\(Vθ​\(st\)−R^t\)2L^\{V\}\\leftarrow\\frac\{1\}\{T\_\{\\mathrm\{ro\}\}\}\\sum\_\{t\}\(V\_\{\\theta\}\(s\_\{t\}\)\-\\hat\{R\}\_\{t\}\)^\{2\}⊳\\trianglerightunmasked: recovery\-trigger targets are variance\-free, exact up to critic error at re\-entry

19:

Lcompat←−λcompatNrec​∑t∉ℛσseg⁡\(t\)​log⁡πθ​\(atrec\|st\)L^\{\\mathrm\{compat\}\}\\leftarrow\-\\frac\{\\lambda\_\{\\mathrm\{compat\}\}\}\{N\_\{\\mathrm\{rec\}\}\}\\sum\_\{t\\notin\\mathcal\{R\}\}\\sigma\_\{\\operatorname\{seg\}\(t\)\}\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\\mathrm\{rec\}\}\|s\_\{t\}\)
20:

L←LsafePPO\+cv​LV\+Lcompat−ce​H​\[πθ\]L\\leftarrow L^\{\\mathrm\{PPO\}\}\_\{\\mathrm\{safe\}\}\+c\_\{v\}L^\{V\}\+L^\{\\mathrm\{compat\}\}\-c\_\{e\}H\[\\pi\_\{\\theta\}\]
21:

θ←θ−ηlr​∇θL\\theta\\leftarrow\\theta\-\\eta\_\{\\mathrm\{lr\}\}\\,\\nabla\_\{\\theta\}L
22:endfor

23:endfor

24:endfor

### B\.2Notation

[Table˜2](https://arxiv.org/html/2607.08925#A2.T2)collects the symbols used in[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)and the design rationale below\.

Table 2:Notation used in[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)and throughout the paper\.πθ​\(a∣s\)\\pi\_\{\\theta\}\(a\\mid s\)main policy, stochastic,θ\\theta\-parameterizedπrec​\(a∣s\)\\pi^\{\\mathrm\{rec\}\}\(a\\mid s\)recovery policy; noθ\\theta\-dependence; arbitrary formπθmix\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}mixed policy \([Section˜3](https://arxiv.org/html/2607.08925#S3)\)ℛ\\mathcal\{R\},dd,dmaxd\_\{\\max\}safe region, current boundary, per\-environment safe\-region maximum \(tuned hyperparameter\)pθ​\(τ\),pθmix​\(τ\)p\_\{\\theta\}\(\\tau\),p^\{\\mathrm\{mix\}\}\_\{\\theta\}\(\\tau\)trajectory distributions underπθ\\pi\_\{\\theta\}alone and the mixed policyJ​\(θ\),Jmix​\(θ\)J\(\\theta\),J^\{\\mathrm\{mix\}\}\(\\theta\)main\-policy return \(unobservable during training\) and mixed\-policy returnα​\(θ,d\)\\alpha\(\\theta,d\)recovery rate \(observable\)β​\(θ,d\)\\beta\(\\theta,d\)main\-policy out\-of\-region rate \(unobservable during training\)Cθ​\(τ\)C\_\{\\theta\}\(\\tau\)compatibility score∏t∉ℛπθ​\(atrec∣st\)\\prod\_\{t\\notin\\mathcal\{R\}\}\\pi\_\{\\theta\}\(a^\{\\mathrm\{rec\}\}\_\{t\}\\mid s\_\{t\}\)Vπθmix,Qπθmix,AπθmixV^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\},Q^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\},A^\{\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\}value,QQ, advantage under the mixed policyℓk\\ell\_\{k\}length of recovery segmentkkseg⁡\(t\)\\operatorname\{seg\}\(t\)index of the recovery segment containing stepttσk\\sigma\_\{k\}outcome gate of segmentkk\(11iff it ends with re\-entry intoℛ\\mathcal\{R\}\)NrecN\_\{\\mathrm\{rec\}\}number of recovery\-controlled steps in the current minibatch \(normalizer ofLcompatL^\{\\mathrm\{compat\}\}\)rtermr\_\{\\mathrm\{term\}\}environment reward on the terminal step of a failed recovery segment; per\-environment values in[Section˜C\.2](https://arxiv.org/html/2607.08925#A3.SS2)
### B\.3Operational features of the safe\-region intervention setup

The safe\-region intervention mechanism produces a training\-time setup with four operational features that simultaneously place it outsideCMDPand pointwise\-filter approaches: \(i\) the safety predicate is binary set memberships∈ℛs\\in\\mathcal\{R\}rather than a scalar cost, so there is no Lagrangian dual to optimize; \(ii\)πrec\\pi^\{\\mathrm\{rec\}\}may be deterministic \(MPC, greedySAC\), in which case it has no Lebesgue density and theIScorrection is ill\-defined; \(iii\) recovery takes control for multi\-step segments rather than at a single step, so training rollouts follow a genuine mixed\-policy distribution rather than a one\-step perturbation; \(iv\) the objective is the original main\-policy returnJ​\(θ\)J\(\\theta\), not a constrained surrogate, so safety is enforced operationally rather than penalized in the reward\. Each of the four method components in[Section˜4](https://arxiv.org/html/2607.08925#S4)responds to a specific feature in this list\.

### B\.4Soft outcome gate

In addition to the hard outcome gate of[Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4), we considered a signed, per\-step preference based on whether the post\-segment state was higher\- or lower\-value than the pre\-segment state, withk​\(t\)k\(t\)the length of the segment containingtt:

wtsoft=γk​\(t\)​Vθ​\(st\+k​\(t\)\)−Vθ​\(st\),normalized to zero mean per minibatch\.w\_\{t\}^\{\\mathrm\{soft\}\}=\\gamma^\{k\(t\)\}V\_\{\\theta\}\(s\_\{t\+k\(t\)\}\)\-V\_\{\\theta\}\(s\_\{t\}\),\\quad\\text\{normalized to zero mean per minibatch\}\.\(14\)Empirically, the soft variant is unstable onHalfCheetahandGo1\(the soft\-gate variants in[Section˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4)\) because the regularizer fits failed\-recovery actions, so we drop it from the headline comparison\.

### B\.5Random\-denominator and value\-loss target choices

Two further choices in[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)warrant explanation: the denominator of the masked policy gradient and the set of timesteps the critic regresses on\. The masked PG estimator divides by\|Tℛ\|\|T\_\{\\mathcal\{R\}\}\|, a random variable that depends on the rollout\. Early in training, whenα​\(θ,d\)\\alpha\(\\theta,d\)is high,\|Tℛ\|\|T\_\{\\mathcal\{R\}\}\|can be small: withm=\|Tℛ\|m=\|T\_\{\\mathcal\{R\}\}\|safe steps the masked mean has variance on the order ofσ2/m\\sigma^\{2\}/m, so a smallmminflates the per\-update gradient variance, and the random denominator additionally introduces a small ratio bias, one of the standard departures noted after[Corollary˜3](https://arxiv.org/html/2607.08925#Thmtheorem3)\. We do not apply variance\-control heuristics\. The curriculum schedule \([Section˜6](https://arxiv.org/html/2607.08925#S6)\) keeps\|Tℛ\|\|T\_\{\\mathcal\{R\}\}\|above a working threshold within the first few updates on every environment we evaluate\.

The value\-loss default also depends on whether the analytic recovery value \([Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)\) is in use:

- •Analytic V on:train the critic on*all*timesteps\. Recovery\-state targets under[Proposition˜6](https://arxiv.org/html/2607.08925#Thmtheorem6)carry no Monte\-Carlo variance and are exact over the segment; the only residual error is the critic’s own error at the single re\-entry state \(failure\-segment targets are exact\), so unmasked is principled and gives the critic dense supervision\.
- •Analytic V off:prefer the masked value lossLV​\(θ\)=\|Tℛ\|−1​∑t:st∈ℛ\(Vθ​\(st\)−R^t\)2L^\{V\}\(\\theta\)=\|T\_\{\\mathcal\{R\}\}\|^\{\-1\}\\sum\_\{t:s\_\{t\}\\in\\mathcal\{R\}\}\(V\_\{\\theta\}\(s\_\{t\}\)\-\\hat\{R\}\_\{t\}\)^\{2\}\. Otherwise the critic regresses against bootstrapped values that themselves depend on the critic at unsafe states, creating a self\-consistency loop\.

Our experiments use the analytic\-V variant by default\.[Section˜8](https://arxiv.org/html/2607.08925#S8)ablates this choice\.

### B\.6Where the changes land in the rollout/update loop

The choices above touch a standardPPOloop at a few points; the list below states where each is implemented\.

- •Rollout, recovery branch:whenst∉ℛs\_\{t\}\\notin\\mathcal\{R\}, the recovery action is queried, the action is stored in the buffer,log⁡πθ​\(at∣st\)\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)is computed at the recovery action, and the step is flagged as recovery\-controlled\.
- •Analytic\-V overwrite:for each recovery segment, the segment endpoint is identified and the value at the triggering state is overwritten withγℓ​Vθ​\(st\+ℓ\)\\gamma^\{\\ell\}V\_\{\\theta\}\(s\_\{t\+\\ell\}\)on success orγℓ−1​rterm\\gamma^\{\\ell\-1\}r\_\{\\mathrm\{term\}\}on failure, whereℓ\\ellis the segment length\. This happens in the rollout\-buffer pass, beforeGAE\.
- •GAE\(standard, no modification\): operates on the overwritten value sequence\.
- •Compatibility loss:computeslog⁡πθ​\(atrec∣st\)\\log\\pi\_\{\\theta\}\(a\_\{t\}^\{\\mathrm\{rec\}\}\\mid s\_\{t\}\)at unsafe steps and applies the chosen gate \(σk\\sigma\_\{k\}for the hard gate, the signed advantagewtsoftw\_\{t\}^\{\\mathrm\{soft\}\}for the soft variant\)\. The resulting loss is added to thePPOsurrogate before backpropagation\.
- •Masked PG and value loss:the safe\-step indicator masks both the policy\-gradient term and \(in the learned\-critic variant, without the analytic value\) the value loss\.

### B\.7Baseline ports

The two recovery\-based baselines in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)are ports into the same on\-policy loop as SafeExplorer: both run inside the identicalPPOupdate, share the identical safe\-region triggers∉ℛ​\(d\)s\\notin\\mathcal\{R\}\(d\)with the same curriculum ondd, and query the identical frozenSACrecovery policy\. They differ from SafeExplorer only in how recovery\-controlled transitions enter the rollout buffer\.

- •Recovery RL port\(Thananjeyan et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib58)\): at recovery steps the buffer keeps the*task policy’s*action, value, and log\-probability, the action\-relabeling data\-handling rule of the original method; recovery\-step rewards are zeroed in the learning signal, as for SafeExplorer\. The original method treats the recovery as part of the environment dynamics, which is coherent for its off\-policy Q\-learning objective; the bias corrected in this paper arises only when that relabeling is carried into an on\-policy policy\-gradient update\.
- •Safe Legged port\(Yang et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib65)\): the same relabeling, plus a reward ofrt−1r\_\{t\}\-1stored at recovery\-triggered steps, the recovery\-penalty shaping component of the published method\.

What is deliberately*not*ported: Recovery RL’s learned safety critic and off\-policy training, and the model\-based switching criterion ofYang et al\. \([2022](https://arxiv.org/html/2607.08925#bib.bib65)\)\. Replacing both with the shared distance trigger isolates each method’s data\-handling rule under a matched on\-policy setup with an identical trigger and identical recovery controller, which is what makes the comparison interpretable\. No baseline\-specific tuning was performed beyond the sharedPPOconfiguration of[Appendix˜C](https://arxiv.org/html/2607.08925#A3)\.

### B\.8Pseudocode parameters

[Table˜3](https://arxiv.org/html/2607.08925#A2.T3)gives the default values that instantiate[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)as run in our experiments\.

Table 3:Default parameter values for[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\.Symbol in[Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)Default valueNN\(total updates\)total​\_​timesteps/\(num​\_​envs×num​\_​steps\)\\mathrm\{total\\\_timesteps\}/\(\\mathrm\{num\\\_envs\}\\times\\mathrm\{num\\\_steps\}\)EE\(parallel envs\)4TroT\_\{\\mathrm\{ro\}\}\(rollout length\)2048KK\(PPOepochs\)10cvc\_\{v\}\(value coef\)0\.5cec\_\{e\}\(entropy coef\)0λcompat\\lambda\_\{\\mathrm\{compat\}\}\(compat coef\)10−310^\{\-3\}ϵ\\epsilon\(PPOclip\)0\.2γ,GAE​λ\\gamma,\\mathrm\{GAE\}\\,\\lambda0\.99, 0\.95rtermr\_\{\\mathrm\{term\}\}\(terminal reward\)environment terminal\-step reward \([Section˜C\.2](https://arxiv.org/html/2607.08925#A3.SS2)\)

## Appendix CHyperparameters

### C\.1Training hyperparameters \(shared across all variants\)

Every method in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)except the twoCMDPbaselines is trained with the singlePPOconfiguration of[Table˜4](https://arxiv.org/html/2607.08925#A3.T4), so that any difference among them reflects the method rather than per\-variant tuning\.CPOandPPO\-Lagrangian additionally carry the constraint hyperparameters their objective requires \([Section˜C\.3](https://arxiv.org/html/2607.08925#A3.SS3)\)\. The compatibility regularizerλcompat\\lambda\_\{\\mathrm\{compat\}\}is the only entry active for SafeExplorer alone\. All remaining hyperparameters are shared\. The per\-environmentdmaxd\_\{\\max\}values were set by a qualitative criterion, large enough that the recovery policy stops triggering under normal task operation asddapproachesdmaxd\_\{\\max\}; no systematic search overdmaxd\_\{\\max\}was run\.

Table 4:PPOtraining hyperparameters for every variant in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\. Per\-environmentdmaxd\_\{\\max\}values are in env\-specific task\-space units, not comparable across environments\.ParameterValueOptimizerAdam \(eps=10−5\\mathrm\{eps\}=10^\{\-5\}\)Learning rate3×10−43\\times 10^\{\-4\}Anneal LR linearly to 0yesNumber of parallel envs4Rollout length per env2048Batch size \(num​\_​envs×num​\_​steps\\mathrm\{num\\\_envs\}\\times\\mathrm\{num\\\_steps\}\)8192Mini\-batch sizebatch size//32GAEλ\\lambda0\.95Discount factorγ\\gamma0\.99PPOratio clippingϵ\\epsilon0\.2Update epochs per batch10Value loss coefficientcvc\_\{v\}0\.5Max gradient norm0\.5Entropy coefficientcec\_\{e\}0Compatibility regularizerλcompat\\lambda\_\{\\mathrm\{compat\}\}\(SafeExplorer only\)10−310^\{\-3\}\(see[Section˜E\.5](https://arxiv.org/html/2607.08925#A5.SS5)for sweep\)Initial safe\-region radiusd0d\_\{0\}0\.01 \(Go1: 0\.05\)Per\-envdmaxd\_\{\\max\}HalfCheetah: 2\.0;Ant: 0\.4;Go1: 0\.15Per\-step recovery reward in the learning signal0\(zeroed;[Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3)\)Terminal\-step rewardrtermr\_\{\\mathrm\{term\}\}environment terminal reward \([Section˜C\.2](https://arxiv.org/html/2607.08925#A3.SS2)\)
### C\.2Terminal\-step rewardrtermr\_\{\\mathrm\{term\}\}

Intermediate recovery\-step rewards are zeroed in the learning signal \([Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3)\), but the terminal\-step reward that the analytic\-VVfailure branch propagates asrtermr\_\{\\mathrm\{term\}\}\([Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\) is the environment’s*own*reward on the terminating step, preserved in the buffer\. Concretely: onHalfCheetahthe terminal step includes an explicit−1\-1penalty; onGo1the per\-step reward is floored at zero and the healthy bonus vanishes on an unhealthy step, sorterm=0r\_\{\\mathrm\{term\}\}=0\(a configurable termination\-penalty flag exists but is0in all main\-table runs\); onAntthe constant healthy bonus of1\.01\.0is paid even on the terminal step, so its terminal reward is that bonus minus the step’s control and contact costs\.

### C\.3CMDPbaseline hyperparameters

BothCMDPbaselines run on OmniSafe 0\.5\.x\. The constraint cost is11on each unhealthy termination and0otherwise; time\-limit truncations incur no cost, matching the fall\-counting convention of[Appendix˜D](https://arxiv.org/html/2607.08925#A4)\. The cost limit is0\.050\.05\. Shared solver settings: steps per epoch20482048,1010update iterations, minibatch size6464, target KL0\.020\.02,γ=γcost=0\.99\\gamma=\\gamma\_\{\\mathrm\{cost\}\}=0\.99,GAEλ=λcost=0\.95\\lambda=\\lambda\_\{\\mathrm\{cost\}\}=0\.95, actor and critic MLPs of two hidden layers of size 64 withtanh\\tanhactivation, learning rate3×10−43\\times 10^\{\-4\}for both with linear decay, observation normalization on, reward normalization on, cost normalization off\.PPO\-Lagrangian additionally uses ratio clip0\.20\.2, Lagrange multiplier initialized at0\.0010\.001, and multiplier learning rate0\.0350\.035\.CPOuses the OmniSafe defaults for its trust\-region settings plus the cost limit above\. No baseline\-specific tuning was performed beyond these settings\.

### C\.4Per\-environment training budget

Table 5:Per\-environment training budget and SafeExplorer wall\-clock cost\.EnvironmentTotal env stepsWall\-clock \(4 CPU cores\)HalfCheetah8×1068\\times 10^\{6\}∼\\sim3\.5 hAnt8×1068\\times 10^\{6\}∼\\sim5 hGo116×10616\\times 10^\{6\}∼\\sim12 hThe one quantity that does vary across environments is the training budget, which we fix before any rollout begins and report in[Table˜5](https://arxiv.org/html/2607.08925#A3.T5)\. The larger budget onGo1reflects its 12\-DoFaction space and longer main\-task time\-to\-converge; the frozenSACrecovery policy \([Section˜D\.3](https://arxiv.org/html/2607.08925#A4.SS3)\) does not enter this budget\. Matching all environments at1616M, or repeatingGo1at88M for a fully matched comparison, is left to follow\-up work\.

### C\.5Network architecture

Within each run, the network thatPPOoptimizes is the standard CleanRL actor/critic: the actor and critic are independent MLPs with two hidden layers of size 64 andtanh\\tanhactivation\. The actor outputs a mean vector and a learnable globallog⁡σ\\log\\sigmafor a diagonal\-Gaussian policy\. Layer weights are initialized orthogonally\. The critic’s output layer uses gain 1\.0 and the actor’s mean output uses gain 0\.01\.

### C\.6Curriculum schedule

The remaining quantity that changes during a run is the safe\-region radius, which is not held fixed but annealed along the linear schedule of[Section˜5](https://arxiv.org/html/2607.08925#S5), interpolatingd0→dmaxd\_\{0\}\\to d\_\{\\max\}over theN=total​\_​timesteps/\(num​\_​envs×num​\_​steps\)N=\\mathrm\{total\\\_timesteps\}/\(\\mathrm\{num\\\_envs\}\\times\\mathrm\{num\\\_steps\}\)training updates\.[Section˜E\.7](https://arxiv.org/html/2607.08925#A5.SS7)ablates the schedule shape \(linear / log / step / constant\) onAnt; the headline results all use the linear schedule\.

## Appendix DEnvironment and Recovery\-Policy Details

### D\.1Environment summary

We evaluate on three MuJoCo locomotion environments that differ in dimensionality and in how the safe region is sensed\.[Table˜6](https://arxiv.org/html/2607.08925#A4.T6)lists, for each one, the degrees of freedom, the observation and action dimensions, the indicator that the recovery policy uses to decide whether the agent has left the safe regionℛ\\mathcal\{R\}, and the termination predicate that defines a fall\. The indicator is a function of torso or base height together with tilt to the nominal pose, the signal the recovery policy is trained to restore\.

Table 6:Per\-environment dimensions, the safe\-region indicator the recovery policy uses, and the termination predicate that defines a fall\.EnvironmentDoFObs dimAction dimSafe\-region indicatorFall \(termination\) iffHalfCheetah6176torso heightzz\+ tilt to nominalz∉\[−0\.5,0\.5\]z\\notin\[\-0\.5,0\.5\]or\|θpitch\|\>1\.57\|\\theta\_\{\\mathrm\{pitch\}\}\|\>1\.57Ant8278torso heightzz\+ tilt to nominalz∉\(0\.3,1\.5\)z\\notin\(0\.3,1\.5\)or non\-finite stateGo1124912basezz\+ base tilt \(roll, pitch\) to nominalz∉\(0\.22,0\.65\)z\\notin\(0\.22,0\.65\)or\|roll\|\|\\mathrm\{roll\}\|or\|pitch\|\>60∘\|\\mathrm\{pitch\}\|\>60^\{\\circ\}

Termination, horizon, and fall counting\.TheHalfCheetahtermination predicate is a custom addition; the stock GymnasiumHalfCheetahnever terminates, which is why our fall counts onHalfCheetahare nonzero\. All three environments run under a10001000\-step episode time limit\. The control timestep is0\.050\.05s onHalfCheetahandAnt\(frame skip55at a0\.010\.01s simulation step\) and0\.010\.01s onGo1\(frame skip55at a0\.0020\.002s simulation step\), so the horizons are5050s and1010s of simulated time respectively\. A*fall*is an unhealthy termination as defined in[Table˜6](https://arxiv.org/html/2607.08925#A4.T6); time\-limit truncations are never counted as falls, in training and evaluation alike\.

### D\.2Safe\-region distance and curriculum

The membership test iss∈ℛ​\(d\)s\\in\\mathcal\{R\}\(d\)iffD​\(s\)≤dD\(s\)\\leq d, with an environment\-specific distanceDDcomputed from the base height and orientation:

- •HalfCheetah:D​\(s\)=‖\(z,θpitch\)‖2D\(s\)=\\\|\(z,\\theta\_\{\\mathrm\{pitch\}\}\)\\\|\_\{2\}against the nominal\(0,0\)\(0,0\), withzzin meters and pitch in radians, unweighted\.
- •Ant:D​\(s\)=‖\(z−0\.75,tilt\)‖2D\(s\)=\\\|\(z\-0\.75,\\mathrm\{tilt\}\)\\\|\_\{2\}, wheretilt=π180​arccos⁡\(axisz\)⋅ϕ/π\\mathrm\{tilt\}=\\frac\{\\pi\}\{180\}\\arccos\(\\mathrm\{axis\}\_\{z\}\)\\cdot\\phi/\\piwithϕ=2​arccos⁡\(qw\)\\phi=2\\arccos\(q\_\{w\}\), a rescaled axis\-angle deviation from upright \(qwq\_\{w\}is the scalar quaternion component andaxisz\\mathrm\{axis\}\_\{z\}the vertical component of the rotation axis\)\. This scaling strongly downweights orientation relative to height onAnt; it is an implementation choice that we state plainly and keep for fidelity with the trained runs\.
- •Go1:D​\(s\)=‖\(z−0\.3,roll,pitch\)‖2D\(s\)=\\\|\(z\-0\.3,\\mathrm\{roll\},\\mathrm\{pitch\}\)\\\|\_\{2\}, withzzin meters and roll and pitch in radians\.

Because eachDDmixes meters and \(rescaled\) radians in a single Euclidean norm,ddanddmaxd\_\{\\max\}carry the same mixed units and are therefore not comparable across environments, the reason[Table˜4](https://arxiv.org/html/2607.08925#A3.T4)reports them as env\-specific values\. During training,ddfollows the linear curriculumd=d0\+u−1N​dmaxd=d\_\{0\}\+\\frac\{u\-1\}\{N\}\\,d\_\{\\max\}at updateuuofNNupdates \([Algorithm˜1](https://arxiv.org/html/2607.08925#alg1)\), with the per\-environment\(d0,dmax\)\(d\_\{0\},d\_\{\\max\}\)of[Table˜4](https://arxiv.org/html/2607.08925#A3.T4)\.

### D\.3Recovery\-policy training

For each environment, the recovery policyπrec\\pi^\{\\mathrm\{rec\}\}is a separately pre\-trainedSACactor, trained on the same MuJoCo dynamics in a recovery\-flavored configuration: the forward and velocity\-tracking task terms of the environment reward are zeroed, so the remaining reward favors surviving and staying near the nominal pose\. The shared recipe is CleanRLSACwith twin256256\-256256QQnetworks, a256256\-256256tanh\-squashed Gaussian actor, replay buffer of10610^\{6\}transitions,γ=0\.99\\gamma=0\.99, batch size256256, and an auto\-tuned entropy coefficient; recovery\-training episodes are capped at250250steps, and at deployment we use the actor’s deterministic tanh\-mean action\. Recovery training is*not*reset\-free\. At every episode reset, the initial state is aggressively randomized to mimic the failure modes a learning task policy will produce\.

Initial\-state randomization \(per env reset\):

- •Joint positions:sampled uniformly over the full mechanical joint range \(or a large subset\)\. ForAnt, each joint has env\-specific bounds \(e\.g\.,±0\.52\\pm 0\.52to±1\.23\\pm 1\.23rad from nominal\)\. ForGo1, all 12 joint positions are drawn from the full actuator range\.
- •Base height \(and optionallyx,yx,y\):basezzis randomized over a range that includes low and near\-fall configurations \(e\.g\.,z∈\[0\.22,0\.45\]z\\in\[0\.22,0\.45\]forGo1\)\.
- •Base orientation:the base quaternion is randomized to produce varied roll, pitch, and yaw\. ForAnt, a random quaternion is generated and constrained so the forward vector has non\-negativezz\(“face up”\)\. ForGo1, roll and pitch are drawn in±π/10\\pm\\pi/10and yaw in\[0,2​π\]\[0,2\\pi\], again with a face\-up constraint\.
- •Velocities:initialized with small noise around zero\.

Per\-environment training and cost\.OnHalfCheetahthe recovery policy was trained on the stock environment with the forward reward disabled, for55M environment steps; the deployed checkpoint is the44M\-step snapshot\. This policy was trained with an earlier revision of the trainer, so we report its checkpoint provenance rather than a reconstructed reward\. OnAntit was trained for33M steps \(22M\-step snapshot deployed; policy learning rate10−410^\{\-4\},QQlearning rate3×10−43\\times 10^\{\-4\}, target\-smoothingτ=0\.002\\tau=0\.002\) on the recovery\-flavored environment, whose effective reward is the healthy bonus minus control\-magnitude and contact costs, so theAntrecovery is torque\-limited\. OnGo1it was trained for1010M steps \(55M\-step snapshot deployed;τ=0\.005\\tau=0\.005,QQlearning rate10−310^\{\-3\}, policy learning rate3×10−43\\times 10^\{\-4\}\) with the velocity\-tracking terms zeroed; the effective reward is the healthy indicator minus quadratic pose, height, and joint\-deviation costs, floored at zero\. Recovery pre\-training therefore costs55M \(HalfCheetah\),33M \(Ant\), and1010M \(Go1\) environment steps\. These steps are not included in the training budgets of[Table˜5](https://arxiv.org/html/2607.08925#A3.T5), and the falls incurred during recovery pre\-training are not tracked\. The cost is nonetheless bounded: at most62\.5%62\.5\\%of a single88M\-stepHalfCheetahorAnttraining run \(55M of88M\) and62\.5%62\.5\\%of the1616M\-stepGo1budget \(1010M of1616M\), which bounds the sense in which the recovery policy is cheap to obtain\.

Per\-environment recovery quality\.How well this training regime works depends on the environment\. Empirically, theHalfCheetahandGo1recoveries succeed at stabilization with high probability across the randomized initial states\. TheAntrecovery policy does not, producing the unreliable\-recovery regime that is the most informative axis of variation in our results \(see[Section˜7](https://arxiv.org/html/2607.08925#S7)\)\. We attribute this to the recovery policy’s reward design rather than to undertraining: the control\-magnitude penalty \(0\.5​‖a‖20\.5\\\|a\\\|^\{2\}in theAntreward\) caps how much torque the policy can apply, and the harder 3DAntstabilization needs exactly those aggressive corrections\. Relaxing that penalty or adopting a stronger recovery class forAnt, and confirming the attribution with a longer\-training control, is left to follow\-up work\.

Recovery quality on the operational distribution\.The randomized\-reset measurement above grades the recovery on states the designer samples, not on the states the learning policy actually drives it into \(which, unlike the face\-up\-constrained reset distribution, can include face\-down configurations\), so the operational grade is, if anything, the harder of the two\. To measure the latter, during a trained run we log every first exit fromℛ\\mathcal\{R\}\(the state at which recovery is triggered\), then replay theSACrecovery from each logged state and record the fraction that*survive*, meaning the recovery completes the segment without an episode termination\. We do this for SafeExplorer and for the masked, analytic\-VVablation rung of[Section˜8](https://arxiv.org/html/2607.08925#S8), which shares the recovery policy and the curriculum with SafeExplorer and differs from it only by the hard outcome gateCθ​\(τ\)C\_\{\\theta\}\(\\tau\)\([Equation˜8](https://arxiv.org/html/2607.08925#S4.E8)\)\.[Table˜7](https://arxiv.org/html/2607.08925#A4.T7)aggregates all logged triggers of one representative run per cell\. Two effects stand out\. First, SafeExplorer leavesℛ\\mathcal\{R\}between2\.1×2\.1\\timesand4\.2×4\.2\\timesless often, so it calls on the recovery far less\. Second, onAnt, where the recovery policy is unreliable, the exits SafeExplorer produces survive91\.3%91\.3\\%of the time against70\.4%70\.4\\%for the ungated rung, a gap of20\.920\.9points\. OnHalfCheetahandGo1, where the recovery policy is reliable, both variants survive at essentially the same rate \(about100%100\\%and97%97\\%\)\. TheAntgap offers an operational reading of the\+2683\+2683reward the hard gate adds onAntin[Section˜8](https://arxiv.org/html/2607.08925#S8)\. The gate biases the main policy toward the part of state space where the weakAntrecovery policy actually succeeds\. The trigger counts in[Table˜7](https://arxiv.org/html/2607.08925#A4.T7)place the binomial95%95\\%confidence interval on each survival rate within±0\.5\\pm 0\.5points\. These intervals reflect sampling over triggers within one run, not variation across training seeds, which we do not characterize here; the table is therefore a single\-run illustration rather than a multi\-seed estimate\.

Table 7:Recovery quality on the*operational*trigger\-state distribution\. We replay theSACrecovery from every logged first exit fromℛ\\mathcal\{R\}of one representative trained run per cell\. “Exits” counts the logged recovery triggers; “Survived” is the percentage of those exits the recovery completes without termination\. The ungated rung is the masked, analytic\-VVvariant of[Section˜8](https://arxiv.org/html/2607.08925#S8)\.Exits \(triggers\)Survived \(%\)EnvironmentSafeExplorerungatedSafeExplorerungatedHalfCheetah80,546338,67199\.8100\.0Ant16,36635,15091\.370\.4Go194,212255,88197\.097\.4
### D\.4Go1environment specification

TheGo1observation is4949\-dimensional, in order: base linear velocity scaled by2\.02\.0\(33\), base angular velocity scaled by0\.250\.25\(33\), base heightzz\(11\), roll, pitch, yaw \(33\), commanded velocity scaled by2\.02\.0\(33\), joint positions minus their defaults \(1212\), joint velocities scaled by0\.050\.05\(1212\), and the previous action \(1212\); the vector is clipped to±100\\pm 100\. The action is1212absolute joint\-position targets driving MuJoCo position servos with proportional gainkp=20k\_\{p\}=20, joint damping1\.01\.0, and an actuator force range of±23\.7\\pm 23\.7N m \(±35\.55\\pm 35\.55N m at the knees\)\. The commanded velocity\(vx,vy,ωz\)\(v\_\{x\},v\_\{y\},\\omega\_\{z\}\)is resampled once per episode, each component uniform in\[−0\.5,0\.5\]\[\-0\.5,0\.5\]\. The reward is4\.0​exp⁡\(−‖vx​ycmd−vx​y‖2/0\.25\)\+2\.0​exp⁡\(−\(ωzcmd−ωz\)2/0\.25\)4\.0\\,\\exp\(\-\\\|v^\{\\mathrm\{cmd\}\}\_\{xy\}\-v\_\{xy\}\\\|^\{2\}/0\.25\)\+2\.0\\,\\exp\(\-\(\\omega^\{\\mathrm\{cmd\}\}\_\{z\}\-\\omega\_\{z\}\)^\{2\}/0\.25\), plus a healthy bonus of1\.01\.0and a feet\-air\-time term of weight1\.01\.0, minus costs with weights: torque2×10−42\\times 10^\{\-4\}, vertical velocity2\.02\.0,x​yxyangular velocity0\.050\.05, action rate0\.0010\.001, joint\-limit violation0\.010\.01, joint acceleration1\.25×10−91\.25\\times 10^\{\-9\}, orientation0\.40\.4, height4\.0​\|z−0\.3\|4\.0\\,\|z\-0\.3\|, and joint deviation0\.10\.1; the total is floored at0\.

### D\.5Custom XML modifications

Supporting the recovery training and value conventions above requires three changes to the environment definitions\. Our custom MuJoCo XMLs extend the standard Gymnasium MuJoCo environments with: \(i\) explicit joint\-range attributes used by the recovery policy’s randomized\-reset training \([Section˜D\.3](https://arxiv.org/html/2607.08925#A4.SS3)\), \(ii\) a configurable flag that zeros the training reward outsideℛ\\mathcal\{R\}\(the learning\-signal convention of[Section˜4\.3](https://arxiv.org/html/2607.08925#S4.SS3); reported returns are unmodified\), and \(iii\) forGo1, themujoco\_menagerieUnitree Go1 model\(Zakka et al\.,[2022](https://arxiv.org/html/2607.08925#bib.bib66)\)with a custom 12\-DoFposition\-actuator configuration and a velocity\-tracking reward adapted fromlegged\_gym\(Rudin et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib43)\)\.

## Appendix EAdditional Results

Run\-batch provenance\.The sensitivity studies in this appendix come from dedicated sweep batches run independently of the[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)runs; each caption states its batch, operating point, and seed count \(first55of1010seeds, i\.e\. seeds11to55, unless noted otherwise\)\. Cumulative falls are heavy\-tailed across seeds, so independent55\-seed batches at identical settings can differ noticeably in absolute falls while preserving the orderings the text relies on\. Where a table’s operating point differs from the body protocol of[Section˜6](https://arxiv.org/html/2607.08925#S6), specifically the wider radiidmax=3\.2d\_\{\\max\}=3\.2onHalfCheetahand0\.560\.56onAntused by the v10 sweep family, the caption says so; within\-table comparisons remain matched\.

### E\.1Headline comparison: rliable interquartile\-mean intervals

This appendix supplies the evidence behind every claim made in the body, ordered as a descent from the headline numbers to the design choices that produce them, the robustness checks that stress them, and the theory and baselines that justify them\. We begin with the headline\.[Figure˜2](https://arxiv.org/html/2607.08925#A5.F2)gives the per\-metric reward and total\-falls intervals for the six comparison\-set methods, complementing the falls\-to\-success headline \([Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)\)\. It shows rliable interquartile means with95%95\\%stratified\-bootstrap confidence intervals\(Agarwal et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib3)\), drawn as ranked intervals so the falls and reward orderings read directly\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/headline_iqm.png)Figure 2:rliable interquartile\-mean intervals of reward and total falls for the six methods in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1), with95%95\\%stratified\-bootstrap confidence intervals; a per\-metric companion to the falls\-to\-success headline \([Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)\)\.
### E\.2Per\-environment learning curves

The interval plot summarizes the endpoints; the learning curves show how each method gets there\.[Figure˜3](https://arxiv.org/html/2607.08925#A5.F3)traces reward and cumulative falls across the full training run, the trajectory view of the headline endpoints just reported\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/learning_curves_grid.png)Figure 3:Per\-environment learning curves: rows are episodic reward and cumulative training\-time falls; columns areHalfCheetah,Ant,Go1\. Solid = mean across seeds, shaded = \[min, max\] envelope; curves smoothed with a moving average for display\. Recovery\-rate curves are deferred to[Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\.
### E\.3Additional ablations

Having established what the headline shows, we now ask which design choices produce it, starting from the component ablation\.[Section˜8](https://arxiv.org/html/2607.08925#S8)analyzes that ablation in the body\.[Figure˜4](https://arxiv.org/html/2607.08925#A5.F4)gives its reward\-vs\-falls view, the trade\-off complement to the falls\-to\-success ranking of[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1); here we also report the remaining variants\. The two threads opened here are picked up in turn below: the soft\-gate variants \([Section˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4)\) help marginally onAntand hurt onHalfCheetah, and theλcompat\\lambda\_\{\\mathrm\{compat\}\}sweep is in[Section˜E\.5](https://arxiv.org/html/2607.08925#A5.SS5)\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/ladder_pareto.png)Figure 4:Ablation ladder on the reward\-vs\-falls plane:PPOand the four rungs, Unmasked PG→\\tomasked learned\-VV→\\tomasked analytic\-VV→\\toSafeExplorer\. Markers are rliable interquartile means with95%95\\%stratified\-bootstrap confidence intervals\(Agarwal et al\.,[2021](https://arxiv.org/html/2607.08925#bib.bib3)\); total training falls on a log axis, upper\-left is better \(fewer falls, higher reward\)\. The reward\-vs\-falls view of the variants[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)ranks by falls\-to\-success; SafeExplorer \(red star\) is on the Pareto front in every environment\.
### E\.4Soft compatibility\-gate variants

The body ablation \([Section˜8](https://arxiv.org/html/2607.08925#S8),[Figure˜1](https://arxiv.org/html/2607.08925#S7.F1)\) analyzes the clean ablation ladder Unmasked PG→\\tomasked learnedVV→\\toanalyticVV→\\tohardCθ​\(τ\)C\_\{\\theta\}\(\\tau\)\(SafeExplorer\);[Table˜8](https://arxiv.org/html/2607.08925#A5.T8)tabulates its full numbers\. The two soft\-gate variants, a soft re\-entry\-advantage gatewtsoftw\_\{t\}^\{\\mathrm\{soft\}\}\([Equation˜14](https://arxiv.org/html/2607.08925#A2.E14)\) applied on the analytic\-VVmodel, and the same soft gate applied on a decomposed value, are negative results\. They help marginally onAntand hurt onHalfCheetah, and never approach SafeExplorer’s safety\. We report their full numbers here and omit them from the body ablation to keep it focused on variants that isolate a single design choice\.[Table˜8](https://arxiv.org/html/2607.08925#A5.T8)places them alongside the ladder rungs so the off→\\tosoft→\\tohard progression is visible \(same five\-seed protocol as[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\)\.

Table 8:Soft\-gate ablations against the full ladder: Unmasked PG \(the biased no\-masking baseline of[Section˜8](https://arxiv.org/html/2607.08925#S8), added here so all rungs appear in one table\), masked learned\-VV, analytic\-VV, and the hard gate \(same protocol as[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\)\. The soft gate iswtsoft=γk​V​\(st\+k\)−V​\(st\)w\_\{t\}^\{\\mathrm\{soft\}\}=\\gamma^\{k\}V\(s\_\{t\+k\}\)\-V\(s\_\{t\}\)\.Boldmarks the best value per column\.HalfCheetahAntGo1MethodRewardFallsRewardFallsRewardFallsUnmasked PG𝟒𝟗𝟐𝟐±𝟐𝟒𝟑\\mathbf\{4922\\pm 243\}194±40194\\pm 401253±4271253\\pm 4275177±13995177\\pm 1399𝟓𝟕𝟖𝟖±𝟒𝟐𝟓\\mathbf\{5788\\pm 425\}2936±6292936\\pm 629Masked, learnedVV1982±10581982\\pm 1058306±360306\\pm 360222±275222\\pm 27550011±3082850011\\pm 308284374±7584374\\pm 758972±206972\\pm 206AnalyticVV\(no gate\)3870±4473870\\pm 44773±1973\\pm 19174±150174\\pm 15035481±2185135481\\pm 218514473±6284473\\pm 628926±259926\\pm 259SoftCθ​\(τ\)C\_\{\\theta\}\(\\tau\)gate2645±8292645\\pm 8291215±18431215\\pm 1843663±323663\\pm 32311745±1843411745\\pm 184344142±14714142\\pm 14712044±26472044\\pm 2647SoftCθ​\(τ\)C\_\{\\theta\}\(\\tau\), decomposedVV3569±10743569\\pm 1074721±985721\\pm 985505±319505\\pm 3198183±80648183\\pm 80644374±11914374\\pm 11911242±4721242\\pm 472SafeExplorer \(hardCθ​\(τ\)C\_\{\\theta\}\(\\tau\)\)3581±9713581\\pm 971𝟏𝟒±𝟖\\mathbf\{14\\pm 8\}𝟐𝟖𝟓𝟕±𝟏𝟐𝟗\\mathbf\{2857\\pm 129\}𝟕𝟓𝟒±𝟒𝟔𝟑\\mathbf\{754\\pm 463\}5594±1775594\\pm 177𝟕𝟑𝟏±𝟏𝟕𝟕\\mathbf\{731\\pm 177\}

The mechanism behind the soft gate’s failure is given in[Section˜4\.4](https://arxiv.org/html/2607.08925#S4.SS4): on a failed recovery segment the soft gate still emits a negative\-signed imitation pressure, teachingπθ\\pi\_\{\\theta\}to*un*learn the recovery action at those states, a noisy signal that the hard gate removes by zeroing the term on failures\.

### E\.5λcompat\\lambda\_\{\\mathrm\{compat\}\}sensitivity \(SafeExplorer\)

With the gate fixed, the remaining knob on the regularizer is its weight\. We sweepλcompat∈\{10−4,10−3,10−2,10−1,1\.0\}\\lambda\_\{\\mathrm\{compat\}\}\\in\\\{10^\{\-4\},10^\{\-3\},10^\{\-2\},10^\{\-1\},1\.0\\\}onHalfCheetahandAnt\.

Table 9:Compatibility\-regularizer weight sweep onHalfCheetahandAnt\. Reward and falls are training\-time \(last\-10%10\\%mean reward, cumulative falls\); mean±\\pmstd over the first55of1010seeds, dedicated sweep batch at the body operating point, run independently of the[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)runs\.Boldmarks the best value per column\.λcompat\\lambda\_\{\\mathrm\{compat\}\}HalfCheetahrewardHalfCheetahfallsAntrewardAntfalls10−410^\{\-4\}2654±18332654\\pm 1833211±266211\\pm 2661803±11761803\\pm 11763557±37953557\\pm 379510−310^\{\-3\}4768±5834768\\pm 583𝟒𝟕±𝟒𝟏\\mathbf\{47\\pm 41\}2908±1572908\\pm 157428±454428\\pm 45410−210^\{\-2\}𝟓𝟕𝟖𝟑±𝟒𝟖𝟏\\mathbf\{5783\\pm 481\}374±548374\\pm 548𝟑𝟏𝟐𝟒±𝟑𝟓𝟓\\mathbf\{3124\\pm 355\}639±517639\\pm 51710−110^\{\-1\}5288±6675288\\pm 66764±5764\\pm 572281±8942281\\pm 894𝟏𝟒𝟗±𝟏𝟒𝟎\\mathbf\{149\\pm 140\}1\.01\.05536±3335536\\pm 333137±151137\\pm 1511352±10361352\\pm 10362798±56582798\\pm 5658Theλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}row is an independent replication of[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)’s configuration: it lands atHalfCheetah4768±5834768\\pm 583/47±4147\\pm 41against[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)’s3581±9713581\\pm 971/14±814\\pm 8, consistent within seed noise for a heavy\-tailed falls count\. The same runs also back[Table˜15](https://arxiv.org/html/2607.08925#A5.T15)\(HalfCheetah/Ant\)\.

*The optimum is environment\-specific\.*OnHalfCheetahthe reward peaks sharply atλcompat=10−2\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-2\}\(57835783\)\. OnAntthe reward is within seed noise fromλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}to10−210^\{\-2\}, with10−310^\{\-3\}the safer choice\. Both environments are robust within a∼10×\\sim 10\\timesband around the optimum\.

*The main results use a singleλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}\.*SafeExplorer in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)and[Figure˜3](https://arxiv.org/html/2607.08925#A5.F3)fixesλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}across all three environments rather than a per\-environment optimum\. We make this choice on*safety*grounds\. In the sweep above,λcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}attains the lowestHalfCheetahtraining\-fall count \(4747, against374374at10−210^\{\-2\}\) and a near\-lowest count onAnt\. A largerλcompat=10−2\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-2\}would raiseHalfCheetahreward substantially \(57835783vs47684768\) and is marginally better onAnt, but it does so at roughly8×8\\timestheHalfCheetahfalls\. We therefore keep the uniform10−310^\{\-3\}default so that the headline fall reductions are not traded away for extra falls, and we note10−210^\{\-2\}as the reward\-optimal setting forHalfCheetahfor practitioners who can tolerate the higher fall count\.

### E\.6Fixed\-ddablation: the safe\-region schedule vs\. a constantdd

With the gate and its weight fixed, the last training\-time design choice is the safe\-region radius, which SafeExplorer anneals on a curriculum\. A natural counterfactual is to pick a smallerdmaxd\_\{\\max\}and holdddconstant there\.[Table˜10](https://arxiv.org/html/2607.08925#A5.T10)sweeps fixedd∈\{0\.25,0\.5,0\.75,1\.0\}⋅dmaxd\\in\\\{0\.25,0\.5,0\.75,1\.0\\\}\\cdot d\_\{\\max\}on each of the three environments, with otherwise\-identical SafeExplorer\.

Table 10:Fixed\-ddsweep at the body operating point \(SafeExplorer\), dedicated sweep batch, mean±\\pmstd over the first55of1010seeds onHalfCheetah/Antand over all55seeds onGo1\.ddis held constant at the listed percentage of the per\-envdmaxd\_\{\\max\}\(2\.02\.0/0\.40\.4/0\.150\.15\) throughout training\. Cells areReward / Falls;boldmarks the best reward and lowest falls per environment\. Bottom row: linear\-curriculum reference at the samedmaxd\_\{\\max\}, reproducing[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\(a separate run batch\); the sweep’s own linear arm at the same settings givesHalfCheetah4773±5364773\\pm 536/46±4146\\pm 41,Ant2929±1572929\\pm 157/419±446419\\pm 446,Go15613±2345613\\pm 234/750±231750\\pm 231, an independent replication consistent within seed noise\.d/dmaxd/d\_\{\\max\}HalfCheetahAntGo125%25\\%3344±1642/1189±12743344\\pm 1642/1189\\pm 1274990±2/𝟓±𝟒990\\pm 2/\\mathbf\{5\\pm 4\}2898±147/1895±4092898\\pm 147/1895\\pm 40950%50\\%2476±668/2623±6642476\\pm 668/2623\\pm 6641602±463/5522±15051602\\pm 463/5522\\pm 15055423±119/𝟑𝟗𝟕±𝟑𝟑5423\\pm 119/\\mathbf\{397\\pm 33\}75%75\\%𝟑𝟔𝟐𝟗±𝟓𝟖𝟐/3489±889\\mathbf\{3629\\pm 582\}/3489\\pm 889555±178/19941±4063555\\pm 178/19941\\pm 40635572±179/888±3725572\\pm 179/888\\pm 372100%100\\%3535±1053/3632±6413535\\pm 1053/3632\\pm 641565±301/21837±8376565\\pm 301/21837\\pm 8376𝟓𝟕𝟓𝟐±𝟏𝟓𝟖/1662±272\\mathbf\{5752\\pm 158\}/1662\\pm 272Linear curr\.\(SafeExplorer, ref\.\)dmax=2\.0d\_\{\\max\}\{=\}2\.0:3581±971/𝟏𝟒±𝟖3581\\pm 971/\\mathbf\{14\\pm 8\}dmax=0\.4d\_\{\\max\}\{=\}0\.4:𝟐𝟖𝟓𝟕±𝟏𝟐𝟗/754±463\\mathbf\{2857\\pm 129\}/754\\pm 463dmax=0\.15d\_\{\\max\}\{=\}0\.15:5594±177/731±1775594\\pm 177/731\\pm 177

HalfCheetah\.Every fixed\-ddcell pays∼1,200\\sim 1\{,\}200to3,6003\{,\}600training falls and tops out near2,5002\{,\}500to3,6003\{,\}600reward, while the linear\-curriculum reference reaches3,5813\{,\}581reward at only1414falls \(an85×85\\timesto260×260\\timesfalls advantage\)\. No constantddmatches the curriculum’s reward\-at\-low\-falls\. The small\-ddcells stay safer but low\-reward and the large\-ddcells climb in falls without gaining reward\.

Ant\.The smallest fixed\-dd\(p25,d=0\.1d=0\.1\) is very safe \(55falls\) but useless \(990990reward\)\. Larger constantddaccumulates5,5005\{,\}500to21,80021\{,\}800falls without reaching the curriculum’s reward\. The linear curriculum gets28572857reward at754754falls, which no constantddmatches\.

Go1\.OnGo1the comparison is closer, and the curriculum does*not*strictly dominate\. A tuned constantddis competitive\. Fixed\-dd\-p50 \(5423/3975423/397\) attains essentially the curriculum’s reward \(55945594\) at*fewer*falls \(397397vs731731\), and p75 and p100 trade modest extra falls for similar reward\. The curriculum’s value onGo1is therefore robustness to the choice ofdd, it reaches high reward at low falls without a per\-environmentddsearch, rather than a strict win over the best constantdd\.

Summary\.OnHalfCheetahandAntno constantddmatches the schedule’s high reward at comparable safety\. OnGo1a tuned fixed\-dd\(p50\) is competitive on both axes, so there the schedule’s contribution is sparing the practitioner a per\-environmentddsearch rather than a strict dominance\. The safe\-region schedule is a supporting mechanism for the objective\-gap bound \([Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\)\. The method’s correctness rests on the unbiased masked gradient \([Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)\), not on the schedule\.

### E\.7Curriculum\-schedule sensitivity \(SafeExplorer onAnt\)

Table 11:Schedule shape onAntSafeExplorer\. Reward / Falls, mean±\\pmstd over the first55of1010seeds; v10 sweep batch at the wider radius \(dmax=0\.56d\_\{\\max\}=0\.56\) rather than the body0\.40\.4, so within\-table comparisons are matched but absolute values differ from[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\.Boldmarks the best value per column; Linear and Log rewards tie within seed noise and are both bolded\.ScheduleRewardFallsLinear𝟐𝟖𝟖𝟖±𝟑𝟓𝟏\\mathbf\{2888\\pm 351\}𝟗𝟓𝟐±𝟏𝟏𝟐𝟏\\mathbf\{952\\pm 1121\}Log𝟐𝟗𝟎𝟔±𝟑𝟕𝟒\\mathbf\{2906\\pm 374\}1086±4381086\\pm 438Step \(4 jumps\)1537±5491537\\pm 5495998±27885998\\pm 2788Constant \(d=dmaxd=d\_\{\\max\}\)294±74294\\pm 7429855±474729855\\pm 4747Granting that some schedule beats a constantdd, the next question is which one\.[Table˜11](https://arxiv.org/html/2607.08925#A5.T11)reports three alternatives against the linear default\. Linear and log are tied within seed noise on every metric\. The step schedule is worse on both, with lower reward and6×6\\timesmore falls\. Constant\-dd\(no curriculum\) collapses, accumulating31×31\\timesthe falls of the linear curriculum at a tenth of its reward\. This is consistent with the fixed\-ddablation above \([Section˜E\.6](https://arxiv.org/html/2607.08925#A5.SS6)\), whose constantd=dmaxd\{=\}d\_\{\\max\}cell shows the same qualitative collapse \(at the body radius rather than this table’s wider one\), and with the prediction of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)that the gap closes asβ\\betashrinks, which the curriculum drives\.

### E\.8Compute\-scaling:2×2\\timesmatched\-budget onHalfCheetahandAnt

Table 12:Compute\-scaling: SafeExplorer vsPPOat the main budget \(88M\) and2×2\\times\(1616M\) onHalfCheetahandAnt\. Reward / Falls, mean±\\pmstd over the first55of1010seeds\. All cells are the v10 batch at the wider radii \(dmax=3\.2d\_\{\\max\}=3\.2onHalfCheetah,0\.560\.56onAnt\) rather than the body protocol; the1616M runs stretch thedd\-curriculum over the doubled budget\.[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)’s body\-radius values \(HalfCheetahSafeExplorer3581±9713581\\pm 971/14±814\\pm 8\) therefore differ from the Main column here; see the preamble of this appendix\.Boldmarks the better method per environment and budget\.MethodMain \(88M\)2×2\\times\(1616M\)HalfCheetahPPO3163±966/5266±25393163\\pm 966/5266\\pm 25393590±1969/15112±85003590\\pm 1969/15112\\pm 8500SafeExplorer𝟒𝟎𝟐𝟕±𝟏𝟐𝟑𝟗/𝟐𝟒𝟖±𝟐𝟖𝟏\\mathbf\{4027\\pm 1239/248\\pm 281\}𝟓𝟗𝟏𝟐±𝟖𝟖𝟏/𝟐𝟏𝟕±𝟑𝟒𝟎\\mathbf\{5912\\pm 881/217\\pm 340\}AntPPO296±35/32369±5625296\\pm 35/32369\\pm 5625197±92/88900±35552197\\pm 92/88900\\pm 35552SafeExplorer𝟐𝟖𝟖𝟖±𝟑𝟓𝟏/𝟗𝟓𝟐±𝟏𝟏𝟐𝟏\\mathbf\{2888\\pm 351/952\\pm 1121\}𝟐𝟗𝟓𝟎±𝟒𝟏𝟗/𝟑𝟑𝟑𝟐±𝟐𝟔𝟐𝟎\\mathbf\{2950\\pm 419/3332\\pm 2620\}

With the design choices settled, we turn to whether the chosen configuration holds up under conditions the body did not exercise, starting with more compute\. We re\-runPPOand SafeExplorer at2×2\\timesthe main budget \(8​M→16​M8\\mathrm\{M\}\\to 16\\mathrm\{M\}\) onHalfCheetahandAnt\([Table˜12](https://arxiv.org/html/2607.08925#A5.T12); wider radii than the body protocol, see the caption\)\. SafeExplorer’s reward improves or holds with2×2\\timescompute \(HalfCheetah4027→59124027\\to 5912,Ant2888→29502888\\to 2950\) while its falls stay low\.PPO’s falls instead grow sharply \(HalfCheetah∼2\.9×\\sim 2\.9\\times,Ant∼2\.7×\\sim 2\.7\\times\), so the safety gap*widens*with budget rather than closing\. Re\-tuningdmaxd\_\{\\max\}for longer horizons or a multi\-stage curriculum is left to follow\-up work\.

### E\.9Compatibility\-regularizer normalization \(global vs\. per\-segment\)

Table 13:Compat\-norm: global vs\. per\-segment\. Reward / Falls, mean±\\pmstd over the first55of1010seeds; v10 batch at the wider radii \(dmax=3\.2d\_\{\\max\}=3\.2HalfCheetah,0\.560\.56Ant\)\.Boldmarks the arm better on both metrics per environment\.Envglobal \(default\)per\-segmentHalfCheetah4027±1239/248±2814027\\pm 1239/248\\pm 281𝟓𝟎𝟑𝟎±𝟔𝟔𝟒/𝟒𝟖±𝟑𝟔\\mathbf\{5030\\pm 664/48\\pm 36\}Ant𝟐𝟖𝟖𝟖±𝟑𝟓𝟏/𝟗𝟓𝟐±𝟏𝟏𝟐𝟏\\mathbf\{2888\\pm 351/952\\pm 1121\}2474±489/2625±15832474\\pm 489/2625\\pm 1583TheCθ​\(τ\)C\_\{\\theta\}\(\\tau\)loss in[Equation˜9](https://arxiv.org/html/2607.08925#S4.E9)averages across all recovery\-active timesteps in the current minibatch \(“global”\)\. An alternative “per\-segment” normalization averages within each contiguous safe→\\tounsafe→\\tosafe segment, then over segments, which prevents long unsafe excursions from dominating the loss\.[Table˜13](https://arxiv.org/html/2607.08925#A5.T13)reports both onHalfCheetahandAnt\. Per\-segment helps onHalfCheetah\(higher reward, fewer falls\) but hurts onAnt\(lower reward, more falls\)\. With no consistent winner across environments and the means within roughly one standard deviation, we keepglobalas the simpler default\.

### E\.10Action\-noise robustness \(HalfCheetahSafeExplorer\)

Table 14:Action\-noise onHalfCheetahSafeExplorer atλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}\. Reward / Falls, mean±\\pmstd over the first55of1010seeds; v10 batch at the widerHalfCheetahradius \(dmax=3\.2d\_\{\\max\}=3\.2\)\. Theσa=0\\sigma\_\{a\}=0arm is the same runs as the global arm of[Table˜13](https://arxiv.org/html/2607.08925#A5.T13)and the Main SafeExplorer cell of[Table˜12](https://arxiv.org/html/2607.08925#A5.T12)\.σa\\sigma\_\{a\}RewardFalls0\.000\.004027±12394027\\pm 1239248±281248\\pm 2810\.050\.054482±4194482\\pm 419108±73108\\pm 730\.100\.102346±9642346\\pm 964546±395546\\pm 395A second stress axis perturbs the actions themselves\. We re\-train SafeExplorer onHalfCheetahwith Gaussian action noise of standard deviationσa∈\{0\.0,0\.05,0\.10\}\\sigma\_\{a\}\\in\\\{0\.0,0\.05,0\.10\\\}added to every action \(recovery and main\) at theλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}default \([Table˜14](https://arxiv.org/html/2607.08925#A5.T14)\)\. Reward and falls stay within seed noise up toσa=0\.05\\sigma\_\{a\}=0\.05, then degrade atσa=0\.10\\sigma\_\{a\}=0\.10\(−42%\-42\\%reward, more than double the falls\)\. An action\-noise sweep at the HC\-tunedλcompat=10−2\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-2\}is left to follow\-up work\. We also ranσa=0\.20\\sigma\_\{a\}=0\.20as a stress test\. SafeExplorer collapses \(1863±5421863\\pm 542reward,1659±18071659\\pm 1807falls\) when the injected noise is comparable in scale to the recovery’s action magnitude, and we omit it from the table as a known out\-of\-regime failure mode\.

### E\.11Failure\-mode catalog

The stress tests above probe how the chosen configuration degrades; this catalog records the qualitative failure each rejected variant exhibits, so the quantitative gaps in the ablations have a concrete behavioral reading\.

- •PPOonGo1: agent falls within first 200 steps; recovery is unavailable; episode terminates\.
- •Analytic\-VVvariantonAnt: agent learns to balance briefly but never escapes the recovery policy’s basin;α\\alphastays near 1 for the full training run\.
- •Soft\-gate variantonHalfCheetah: regularizer fits failed\-recovery actions; reward oscillates; final policy is unreliable\.
- •SafeExploreronAnt: clean separation,α\\alphadrops to near zero around update 1000, reward climbs steadily thereafter, falls plateau\.

### E\.12Empirical check of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)

We now turn from the method’s empirical robustness to its theory, checking each of the paper’s two theorems against the runs, starting with the objective\-gap bound\.[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)bounds the deployment gap\|Jmix​\(θk\)−J​\(θk\)\|\|J^\{\\mathrm\{mix\}\}\(\\theta\_\{k\}\)\-J\(\\theta\_\{k\}\)\|, between the during\-training mixed\-policy return and the return ofπθ\\pi\_\{\\theta\}deployed alone, by the main\-policy out\-of\-region rateβ​\(θ,d\)\\beta\(\\theta,d\)\. The bound is a proportionality, not a vanishing guarantee\. It predicts a small gap exactly whenβ​\(θ,d\)\\beta\(\\theta,d\)is small, and[Corollary˜5](https://arxiv.org/html/2607.08925#Thmtheorem5)sharpens this to an exact fixed point only in the idealizedβ​\(θ,d\)→0\\beta\(\\theta,d\)\\to 0limit\. We test that prediction directly\. Every 4PPOupdates we run a recovery\-disabled evaluation pass, averaging55episodes per pass, that samples from the same stochastic policyπθ\(⋅\|s\)=𝒩\(μθ\(s\),σθ2\)\\pi\_\{\\theta\}\(\\cdot\|s\)=\\mathcal\{N\}\(\\mu\_\{\\theta\}\(s\),\\sigma\_\{\\theta\}^\{2\}\)used in training, the operativeJ​\(θk\)J\(\\theta\_\{k\}\), and compare it to the mixed\-policy rollout returnJmix​\(θk\)J^\{\\mathrm\{mix\}\}\(\\theta\_\{k\}\)\. Becauseβ​\(θ,d\)\\beta\(\\theta,d\)is an out\-of\-region rate underπθ\\pi\_\{\\theta\}alone and is not directly observable, we report alongside it the observable recovery rateα\\alpha, which[Section˜4\.2](https://arxiv.org/html/2607.08925#S4.SS2)identifies as a diagnostic forβ​\(θ,d\)\\beta\(\\theta,d\)rather than a certified upper bound\. A deterministic pass usingμθ​\(s\)\\mu\_\{\\theta\}\(s\)alone is also logged as a noise\-reduced deployment diagnostic but is not the quantity the theorem concerns\.

Table 15:End\-of\-training values for SafeExplorer at the body operating point, first55of1010seeds, over the last20%20\\%of training; an independent batch shared with theλcompat=10−3\\lambda\_\{\\mathrm\{compat\}\}=10^\{\-3\}arm of[Table˜9](https://arxiv.org/html/2607.08925#A5.T9)onHalfCheetah/Ant, withGo1from the body\-1616M family\. Columns: the mixed\-policy returnJmixJ^\{\\mathrm\{mix\}\}, theπθ\\pi\_\{\\theta\}\-only stochastic eval returnJJ, their signed difference, the recovery rateα\\alpha\(observable diagnostic forβ​\(θ,d\)\\beta\(\\theta,d\)\), cumulative training\-time falls, and per\-episode eval\-time falls underπθ\\pi\_\{\\theta\}alone \(recovery disabled\)\.EnvJmixJ^\{\\mathrm\{mix\}\}JJJmix−JJ^\{\\mathrm\{mix\}\}\-Jα\\alphaFalls \(train\)Falls \(eval/ep\)HalfCheetah4677±5424677\\pm 5424676±5524676\\pm 5520±120\\pm 120\.0000\.00047±4147\\pm 410\.000\.00Ant2904±1452904\\pm 1452921±1512921\\pm 151−17±25\-17\\pm 250\.0000\.000428±454428\\pm 4540\.120\.12Go15576±2135576\\pm 2135112±3405112\\pm 340463±150463\\pm 1500\.044±0\.0050\.044\\pm 0\.005768±241768\\pm 2410\.360\.36The result follows the bound’s structure precisely, and the key observation is that*the gap tracksβ​\(θ,d\)\\beta\(\\theta,d\)*\. OnHalfCheetahandAntthe recovery rate falls to essentially zero \(α=0\.000\\alpha=0\.000on both at end\-of\-training\), and the gap closes to within seed noise of zero \(0±120\\pm 12and−17±25\-17\\pm 25\)\. These are the self\-stable environments\. Hereπθ\\pi\_\{\\theta\}holds the safe region unaided, soπθmix≡πθ\\pi^\{\\mathrm\{mix\}\}\_\{\\theta\}\\equiv\\pi\_\{\\theta\}along the trajectory,β​\(θ,d\)→0\\beta\(\\theta,d\)\\to 0, and the fixed point of[Corollary˜5](https://arxiv.org/html/2607.08925#Thmtheorem5)is reached up to the invariance\-slack residualη⋆\\eta\_\{\\star\}the corollary anticipates \(the residuals here are statistically indistinguishable from zero; empirically it is the stochastic policy’s entropy floor that keepsη⋆\\eta\_\{\\star\}from being exactly zero\)\. OnGo1the policy does not become fully self\-stable, because the quadruped’s gait cannot stay inside the tightℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)tube\. The recovery rate therefore settles at a small positive floor \(α=0\.044±0\.005\\alpha=0\.044\\pm 0\.005\) rather than at zero, and the gap settles at a correspondingly positive residual \(463±150463\\pm 150, about8%8\\%ofJmixJ^\{\\mathrm\{mix\}\}\) rather than vanishing\. After the early\-curriculum transient the gap collapses from its peak onHalfCheetah, stays within seed noise of zero onAntin the final average despite a late upward drift in the plotted per\-seed magnitude, and declines but stays positive onGo1\([Figure˜5](https://arxiv.org/html/2607.08925#A5.F5)\)\. OnGo1it falls from∼1950\\sim 1950when the safe region is small to a small positive residual at1616M env\-steps\. The end\-of\-training gap and recovery rate in[Table˜15](https://arxiv.org/html/2607.08925#A5.T15)make the proportionality explicit\. The gap is near zero exactly whereα\\alphais, and the only environment with a residual gap is the only one with a residualα\\alpha\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/thm2_gap_validation.png)Figure 5:Objective gap over training for the SafeExplorer runs of[Table˜15](https://arxiv.org/html/2607.08925#A5.T15): the curve is the seed\-mean of the per\-seed absolute gap\|Jmix​\(θk\)−J​\(θk\)\|\|J^\{\\mathrm\{mix\}\}\(\\theta\_\{k\}\)\-J\(\\theta\_\{k\}\)\|, smoothed with a 9\-point moving average for display; shaded =±1\\pm 1std\. Because the curve averages magnitudes while[Table˜15](https://arxiv.org/html/2607.08925#A5.T15)reports the signed across\-seed mean over the final20%20\\%of training, the curve’s end value can sit above the table’s near\-zero signed gap \(as onAnt\)\.This two\-regime behavior is a stronger test of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)than a uniform collapse to zero would be\. The bound does not assert that the gap vanishes; it asserts that the gap is governed byβ​\(θ,d\)\\beta\(\\theta,d\)\.HalfCheetahandAntrealize the vanishing regime \(β​\(θ,d\)→0\\beta\(\\theta,d\)\\to 0\), andGo1realizes the residual regime \(β​\(θ,d\)\\beta\(\\theta,d\)bounded away from zero because[˜2](https://arxiv.org/html/2607.08925#Thmassumption2)holds only approximately for a quadruped confined to a tight tube\), with the observableα\\alphamoving in lockstep with the gap in both\. The sign of the gap is itself interpretable\. OnGo1,Jmix−J\>0J^\{\\mathrm\{mix\}\}\-J\>0means that deployingπθ\\pi\_\{\\theta\}alone forfeits the return that the recovery policy would otherwise secure, so the gap measures the policy’s residual reliance on the recovery policy, whichHalfCheetahandAntdrive to zero andGo1reduces to about8%8\\%\.

The absolute prefactor of[Theorem˜4](https://arxiv.org/html/2607.08925#Thmtheorem4)\(2​rmax/\(1−γ\)2≈2×1052r\_\{\\max\}/\(1\-\\gamma\)^\{2\}\\approx 2\\times 10^\{5\}forrmax=10r\_\{\\max\}=10,γ=0\.99\\gamma=0\.99\) is loose, as is standard for performance\-difference\-lemma bounds, so we do not read the inequality quantitatively\. Its operative content is the structural claim that any mechanism reducingβ​\(θ,d\)\\beta\(\\theta,d\), here the radius curriculum together with the compatibility regularizer, tightens the gap\.[Figure˜5](https://arxiv.org/html/2607.08925#A5.F5)confirms that claim across the full range ofβ​\(θ,d\)\\beta\(\\theta,d\)our three environments realize\. Eval\-time falls underπθ\\pi\_\{\\theta\}alone \(recovery disabled\) corroborate the reading\. They are0\.000\.00,0\.120\.12, and0\.360\.36per episode onHalfCheetah,Ant, andGo1, ordered exactly as the gaps and the recovery rates are\.[Figure˜6](https://arxiv.org/html/2607.08925#A5.F6)plots the full recovery\-rate trajectories that this diagnostic summarizes\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/alpha_curves_appendix.png)Figure 6:Recovery rateα\\alpha\(fraction of rollout steps on which recovery fired\) over training, per environment, for the recovery\-using methods \(theCMDPsolvers act with a single policy and never invoke recovery, so they have noα\\alpha\)\. Solid = mean across seeds, shaded = \[min, max\] envelope\.
### E\.13Masked gradient versusISunder a stochastic recovery

With the objective\-gap bound checked, we turn to the gradient estimator:[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)makes the masked estimator exactly unbiased at recovery steps, while per\-stepIS, where it applies at all, is biased in general \([Remark˜1](https://arxiv.org/html/2607.08925#Thmremark1)\); what remains empirical is how the two compare in practice\.[Theorem˜1](https://arxiv.org/html/2607.08925#Thmtheorem1)holds for anyθ\\theta\-independent recovery; for a deterministic one, not evenISapplies \([Section˜4\.1](https://arxiv.org/html/2607.08925#S4.SS1)\)\. Here we ask what happens when the recovery*is*stochastic, soISbecomes an option\. Writingwt=πθ​\(at∣st\)/πrec​\(at∣st\)w\_\{t\}=\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)/\\pi^\{\\mathrm\{rec\}\}\(a\_\{t\}\\mid s\_\{t\}\)for the importance weight at a recovery step, theISestimator of∇θJmix​\(θ\)\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\(\\theta\)\([Corollary˜2](https://arxiv.org/html/2607.08925#Thmtheorem2)\) decomposes as

g^IS=∑t:st∈ℛ∇θlog⁡πθ​\(at∣st\)​At⏟g^masked\+∑t:st∉ℛwt​∇θlog⁡πθ​\(at∣st\)​At,\\hat\{g\}\_\{\\mathrm\{IS\}\}=\\underbrace\{\\sum\_\{t:\\,s\_\{t\}\\in\\mathcal\{R\}\}\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,A\_\{t\}\}\_\{\\hat\{g\}\_\{\\mathrm\{masked\}\}\}\\;\+\\;\\sum\_\{t:\\,s\_\{t\}\\notin\\mathcal\{R\}\}w\_\{t\}\\,\\nabla\_\{\\theta\}\\log\\pi\_\{\\theta\}\(a\_\{t\}\\mid s\_\{t\}\)\\,A\_\{t\},\(15\)withAtA\_\{t\}thePPOadvantage estimate\. The second sum would be mean zero by the score\-function identity only ifAtA\_\{t\}were replaced by an action\-independent baselineb​\(st\)b\(s\_\{t\}\); with the actual advantage, which depends onata\_\{t\}through the reward and the successor state, its conditional mean is the off\-policy policy\-gradient contribution at the unsafe states, which is generically nonzero\. The per\-stepISestimator is therefore biased for∇θJmix\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}, and masking removes a bias\-carrying term, not merely a noisy one\. A recovery policy differs sharply from the main policy by design, so the term’s weights are heavy\-tailed; truncating them\(Espeholt et al\.,[2018](https://arxiv.org/html/2607.08925#bib.bib13); Munos et al\.,[2016](https://arxiv.org/html/2607.08925#bib.bib34)\)trades variance for further bias\. The measurement below therefore compares two estimators that do not share a mean: the recorded explosion of theISweights is evidence about the conditioning of the added term, not a like\-for\-like variance comparison between unbiased estimators\.

Concretely, we rerun SafeExplorer on all three environments \(settings of[Section˜6](https://arxiv.org/html/2607.08925#S6)\) with the recovery action*sampled*from theSACrecovery rather than taken greedily at its mean, and shadow\-compute, per rollout, the weights’ effective sample size and each estimator’s gradient variance\. Training itself always uses the masked gradient\.

[Figure˜7](https://arxiv.org/html/2607.08925#A5.F7)shows that the three regimes expose complementary failure modes\. OnHalfCheetahthe strong recovery makes the weights degenerate from the first update\. The effective sample size never exceeds0\.5%0\.5\\%of the recovery steps, and theISgradient variance exceeds the masked variance by a factor of up to8×10128\\times 10^\{12\}\. OnAntthe weak recovery initially overlaps the main policy andISstays formally usable for longer, but the variance ratio holds near101310^\{13\}for the first1\.21\.2M steps\.Go1completes the spectrum\. Over its 12\-dimensional action space the recovery’s actions are so unlikely under the main policy that the weights die at once \(effective sample size always below3%3\\%, reaching7×10−57\\times 10^\{\-5\}\), so the added term vanishes before its variance can build, and the ratio never rises above∼40\\sim\\\!40and sits at one for the full1616M steps, over which the quadruped’s recovery rate never reaches zero \([Section˜E\.12](https://arxiv.org/html/2607.08925#A5.SS12)\)\. Once the main policy separates from the recovery the weights vanish and the ratio returns to one\.ISproves either catastrophically noisy or identical to the masked estimator, never better in any regime we measure\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/fig_is_variance.png)Figure 7:Masked gradient versusISon a stochastic recovery, SafeExplorer \(mean±\\pmstd\)\.*Left*: the effective sample size of the importance weights collapses toward zero when the recovery is active, the signature of weight degeneracy\.*Right*: the ratio ofISto masked gradient variance \(log scale; dashed line at one\)\. The masked estimator targets∇θJmix\\nabla\_\{\\theta\}J^\{\\mathrm\{mix\}\}\([Equation˜15](https://arxiv.org/html/2607.08925#A5.E15)\) without theISterm’s bias and variance pathologies\. TheHalfCheetahandAntcurves end where the recovery stops engaging, because the diagnostics exist only on updates with recovery steps; onGo1recovery stays mildly engaged throughout, so the curve spans the full run\.Training with the truncated\-ISgradient \(wtw\_\{t\}clipped atc=10c=10\) confirms the cost \([Figure˜8](https://arxiv.org/html/2607.08925#A5.F8)\)\. These runs use the sampled\-recovery setup of this appendix, so their absolute rewards are not comparable to[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)’s greedy\-recovery numbers; only the within\-setup masked\-vs\-ISdifferences are meaningful\. OnHalfCheetahthe weights die immediately and the two variants are statistically indistinguishable \(reward4675±6424675\\pm 642versus4421±6474421\\pm 647\)\. OnAntthe million\-step degeneracy window is expensive:793±214793\\pm 214reward against1325±2891325\\pm 289for the masked gradient, a40%40\\%drop, and its lower fall count \(2066±15822066\\pm 1582versus4682±10474682\\pm 1047\) reflects the weaker policy rather than safer behavior\. OnGo1the truncated\-ISvariant again trails the masked gradient \(4884±2414884\\pm 241versus5243±2195243\\pm 219reward, a7%7\\%drop\) at statistically indistinguishable falls \(3293±5893293\\pm 589versus3401±3513401\\pm 351\)\.

![Refer to caption](https://arxiv.org/html/2607.08925v1/figures/fig_is_train.png)Figure 8:Training SafeExplorer with the masked gradient versus the truncated\-ISgradient \(mean±\\pmstd\)\.*Left*: final reward\.*Right*: total training falls\.
### E\.14CMDPbaselines: constraint feasibility

Having checked the method’s own theory, we finally audit the baselines it is compared against, asking whether theCMDPsolvers even satisfy the constraint they are given\. This appendix supports theCMDPcomparison of[Section˜7\.3](https://arxiv.org/html/2607.08925#S7.SS3)\. For the comparison to be fair, both solvers use the OmniSafe implementation\(Ji et al\.,[2024](https://arxiv.org/html/2607.08925#bib.bib23)\)and run on the identical environments, reward functions, network architecture, optimizer, and step budget as the variants in[Table˜1](https://arxiv.org/html/2607.08925#S7.T1)\. We use the direct cost encoding, a cost of11on a fall \(a termination\), the actual safety signal the constraint should bound\. The only incidental differences from SafeExplorer are observation and reward normalization, applied to theCMDPbaselines and not to SafeExplorer, so they favor the baselines\. Reported rewards for both solvers are the raw episode return \(EpRetin OmniSafe\), un\-normalized and therefore on the same task\-reward scale as the other methods\. The normalization affects only the solvers’ internal optimization, not the reported numbers\.

On this matched footing,[Table˜16](https://arxiv.org/html/2607.08925#A5.T16)reports the final episodic cost each solver converges to, against the cost budget of0\.050\.05it was given \(a per\-episode fall rate\), averaged over the same seeds\. The pattern is clean\. Both solvers reach feasibility onHalfCheetah, where staying safe is easy, and both*fail*to reach feasibility onAntandGo1, where the converged cost exceeds the budget by up to19×19\\times\. The poor safety of theCMDPbaselines is therefore not a tuning artifact\. The constrained optimization does not find a feasible policy at all in the regimes where the safe\-region intervention is needed\.

Table 16:Converged episodic cost vs\. the cost budget \(≤0\.05\\leq 0\.05\) for theCMDPbaselines \(final5%5\\%of training\)\.✓feasible,×\\timesbudget violated\.MethodHalfCheetahAntGo1CPO0\.007±0\.0110\.007\\pm 0\.011✓0\.103±0\.0290\.103\\pm 0\.029×\\times0\.956±0\.0250\.956\\pm 0\.025×\\timesPPO\-Lagrangian0\.001±0\.0010\.001\\pm 0\.001✓0\.326±0\.1770\.326\\pm 0\.177×\\times0\.377±0\.1260\.377\\pm 0\.126×\\timesA safe\-region indicator cost \(cost11at every step outsideℛ​\(dmax\)\\mathcal\{R\}\(d\_\{\\max\}\)\) is a second natural encoding, but it is comparable to the recovery\-using methods only when the recovery policy is active, since that is the setting in which an out\-of\-region penalty reduces to recovery\-active reward shaping\. We therefore report the direct termination encoding in the main comparison and leave the recovery\-active safe\-region variant to future work\.

## Appendix FBroader Impact

The goal of this work is to make reinforcement learning on physical robots safer to train\. Achieving that goal would reduce hardware damage during training, lower the experimental cost borne by robot\-learning research groups, and provide a foundation for on\-robot continual learning that does not depend on extensive pre\-training in simulation\.

Set against that intent, the dual\-use surface of the technique is narrow\. It is dual\-use only in the trivial sense that any improvement in robotic control technology is so, because the method does not change the capabilities a deployed policy can express, only the training\-time safety profile of how that policy is reached\. For this reason we do not foresee specific misuse pathways introduced by this work that are not already present in the underlyingPPO\+SAC\+ MuJoCo / Go1 stack\.

The experiments themselves are deliberately scoped to limit any training\-time harm\. We do not collect or release human\-subject data and do not perform experiments that could affect bystanders during training\. All experiments are in simulation \(MuJoCo and the Go1 model\), so the training\-time falls we reduce are simulated rather than physical; carrying that reduction onto hardware is future work \([Section˜9](https://arxiv.org/html/2607.08925#S9)\)\.

Because the contribution is methodological, its value depends on being reproducible and affordable, and we account for both: the full algorithm, hyperparameter, and environment specifications needed to rerun the experiments are given in[Appendices˜B](https://arxiv.org/html/2607.08925#A2),[C](https://arxiv.org/html/2607.08925#A3)and[D](https://arxiv.org/html/2607.08925#A4)\. We report negative results alongside the positive ones, the soft\-gate variants, the fixed\-ddconcessions, and the action\-noise stress failures \([Sections˜E\.4](https://arxiv.org/html/2607.08925#A5.SS4),[E\.6](https://arxiv.org/html/2607.08925#A5.SS6)and[E\.11](https://arxiv.org/html/2607.08925#A5.SS11)\), so that future work can avoid repeating them\. The compute budget for the component ablation \(99variants×\\times33environments×\\times55seeds, 4 cores each, weighted by per\-environment wall\-clock\) is approximately3,7003\{,\}700CPU\-core\-hours, modest relative to contemporary safe\-RLbenchmarks; the additional sensitivity sweeps in[Appendix˜E](https://arxiv.org/html/2607.08925#A5)add further compute\.