Safe Online Learning via Smooth Safety-Structured Policy Composition

# Safe Online Learning via Smooth Safety-Structured Policy Composition
Source: [https://arxiv.org/html/2606.31320](https://arxiv.org/html/2606.31320)
Hongpeng Cao†cao\.hongpeng@tum\.de School of Engineering and Design Technical University of Munich Garching, Munich 85748, GermanyLiqun Zhao†liqun\.zhao@eng\.ox\.ac\.uk Department of Engineering Science University of Oxford Oxford OX1 3PJ, United KingdomYuliang Gu†yuliang3@illinois\.edu Department of Mechanical Science and Engineering University of Illinois Urbana\-Champaign Urbana, IL 61801, USANaira Hovakimyannhovakim@illinois\.edu Department of Mechanical Science and Engineering University of Illinois Urbana\-Champaign Urbana, IL 61801, USALui Shalrs@illinois\.edu Department of Computer Science University of Illinois Urbana\-Champaign Urbana, IL 61801, USAMarco Caccamomcaccamo@tum\.de School of Engineering and Design Technical University of Munich Garching, Munich 85748, Germany

###### Abstract

Safe online reinforcement learning requires policies to respect safety constraints while maintaining smooth optimization dynamics\. Existing approaches typically rely on either strict safety enforcement via action interventions, which introduce discontinuities in system interaction and learning, or soft safety constraint formulations, which preserve smooth learning but provide limited safety assurance\. We propose*AutoSafe*, a safety\-aware policy architecture that integrates structured safety monitoring and intervention directly into the action generation process\. This design enables smooth, risk\-dependent transitions between performance\-driven and safety\-preserving behaviors, resulting in continuous online interaction and learning dynamics\. Empirical results across a suite of continuous\-control benchmarks demonstrate strong safety enforcement without sacrificing learning smoothness\. We further validate AutoSafe on a physical cart\-pole system, highlighting its practical effectiveness for safe online learning in the real world\.

†\\daggerEqual contribution\.

## 1Introduction

Enabling deep reinforcement learning \(DRL\) agents to interact safely with their environment while maintaining stable and continual policy improvement remains a central challenge for real\-world learning systems\(Ibarzet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib134)\)\. In safety\-critical applications, however, safety requirements must be enforced as hard constraints, significantly complicating online learning\.

Among existing safe reinforcement learning \(RL\) approaches,*constrained policy optimization*methods\(Altman,[2021](https://arxiv.org/html/2606.31320#bib.bib121); Achiamet al\.,[2017](https://arxiv.org/html/2606.31320#bib.bib105)\)and*safety\-aware reward shaping*methods\(Westenbroeket al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib108);[Caoet al\.,](https://arxiv.org/html/2606.31320#bib.bib18)\)typically formulate safety requirements as*soft constraints*, allowing temporary violations during learning in exchange for improved long\-term performance\. An advantage of this formulation is that it preserves smooth gradient propagation during policy optimization\. While effective in some settings, this paradigm implicitly assumes that the agent learns to be safe through experience with safety violations, which may be unacceptable in safety\-critical applications\.

In contrast,*safety filter\-based*methods\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6); Zhonget al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib27); Alshiekhet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib11); Chenget al\.,[2019b](https://arxiv.org/html/2606.31320#bib.bib10)\)enforce safety through explicit intervention mechanisms that preemptively modify unsafe actions\. While such hard enforcement can provide stronger formal safety guarantees, it may introduce abrupt action corrections\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\)that disrupt gradient flow and destabilize online and continual learning, particularly under frequent interventions\. Recent work has explored differentiable safety filters\(Amos and Kolter,[2017](https://arxiv.org/html/2606.31320#bib.bib146); Xiaoet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib147); Jinet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib148); Markgrafet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib140); Suttleet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib149)\)that have helped address the gradient\-flow issue in safe learning\. However, many existing approaches still rely on expensive online optimization or are mainly evaluated in simplified or offline settings\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib147); Suttleet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib149)\)\. As a result, scalability to higher\-dimensional systems remains an important challenge\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\), particularly in safe online learning scenarios that require efficient safety intervention in time\.

To resolve the fundamental trade\-off between smooth policy optimization and strict safety enforcement, we propose*AutoSafe*, a simple yet effective*safety\-aware policy architecture*that combines strong real\-time safety assurance with smooth learning dynamics\.

*AutoSafe*introduces two key innovations compared to existing safe RL approaches\. First, it incorporates risk monitoring and safe intervention as*structural inductive biases*through a differentiable policy composition\. Rather than treating safety as an external post\-processing step\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\), AutoSafe integrates safety awareness directly into the action\-generation process, thereby reshaping the policy parameterization and its learning dynamics\. Crucially, the differentiable composition allows learning signals to backpropagate through the safety mechanism, enabling stable and efficient policy optimization under safety constraints\. As a result, the resulting policy exhibits inherently safety\-aware interactions with the environment: it behaves conservatively in high\-risk regions while gradually releasing safety constraints and reverting to nominal performance\-driven behavior as risk diminishes\.

Second, in contrast to typical*last\-moment*intervention mechanisms used in safety filters, where corrective actions are applied only when a safety condition is triggered, inducing hard intervention\. AutoSafe embeds a*safe policy prior*defined over the entire state space and explicitly guides the intervention process\. This prior provides a reliable fallback behavior for safety enforcement and can be constructed using well\-established safe controller design methods\(Freeman and Kokotovic,[2008](https://arxiv.org/html/2606.31320#bib.bib2); Grandiaet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib124); Sha,[2001](https://arxiv.org/html/2606.31320#bib.bib107)\)\. By supplying well\-defined safe actions at all times, AutoSafe enables*precautionary*interventions before the system reaches the safety boundary, resulting in smoother state trajectories and more stable online learning\.

We compare AutoSafe against representative safety filter–based approaches and safe reinforcement learning baselines across multiple simulated benchmarks\. AutoSafe consistently exhibits stable learning dynamics, achieves strong safety assurance as standard safety filters, and matches or outperforms state\-of\-the\-art safe learning methods in task performance\. We further show the practical value of AutoSafe through a real\-world cart\-pole training experiment\.

Our contributions are summarized as follows: \(i\) We propose a safety\-aware policy architecture that embeds safety monitoring and intervention as*structural inductive biases*, enabling smooth action generation and stable online learning\. \(ii\) We provide theoretical insights into key properties of the proposed architecture, including its smooth intervention behavior and learning dynamics\. \(iii\) We empirically evaluate AutoSafe against representative safety filters and safety\-aware learning baselines in online policy learning settings across a suite of simulated continuous\-control tasks\. \(iv\) We demonstrate the practical applicability of AutoSafe through real\-world experiments on a cart\-pole system\.

## 2Preliminaries

### 2\.1Policy Learning under Hard Safety Constraints

We formulate the safe learning problem as an infinite\-horizon discounted Markov decision process \(MDP\), defined asℳ=\{𝒮,𝒜,P,R,γ\}\\mathcal\{M\}=\\\{\\mathcal\{S\},\\mathcal\{A\},P,R,\\gamma\\\}\. Here,𝒮⊆ℝn\\mathcal\{S\}\\subseteq\\mathbb\{R\}^\{n\}denotes thenn\-dimensional state space,𝒜⊆ℝm\\mathcal\{A\}\\subseteq\\mathbb\{R\}^\{m\}denotes themm\-dimensional action space,P:𝒮×𝒜→𝒮P:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathcal\{S\}is the state transition function,R:𝒮×𝒜→ℝR:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}is a bounded reward function, andγ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor\.

The objective of safe deep reinforcement learning under hard safety constraints is to find a policyπθ:𝒮→𝒜\\pi^\{\\theta\}:\\mathcal\{S\}\\rightarrow\\mathcal\{A\}that maximizes the expected discounted return while ensuring constraint satisfaction at all time steps:

maxπθ\\displaystyle\\max\_\{\\pi^\{\\theta\}\}\\quadVπθ​\(𝐬\)=𝔼τ∼πθ​\[∑t=0∞γt​R​\(𝐬t,𝐚t\)\|𝐬0=𝐬\],\\displaystyle V^\{\\pi^\{\\theta\}\}\(\\mathbf\{s\}\)=\\mathbb\{E\}\_\{\\tau\\sim\\pi^\{\\theta\}\}\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}R\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)~\\bigg\|~\\mathbf\{s\}\_\{0\}=\\mathbf\{s\}\\right\],\(1\)s\.t\.𝐬t∈𝒮c,∀t∈ℕ\.\\displaystyle\\mathbf\{s\}\_\{t\}\\in\\mathcal\{S\}\_\{c\},\\quad\\forall t\\in\\mathbb\{N\}\.\(2\)Here,τ=\{𝐬0,𝐚0,𝐬1,𝐚1,…\}\\tau=\\\{\\mathbf\{s\}\_\{0\},\\mathbf\{a\}\_\{0\},\\mathbf\{s\}\_\{1\},\\mathbf\{a\}\_\{1\},\\ldots\\\}denotes the trajectory induced by policyπθ\\pi^\{\\theta\}, and𝒮c⊆𝒮\\mathcal\{S\}\_\{c\}\\subseteq\\mathcal\{S\}denotes the set of admissible \(safe\) states\. Throughout this work, we consider a fully observable MDP with continuous state and action spaces\.

### 2\.2Safety Filter

A safety filter is an automatic mechanism that continuously monitors the state of an autonomous system and intervenes by modifying intended actions when safety risks are detected\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\)\. The goal of a safety filter is to enforce hard safety constraints during interaction, while allowing a learning\-based policy to optimize task performance whenever possible\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x1.png)\(a\)A safety filter that monitors system states and modifies actions when safety risks are detected\.
![Refer to caption](https://arxiv.org/html/2606.31320v1/x2.png)\(b\)AutoSafe policy architecture with differentiable convex composition of a learning\-based policy and a certified safe policy via a state\-dependent risk\-aware weightλ\\lambda\.

Figure 1:Diagram of a conventional safety filter architecture and the proposedAutoSafepolicy architecture\.As shown in Fig\.[1\(a\)](https://arxiv.org/html/2606.31320#S2.F1.sf1), typically, a safety filter consists of three components\. First, a*safety monitor*Δ:𝒮×\(𝒜\)→ℝ\\Delta:\\mathcal\{S\}\\times\(\\mathcal\{A\}\)\\rightarrow\\mathbb\{R\}evaluates the risk associated with a given state and \(possibly\) intended action by producing a scalar safety margin measurement\. Second, a*fallback \(safe\) policy*πsafe:𝒮×\(𝒜\)→𝒜\\pi^\{\\mathrm\{safe\}\}:\\mathcal\{S\}\\times\(\\mathcal\{A\}\)\\rightarrow\\mathcal\{A\}provides safe actions based on the state and the \(possibly\) intended unsafe action, which is usually deterministic and designed to keep the system state within the constraint set𝒮c\\mathcal\{S\}\_\{c\}\. Finally, an*intervention mechanism*η:𝒮×𝒜→𝒜\\eta:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathcal\{A\}determines when and how the fallback policy is applied\.

Most existing work on safety filters focuses on the design of safe policiesπsafe​\(⋅\)\\pi^\{\\mathrm\{safe\}\}\(\\cdot\), including*action replacement*\(Sha,[2001](https://arxiv.org/html/2606.31320#bib.bib107); Zhonget al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib27); Phanet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib112); Alshiekhet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib11)\)and*action projection*\(Chenget al\.,[2019a](https://arxiv.org/html/2606.31320#bib.bib19); Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Grandiaet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib124)\), as well as on the design of the safety monitorΔ​\(⋅\)\\Delta\(\\cdot\), which typically falls into*value\-based*\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Phanet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib112)\)or*rollout\-based*\(Chenet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib135); Bastani,[2021](https://arxiv.org/html/2606.31320#bib.bib136)\)approaches\.

In contrast, relatively little attention has been paid to the design of the intervention mechanism itself\. As a result, most existing safety filters adopt a binary intervention logic, either explicitly via action replacement or implicitly via optimization\-based action projection, of the following form:

η​\(𝐬t,𝐚tθ\)=\{𝐚tθ∼πθ\(⋅∣𝐬t\),if​Δ​\(𝐬t,𝐚tθ\)\>Δmin,𝐚tsafe=πsafe​\(𝐬t\),otherwise\.\\displaystyle\\eta\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}^\{\\theta\}\_\{t\}\)=\\begin\{cases\}\\mathbf\{a\}^\{\\theta\}\_\{t\}\\sim\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\_\{t\}\),&\\text\{if \}\\Delta\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}^\{\\theta\}\_\{t\}\)\>\\Delta\_\{\\text\{min\}\},\\\\\[4\.0pt\] \\mathbf\{a\}^\{\\mathrm\{safe\}\}\_\{t\}=\\pi^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\_\{t\}\),&\\text\{otherwise\.\}\\end\{cases\}Here,𝐚tθ\\mathbf\{a\}^\{\\theta\}\_\{t\}denotes the action generated by the learning\-based policyπθ\\pi^\{\\theta\}\.Δ​\(𝐬t,𝐚t\)\\Delta\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}\_\{t\}\)denotes the safety margin at time steptt, where the safety conditionΔ​\(𝐬t,𝐚tθ\)\>Δmin\\Delta\(\\mathbf\{s\}\_\{t\},\\mathbf\{a\}^\{\\theta\}\_\{t\}\)\>\\Delta\_\{\\text\{min\}\}is induced by a state–action–dependent safety certificate, such as a control barrier function, or by a state\-dependent safety measureΔ​\(𝐬t\)\\Delta\(\\mathbf\{s\}\_\{t\}\)defined with respect to the closed\-loop dynamics, e\.g\., a Lyapunov\-based safety value\. Satisfying this condition ensures that the system remains within a predefined safe set\.

## 3Safety\-filtered Actor\-Critic DRL

A key motivation for AutoSafe follows from the fact that online DRL interacts with the environment through*samples*from a stochastic policy\. With a conventional safety filter, unsafe samples are handled by a hard intervention to a deterministic safety fallback, which indeed enforces safety but creates an undesirable learning behavior: whenever the filter triggers, the executed action becomes*independent*ofπθ\\pi^\{\\theta\}\. As a result, the agent receives little learning signal about*how*to update the policy to avoid future safety intervention\.

To be more specific, most DRL algorithms for continuous control adopt an actor–critic architecture, in which a parameterized policy \(actor\) is optimized using learning signals from one or more critic networks\. Let’s denote the policy asπθ:𝒮→𝒜\\pi^\{\\theta\}:\\mathcal\{S\}\\rightarrow\\mathcal\{A\}and a function of some form of critic parameterized byψ\\psiforπθ\\pi^\{\\theta\}asCπθψ:𝒮×𝒜→ℝC^\{\\psi\}\_\{\\pi^\{\\theta\}\}:\\mathcal\{S\}\\times\\mathcal\{A\}\\rightarrow\\mathbb\{R\}\. The update gradient of the actor in a normal case without safety filter can be generally expressed as:

∇θJ​\(θ\)=𝔼𝐬∼𝒟πθ​\[∇θπθ​\(𝐬\)​∇𝐚θCπθψ​\(𝐬,𝐚θ\)\],\\nabla\_\{\\theta\}J\(\\theta\)=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}^\{\\pi^\{\\theta\}\}\}\\Big\[\\nabla\_\{\\theta\}\\pi^\{\\theta\}\(\\mathbf\{s\}\)\\,\\nabla\_\{\\mathbf\{a\}^\{\\theta\}\}C^\{\\psi\}\_\{\\pi^\{\\theta\}\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\Big\],\(3\)where𝒟πθ\\mathcal\{D\}^\{\\pi^\{\\theta\}\}denote samples generated byπθ\\pi^\{\\theta\}under the dynamic transition𝐬′∼P\(⋅∣𝐬,𝐚θ\)\\mathbf\{s\}^\{\\prime\}\\sim P\(\\cdot\\mid\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\. The specific form of the criticCπθψC^\{\\psi\}\_\{\\pi^\{\\theta\}\}depends on the underlying actor\-critic algorithm, such as advantage estimation with a state\-value critic\(Schulmanet al\.,[2017](https://arxiv.org/html/2606.31320#bib.bib46)\), action\-value critics\(Lillicrapet al\.,[2015](https://arxiv.org/html/2606.31320#bib.bib47); Fujimotoet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib130)\), or entropy\-regularized action\-value critics\(Haarnojaet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib76)\)\.

When a safety filter is applied to the interaction process, the action sampled from the policy𝐚θ∼πθ​\(𝐬\)\\mathbf\{a\}^\{\\theta\}\\sim\\pi^\{\\theta\}\(\\mathbf\{s\}\)is modified according to a safety intervention ruleη​\(⋅\)\\eta\(\\cdot\), yielding the executed action𝐚~=η​\(𝐬,πθ​\(𝐬\)\)\\tilde\{\\mathbf\{a\}\}=\\eta\\\!\\left\(\\mathbf\{s\},\\pi^\{\\theta\}\(\\mathbf\{s\}\)\\right\)\. We denote the resulting safety\-filtered policy byπ~θ\\tilde\{\\pi\}^\{\\theta\}\. Under this intervention, the actor update in Eq\.[3](https://arxiv.org/html/2606.31320#S3.E3)becomes

∇θJ​\(θ\)\\displaystyle\\nabla\_\{\\theta\}J\(\\theta\)=𝔼𝐬∼𝒟π~θ​\[∇θπ~θ​\(𝐬\)​∇𝐚~Cπ~θψ​\(𝐬,𝐚~\)\],\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}^\{\\tilde\{\\pi\}^\{\\theta\}\}\}\\Big\[\\nabla\_\{\\theta\}\\tilde\{\\pi\}^\{\\theta\}\(\\mathbf\{s\}\)\\,\\nabla\_\{\\tilde\{\\mathbf\{a\}\}\}C^\{\\psi\}\_\{\\tilde\{\\pi\}^\{\\theta\}\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\)\\Big\],=𝔼𝐬∼𝒟π~θ​\[∇θη​\(𝐬,πθ​\(𝐬\)\)​∇𝐚~Cπ~θψ​\(𝐬,𝐚~\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\}^\{\\tilde\{\\pi\}^\{\\theta\}\}\}\\Big\[\\nabla\_\{\\theta\}\\eta\(\\mathbf\{s\},\\pi^\{\\theta\}\(\\mathbf\{s\}\)\)\\,\\nabla\_\{\\tilde\{\\mathbf\{a\}\}\}C^\{\\psi\}\_\{\\tilde\{\\pi\}^\{\\theta\}\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\)\\Big\]\.\(4\)From Eq\.[3](https://arxiv.org/html/2606.31320#S3.Ex2), it is evident that when the safety intervention ruleη\\etais non\-differentiable, the safety filter blocks the gradient backpropagation to the actor, preventing effective policy updates\.

One possible workaround is to treat the safety filter as part of the environment dynamics\(Markgrafet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib140)\), such that the actor update formally retains the same structure as in Eq\.[3](https://arxiv.org/html/2606.31320#S3.E3)\. This modeling choice, however, induces a modified transition kernel when safety intervention occurs:Pη=P​\(𝐬′∣𝐬,η​\(𝐬,πθ​\(𝐬\)\)\),P^\{\\eta\}=P\\\!\\left\(\\mathbf\{s\}^\{\\prime\}\\mid\\mathbf\{s\},\\eta\\\!\\left\(\\mathbf\{s\},\\pi^\{\\theta\}\(\\mathbf\{s\}\)\\right\)\\right\),where the next state is not*uniquely*dependent on the action output ofπθ\\pi^\{\\theta\}\.

When safety interventions occur, distinct intended unsafe actions may map to the same executed filtered action throughη​\(⋅\)\\eta\(\\cdot\)\. This many\-to\-one mapping yields identical transition outcomes for different\(𝐬,𝐚θ\)\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)pairs, introducing ambiguity in the critic’s learning and weakening its ability to capture action\-dependent value differences\(Markgrafet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib140)\)\. Consequently, value estimates may become biased and policy gradients less informative, slowing convergence\.

Moreover, hard safety interventions often rely on last\-moment switching between the learned policy and a fallback controller\. The abrupt change may introduce non\-smooth transition dynamics near the safety decision boundary\. These discontinuities further degrade learning by inflating gradient variance and biasing value estimation in actor–critic methods\(Achiamet al\.,[2017](https://arxiv.org/html/2606.31320#bib.bib105); Rayet al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib118)\)\.

These considerations motivate the design of a safe online learner that \(i\) transitions smoothly between regimes with inactive and active safety intervention, thereby preserving informative gradients and stable learning signals for policy optimization, and \(ii\) learns a natural safe behavior by acting*cautiously*as the safety margin tightens: the policy should progressively bias its actions toward safe behavior while reducing exploratory variance to mitigate accidental safety violations\.

## 4Smooth Intervention via Convex Policy Composition

We introduce*AutoSafe*\(Fig\.[1\(b\)](https://arxiv.org/html/2606.31320#S2.F1.sf2)\), a safety\-aware policy architecture that integrates safety directly into action generation\. Rather than treating safety as a rejection or correction step applied after action selection, AutoSafe structures the executable policy as a differentiable convex composition of a performance\-driven learner and a safety prior\.

### 4\.1Structural Policy Composition

At each time step, the executed action𝐚~\\tilde\{\\mathbf\{a\}\}is synthesized by interpolating between the stochastic action proposed by the learning policy,𝐚θ∼πθ\(⋅∣𝐬\)\\mathbf\{a\}^\{\\theta\}\\sim\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\), and a deterministic safe action provided by the safety prior,𝐚safe=πsafe​\(𝐬\)\\mathbf\{a\}^\{\\mathrm\{safe\}\}=\\pi^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\):

𝐚~=\(1−λ​\(𝐬\)\)​𝐚θ\+λ​\(𝐬\)​𝐚safe,λ​\(𝐬\)∈\[0,1\]\.\\tilde\{\\mathbf\{a\}\}=\(1\-\\lambda\(\\mathbf\{s\}\)\)\\,\\mathbf\{a\}^\{\\theta\}\+\\lambda\(\\mathbf\{s\}\)\\,\\mathbf\{a\}^\{\\mathrm\{safe\}\},\\quad\\lambda\(\\mathbf\{s\}\)\\in\[0,1\]\.\(5\)Here,λ​\(s\)\\lambda\(s\)acts as a dynamic mixing that balances between reward maximization and risk mitigation\.

This design leverages the geometry of continuous control tasks\. Assuming the feasible action space𝒜\\mathcal\{A\}is convex \(e\.g\., standard box constraints\), any convex combination of two valid actions is guaranteed to remain valid \(𝐚θ,𝐚safe∈𝒜→𝐚~∈𝒜\\mathbf\{a\}^\{\\theta\},\\mathbf\{a\}^\{\\mathrm\{safe\}\}\\in\\mathcal\{A\}\\to\\tilde\{\\mathbf\{a\}\}\\in\\mathcal\{A\}\(Boyd and Vandenberghe,[2004](https://arxiv.org/html/2606.31320#bib.bib139)\)\)\. Physically, this means the intervention does not arbitrarily “clip” the action, but rather pulls it along the line segment connecting the agent’s desired action to the safe anchor\.

This formulation secures two key properties for online learning:Differentiability:Unlike discrete safety filters, Eq\.[5](https://arxiv.org/html/2606.31320#S4.E5)is fully differentiable with respect toλ\\lambda\. This preserves gradient flow even during intervention\.Expressiveness:The composite policy strictly contains the original policy class\. In safe regions \(λ→0\\lambda\\to 0\), the agent recovers full autonomy to learn optimal behaviors; as risk escalates \(λ→1\\lambda\\to 1\), behavior smoothly collapses to the safe prior\.

### 4\.2Distributional Shaping & Variance Control

The structural composition in Eq\.[5](https://arxiv.org/html/2606.31320#S4.E5)fundamentally reshapes the policy’s stochastic profile\. By viewing the intervention as a transformation \(a pushforward under the affine map\) of the learner’s distributionπθ\(⋅∣𝐬\)\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\), we can quantify exactly how the geometric interpolation alters the agent’s behavior\.

For Gaussian policies, standard in continuous control whereπθ\(⋅∣𝐬\)=𝒩\(μθ\(𝐬\),Σθ\(𝐬\)\)\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)=\\mathcal\{N\}\(\\mu\_\{\\theta\}\(\\mathbf\{s\}\),\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\), the executed policyπ~\\tilde\{\\pi\}remains Gaussian with moments:

μ~​\(𝐬\)=\(1−λ\)​μθ​\(𝐬\)\+λ​𝐚safe,Σ~​\(𝐬\)=\(1−λ\)2​Σθ​\(𝐬\)\.\\tilde\{\\mu\}\(\\mathbf\{s\}\)=\(1\-\\lambda\)\\mu\_\{\\theta\}\(\\mathbf\{s\}\)\+\\lambda\\mathbf\{a\}^\{\\mathrm\{safe\}\},~~\\tilde\{\\Sigma\}\(\\mathbf\{s\}\)=\(1\-\\lambda\)^\{2\}\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\.\(6\)This result reveals that the convex structure of*AutoSafe*automatically couples safety with uncertainty reduction\. Asλ\\lambdaincreases, the intervention induces:

- •Mean bias:The expected actionμ~\\tilde\{\\mu\}shifts toward the safe prior𝐚safe\\mathbf\{a\}^\{\\mathrm\{safe\}\}, correcting the learner’s intent\.
- •Variance damping:Exploration noise is reduced by a quadratic factor\(1−λ\)2\(1\-\\lambda\)^\{2\}\.

This coupling is the first\-principles mechanism that stabilizes learning: as the agent approaches risk \(highλ\\lambda\), it does not merely steer away; it effectively restricts stochastic exploration to prevent accidental boundary violations\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x3.png)Figure 2:\(Left\)Distributional Shaping \(Sec\. 4\.2\): As the intervention weightλ\\lambdaincreases, the policy distribution gradually shifts its mean toward the safe anchorasafea^\{\\mathrm\{safe\}\}, while its variance decreases quadratically \( Eq\.[6](https://arxiv.org/html/2606.31320#S4.E6)\)\.\(Right\)Learnable Sharpness \(Sec\. 4\.4\): The sharpness parameterppcontrols the intervention profile\. A largerppcreates a “sharper” boundary that delays intervention to prioritize performance, while a smallerppinduces earlier and more gradual intervention\.
### 4\.3Risk\-Calibrated Learnable Intervention

With the intervention mechanism in Sec\.[4\.1](https://arxiv.org/html/2606.31320#S4.SS1)and its variance\-damping effect in Sec\.[4\.2](https://arxiv.org/html/2606.31320#S4.SS2)established, we now specify how the intervention weightλ​\(𝐬\)\\lambda\(\\mathbf\{s\}\)is chosen\. Our guiding principle is*minimal intervention*: the learning policy should retain as much control authority as possible, while the executed policy should place only negligible probability mass in unsafe regions\. For the smooth rate analyzed below,λ​\(𝐬\)\\lambda\(\\mathbf\{s\}\)is treated as fixed after conditioning on𝐬\\mathbf\{s\}, which preserves the Gaussian pushforward identity in Eq\.[6](https://arxiv.org/html/2606.31320#S4.E6)\. Action\-dependent safety checks are still allowed, but are used only as hard triggers\.

Let

Δ^​\(𝐬,𝐚\)≔Δ​\(𝐬,𝐚\)−Δmin\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\coloneqq\\Delta\(\\mathbf\{s\},\\mathbf\{a\}\)\-\\Delta\_\{\\min\}\(7\)denote the relative safety margin, so thatΔ^​\(𝐬,𝐚\)≥0\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\geq 0indicates satisfaction of the safety condition\. For a fixed state𝐬\\mathbf\{s\}, the executed action𝐚~λ\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}induces the random margin

Z​\(λ\)≔Δ^​\(𝐬,𝐚~λ\)\.Z\(\\lambda\)\\coloneqq\\hat\{\\Delta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}\)\.We require the executed policy to satisfy the state\-wise chance constraint

Pr⁡\(Z​\(λ\)​<0∣​𝐬\)=Pr⁡\(Δ​\(𝐬,𝐚~λ\)​<Δmin∣​𝐬\)≤δ​\(𝐬\),\\Pr\\\!\\left\(Z\(\\lambda\)<0\\mid\\mathbf\{s\}\\right\)=\\Pr\\\!\\left\(\\Delta\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}\)<\\Delta\_\{\\min\}\\mid\\mathbf\{s\}\\right\)\\leq\\delta\(\\mathbf\{s\}\),\(8\)whereδ​\(𝐬\)\\delta\(\\mathbf\{s\}\)is a risk tolerance\. This constraint bounds the unsafe tail probability of the executed action distribution\.

A tractable intervention template follows by locally approximating the relative safety margin as affine along the interpolation segment between the nominal action𝐚θ\\mathbf\{a\}^\{\\theta\}and the safe prior action𝐚safe\\mathbf\{a\}^\{\\mathrm\{safe\}\}\. Under this approximation, and using a one\-sided tail condition, the smallest feasible intervention weight has the closed form

λ∗​\(𝐬\)=clip\[0,1\]​\(βδ​σΔ​\(𝐬\)−μΔ​\(𝐬\)Δsafe​\(𝐬\)−μΔ​\(𝐬\)\+βδ​σΔ​\(𝐬\)\),\\lambda^\{\*\}\(\\mathbf\{s\}\)=\\mathrm\{clip\}\_\{\[0,1\]\}\\left\(\\frac\{\\beta\_\{\\delta\}\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\-\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\}\{\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\-\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\+\\beta\_\{\\delta\}\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\}\\right\),\(9\)where

μΔ​\(𝐬\)=𝔼​\[Δ^​\(𝐬,𝐚θ\)∣𝐬\],σΔ​\(𝐬\)=Std⁡\[Δ^​\(𝐬,𝐚θ\)∣𝐬\],\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)=\\mathbb\{E\}\\\!\\left\[\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\mid\\mathbf\{s\}\\right\],\\qquad\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)=\\operatorname\{Std\}\\\!\\left\[\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\mid\\mathbf\{s\}\\right\],and

Δsafe​\(𝐬\)=Δ^​\(𝐬,𝐚safe​\(𝐬\)\)\.\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)=\\hat\{\\Delta\}\\\!\\left\(\\mathbf\{s\},\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\right\)\.Hereβδ\\beta\_\{\\delta\}is a risk quantile, e\.g\.,βδ=Φ−1​\(1−δ​\(𝐬\)\)\\beta\_\{\\delta\}=\\Phi^\{\-1\}\(1\-\\delta\(\\mathbf\{s\}\)\)for Gaussian margins, or the conservative sub\-Gaussian choiceβδ=2​log⁡\(1/δ​\(𝐬\)\)\\beta\_\{\\delta\}=\\sqrt\{2\\log\(1/\\delta\(\\mathbf\{s\}\)\)\}\. The derivation and approximation assumptions are given in Appendix[A](https://arxiv.org/html/2606.31320#A1)\.

Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9)provides the key design insight\. If the risk\-adjusted nominal margin

Lθ​\(𝐬\)≔μΔ​\(𝐬\)−βδ​σΔ​\(𝐬\)L\_\{\\theta\}\(\\mathbf\{s\}\)\\coloneqq\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\-\\beta\_\{\\delta\}\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)is nonnegative, the nominal policy already satisfies the tail\-risk condition and no intervention is needed\. Otherwise, the numerator in Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9)measures the nominal safety deficit, while the denominator measures the safety improvement available by moving toward the safe prior\. Thus,λ\\lambdashould increase as the safety margin decreases, as uncertainty increases, or as the desired risk tolerance becomes stricter\.

In online deep RL, directly estimatingσΔ​\(𝐬\)\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)and selecting a state\-dependentβδ\\beta\_\{\\delta\}can be unreliable\. AutoSafe therefore uses Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9)as a structural template rather than as an online estimator\. The implemented rate enforces three properties: monotonicity with respect to the safety margin, endpoint safety at the boundary, and a learnable sharpness controlling how quickly intervention rises\.

The monitor provides a normalized state\-conditioned marginΔ~​\(𝐬\)∈\[0,1\]\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\\in\[0,1\], whereΔ~​\(𝐬\)=1\\tilde\{\\Delta\}\(\\mathbf\{s\}\)=1denotes a clearly safe interior state andΔ~​\(𝐬\)=0\\tilde\{\\Delta\}\(\\mathbf\{s\}\)=0denotes the safety boundary\. We instantiate the intervention weight as

λ​\(Δ~​\(𝐬\),p​\(𝐬\)\)=exp⁡\(p​\(𝐬\)​\(1−Δ~​\(𝐬\)\)\)−1exp⁡\(p​\(𝐬\)\)−1,\\lambda\\\!\\left\(\\tilde\{\\Delta\}\(\\mathbf\{s\}\),p\(\\mathbf\{s\}\)\\right\)=\\frac\{\\exp\\\!\\left\(p\(\\mathbf\{s\}\)\(1\-\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\)\\right\)\-1\}\{\\exp\\\!\\left\(p\(\\mathbf\{s\}\)\\right\)\-1\},\(10\)wherep​\(𝐬\)\>0p\(\\mathbf\{s\}\)\>0is a learnable sharpness parameter\. This map satisfies

λ​\(1,p\)=0,λ​\(0,p\)=1,∂λ∂Δ~<0\.\\lambda\(1,p\)=0,\\qquad\\lambda\(0,p\)=1,\\qquad\\frac\{\\partial\\lambda\}\{\\partial\\tilde\{\\Delta\}\}<0\.Hence, intervention vanishes in the safe interior and increases monotonically as the system approaches the safety boundary\.

The sharpness parameter controls the conservatism of this transition\. Smallerp​\(𝐬\)p\(\\mathbf\{s\}\)produces earlier and smoother intervention, while largerp​\(𝐬\)p\(\\mathbf\{s\}\)delays intervention until closer to the boundary, preserving more autonomy for the learning policy\. We parameterize it using a small prediction head,

p​\(𝐬;ϕ\)=pmin\+\(pmax−pmin\)​sigmoid⁡\(hϕ​\(𝐬\)\),0<pmin<pmax\.p\(\\mathbf\{s\};\\phi\)=p\_\{\\min\}\+\(p\_\{\\max\}\-p\_\{\\min\}\)\\operatorname\{sigmoid\}\\\!\\left\(h\_\{\\phi\}\(\\mathbf\{s\}\)\\right\),\\qquad 0<p\_\{\\min\}<p\_\{\\max\}\.\(11\)The parametersϕ\\phiare optimized jointly with the actor through the executed action

𝐚~=\(1−λ​\(𝐬\)\)​𝐚θ\+λ​\(𝐬\)​𝐚safe\.\\tilde\{\\mathbf\{a\}\}=\(1\-\\lambda\(\\mathbf\{s\}\)\)\\mathbf\{a\}^\{\\theta\}\+\\lambda\(\\mathbf\{s\}\)\\mathbf\{a\}^\{\\mathrm\{safe\}\}\.
Finally, near\-boundary determinism follows directly from Eq\.[6](https://arxiv.org/html/2606.31320#S4.E6):

Σ~​\(𝐬\)=\(1−λ​\(𝐬\)\)2​Σθ​\(𝐬\)\.\\tilde\{\\Sigma\}\(\\mathbf\{s\}\)=\(1\-\\lambda\(\\mathbf\{s\}\)\)^\{2\}\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\.Asλ​\(𝐬\)→1\\lambda\(\\mathbf\{s\}\)\\to 1, the executed covariance collapses, suppressing exploratory variance near the safety boundary\. If the monitor detects an imminent safety violation, e\.g\.,Δ^​\(𝐬,𝐚θ\)≤0\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\leq 0for a state\-action monitor orΔ^​\(𝐬\)≤0\\hat\{\\Delta\}\(\\mathbf\{s\}\)\\leq 0for a state monitor, thenΔ~​\(𝐬\)=0\\tilde\{\\Delta\}\(\\mathbf\{s\}\)=0, andAutoSafesetsλ=1\\lambda=1to fully execute the certified safe action𝐚safe\\mathbf\{a\}^\{\\mathrm\{safe\}\}\. Thus, the learnable rate shapes precautionary behavior before the boundary is reached, while the hard trigger preserves the endpoint safety inherited from the certified safe prior\.

## 5Experiments and Results

### 5\.1Backbone Algorithms

![Refer to caption](https://arxiv.org/html/2606.31320v1/x4.png)Figure 3:Illustration of a Simplex\-style safety mechanism with an inner safe setΩΔmin\\Omega\_\{\\Delta\_\{\\min\}\}and switching to a certified safe policy at the safety boundary in a static 2D case\.The proposed policy composition diagram is*safety\-filter agnostic*\. As a representative instantiation, we adopt the widely used*Simplex*architecture\(Sha,[2001](https://arxiv.org/html/2606.31320#bib.bib107)\)as the baseline safety design\. Simplex employs a robust,*certified\-safe fallback policy*obtained by solving an offline model\-based optimization problem that explicitly accounts for closed\-loop stability under safety constraints\.

The safe action is given by a state\-feedback control law𝐚safe≔𝐅𝐬,\\mathbf\{a\}^\{\\mathrm\{safe\}\}\\coloneqq\\mathbf\{F\}\\mathbf\{s\},where𝐅\\mathbf\{F\}denotes the feedback gain matrix\. Safety monitoring is performed using a state\-dependent Lyapunov\-like functionΔ​\(𝐬\)≔1−𝐬⊤​𝐏𝐬,\\Delta\(\\mathbf\{s\}\)\\coloneqq 1\-\\mathbf\{s\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\},where𝐏≻0\\mathbf\{P\}\\succ 0is a positive definite Lyapunov matrix associated with the closed\-loop dynamics under the fallback controller, obtained via standard Lyapunov or LMI\-based synthesis\. Following the convention introduced in[Section˜2\.2](https://arxiv.org/html/2606.31320#S2.SS2), control switches toπθ\\pi^\{\\theta\}whenΔ​\(𝐬\)\>Δmin\\Delta\(\\mathbf\{s\}\)\>\\Delta\_\{\\min\}and toπsafe\\pi^\{\\mathrm\{safe\}\}otherwise\.

As illustrated in a two\-dimensional setting in Fig\.[3](https://arxiv.org/html/2606.31320#S5.F3), the conditionΔ​\(𝐬\)\>Δmin\\Delta\(\\mathbf\{s\}\)\>\\Delta\_\{\\min\}defines an inner safe region\.

ΩΔmin≔\{𝐬\|1−𝐬⊤​𝐏𝐬\>Δmin\},\\Omega\_\{\\Delta\_\{\\min\}\}\\coloneqq\\left\\\{\\mathbf\{s\}\\;\\middle\|\\;1\-\\mathbf\{s\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\}\>\\Delta\_\{\\min\}\\right\\\},shown as the ellipsoidal set with yellow boundary\. When the system state lies withinΩΔmin\\Omega\_\{\\Delta\_\{\\min\}\}, control authority is assigned to the learning\-based policyπθ\\pi^\{\\theta\}\. Once the monitored safety marginΔ​\(𝐬\)\\Delta\(\\mathbf\{s\}\)decreases to the thresholdΔmin\\Delta\_\{\\min\}, the controller immediately switches to the certified safe policyπsafe\\pi^\{\\mathrm\{safe\}\}\. The black dashed boundary denotes the largest invariant safety envelope, corresponding to the zero level setΔ​\(𝐬\)=0\\Delta\(\\mathbf\{s\}\)=0\(i\.e\.,𝐬⊤​𝐏𝐬=1\\mathbf\{s\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\}=1\), within which the fallback controller guarantees forward invariance for safety\. Notably, Simplex avoids solving optimization problems at runtime as in\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Chenget al\.,[2019b](https://arxiv.org/html/2606.31320#bib.bib10)\)and always provides a valid safe action as a backup, making it particularly attractive for real\-time safety\-critical applications\.

In this work, we adopt*soft\-actor\-critic*algorithm \(SAC\)\(Haarnojaet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib76)\)for policy learning, where the policy parametersθ\\thetaare optimized via gradient ascent on the objective

J​\(θ\)\\displaystyle J\_\{\\text\{\}\}\(\\theta\)=𝔼𝐬∼𝒟,𝐚~∼π~θ​\[Qψ​\(𝐬,𝐚~\)−α​log⁡π~θ​\(𝐚~∣𝐬\)\],\\displaystyle=\\mathbb\{E\}\_\{\\mathbf\{s\}\\sim\\mathcal\{D\},\\,\\tilde\{\\mathbf\{a\}\}\\sim\\tilde\{\\pi\}^\{\\theta\}\}\\\!\\left\[Q^\{\\psi\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\)\-\\alpha\\log\\tilde\{\\pi\}^\{\\theta\}\(\\tilde\{\\mathbf\{a\}\}\\mid\\mathbf\{s\}\)\\right\],\(12\)whereQψQ^\{\\psi\}denotes the action\-value critic that estimates the expected return of executing the composite policyπ~θ\\tilde\{\\pi\}^\{\\theta\}, i\.e\., taking action𝐚~∼π~θ\\tilde\{\\mathbf\{a\}\}\\sim\\tilde\{\\pi\}^\{\\theta\}as defined in Eq\.[5](https://arxiv.org/html/2606.31320#S4.E5), andα\\alphais the temperature parameter that weights the entropy regularization term\. AutoSafe does not modify the critic or the learning objective\. Instead, it alters only how the actor produces executable actions by composing the learning\-based action with a safe fallback action\. The remainder of the interaction and training scheme remains identical to the original SAC algorithm\.

### 5\.2Experiment Setup

We evaluate our method against safety\-filter\-based baselines and safe RL baselines in the online safe learning context, including:Safety Filter:i\) SimplexRL\(Alshiekhet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib11); Phanet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib112); Caiet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib129)\): runtime safety filter by safe action replacement based onSimplex; ii\) CBF\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Chenget al\.,[2019a](https://arxiv.org/html/2606.31320#bib.bib19)\): runtime safety filter by safe action projection using control barrier function\.Safe Learning with Prior:i\) AdaLam\(Chenget al\.,[2019c](https://arxiv.org/html/2606.31320#bib.bib35); Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\): weighted summation of safe policy prior and DRL policy, where the weights are adaptively adjusted based on context and safety; ii\) Residual\(Johanninket al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib116)\): learns a residual policy on top of a fixed safe policy prior;Constrained RL:i\) Lyapunov\(Westenbroeket al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib108);[Caoet al\.,](https://arxiv.org/html/2606.31320#bib.bib18)\): adds Lyapunov\-based penalties for reward shaping for safe exploration; ii\) Lagrangian\(Haet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib127); Achiamet al\.,[2017](https://arxiv.org/html/2606.31320#bib.bib105)\): constrained policy optimization via dual variables\.

We consider four simulated and one real\-world case studies, including: i\) CartPole Balancing \(sim & real\)\(Towerset al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib14)\), with continuous force control under position and angle constraints; ii\) Glucose Regulation\(Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\), with continuous insulin control under glucose level constraints; iii\) 3D Quadrotor Goal Reaching\(Yuanet al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib78)\), with four\-dimensional thrust control under position and attitude constraints; and iv\) Quadruped Navigation\(Yanget al\.,[2022a](https://arxiv.org/html/2606.31320#bib.bib100)\), with six\-dimensional acceleration control on uneven terrain under height and velocity constraints\. Implementation details are summarized in Appendix[C](https://arxiv.org/html/2606.31320#A3)\. All experiments are run with five random seeds\.

### 5\.3Main Results

Table 1:Main Results\.Performance and safety comparison across four continuous control tasks\. We report mean±\\pmstd over five random seeds\. Higher return is better \(↑\\uparrow\), while fewer safety violations are better \(↓\\downarrow\)\. Best results are marked inbold\.Overall, AutoSafe outperforms the baselines in the majority of tasks and runs, achieving higher returns while maintaining strong safety assurance, as shown in[Table˜1](https://arxiv.org/html/2606.31320#S5.T1)\. In relatively low\-dimensional settings such as Cartpole and Glucose, hard\-intervention–based safety filters \(e\.g\., SimplexRL and CBF\) achieve performance similar to AutoSafe\. However, their performance degrades substantially on higher\-dimensional, more dynamically complex tasks, such as Quadrotor and Quadruped\. In these environments, frequent safety interventions introduce abrupt distribution shifts that hinder smooth function approximation, leading to unstable learning and occasional divergence\.

Residual learning combines learning\-based and safe actions through direct summation and performs well in simpler tasks such as Cartpole and Glucose\. However, its effectiveness depends strongly on the quality and alignment of the safe prior; mismatches can bias learning and slow convergence in more complex domains\. Similarly, AdaLam adaptively blends learning\-based and safe actions via a weighting parameterλ\\lambda\. However, safety is not explicitly enforced, and the weighting parameterλ\\lambdais optimized solely for task performance\. In practice, we observe that asλ\\lambdadecreases, safety violations increase dramatically, causing learning to make little further progress\.

Lastly, approaches that learn using safe priors benefit from incorporating domain knowledge and tend to achieve stronger safety and performance within limited training budgets\. In contrast, constrained RL methods typically rely on learning safety from constraint violations, which can lead to increased violations during training and require substantially more interaction to converge\. We summarize the computational time comparison for all methods in[Table˜5](https://arxiv.org/html/2606.31320#A4.T5)at[Section˜D\.7](https://arxiv.org/html/2606.31320#A4.SS7), showing thatAutoSafeachieves computational efficiency comparable to vanilla SAC\.

### 5\.4Influences of hard intervention on learning

![Refer to caption](https://arxiv.org/html/2606.31320v1/x5.png)\(a\)Performance return and critic loss curves \(logarithmic\) for tasks with different levels of goal conflicts\.
![Refer to caption](https://arxiv.org/html/2606.31320v1/x6.png)\(b\)Learning to Be Safe: Smooth vs\. Hard Intervention\.\(Left\): Smooth policy composition yields stable and consistent safety learning across different levels of task conflict\.\(Right\): Hard safety intervention initially reduces interventions but later stalls, leading to repeated safety interventions\.

Figure 4:Comparison of learning dynamics and safety intervention behavior\.In this section, we study the impact of*hard safety interventions*on learning dynamics using a controlled CartPole experiment based on*Simplex*\. We consider a position\-tracking task in which the learning policy must stabilize the system while reaching a target cart position\. The safe policy regulates the system towardxgoal=0\.0​mx\_\{\\mathrm\{goal\}\}=0\.0~\\mathrm\{m\}\. To induce different levels of goal conflict, the learning policy is tasked with reaching target positionsxgoal∈\{0\.0,0\.1,0\.2\}​mx\_\{\\mathrm\{goal\}\}\\in\\\{0\.0,0\.1,0\.2\\\}~\\mathrm\{m\}, with larger offsets corresponding to stronger conflicts between task objectives and safety enforcement\.

As shown in Fig\.[4\(a\)](https://arxiv.org/html/2606.31320#S5.F4.sf1), our method robustly handles different levels of goal conflict and consistently outperforms the*hard\-intervention*baseline\. In contrast, learning under the hard\-intervention mechanism becomes increasingly sensitive as the goal conflict intensifies\. Frequent safety interventions lead to unstable critic estimation, manifested by rapidly increasing critic loss\. As a result, policy optimization stagnates, and overall task performance degrades significantly\.

Fig\.[4\(b\)](https://arxiv.org/html/2606.31320#S5.F4.sf2)further illustrates the learning dynamics under different goal\-reaching tasks\. With smooth policy composition, the agent gradually reduces the intervention weightλ\\lambda, relying less on the safe policy and increasingly exploiting the learning\-based policy to maximize task performance\. By contrast, while the hard\-intervention mechanism initially reduces the intervention frequency, this trend breaks down under stronger goal conflicts\. Forxgoal=0\.1​m​and​0\.2​mx\_\{\\mathrm\{goal\}\}=0\.1~\\mathrm\{m\}~\\text\{and\}~0\.2~\\mathrm\{m\}, the intervention rate plateaus after approximately20​k20\\text\{k\}steps, coinciding with a sharp rise in critic loss \(Fig\.[4\(a\)](https://arxiv.org/html/2606.31320#S5.F4.sf1)\)\. This behavior suggests that excessive and persistent interventions induce extrapolation errors in the critic, preventing the policy from receiving informative learning signals and ultimately halting learning progress\.

Further analysis on the sensitivity to the choice ofΔmin\\Delta\_\{\\mathrm\{min\}\}for both SimplexRL andAutoSafeis provided in[Table˜4](https://arxiv.org/html/2606.31320#A4.T4)at the[D\.6](https://arxiv.org/html/2606.31320#A4.SS6)\. The results show the expected trade\-off: a largerΔmin\\Delta\_\{\\mathrm\{min\}\}leads to earlier safety intervention and more conservative behavior, while a smallerΔmin\\Delta\_\{\\mathrm\{min\}\}allows more aggressive exploration at the cost of increased safety violations\.

### 5\.5Adaptation of learnedpp

![Refer to caption](https://arxiv.org/html/2606.31320v1/x7.png)Figure 5:CartPole safety envelope visualization\.Visualization ofλ​\(Δ~​\(𝐬\)\)\\lambda\(\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\)over thexxandθ\\thetadimensions for different learned values ofppacross CartPole tasks\. The parameterppadapts to the level of goal conflict: after the agent has learned to remain safe, increasingppreducesλ\\lambdaat the same state, downweighting the safety policy to favor higher task performance within the safety envelope\.Making the sharpness parameterpplearnable enables the agent to adaptively adjust its value, thereby maximizing performance across different system dynamics and tasks\. To validate this hypothesis, we visualize the learnedppfrom the a more fine\-grained CartPole experiment in Fig\.[5](https://arxiv.org/html/2606.31320#S5.F5)\. As the goal mismatch increases, the suboptimality of the safe prior becomes more pronounced\. Correspondingly, the value of learnedppincreases, which in turn reduces the blending weightλ\\lambda\. This indicates that the agent adaptively learns to rely less on the suboptimal safe policy to seek higher return\.

We observe that the converged sharpness parameterppvaries substantially across tasks \(see Fig\.[12](https://arxiv.org/html/2606.31320#A4.F12)at Appendix[D](https://arxiv.org/html/2606.31320#A4)\) and is not analytically tractable to determine\. Heuristic schedules, therefore, require task\-specific tuning and may introduce design bias, as seen in the Quadrotor setting \(see Fig\.[13](https://arxiv.org/html/2606.31320#A4.F13)in Appendix[D](https://arxiv.org/html/2606.31320#A4)\)\. Learningppalong with the policy avoids this issue and yields robust performance across all tasks\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x8.png)

![Refer to caption](https://arxiv.org/html/2606.31320v1/x9.png)

Figure 6:AutoSafe\-controlled trajectories for CartPole \(Top\) and Quadrotor \(Bottom\)\. The intervention weightλ\\lambdarises as the system approaches safety boundaries, enabling precautionary control, and remains low elsewhere to preserve performance\-driven behavior\.
### 5\.6Precautionary safety intervention

The leverage of a safety prior enables AutoSafe to perform*precautionary*safety intervention, rather than relying on last\-moment triggering\. As visualized in Fig\.[6](https://arxiv.org/html/2606.31320#S5.F6), when the system approaches the safety boundary, the agent learns when and how strongly to intervene \(e\.g\., in CartPole,λ\\lambdaincreases primarily once the safety margin falls below≈0\.3\\approx 0\.3\)\.

Compared to CartPole, the Quadrotor requires more frequent and stronger interventions because the safety margin is more sensitive to state variations in the higher\-dimensional state space \(𝐬∈ℝ12\\mathbf\{s\}\\in\\mathbb\{R\}^\{12\}vs\.𝐬∈ℝ4\\mathbf\{s\}\\ \\in\\mathbb\{R\}^\{4\}\), where deviations across multiple state components can jointly cause the safety margin to decrease more sharply\. In both cases,λ\\lambdaremains small most of the time, indicating that the learned policy largely maintains performance\-driven behavior while being safe\.

### 5\.7Real world applicability

The proposed architecture is favorable for safe online policy learning in real\-world settings\. By smoothly intervening, it avoids abrupt changes in system dynamics that can increase stochasticity and destabilize learning\. Moreover, unlike CBF\- or MPC\-based methods\(Chenget al\.,[2019a](https://arxiv.org/html/2606.31320#bib.bib19); Grandiaet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib124)\), it does not require online optimization, resulting in higher computational efficiency suitable for high\-frequency control tasks\.

We demonstrate this feature in a real\-world CartPole learning task \(Fig\.[7\(a\)](https://arxiv.org/html/2606.31320#S5.F7.sf1)\), where the AutoSafe policy is deployed on an embedded device \(Raspberry Pi\) to ensure safe interaction, while policy parameters are periodically updated from a remote workstation\. As shown in Fig\.[7\(b\)](https://arxiv.org/html/2606.31320#S5.F7.sf2), AutoSafe enables the agent to interact safely with the environment while continuously improving task performance\. Over time, the parameterλ\\lambdadecreases, indicating that the policy gradually reduces its reliance on the safe prior as it learns safer and more performant behaviors\. Additionally, we show that AutoSafe can learn safely under dynamically changing obstacle constraints by efficiently adapting the offline\-certified safe recoverable region without online optimization, as demonstrated in the additional CartPole and Quadrotor experiments in[Section˜D\.5](https://arxiv.org/html/2606.31320#A4.SS5)\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x10.png)\(a\)Real\-world safe learning on a cart\-pole system\.
![Refer to caption](https://arxiv.org/html/2606.31320v1/x11.png)\(b\)Experimental results over three runs\.

Figure 7:Real\-world experiments with AutoSafe\. The agent is deployed on an embedded device, enabling safe interaction at a control frequency of50​Hz50\\,\\mathrm\{Hz\}\.

## 6Related Work

Safety as a soft constraint:A large body of work improves safety during learning by formulating safety requirements as soft constraints, including constrained policy optimization\(Wachi and Sui,[2020](https://arxiv.org/html/2606.31320#bib.bib104); Achiamet al\.,[2017](https://arxiv.org/html/2606.31320#bib.bib105)\), policy\-prior aided training\(Xieet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib106)\), and Lyapunov\-based reward shaping\(Westenbroeket al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib108); Zhaoet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib109);[Caoet al\.,](https://arxiv.org/html/2606.31320#bib.bib18)\)\. Safety assurance in these methods is typically asymptotic or expectation\-based after convergence, rather than being hard\-enforced during training and deployment\. An emerging direction incorporates safety priors to guide learning, using residual policies\(Johanninket al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib116)\), regularization\(Chenget al\.,[2019c](https://arxiv.org/html/2606.31320#bib.bib35)\), or policy fusion\(Ranaet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib117)\)\. However, these approaches primarily target data efficiency and performance improvement, rather than explicitly enforcing safety constraints\.

Enforce hard safety constraint:Hard safety guarantees are typically enforced via safety filters\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\), including action projection–based methods\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Chenget al\.,[2019a](https://arxiv.org/html/2606.31320#bib.bib19)\)and action replacement mechanisms\(Alshiekhet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib11); Zhonget al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib27);[Baket al\.,](https://arxiv.org/html/2606.31320#bib.bib138); Nestiet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib9)\)\. While effective at preventing violations, frequent or abrupt interventions can disrupt gradient flow and destabilize online and continual learning\.

Recent efforts on differentiable safety filters\(Amos and Kolter,[2017](https://arxiv.org/html/2606.31320#bib.bib146); Xiaoet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib147); Jinet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib148); Markgrafet al\.,[2025](https://arxiv.org/html/2606.31320#bib.bib140); Suttleet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib149)\)have helped alleviate the gradient\-flow issue in safe learning\. However, many of these approaches still rely on solving online optimization problems\(Jinet al\.,[2021](https://arxiv.org/html/2606.31320#bib.bib148); Amos and Kolter,[2017](https://arxiv.org/html/2606.31320#bib.bib146)\), or are primarily evaluated in simplified taskSuttleet al\.\([2024](https://arxiv.org/html/2606.31320#bib.bib149)\)or offline training settings using pre\-collected datasets\(Xiaoet al\.,[2023](https://arxiv.org/html/2606.31320#bib.bib147)\)\. While these methods demonstrate promising capabilities, their scalability to higher\-dimensional systems remains an active challenge\(Hsuet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib6)\), which may limit their applicability in safe online learning scenarios where computational efficiency and timely safety intervention are critical\.

## 7Conclusions and Limitations

We introducedAutoSafe, a policy architecture that incorporates safety monitoring and intervention directly into the action\-generation process to enable smooth online reinforcement learning under hard safety constraints\. By integrating safety as a structural inductive bias, the proposed framework preserves recoverable safety while maintaining stable learning dynamics and strong task performance\. Experimental results on both simulated and real\-world systems demonstrate thatAutoSafecan provide effective safety assurance without sacrificing learning capability\.

Despite these promising results, several limitations remain for more challenging real\-world training\. First, similar to many provably safe learning architecturesKrasowskiet al\.\([2023](https://arxiv.org/html/2606.31320#bib.bib145)\),AutoSafebuilds on model\-based safety design using prior knowledge of the system dynamics\. The quality of the derived safe policy or safety monitor can affect the overall system performance\. Although the proposed smooth policy composition is designed to remain robust to sub\-optimal safe policies through the adaptive mixing parameterλ\\lambda, inaccurate or overly conservative models may still limit exploration capability and achievable task performance\.

In addition, the current framework primarily considers safety constraints defined in relatively local and structured state spaces\. In more dynamic environments with time\-varying constraints, disturbances, or moving obstacles, the safety filter backbone may need to be adapted accordingly, as shown in[Section˜D\.5](https://arxiv.org/html/2606.31320#A4.SS5)\. Maintaining safety in such scenarios may require online adaptation of the safe policy for disturbance rejection\(Wanget al\.,[2013](https://arxiv.org/html/2606.31320#bib.bib143)\)or safe set\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8); Wabersich and Zeilinger,[2019](https://arxiv.org/html/2606.31320#bib.bib119)\)\.

Future work will therefore focus on extendingAutoSafetoward more complex real\-world settings, including higher\-dimensional robotic systems, adaptive safety mechanisms under uncertainty, and dynamic environments with evolving safety constraints\.

## References

- Constrained policy optimization\.InInternational conference on machine learning,pp\. 22–31\.External Links:[Link](https://proceedings.mlr.press/v70/achiam17a)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p2.1),[§3](https://arxiv.org/html/2606.31320#S3.p6.1),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- M\. Alshiekh, R\. Bloem, R\. Ehlers, B\. Könighofer, S\. Niekum, and U\. Topcu \(2018\)Safe reinforcement learning via shielding\.Vol\.32\.Note:Issue: 1Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p2.1)\.
- E\. Altman \(2021\)Constrained Markov decision processes\.Routledge\.Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p2.1)\.
- A\. D\. Ames, S\. Coogan, M\. Egerstedt, G\. Notomista, K\. Sreenath, and P\. Tabuada \(2019\)Control Barrier Functions: Theory and Applications\.arXiv\.Note:arXiv:1903\.11199 \[eess\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.1903.11199)Cited by:[§C\.1\.4](https://arxiv.org/html/2606.31320#A3.SS1.SSS4.p1.2),[Table 5](https://arxiv.org/html/2606.31320#A4.T5.36.32.32.9),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.1](https://arxiv.org/html/2606.31320#S5.SS1.p3.8),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.38.32.32.9),[§6](https://arxiv.org/html/2606.31320#S6.p2.1),[§7](https://arxiv.org/html/2606.31320#S7.p3.1)\.
- B\. Amos and J\. Z\. Kolter \(2017\)OptNet: Differentiable Optimization as a Layer in Neural Networks\.InProceedings of the 34th International Conference on Machine Learning,pp\. 136–145\(en\)\.External Links:ISSN 2640\-3498,[Link](https://proceedings.mlr.press/v70/amos17a.html)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- \[6\]S\. Bak, T\. T\. Johnson, M\. Caccamo, and L\. ShaReal\-Time Reachability for Verified Simplex Design\.\(en\)\.Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p2.1)\.
- O\. Bastani \(2021\)Safe reinforcement learning with nonlinear dynamics via model predictive shielding\.In2021 American control conference \(ACC\),pp\. 3488–3494\.Cited by:[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2)\.
- S\. P\. Boyd and L\. Vandenberghe \(2004\)Convex optimization\.Cambridge university press\.Cited by:[§4\.1](https://arxiv.org/html/2606.31320#S4.SS1.p2.2)\.
- Y\. Cai, Y\. Mao, L\. Sha, H\. Cao, and M\. Caccamo \(2025\)Runtime Learning Machine\.ACM Trans\. Cyber\-Phys\. Syst\.\.Note:Just AcceptedExternal Links:ISSN 2378\-962X,[Document](https://dx.doi.org/10.1145/3744351)Cited by:[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1)\.
- \[10\]H\. Cao, Y\. Mao, L\. Sha, and M\. CaccamoPhysics\-regulated deep reinforcement learning: invariant embeddings\.InThe Twelfth International Conference on Learning Representations,Cited by:[§C\.2\.1](https://arxiv.org/html/2606.31320#A3.SS2.SSS1.p4.2),[§1](https://arxiv.org/html/2606.31320#S1.p2.1),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- H\. Cao, M\. Theile, F\. G\. Wyrwal, and M\. Caccamo \(2022\)Cloud\-edge training architecture for sim\-to\-real deep reinforcement learning\.In2022 IEEE/RSJ International Conference on Intelligent Robots and Systems \(IROS\),pp\. 9363–9370\.Cited by:[§C\.2\.1](https://arxiv.org/html/2606.31320#A3.SS2.SSS1.p2.1),[§C\.2\.1](https://arxiv.org/html/2606.31320#A3.SS2.SSS1.p2.6)\.
- M\. Chen, S\. L\. Herbert, H\. Hu, Y\. Pu, J\. F\. Fisac, S\. Bansal, S\. Han, and C\. J\. Tomlin \(2021\)Fastrack: a modular framework for real\-time motion planning and guaranteed safe tracking\.IEEE Transactions on Automatic Control66\(12\),pp\. 5861–5876\.Note:Publisher: IEEECited by:[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2)\.
- R\. Cheng, G\. Orosz, R\. M\. Murray, and J\. W\. Burdick \(2019a\)End\-to\-end safe reinforcement learning through barrier functions for safety\-critical continuous control tasks\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.33,pp\. 3387–3395\.Cited by:[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§5\.7](https://arxiv.org/html/2606.31320#S5.SS7.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p2.1)\.
- R\. Cheng, G\. Orosz, R\. M\. Murray, and J\. W\. Burdick \(2019b\)End\-to\-end safe reinforcement learning through barrier functions for safety\-critical continuous control tasks\.Vol\.33,pp\. 3387–3395–3387–3395\.Note:Issue: 01Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§5\.1](https://arxiv.org/html/2606.31320#S5.SS1.p3.8)\.
- R\. Cheng, A\. Verma, G\. Orosz, S\. Chaudhuri, Y\. Yue, and J\. Burdick \(2019c\)Control regularization for reduced variance reinforcement learning\.InInternational Conference on Machine Learning,pp\. 1141–1150\.Cited by:[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- R\. V\. Florian \(2007\)Correct equations for the dynamics of the cart\-pole system\.Center for Cognitive and Neural Studies \(Coneural\), Romania\.External Links:[Document](https://dx.doi.org/https%3A//coneural.org/florian/papers/05%5Fcart%5Fpole.pdf)Cited by:[§C\.2\.1](https://arxiv.org/html/2606.31320#A3.SS2.SSS1.p4.2)\.
- I\. Fox, J\. Lee, R\. Pop\-Busui, and J\. Wiens \(2020\)Deep Reinforcement Learning for Closed\-Loop Blood Glucose Control\.InProceedings of the 5th Machine Learning for Healthcare Conference,pp\. 508–536\(en\)\.Note:ISSN: 2640\-3498External Links:[Link](https://proceedings.mlr.press/v126/fox20a.html)Cited by:[§C\.2\.2](https://arxiv.org/html/2606.31320#A3.SS2.SSS2.p1.2)\.
- R\. Freeman and P\. V\. Kokotovic \(2008\)Robust nonlinear control design: state\-space and lyapunov techniques\.Springer Science & Business Media\.Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p6.1)\.
- S\. Fujimoto, H\. Hoof, and D\. Meger \(2018\)Addressing function approximation error in actor\-critic methods\.pp\. 1587–1596–1587–1596\.Cited by:[§C\.1\.1](https://arxiv.org/html/2606.31320#A3.SS1.SSS1.p1.1),[§3](https://arxiv.org/html/2606.31320#S3.p2.8)\.
- R\. Grandia, A\. J\. Taylor, A\. D\. Ames, and M\. Hutter \(2021\)Multi\-Layered Safety for Legged Robots via Control Barrier Functions and Model Predictive Control\.In2021 IEEE International Conference on Robotics and Automation \(ICRA\),Xi’an, China,pp\. 8352–8358\(en\)\.External Links:ISBN 978\-1\-72819\-077\-8,[Link](https://ieeexplore.ieee.org/document/9561510/),[Document](https://dx.doi.org/10.1109/ICRA48506.2021.9561510)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p6.1),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.7](https://arxiv.org/html/2606.31320#S5.SS7.p1.1)\.
- S\. Ha, P\. Xu, Z\. Tan, S\. Levine, and J\. Tan \(2020\)Learning to Walk in the Real World with Minimal Human Effort\.arXiv\.Note:arXiv:2002\.08550 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2002.08550)Cited by:[Table 5](https://arxiv.org/html/2606.31320#A4.T5.68.64.64.9),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.70.64.64.9)\.
- T\. Haarnoja, A\. Zhou, P\. Abbeel, and S\. Levine \(2018\)Soft actor\-critic: off\-policy maximum entropy deep reinforcement learning with a stochastic actor\.InProceedings of the 35th International Conference on Machine Learning,J\. Dy and A\. Krause \(Eds\.\),Proceedings of Machine Learning Research, Vol\.80,pp\. 1861–1870\.Cited by:[§C\.1\.1](https://arxiv.org/html/2606.31320#A3.SS1.SSS1.p1.1),[Table 5](https://arxiv.org/html/2606.31320#A4.T5.20.16.16.9),[§3](https://arxiv.org/html/2606.31320#S3.p2.8),[§5\.1](https://arxiv.org/html/2606.31320#S5.SS1.p4.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.22.16.16.9)\.
- K\. Hsu, H\. Hu, and J\. F\. Fisac \(2024\)The Safety Filter: A Unified View of Safety\-Critical Control in Autonomous Systems\.Annual Review of Control, Robotics, and Autonomous Systems7\(1\),pp\. 47–72\(en\)\.External Links:ISSN 2573\-5144,[Document](https://dx.doi.org/10.1146/annurev-control-071723-102940)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§1](https://arxiv.org/html/2606.31320#S1.p5.1),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p1.1),[§6](https://arxiv.org/html/2606.31320#S6.p2.1),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- J\. Ibarz, J\. Tan, C\. Finn, M\. Kalakrishnan, P\. Pastor, and S\. Levine \(2021\)How to train your robot with deep reinforcement learning: lessons we have learned\.The International Journal of Robotics Research40\(4\-5\),pp\. 698–721\(en\)\.External Links:ISSN 0278\-3649, 1741\-3176,[Document](https://dx.doi.org/10.1177/0278364920987859)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p1.1)\.
- W\. Jin, S\. Mou, and G\. J\. Pappas \(2021\)Safe pontryagin differentiable programming\.Advances in Neural Information Processing Systems34,pp\. 16034–16050\.External Links:[Link](https://proceedings.neurips.cc/paper/2021/hash/85ea6fd7a2ca3960d0cf5201933ac998-Abstract.html)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- T\. Johannink, S\. Bahl, A\. Nair, J\. Luo, A\. Kumar, M\. Loskyll, J\. A\. Ojea, E\. Solowjow, and S\. Levine \(2019\)Residual reinforcement learning for robot control\.pp\. 6023–6029–6023–6029\.Cited by:[Table 5](https://arxiv.org/html/2606.31320#A4.T5.52.48.48.9),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.54.48.48.9),[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- H\. Krasowski, J\. Thumm, M\. Müller, L\. Schäfer, X\. Wang, and M\. Althoff \(2023\)Provably Safe Reinforcement Learning: Conceptual Analysis, Survey, and Benchmarking\.arXiv\.Note:arXiv:2205\.06750 \[cs\]External Links:[Link](http://arxiv.org/abs/2205.06750),[Document](https://dx.doi.org/10.48550/arXiv.2205.06750)Cited by:[§7](https://arxiv.org/html/2606.31320#S7.p2.1)\.
- T\. P\. Lillicrap, J\. J\. Hunt, A\. Pritzel, N\. Heess, T\. Erez, Y\. Tassa, D\. Silver, and D\. Wierstra \(2015\)Continuous control with deep reinforcement learning\.arXiv preprint arXiv:1509\.02971\.Cited by:[§3](https://arxiv.org/html/2606.31320#S3.p2.8)\.
- H\. Markgraf, S\. Sawant, H\. Krasowski, L\. Schäfer, S\. Gros, and M\. Althoff \(2025\)Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?\.arXiv\.Note:arXiv:2509\.12833 \[cs\]External Links:[Link](http://arxiv.org/abs/2509.12833),[Document](https://dx.doi.org/10.48550/arXiv.2509.12833)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§3](https://arxiv.org/html/2606.31320#S3.p4.2),[§3](https://arxiv.org/html/2606.31320#S3.p5.2),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- F\. Nesti, N\. Salamini, M\. Marinoni, G\. M\. Cicero, G\. Serra, A\. Biondi, and G\. Buttazzo \(2025\)The Use of the Simplex Architecture to Enhance Safety in Deep\-Learning\-Powered Autonomous Systems\.arXiv\.Note:arXiv:2509\.21014 \[eess\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2509.21014)Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p2.1)\.
- D\. T\. Phan, R\. Grosu, N\. Jansen, N\. Paoletti, S\. A\. Smolka, and S\. D\. Stoller \(2020\)Neural Simplex Architecture\.InNASA Formal Methods,R\. Lee, S\. Jha, A\. Mavridou, and D\. Giannakopoulou \(Eds\.\),Vol\.12229,pp\. 97–114\(en\)\.Note:Series Title: Lecture Notes in Computer ScienceExternal Links:ISBN 978\-3\-030\-55753\-9 978\-3\-030\-55754\-6,[Document](https://dx.doi.org/10.1007/978-3-030-55754-6%5F6)Cited by:[§C\.1\.3](https://arxiv.org/html/2606.31320#A3.SS1.SSS3.p1.5),[Table 5](https://arxiv.org/html/2606.31320#A4.T5.28.24.24.9),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.30.24.24.9)\.
- K\. Rana, V\. Dasagi, J\. Haviland, B\. Talbot, M\. Milford, and N\. Sünderhauf \(2023\)Bayesian controller fusion: Leveraging control priors in deep reinforcement learning for robotics\.The International Journal of Robotics Research42\(3\),pp\. 123–146\.Note:Publisher: SAGE Publications Ltd STMExternal Links:ISSN 0278\-3649,[Document](https://dx.doi.org/10.1177/02783649231167210)Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- A\. Ray, J\. Achiam, and D\. Amodei \(2019\)Benchmarking safe exploration in deep reinforcement learning\.arXiv preprint arXiv:1910\.017087\(1\),pp\. 2\.Cited by:[§3](https://arxiv.org/html/2606.31320#S3.p6.1)\.
- J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov \(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§3](https://arxiv.org/html/2606.31320#S3.p2.8)\.
- D\. Seto and L\. Sha \(1999\)A case study on analytical analysis of the inverted pendulum real\-time control system\.Citeseer\.Cited by:[Definition B\.2](https://arxiv.org/html/2606.31320#A2.Thmtheorem2),[Appendix B](https://arxiv.org/html/2606.31320#A2.p1.1),[§C\.2\.1](https://arxiv.org/html/2606.31320#A3.SS2.SSS1.p4.2)\.
- L\. Sha \(2001\)Using simplicity to control complexity\.IEEE Software18\(4\),pp\. 20–28\.External Links:[Document](https://dx.doi.org/10.1109/MS.2001.936213)Cited by:[Appendix B](https://arxiv.org/html/2606.31320#A2.p5.1),[§C\.1\.3](https://arxiv.org/html/2606.31320#A3.SS1.SSS3.p1.5),[§1](https://arxiv.org/html/2606.31320#S1.p6.1),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§5\.1](https://arxiv.org/html/2606.31320#S5.SS1.p1.1)\.
- W\. Suttle, V\. K\. Sharma, K\. C\. Kosaraju, S\. Seetharaman, J\. Liu, V\. Gupta, and B\. M\. Sadler \(2024\)Sampling\-based safe reinforcement learning for nonlinear dynamical systems\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 4420–4428\.External Links:[Link](https://proceedings.mlr.press/v238/suttle24a.html)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- H\. Tian, H\. Hamedmoghadam, R\. Shorten, and P\. Ferraro \(2024\)Reinforcement Learning with Adaptive Regularization for Safe Control of Critical Systems\.arXiv\.Note:arXiv:2404\.15199 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.2404.15199)Cited by:[§C\.2\.2](https://arxiv.org/html/2606.31320#A3.SS2.SSS2.p1.2),[§C\.2\.2](https://arxiv.org/html/2606.31320#A3.SS2.SSS2.p2.3),[§C\.2\.2](https://arxiv.org/html/2606.31320#A3.SS2.SSS2.p3.2),[Table 5](https://arxiv.org/html/2606.31320#A4.T5.44.40.40.9),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p2.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.46.40.40.9)\.
- M\. Towers, A\. Kwiatkowski, J\. Terry, J\. U\. Balis, G\. De Cola, T\. Deleu, M\. Goulão, A\. Kallinteris, M\. Krimmel, A\. KG,et al\.\(2024\)Gymnasium: a standard interface for reinforcement learning environments\.arXiv preprint arXiv:2407\.17032\.Cited by:[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p2.1)\.
- R\. Vershynin \(2018\)High\-dimensional probability: an introduction with applications in data science\.Vol\.47,Cambridge university press\.Cited by:[§A\.2](https://arxiv.org/html/2606.31320#A1.SS2.1.p1.1)\.
- K\. P\. Wabersich and M\. N\. Zeilinger \(2019\)Linear model predictive safety certification for learning\-based control\.arXiv\.Note:arXiv:1803\.08552 \[cs\]External Links:[Document](https://dx.doi.org/10.48550/arXiv.1803.08552)Cited by:[§7](https://arxiv.org/html/2606.31320#S7.p3.1)\.
- A\. Wachi and Y\. Sui \(2020\)Safe reinforcement learning in constrained Markov decision processes\.pp\. 9797–9806–9797–9806\.Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- X\. Wang, N\. Hovakimyan, and L\. Sha \(2013\)L1Simplex: fault\-tolerant control of cyber\-physical systems\.InProceedings of the ACM/IEEE 4th International Conference on Cyber\-Physical Systems,Philadelphia Pennsylvania,pp\. 41–50\(en\)\.External Links:ISBN 978\-1\-4503\-1996\-6,[Link](https://dl.acm.org/doi/10.1145/2502524.2502531),[Document](https://dx.doi.org/10.1145/2502524.2502531)Cited by:[§7](https://arxiv.org/html/2606.31320#S7.p3.1)\.
- T\. Westenbroek, F\. Castaneda, A\. Agrawal, S\. Sastry, and K\. Sreenath \(2022\)Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning\.arXiv preprint arXiv:2208\.06721\.Cited by:[Table 5](https://arxiv.org/html/2606.31320#A4.T5.60.56.56.9),[§1](https://arxiv.org/html/2606.31320#S1.p2.1),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p1.1),[Table 1](https://arxiv.org/html/2606.31320#S5.T1.62.56.56.9),[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- W\. Xiao, T\. Wang, R\. Hasani, M\. Chahine, A\. Amini, X\. Li, and D\. Rus \(2023\)Barriernet: Differentiable control barrier functions for learning of safe robot control\.IEEE Transactions on Robotics39\(3\),pp\. 2289–2307\.External Links:[Link](https://ieeexplore.ieee.org/abstract/document/10077790/)Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§6](https://arxiv.org/html/2606.31320#S6.p3.1)\.
- L\. Xie, S\. Wang, S\. Rosa, A\. Markham, and N\. Trigoni \(2018\)Learning with training wheels: speeding up training with a simple controller for deep reinforcement learning\.pp\. 6276–6283–6276–6283\.Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- R\. Yang, M\. Zhang, N\. Hansen, H\. Xu, and X\. Wang \(2022a\)Learning vision\-guided quadrupedal locomotion end\-to\-end with cross\-modal transformers\.InInternational Conference on Learning Representations,Cited by:[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p2.1)\.
- Y\. Yang, T\. Zhang, E\. Coumans, J\. Tan, and B\. Boots \(2022b\)Fast and efficient locomotion via learned gait transitions\.pp\. 773–783–773–783\.Cited by:[§C\.2\.4](https://arxiv.org/html/2606.31320#A3.SS2.SSS4.p2.7)\.
- Z\. Yuan, A\. W\. Hall, S\. Zhou, L\. Brunke, M\. Greeff, J\. Panerati, and A\. P\. Schoellig \(2022\)Safe\-control\-gym: a unified benchmark suite for safe learning\-based control and reinforcement learning in robotics\.IEEE Robotics and Automation Letters7\(4\),pp\. 11142–11149\.Cited by:[§C\.2\.3](https://arxiv.org/html/2606.31320#A3.SS2.SSS3.p1.4),[§C\.2\.3](https://arxiv.org/html/2606.31320#A3.SS2.SSS3.p3.2),[§5\.2](https://arxiv.org/html/2606.31320#S5.SS2.p2.1)\.
- L\. Zhao, K\. Gatsis, and A\. Papachristodoulou \(2023\)A Barrier\-Lyapunov Actor\-Critic Reinforcement Learning Approach for Safe and Stable Control\.arXiv preprint arXiv:2304\.04066\.Cited by:[§6](https://arxiv.org/html/2606.31320#S6.p1.1)\.
- B\. Zhong, H\. Cao, M\. Zamani, and M\. Caccamo \(2023\)Towards safe ai: sandboxing dnns\-based controllers in stochastic games\.InProceedings of the AAAI Conference on Artificial Intelligence,Vol\.37,pp\. 15340–15349\.Cited by:[§1](https://arxiv.org/html/2606.31320#S1.p3.1),[§2\.2](https://arxiv.org/html/2606.31320#S2.SS2.p3.2),[§6](https://arxiv.org/html/2606.31320#S6.p2.1)\.

## Appendix ASupplementary Results for Risk\-Calibrated Intervention

This appendix provides the assumptions and derivations supporting Sec\.[4\.3](https://arxiv.org/html/2606.31320#S4.SS3)–Sec\.[4\.3](https://arxiv.org/html/2606.31320#S4.SS3)\. The purpose is not to require online estimation of all quantities in Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9), but to justify the structure of the intervention rule used by AutoSafe\.

### A\.1Setup and Assumptions

Fix a state𝐬∈𝒮\\mathbf\{s\}\\in\\mathcal\{S\}\. The learner samples a nominal action𝐚θ∼πθ\(⋅∣𝐬\)\\mathbf\{a\}^\{\\theta\}\\sim\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\), and the safe prior produces a deterministic action

𝐚safe​\(𝐬\)≔πsafe​\(𝐬\)\.\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\coloneqq\\pi^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\.\(13\)For a state\-conditioned intervention weightλ∈\[0,1\]\\lambda\\in\[0,1\], AutoSafe executes

𝐚~λ=\(1−λ\)​𝐚θ\+λ​𝐚safe​\(𝐬\)\.\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}=\(1\-\\lambda\)\\mathbf\{a\}^\{\\theta\}\+\\lambda\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\.\(14\)Throughout this appendix,λ\\lambdais treated as fixed after conditioning on𝐬\\mathbf\{s\}\. Ifλ\\lambdais made dependent on the sampled action𝐚θ\\mathbf\{a\}^\{\\theta\}, then Eq\.[14](https://arxiv.org/html/2606.31320#A1.E14)is no longer an affine map with deterministic coefficient, and the Gaussian pushforward identities below become approximations rather than exact distributional identities\.

Assume the nominal policy is Gaussian,

πθ\(⋅∣𝐬\)=𝒩\(μθ\(𝐬\),Σθ\(𝐬\)\)\.\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)=\\mathcal\{N\}\\\!\\left\(\\mu\_\{\\theta\}\(\\mathbf\{s\}\),\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\\right\)\.\(15\)Then𝐚~λ\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}is Gaussian with

𝐚~λ∼𝒩​\(μ~​\(𝐬\),Σ~​\(𝐬\)\),μ~​\(𝐬\)=\(1−λ\)​μθ​\(𝐬\)\+λ​𝐚safe​\(𝐬\),Σ~​\(𝐬\)=\(1−λ\)2​Σθ​\(𝐬\)\.\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}\\sim\\mathcal\{N\}\\\!\\left\(\\tilde\{\\mu\}\(\\mathbf\{s\}\),\\tilde\{\\Sigma\}\(\\mathbf\{s\}\)\\right\),\\qquad\\tilde\{\\mu\}\(\\mathbf\{s\}\)=\(1\-\\lambda\)\\mu\_\{\\theta\}\(\\mathbf\{s\}\)\+\\lambda\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\),\\qquad\\tilde\{\\Sigma\}\(\\mathbf\{s\}\)=\(1\-\\lambda\)^\{2\}\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\.\(16\)This recovers Eq\.[6](https://arxiv.org/html/2606.31320#S4.E6)in the main text\. For0≤λ<10\\leq\\lambda<1, the differential entropy satisfies

ℋ\(π~λ\(⋅∣𝐬\)\)=ℋ\(πθ\(⋅∣𝐬\)\)\+mlog\(1−λ\),\\mathcal\{H\}\\\!\\left\(\\tilde\{\\pi\}\_\{\\lambda\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\)=\\mathcal\{H\}\\\!\\left\(\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{s\}\)\\right\)\+m\\log\(1\-\\lambda\),\(17\)wheremmis the action dimension\. Atλ=1\\lambda=1, the executed distribution collapses to a Dirac distribution at𝐚safe​\(𝐬\)\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)and does not have finite differential entropy\.

Safety is specified by a margin functionΔ​\(𝐬,𝐚\)\\Delta\(\\mathbf\{s\},\\mathbf\{a\}\)and thresholdΔmin\\Delta\_\{\\min\}\. We define the relative margin

Δ^​\(𝐬,𝐚\)≔Δ​\(𝐬,𝐚\)−Δmin,Δ^​\(𝐬,𝐚\)≥0⟺safe at​𝐬\.\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\coloneqq\\Delta\(\\mathbf\{s\},\\mathbf\{a\}\)\-\\Delta\_\{\\min\},\\qquad\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\geq 0\\Longleftrightarrow\\text\{safe at \}\\mathbf\{s\}\.\(18\)The random margin under the executed action is

Z​\(λ\)≔Δ^​\(𝐬,𝐚~λ\)\.Z\(\\lambda\)\\coloneqq\\hat\{\\Delta\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}\)\.\(19\)The chance constraint used in the main text is

Pr⁡\(Z​\(λ\)​<0∣​𝐬\)≤δ​\(𝐬\)\.\\Pr\\\!\\left\(Z\(\\lambda\)<0\\mid\\mathbf\{s\}\\right\)\\leq\\delta\(\\mathbf\{s\}\)\.\(20\)
For convenience, define

μΔ​\(𝐬\)≔𝔼​\[Δ^​\(𝐬,𝐚θ\)∣𝐬\],σΔ2​\(𝐬\)≔Var⁡\(Δ^​\(𝐬,𝐚θ\)∣𝐬\),Δsafe​\(𝐬\)≔Δ^​\(𝐬,𝐚safe​\(𝐬\)\)\.\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\\coloneqq\\mathbb\{E\}\\\!\\left\[\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\mid\\mathbf\{s\}\\right\],\\qquad\\sigma\_\{\\Delta\}^\{2\}\(\\mathbf\{s\}\)\\coloneqq\\operatorname\{Var\}\\\!\\left\(\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\)\\mid\\mathbf\{s\}\\right\),\\qquad\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\coloneqq\\hat\{\\Delta\}\\\!\\left\(\\mathbf\{s\},\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\right\)\.\(21\)All expectations and variances are conditioned on the fixed state𝐬\\mathbf\{s\}\.

The following assumptions are used only to justify the closed\-form template in Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9)\.

###### Assumption A\.1\.

For fixed𝐬\\mathbf\{s\}, the relative marginΔ^​\(𝐬,𝐚\)\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)is locally approximated by an affine function of𝐚\\mathbf\{a\}on a neighborhood containing the interpolation segment between𝐚θ\\mathbf\{a\}^\{\\theta\}and𝐚safe​\(𝐬\)\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\. That is, there existb​\(𝐬\)∈ℝb\(\\mathbf\{s\}\)\\in\\mathbb\{R\}and𝐠​\(𝐬\)∈ℝm\\mathbf\{g\}\(\\mathbf\{s\}\)\\in\\mathbb\{R\}^\{m\}such that

Δ^​\(𝐬,𝐚\)=b​\(𝐬\)\+𝐠​\(𝐬\)⊤​𝐚\+εlin​\(𝐬,𝐚\),\|εlin​\(𝐬,𝐚\)\|≤ϵlin​\(𝐬\)\\hat\{\\Delta\}\(\\mathbf\{s\},\\mathbf\{a\}\)=b\(\\mathbf\{s\}\)\+\\mathbf\{g\}\(\\mathbf\{s\}\)^\{\\top\}\\mathbf\{a\}\+\\varepsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\},\\mathbf\{a\}\),\\qquad\\left\|\\varepsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\},\\mathbf\{a\}\)\\right\|\\leq\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)\(22\)for all actions𝐚\\mathbf\{a\}on this segment\. Whenϵlin​\(𝐬\)=0\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)=0, the margin is exactly affine along the interpolation direction\.

###### Assumption A\.2\.

Conditioned on𝐬\\mathbf\{s\}, the nominal action𝐚θ\\mathbf\{a\}^\{\\theta\}is sub\-Gaussian: for any𝐮∈ℝm\\mathbf\{u\}\\in\\mathbb\{R\}^\{m\},𝐮⊤​\(𝐚θ−𝔼​\[𝐚θ\]\)\\mathbf\{u\}^\{\\top\}\(\\mathbf\{a\}^\{\\theta\}\-\\mathbb\{E\}\[\\mathbf\{a\}^\{\\theta\}\]\)is sub\-Gaussian with variance proxy𝐮⊤​Σθ​\(𝐬\)​𝐮\\mathbf\{u\}^\{\\top\}\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\\mathbf\{u\}\. This holds exactly when𝐚θ∼𝒩​\(μθ​\(𝐬\),Σθ​\(𝐬\)\)\\mathbf\{a\}^\{\\theta\}\\sim\\mathcal\{N\}\(\\mu\_\{\\theta\}\(\\mathbf\{s\}\),\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\)\.

###### Assumption A\.3\.

The safe prior has nonnegative relative safety margin at𝐬\\mathbf\{s\}:

Δsafe​\(𝐬\)=Δ^​\(𝐬,𝐚safe​\(𝐬\)\)≥0\.\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)=\\hat\{\\Delta\}\\\!\\left\(\\mathbf\{s\},\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\right\)\\geq 0\.\(23\)When strict reserve is required, we assumeΔsafe​\(𝐬\)\>0\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\>0\.

### A\.2Useful Facts

###### Definition A\.4\(Sub\-Gaussian random variable\)\.

A scalar random variableXXisσ2\\sigma^\{2\}\-sub\-Gaussian if

𝔼​\[exp⁡\(t​\(X−𝔼​\[X\]\)\)\]≤exp⁡\(t2​σ22\),∀t∈ℝ\.\\mathbb\{E\}\\left\[\\exp\\\!\\left\(t\(X\-\\mathbb\{E\}\[X\]\)\\right\)\\right\]\\leq\\exp\\\!\\left\(\\frac\{t^\{2\}\\sigma^\{2\}\}\{2\}\\right\),\\qquad\\forall t\\in\\mathbb\{R\}\.\(24\)

###### Lemma A\.5\.

IfXXisσ2\\sigma^\{2\}\-sub\-Gaussian, then for anyt\>0t\>0,

Pr⁡\(X−𝔼​\[X\]≤−t\)≤exp⁡\(−t22​σ2\)\.\\Pr\\\!\\left\(X\-\\mathbb\{E\}\[X\]\\leq\-t\\right\)\\leq\\exp\\\!\\left\(\-\\frac\{t^\{2\}\}\{2\\sigma^\{2\}\}\\right\)\.\(25\)

###### Proof\.

This is the standard one\-sided sub\-Gaussian Chernoff bound; seeVershynin \([2018](https://arxiv.org/html/2606.31320#bib.bib4)\)\. ∎

###### Corollary A\.6\.

LetXXbeσ2\\sigma^\{2\}\-sub\-Gaussian\. Then

Pr⁡\(X<0\)≤δ⇐𝔼​\[X\]≥2​log⁡1δ​σ\.\\Pr\(X<0\)\\leq\\delta\\quad\\Leftarrow\\quad\\mathbb\{E\}\[X\]\\geq\\sqrt\{2\\log\\\!\\frac\{1\}\{\\delta\}\}\\,\\sigma\.\(26\)Equivalently,𝔼​\[X\]−β​\(δ\)​σ≥0\\mathbb\{E\}\[X\]\-\\beta\(\\delta\)\\sigma\\geq 0withβ​\(δ\)=2​log⁡\(1/δ\)\\beta\(\\delta\)=\\sqrt\{2\\log\(1/\\delta\)\}\.

###### Proof\.

Apply Lemma[A\.5](https://arxiv.org/html/2606.31320#A1.Thmtheorem5)witht=𝔼​\[X\]t=\\mathbb\{E\}\[X\]\. ∎

###### Corollary A\.7\(Gaussian quantile\)\.

IfX∼𝒩​\(μ,σ2\)X\\sim\\mathcal\{N\}\(\\mu,\\sigma^\{2\}\), then

Pr⁡\(X<0\)≤δ⟺μ≥z1−δ​σ,\\Pr\(X<0\)\\leq\\delta\\quad\\Longleftrightarrow\\quad\\mu\\geq z\_\{1\-\\delta\}\\sigma,\(27\)wherez1−δ=Φ−1​\(1−δ\)z\_\{1\-\\delta\}=\\Phi^\{\-1\}\(1\-\\delta\)is the\(1−δ\)\(1\-\\delta\)quantile of the standard normal distribution\.

### A\.3Closed\-Form Intervention Weight

Under Assumption[A\.1](https://arxiv.org/html/2606.31320#A1.Thmtheorem1)withϵlin​\(𝐬\)=0\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)=0, the relative margin is exactly affine along the interpolation segment\. Therefore,

Z​\(λ\)=\(1−λ\)​Z​\(0\)\+λ​Δsafe​\(𝐬\)\.Z\(\\lambda\)=\(1\-\\lambda\)Z\(0\)\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\.\(28\)Taking moments gives

𝔼​\[Z​\(λ\)\]=\(1−λ\)​μΔ​\(𝐬\)\+λ​Δsafe​\(𝐬\),Var⁡\(Z​\(λ\)\)=\(1−λ\)2​σΔ2​\(𝐬\)\.\\mathbb\{E\}\[Z\(\\lambda\)\]=\(1\-\\lambda\)\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\),\\qquad\\operatorname\{Var\}\(Z\(\\lambda\)\)=\(1\-\\lambda\)^\{2\}\\sigma\_\{\\Delta\}^\{2\}\(\\mathbf\{s\}\)\.\(29\)
###### Proposition A\.8\.

Assume the sufficient tail condition

𝔼​\[Z​\(λ\)\]−β​Var⁡\(Z​\(λ\)\)≥0\\mathbb\{E\}\[Z\(\\lambda\)\]\-\\beta\\sqrt\{\\operatorname\{Var\}\(Z\(\\lambda\)\)\}\\geq 0\(30\)for someβ\>0\\beta\>0\. Under the exact interpolation identities in Eq\.[29](https://arxiv.org/html/2606.31320#A1.E29), define the risk\-adjusted nominal margin

Lθ​\(𝐬\)≔μΔ​\(𝐬\)−β​σΔ​\(𝐬\)\.L\_\{\\theta\}\(\\mathbf\{s\}\)\\coloneqq\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\-\\beta\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\.\(31\)Then the smallestλ∈\[0,1\]\\lambda\\in\[0,1\]satisfying Eq\.[30](https://arxiv.org/html/2606.31320#A1.E30)is

λ∗​\(𝐬\)=clip\[0,1\]​\(β​σΔ​\(𝐬\)−μΔ​\(𝐬\)Δsafe​\(𝐬\)−μΔ​\(𝐬\)\+β​σΔ​\(𝐬\)\)=\{0,Lθ​\(𝐬\)≥0,−Lθ​\(𝐬\)Δsafe​\(𝐬\)−Lθ​\(𝐬\),Lθ​\(𝐬\)<0\.\\lambda^\{\*\}\(\\mathbf\{s\}\)=\\mathrm\{clip\}\_\{\[0,1\]\}\\left\(\\frac\{\\beta\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\-\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\}\{\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\-\\mu\_\{\\Delta\}\(\\mathbf\{s\}\)\+\\beta\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\}\\right\)=\\begin\{cases\}0,&L\_\{\\theta\}\(\\mathbf\{s\}\)\\geq 0,\\\\\[6\.0pt\] \\dfrac\{\-L\_\{\\theta\}\(\\mathbf\{s\}\)\}\{\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\-L\_\{\\theta\}\(\\mathbf\{s\}\)\},&L\_\{\\theta\}\(\\mathbf\{s\}\)<0\.\\end\{cases\}\(32\)This is Eq\.[9](https://arxiv.org/html/2606.31320#S4.E9)in the main text withβ=β​\(δ​\(𝐬\)\)\\beta=\\beta\(\\delta\(\\mathbf\{s\}\)\)\.

###### Proof\.

Using Eq\.[29](https://arxiv.org/html/2606.31320#A1.E29),

𝔼​\[Z​\(λ\)\]−β​Var⁡\(Z​\(λ\)\)\\displaystyle\\mathbb\{E\}\[Z\(\\lambda\)\]\-\\beta\\sqrt\{\\operatorname\{Var\}\(Z\(\\lambda\)\)\}=\(1−λ\)​μΔ\+λ​Δsafe−β​\(1−λ\)​σΔ\\displaystyle=\(1\-\\lambda\)\\mu\_\{\\Delta\}\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\-\\beta\(1\-\\lambda\)\\sigma\_\{\\Delta\}\(33\)=\(1−λ\)​Lθ\+λ​Δsafe\.\\displaystyle=\(1\-\\lambda\)L\_\{\\theta\}\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\.\(34\)IfLθ≥0L\_\{\\theta\}\\geq 0, the condition holds atλ=0\\lambda=0, soλ∗=0\\lambda^\{\*\}=0\. IfLθ<0L\_\{\\theta\}<0, solving

\(1−λ\)​Lθ\+λ​Δsafe≥0\(1\-\\lambda\)L\_\{\\theta\}\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\\geq 0gives

λ≥−LθΔsafe−Lθ\.\\lambda\\geq\\frac\{\-L\_\{\\theta\}\}\{\\Delta\_\{\\mathrm\{safe\}\}\-L\_\{\\theta\}\}\.Clamping to\[0,1\]\[0,1\]yields Eq\.[32](https://arxiv.org/html/2606.31320#A1.E32)\. ∎

The expression in Eq\.[32](https://arxiv.org/html/2606.31320#A1.E32)has a geometric interpretation: the intervention weight is the ratio between the nominal safety deficit,max⁡\(0,−Lθ​\(𝐬\)\)\\max\(0,\-L\_\{\\theta\}\(\\mathbf\{s\}\)\), and the total safety improvement available by moving from the nominal policy toward the safe prior,Δsafe​\(𝐬\)−Lθ​\(𝐬\)\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\-L\_\{\\theta\}\(\\mathbf\{s\}\)\.

### A\.4Robustness to Local Approximation Error

The exact derivation above assumes that the safety margin is affine along the interpolation segment\. We next state a conservative perturbation bound when the local affine approximation has bounded error\.

###### Proposition A\.9\.

Under Assumption[A\.1](https://arxiv.org/html/2606.31320#A1.Thmtheorem1), define the ideal interpolated margin

Zint​\(λ\)≔\(1−λ\)​Z​\(0\)\+λ​Δsafe​\(𝐬\)\.Z\_\{\\mathrm\{int\}\}\(\\lambda\)\\coloneqq\(1\-\\lambda\)Z\(0\)\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\.\(35\)Then

\|Z​\(λ\)−Zint​\(λ\)\|≤2​ϵlin​\(𝐬\)\.\\left\|Z\(\\lambda\)\-Z\_\{\\mathrm\{int\}\}\(\\lambda\)\\right\|\\leq 2\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)\.\(36\)Consequently,

\|𝔼​\[Z​\(λ\)\]−\(\(1−λ\)​μΔ\+λ​Δsafe\)\|≤2​ϵlin​\(𝐬\),\\left\|\\mathbb\{E\}\[Z\(\\lambda\)\]\-\\left\(\(1\-\\lambda\)\\mu\_\{\\Delta\}\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\\right\)\\right\|\\leq 2\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\),\(37\)and

Var⁡\(Z​\(λ\)\)≤\(1−λ\)​σΔ​\(𝐬\)\+2​ϵlin​\(𝐬\)\.\\sqrt\{\\operatorname\{Var\}\(Z\(\\lambda\)\)\}\\leq\(1\-\\lambda\)\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\+2\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)\.\(38\)

###### Proof\.

Let

ελ=εlin​\(𝐬,𝐚~λ\),ε0=εlin​\(𝐬,𝐚θ\),εsafe=εlin​\(𝐬,𝐚safe\)\.\\varepsilon\_\{\\lambda\}=\\varepsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\},\\tilde\{\\mathbf\{a\}\}\_\{\\lambda\}\),\\quad\\varepsilon\_\{0\}=\\varepsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\theta\}\),\\quad\\varepsilon\_\{\\mathrm\{safe\}\}=\\varepsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\},\\mathbf\{a\}^\{\\mathrm\{safe\}\}\)\.By Assumption[A\.1](https://arxiv.org/html/2606.31320#A1.Thmtheorem1),

Z​\(λ\)−Zint​\(λ\)=ελ−\(1−λ\)​ε0−λ​εsafe\.Z\(\\lambda\)\-Z\_\{\\mathrm\{int\}\}\(\\lambda\)=\\varepsilon\_\{\\lambda\}\-\(1\-\\lambda\)\\varepsilon\_\{0\}\-\\lambda\\varepsilon\_\{\\mathrm\{safe\}\}\.Since each error term has magnitude at mostϵlin​\(𝐬\)\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\),

\|Z​\(λ\)−Zint​\(λ\)\|≤ϵlin\+\(1−λ\)​ϵlin\+λ​ϵlin=2​ϵlin\.\\left\|Z\(\\lambda\)\-Z\_\{\\mathrm\{int\}\}\(\\lambda\)\\right\|\\leq\\epsilon\_\{\\mathrm\{lin\}\}\+\(1\-\\lambda\)\\epsilon\_\{\\mathrm\{lin\}\}\+\\lambda\\epsilon\_\{\\mathrm\{lin\}\}=2\\epsilon\_\{\\mathrm\{lin\}\}\.Taking expectations gives Eq\.[37](https://arxiv.org/html/2606.31320#A1.E37)\. For the standard deviation bound, write

Z​\(λ\)=Zint​\(λ\)\+ξλ,\|ξλ\|≤2​ϵlin\.Z\(\\lambda\)=Z\_\{\\mathrm\{int\}\}\(\\lambda\)\+\\xi\_\{\\lambda\},\\qquad\|\\xi\_\{\\lambda\}\|\\leq 2\\epsilon\_\{\\mathrm\{lin\}\}\.By the triangle inequality for standard deviation,

Std⁡\[Z​\(λ\)\]≤Std⁡\[Zint​\(λ\)\]\+Std⁡\[ξλ\]≤\(1−λ\)​σΔ\+2​ϵlin\.\\operatorname\{Std\}\[Z\(\\lambda\)\]\\leq\\operatorname\{Std\}\[Z\_\{\\mathrm\{int\}\}\(\\lambda\)\]\+\\operatorname\{Std\}\[\\xi\_\{\\lambda\}\]\\leq\(1\-\\lambda\)\\sigma\_\{\\Delta\}\+2\\epsilon\_\{\\mathrm\{lin\}\}\.∎

It follows that a conservative sufficient condition under local approximation error is

\(1−λ\)​Lθ​\(𝐬\)\+λ​Δsafe​\(𝐬\)≥2​\(1\+β\)​ϵlin​\(𝐬\)\.\(1\-\\lambda\)L\_\{\\theta\}\(\\mathbf\{s\}\)\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\geq 2\(1\+\\beta\)\\epsilon\_\{\\mathrm\{lin\}\}\(\\mathbf\{s\}\)\.\(39\)Indeed, Eq\.[37](https://arxiv.org/html/2606.31320#A1.E37)and Eq\.[38](https://arxiv.org/html/2606.31320#A1.E38)imply

𝔼​\[Z​\(λ\)\]−β​Std⁡\[Z​\(λ\)\]≥\(1−λ\)​Lθ\+λ​Δsafe−2​\(1\+β\)​ϵlin\.\\displaystyle\\mathbb\{E\}\[Z\(\\lambda\)\]\-\\beta\\operatorname\{Std\}\[Z\(\\lambda\)\]\\geq\(1\-\\lambda\)L\_\{\\theta\}\+\\lambda\\Delta\_\{\\mathrm\{safe\}\}\-2\(1\+\\beta\)\\epsilon\_\{\\mathrm\{lin\}\}\.\(40\)Thus, the same risk bufferβ​σΔ\\beta\\sigma\_\{\\Delta\}that controls stochastic tail risk also provides robustness to local certificate mismatch when it dominates the linearization error\.

### A\.5Near\-Determinism at the Boundary

The main text motivates near\-boundary determinism by requiring the tail budget to become more conservative as the system approaches the safety boundary\. The following result formalizes this limiting behavior for the closed\-form template\.

###### Proposition A\.10\.

AssumeσΔ​\(𝐬\)\>0\\sigma\_\{\\Delta\}\(\\mathbf\{s\}\)\>0andΔsafe​\(𝐬\)∈\(0,∞\)\\Delta\_\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\\in\(0,\\infty\)\. Let the risk tolerance be

δ​\(𝐬\)=exp⁡\(−ρ​\(Δ~​\(𝐬\)\)\),\\delta\(\\mathbf\{s\}\)=\\exp\\\!\\left\(\-\\rho\(\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\)\\right\),whereΔ~​\(𝐬\)∈\[0,1\]\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\\in\[0,1\]is a normalized margin withΔ~​\(𝐬\)=0\\tilde\{\\Delta\}\(\\mathbf\{s\}\)=0at the safety boundary, andρ​\(x\)→∞\\rho\(x\)\\to\\inftyasx↓0x\\downarrow 0\. If the sub\-Gaussian quantileβ​\(δ\)=2​log⁡\(1/δ\)\\beta\(\\delta\)=\\sqrt\{2\\log\(1/\\delta\)\}is used, then for any sequence of states withΔ~​\(𝐬\)↓0\\tilde\{\\Delta\}\(\\mathbf\{s\}\)\\downarrow 0,

β​\(δ​\(𝐬\)\)→∞⟹λ∗​\(𝐬\)→1⟹Σ~​\(𝐬\)=\(1−λ∗​\(𝐬\)\)2​Σθ​\(𝐬\)→0\.\\beta\(\\delta\(\\mathbf\{s\}\)\)\\to\\infty\\quad\\Longrightarrow\\quad\\lambda^\{\*\}\(\\mathbf\{s\}\)\\to 1\\quad\\Longrightarrow\\quad\\tilde\{\\Sigma\}\(\\mathbf\{s\}\)=\(1\-\\lambda^\{\*\}\(\\mathbf\{s\}\)\)^\{2\}\\Sigma\_\{\\theta\}\(\\mathbf\{s\}\)\\to 0\.\(41\)

###### Proof\.

Using Eq\.[32](https://arxiv.org/html/2606.31320#A1.E32), for sufficiently largeβ\\beta,

λ∗​\(𝐬\)=β​σΔ−μΔΔsafe−μΔ\+β​σΔ\.\\lambda^\{\*\}\(\\mathbf\{s\}\)=\\frac\{\\beta\\sigma\_\{\\Delta\}\-\\mu\_\{\\Delta\}\}\{\\Delta\_\{\\mathrm\{safe\}\}\-\\mu\_\{\\Delta\}\+\\beta\\sigma\_\{\\Delta\}\}\.Asβ→∞\\beta\\to\\infty, the dominant terms give

λ∗​\(𝐬\)→β​σΔΔsafe\+β​σΔ=1−ΔsafeΔsafe\+β​σΔ→1,\\lambda^\{\*\}\(\\mathbf\{s\}\)\\to\\frac\{\\beta\\sigma\_\{\\Delta\}\}\{\\Delta\_\{\\mathrm\{safe\}\}\+\\beta\\sigma\_\{\\Delta\}\}=1\-\\frac\{\\Delta\_\{\\mathrm\{safe\}\}\}\{\\Delta\_\{\\mathrm\{safe\}\}\+\\beta\\sigma\_\{\\Delta\}\}\\to 1,becauseσΔ\>0\\sigma\_\{\\Delta\}\>0andΔsafe<∞\\Delta\_\{\\mathrm\{safe\}\}<\\infty\. The covariance claim then follows immediately from Eq\.[16](https://arxiv.org/html/2606.31320#A1.E16)\. ∎

Proposition[A\.10](https://arxiv.org/html/2606.31320#A1.Thmtheorem10)motivates the endpoint behavior of the implemented rate, but AutoSafe does not need to estimate the risk potentialρ\\rhoexplicitly\. Instead, Eq\.[10](https://arxiv.org/html/2606.31320#S4.E10)enforces the boundary behavior by construction:λ​\(0,p\)=1\\lambda\(0,p\)=1andλ​\(1,p\)=0\\lambda\(1,p\)=0for allp\>0p\>0\. The learnable sharpness parameterppcontrols the finite\-margin intervention profile, while the covariance collapse at the boundary follows directly from the affine policy composition\.

## Appendix BSafety Design for Simplex

There exist many approaches for safe policy design\. In this work, we adopt the procedure introduced in\(Seto and Sha,[1999](https://arxiv.org/html/2606.31320#bib.bib21)\)to synthesize a robust safe controller and derive a certified safety envelope using an over\-approximated system model\. In this appendix, we summarize the general formulation and solution procedure\. Detailed implementation and experiment\-specific parameters are provided in the supplementary code\.

Consider the nonlinear system

𝐬˙=𝐀𝐬\+𝐁𝐚⏟known\+𝐟​\(𝐬,𝐚\)⏟unknown,\\displaystyle\\dot\{\\mathbf\{s\}\}=\\underbrace\{\\mathbf\{A\}\\mathbf\{s\}\+\\mathbf\{B\}\\mathbf\{a\}\}\_\{\\text\{known\}\}\+\\underbrace\{\\mathbf\{f\}\(\\mathbf\{s\},\\mathbf\{a\}\)\}\_\{\\text\{unknown\}\},\(42\)where𝐬∈ℝn\\mathbf\{s\}\\in\\mathbb\{R\}^\{n\}and𝐚∈ℝm\\mathbf\{a\}\\in\\mathbb\{R\}^\{m\}denote the state and action, respectively\.

The objective is to design a robust safe controller that keeps the system within the safety set

𝒮f≔\{𝐬∈ℝn∣𝐀s​𝐬≤𝐛s\},\\displaystyle\\mathcal\{S\}\_\{f\}\\coloneqq\\left\\\{\\mathbf\{s\}\\in\\mathbb\{R\}^\{n\}\\mid\\mathbf\{A\}\_\{s\}\\mathbf\{s\}\\leq\\mathbf\{b\}\_\{s\}\\right\\\},\(43\)under the action constraints

𝒜f≔\{𝐚∈ℝm∣𝐀a​𝐚≤𝐛a\}\.\\displaystyle\\mathcal\{A\}\_\{f\}\\coloneqq\\left\\\{\\mathbf\{a\}\\in\\mathbb\{R\}^\{m\}\\mid\\mathbf\{A\}\_\{a\}\\mathbf\{a\}\\leq\\mathbf\{b\}\_\{a\}\\right\\\}\.\(44\)
Here,𝐀s∈ℝns×n\\mathbf\{A\}\_\{s\}\\in\\mathbb\{R\}^\{n\_\{s\}\\times n\}and𝐛s∈ℝns\\mathbf\{b\}\_\{s\}\\in\\mathbb\{R\}^\{n\_\{s\}\}define the state constraints, while𝐀a∈ℝna×m\\mathbf\{A\}\_\{a\}\\in\\mathbb\{R\}^\{n\_\{a\}\\times m\}and𝐛a∈ℝna\\mathbf\{b\}\_\{a\}\\in\\mathbb\{R\}^\{n\_\{a\}\}define the action constraints\.

###### Definition B\.1\(Safety Definition\)\.

Consider the safety set𝒮f\\mathcal\{S\}\_\{f\}Eq\.[43](https://arxiv.org/html/2606.31320#A2.E43)\. The system Eq\.[42](https://arxiv.org/html/2606.31320#A2.E42)is said to be safe, if given any𝐬t∈𝒮f\\mathbf\{s\}\_\{t\}\\in\\mathcal\{S\}\_\{f\}, the𝐬t\+1∈𝒮f\\mathbf\{s\}\_\{t\+1\}\\in\\mathcal\{S\}\_\{f\}holds for any timet∈ℕt\\in\\mathbb\{N\}\.

Directly designing a verifiable safe controller for the nonlinear system Eq\.[42](https://arxiv.org/html/2606.31320#A2.E42)is difficult due to the unknown nonlinear term𝐟​\(𝐬,𝐚\)\\mathbf\{f\}\(\\mathbf\{s\},\\mathbf\{a\}\)\. Instead, we consider the linear over\-approximation\(Sha,[2001](https://arxiv.org/html/2606.31320#bib.bib107)\)

𝐬˙=𝐀𝐬\+𝐁𝐚\.\\displaystyle\\dot\{\\mathbf\{s\}\}=\\mathbf\{A\}\\mathbf\{s\}\+\\mathbf\{B\}\\mathbf\{a\}\.\(45\)
We parameterize the safe controller as a linear feedback policy

𝐚=𝐅𝐬,\\displaystyle\\mathbf\{a\}=\\mathbf\{F\}\\mathbf\{s\},\(46\)where𝐅∈ℝm×n\\mathbf\{F\}\\in\\mathbb\{R\}^\{m\\times n\}is the controller gain matrix\. The resulting closed\-loop dynamics become

𝐬˙=𝐀¯​𝐬,𝐀¯=𝐀\+𝐁𝐅\.\\displaystyle\\dot\{\\mathbf\{s\}\}=\\bar\{\\mathbf\{A\}\}\\mathbf\{s\},\\qquad\\bar\{\\mathbf\{A\}\}=\\mathbf\{A\}\+\\mathbf\{B\}\\mathbf\{F\}\.\(47\)
To jointly enforce state and action constraints, we define the unified constraint matrix

𝐃=\[𝐀sdiag\(𝐛s\)−1𝐀a𝐅diag\(𝐛a\)−1\],\\displaystyle\\mathbf\{D\}=\\begin\{bmatrix\}\\mathbf\{A\}\_\{s\}\\operatorname\{diag\}\(\\mathbf\{b\}\_\{s\}\)^\{\-1\}\\\\ \\mathbf\{A\}\_\{a\}\\mathbf\{F\}\\operatorname\{diag\}\(\\mathbf\{b\}\_\{a\}\)^\{\-1\}\\end\{bmatrix\},\(48\)such that the constraints can be compactly written as

𝐃𝐬≤𝟏\.\\displaystyle\\mathbf\{D\}\\mathbf\{s\}\\leq\\mathbf\{1\}\.\(49\)
The objective is to find a controller gain𝐅\\mathbf\{F\}such that the closed\-loop system is asymptotically stable and admits a certified forward\-invariant subset of the safety set\.

###### Definition B\.2\(Quadratic Stability\(Seto and Sha,[1999](https://arxiv.org/html/2606.31320#bib.bib21)\)\)\.

The closed\-loop system Eq\.[47](https://arxiv.org/html/2606.31320#A2.E47)is quadratically stable if there exists a positive definite matrix𝐏≻0\\mathbf\{P\}\\succ 0such that the quadratic Lyapunov function

V​\(𝐬\)=𝐬⊤​𝐏𝐬\\displaystyle V\(\\mathbf\{s\}\)=\\mathbf\{s\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\}\(50\)has negative derivative along all trajectories of the system\.

The derivative of the Lyapunov function is

V˙​\(𝐬\)=𝐬⊤​\(𝐀¯⊤​𝐏\+𝐏​𝐀¯\)​𝐬\.\\displaystyle\\dot\{V\}\(\\mathbf\{s\}\)=\\mathbf\{s\}^\{\\top\}\\left\(\\bar\{\\mathbf\{A\}\}^\{\\top\}\\mathbf\{P\}\+\\mathbf\{P\}\\bar\{\\mathbf\{A\}\}\\right\)\\mathbf\{s\}\.\(51\)
Therefore, the closed\-loop system is asymptotically stable if there exists𝐏≻0\\mathbf\{P\}\\succ 0satisfying

𝐀¯⊤​𝐏\+𝐏​𝐀¯⪯0\.\\displaystyle\\bar\{\\mathbf\{A\}\}^\{\\top\}\\mathbf\{P\}\+\\mathbf\{P\}\\bar\{\\mathbf\{A\}\}\\preceq 0\.\(52\)
The corresponding certified stability region is the ellipsoid

Ω≜\{𝐬∈ℝn∣𝐬⊤​𝐏𝐬≤1\}\.\\displaystyle\\Omega\\triangleq\\left\\\{\\mathbf\{s\}\\in\\mathbb\{R\}^\{n\}\\mid\\mathbf\{s\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\}\\leq 1\\right\\\}\.\(53\)
To maximize the size of this certified region, we introduce the change of variables

𝐐=𝐏−1,𝐑=𝐅𝐐,\\displaystyle\\mathbf\{Q\}=\\mathbf\{P\}^\{\-1\},\\qquad\\mathbf\{R\}=\\mathbf\{FQ\},\(54\)and formulate the following semidefinite program:

min𝐐≻0,𝐑\\displaystyle\\min\_\{\\mathbf\{Q\}\\succ 0,\\mathbf\{R\}\}\\quad−log​det\(𝐐\)\\displaystyle\-\\log\\det\(\\mathbf\{Q\}\)\(55\)s\.t\.𝐃𝐐𝐃⊤⪯𝐈,\\displaystyle\\mathbf\{D\}\\mathbf\{Q\}\\mathbf\{D\}^\{\\top\}\\preceq\\mathbf\{I\},\(56\)\[α​𝐐\(𝐀𝐐\+𝐁𝐑\)⊤𝐀𝐐\+𝐁𝐑𝐐\]⪰0,\\displaystyle\\begin\{bmatrix\}\\alpha\\mathbf\{Q\}&\(\\mathbf\{A\}\\mathbf\{Q\}\+\\mathbf\{B\}\\mathbf\{R\}\)^\{\\top\}\\\\ \\mathbf\{A\}\\mathbf\{Q\}\+\\mathbf\{B\}\\mathbf\{R\}&\\mathbf\{Q\}\\end\{bmatrix\}\\succeq 0,\(57\)\[𝐐𝐑⊤𝐑1β​𝐈m\]⪰0\.\\displaystyle\\begin\{bmatrix\}\\mathbf\{Q\}&\\mathbf\{R\}^\{\\top\}\\\\ \\mathbf\{R\}&\\frac\{1\}\{\\beta\}\\mathbf\{I\}\_\{m\}\\end\{bmatrix\}\\succeq 0\.\(58\)
Solving the SDP in Eq\.[55](https://arxiv.org/html/2606.31320#A2.E55)–Eq\.[58](https://arxiv.org/html/2606.31320#A2.E58)yields𝐐\\mathbf\{Q\}and𝐑\\mathbf\{R\}, from which the Lyapunov matrix and controller gain are recovered as

𝐏=𝐐−1,𝐅=𝐑𝐐−1\.\\displaystyle\\mathbf\{P\}=\\mathbf\{Q\}^\{\-1\},\\qquad\\mathbf\{F\}=\\mathbf\{R\}\\mathbf\{Q\}^\{\-1\}\.\(59\)
The matrices𝐏\\mathbf\{P\}and𝐅\\mathbf\{F\}, together with the system matrices𝐀\\mathbf\{A\}and𝐁\\mathbf\{B\}used in the experiments, are provided in the accompanying code\.

## Appendix CExperimental Details

### C\.1Algorithm Implementations

#### C\.1\.1Soft actor\-critic \(SAC\) Baseline

Our implementation of Soft Actor\-Critic follows\(Haarnojaet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib76)\)and\(Fujimotoet al\.,[2018](https://arxiv.org/html/2606.31320#bib.bib130)\), with default parameters summarized in Table[3](https://arxiv.org/html/2606.31320#A3.T3)\. These parameters are shared across all baseline algorithms unless specified otherwise\.

Table 2:Parameter setting for Soft Actor\-Critic \(SAC\)HyperparameterValueDiscount factor \(γ\\gamma\)0\.99Learning rate \(actor, critic\)3×10−43\\times 10^\{\-4\}OptimizerAdamTarget smoothing coefficient \(τ\\tau\)0\.005Entropy coefficient0\.1Target update interval1Activation functionReLUTraining steps per environment step1Evaluation period \(steps\)10000Neural Network \(MLP\)\[256, 256\]Batch size128Table 3:Task Specific Parameter Setting
#### C\.1\.2AutoSafe

![Refer to caption](https://arxiv.org/html/2606.31320v1/x12.png)Figure 8:AutoSafe: The policy uses the state vector𝐬t\\mathbf\{s\}\_\{t\}, included in the observation, to assess riskΔ~​\(𝐬t\)\\tilde\{\\Delta\}\(\\mathbf\{s\}\_\{t\}\)and generate safe action𝐚safe\\mathbf\{a\}^\{\\mathrm\{safe\}\}\. In parallel, the learning\-based component processes the full observation𝐨t=\(𝐬t,𝐢t\)\\mathbf\{o\}\_\{t\}=\(\\mathbf\{s\}\_\{t\},\\mathbf\{i\}\_\{t\}\)to produce high\-performance but potentially unsafe actions𝐚θ\\mathbf\{a\}^\{\\theta\}\. The two action outputs are fused through a weighted summation, where the weights are determined by a learnable exponential activation function conditioned on the risk value\.AutoSafefuses a standard DRL policyπθ\\pi\_\{\\theta\}with a safe model\-based policyπsafe\\pi\_\{\\mathrm\{safe\}\}through a weighted summation of their action outputs:𝐚~=\(1−λ​\(𝐬\)\)⋅𝐚θ​\(𝐨\)\+λ​\(𝐬\)⋅𝐚safe​\(𝐬\)\\tilde\{\\mathbf\{a\}\}=\(1\-\\lambda\(\\mathbf\{s\}\)\)\\cdot\\mathbf\{a\}^\{\\theta\}\(\\mathbf\{o\}\)\+\\lambda\(\\mathbf\{s\}\)\\cdot\\mathbf\{a\}^\{\\mathrm\{safe\}\}\(\\mathbf\{s\}\)\. Here, we use a more general expression for the input of DRL, because DRL can work on a broader range of information, for instance, images and augmented state information that includes past states\. The specific observation designs can be found in the implementation details of the applications\. The DRL policyπθ\\pi\_\{\\theta\}takes the full observation𝐨t=\(𝐬t,𝐢t\)\\mathbf\{o\}\_\{t\}=\(\\mathbf\{s\}\_\{t\},\\mathbf\{i\}\_\{t\}\), which includes the state vector𝐬t\\mathbf\{s\}\_\{t\}and additional information𝐢t\\mathbf\{i\}\_\{t\}\(e\.g\., images\), and generates an action𝐚θ∼πθ\(⋅∣𝐨t\)\.\\mathbf\{a\}\_\{\\theta\}\\sim\\pi\_\{\\theta\}\(\\cdot\\mid\\mathbf\{o\}\_\{t\}\)\.The safe policyπsafe\\pi\_\{\\mathrm\{safe\}\}only observes the state vector and outputs a safe action𝐚safe=𝐅𝐬\.\\mathbf\{a\}\_\{\\mathrm\{safe\}\}=\\mathbf\{Fs\}\.

The mixing coefficientλ​\(𝐬\)∈\[0,1\]\\lambda\(\\mathbf\{s\}\)\\in\[0,1\]is determined by an exponential\-like activation function that takes as input a non\-negative safety marginΔt∈ℝ≥0\\Delta\_\{t\}\\in\\mathbb\{R\}\_\{\\geq 0\}, computed by a non\-learnable Safety monitor function, together with a learnable sharpness parameterpp\. When the agent operates well inside the safety envelope,λ​\(𝐬\)\\lambda\(\\mathbf\{s\}\)is small and the action is dominated by𝐚θ\\mathbf\{a\}\_\{\\theta\}\. As the riskzzincreases near the boundary of the safety set𝒮c\\mathcal\{S\}\_\{c\},λ​\(𝐬\)\\lambda\(\\mathbf\{s\}\)grows, shifting the action towards𝐚safe\\mathbf\{a\}\_\{\\mathrm\{safe\}\}\. At the boundary, the agent relies solely on the safe policy to ensure constraint satisfaction\. The sharpness parameterppcontrols the sharpness of this transition\. An overview of the entire architecture is shown in Fig\.[8](https://arxiv.org/html/2606.31320#A3.F8)\.

The proposedAutoSafeneural policy additionally has a temperature prediction head compared to the standard actor network in SAC\. This prediction head shares the same backbone as the action prediction head and is activated using thetanhactivation function\. We then apply an affine transformation to map the output range\(−1,1\)\(\-1,1\)into\(pmin,pmax\)\(p\_\{\\text\{min\}\},p\_\{\\text\{max\}\}\)\. We setpmin=1\.0p\_\{\\text\{min\}\}=1\.0andTmax=25\.0T\_\{\\text\{max\}\}=25\.0for all studied case studies\. The selection of the range can be flexible\. We setpmax=25\.0p\_\{\\text\{max\}\}=25\.0to ensure the DRL agent has enough freedom to explore the state using its own action within the safety envelope\. The value ofλ\\lambdais capped to11for the case of exceeding the safety envelope caused by abrupt environmental uncertainties such as disturbance or noise\.

#### C\.1\.3Simplex

The implementation of the Simplex architecture follows the standard setting as in\(Sha,[2001](https://arxiv.org/html/2606.31320#bib.bib107); Phanet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib112)\)\. The system’s safety is backed up by a safe policy\. The switching between the safe and learning\-based policy is determined by a state\-dependent function, detailed as follows:

𝐚~t=\{𝐚tsafe=πsafe​\(𝐬t\),if​1−𝐬t⊤​𝐏𝐬≤Δmin,i\.e,𝐬∉Ω𝐚θt∼πθ\(⋅∣𝐨t\),otherwise\.\\tilde\{\\mathbf\{a\}\}\_\{t\}=\\begin\{cases\}\\mathbf\{a\}^\{\\text\{safe\}\}\_\{t\}=\\pi^\{\\text\{safe\}\}\(\\mathbf\{s\}\_\{t\}\),&\\text\{if\}~1\-\\mathbf\{s\}\_\{t\}^\{\\top\}\\mathbf\{P\}\\mathbf\{s\}\\leq\\Delta\_\{\\mathrm\{min\}\},i\.e,~\\mathbf\{s\}\\notin\\Omega\\\\ \{\\mathbf\{a\}^\{\\theta\}\}\_\{t\}\\sim\\pi^\{\\theta\}\(\\cdot\\mid\\mathbf\{o\}\_\{t\}\),&\\text\{otherwise\}\.\\\\ \\end\{cases\}The matrix𝐏\\mathbf\{P\}is solved from the LMIs problem in[B](https://arxiv.org/html/2606.31320#A2)and identical to the𝐏\\mathbf\{P\}inAutoSafe\. The safe policyπs​a​f​e\\pi\_\{safe\}generates the safe action as𝐚s​a​f​e=𝐅𝐬\\mathbf\{a\}\_\{safe\}=\\mathbf\{Fs\}\.

#### C\.1\.4CBF

In this experiment, we implement a Control\-Barrier Function based approach for the safety back up mechanism\. Our implementation of CBF follows the setup introduced in\(Ameset al\.,[2019](https://arxiv.org/html/2606.31320#bib.bib8)\)\. In particular, given an action from a DRL policy𝐚θ\\mathbf\{a\}\_\{\\theta\}, a CBF\-based safety filter computes a safe action𝐚s​a​f​e\\mathbf\{a\}\_\{safe\}by solving the following quadratic program:

𝐚s​a​f​e\\displaystyle\\mathbf\{a\}\_\{safe\}=arg⁡min𝐚⁡12​‖𝐚−𝐚θ‖2\\displaystyle=\\arg\\min\_\{\\mathbf\{a\}\}\\frac\{1\}\{2\}\\big\\\|\\mathbf\{a\}\-\\mathbf\{a\}\_\{\\theta\}\\big\\\|^\{2\}\(60\)s\.t\.Lf​h​\(𝐬\)\+Lg​h​\(𝐬\)​𝐚≥−α​\(h​\(𝐬\)\)\\displaystyle L\_\{f\}h\(\\mathbf\{s\}\)\+L\_\{g\}h\(\\mathbf\{s\}\)\\mathbf\{a\}\\geq\-\\alpha\(h\(\\mathbf\{s\}\)\)\(61\)𝐀a​𝐚≤𝐛a,\\displaystyle\\mathbf\{A\}\_\{a\}\\mathbf\{a\}\\leq\\mathbf\{b\}\_\{a\},\(62\)whereα​\(h​\(𝐬\)\)=k⋅h​\(𝐬\)\\alpha\(h\(\\mathbf\{s\}\)\)=k\\cdot h\(\\mathbf\{s\}\)andh​\(𝐬\)h\(\\mathbf\{s\}\)is defined as:

h​\(𝐬\)=𝐛s−𝐀s​𝐬h​\(𝐬\)∈ℝns\.\\displaystyle h\(\\mathbf\{s\}\)=\\mathbf\{b\}\_\{s\}\-\\mathbf\{A\}\_\{s\}\\mathbf\{s\}\\quad h\(\\mathbf\{s\}\)\\in\\mathbb\{R\}^\{n\_\{s\}\}\.\(63\)Here,h​\(𝐬\)h\(\\mathbf\{s\}\)is defined in matrix form, yielding a vector\-valued barrier function\. In practice, each component of this vector is enforced as an individual CBF constraint\. The\(𝐀s,𝐛s\)\(\\mathbf\{A\}\_\{s\},\\mathbf\{b\}\_\{s\}\)and\(𝐀a,𝐛a\)\(\\mathbf\{A\}\_\{a\},\\mathbf\{b\}\_\{a\}\)correspond to the state constraints and action constraints, respectively, as introduced in the background section\. For all experiments, we setk=0\.5k=0\.5\.

The results are shown in[Figure˜10](https://arxiv.org/html/2606.31320#A4.F10)\. We observed that the constructed CBF baseline reduces safety violations, but its performance remains below that of the Simplex approach\. In practice, we found that solving the CBF QP online is sensitive to parameter choices and can frequently become infeasible, especially near the safety boundary\. Additional parameter tuning and system\-specific tuning may improve its performance, but this process is nontrivial and beyond the scope of this work\.

### C\.2Applications Details

#### C\.2\.1Cartpole

Task Definition:The goal of this task is to balance the pole at the intended target positionx^\\hat\{x\}by driving the cart using force input\. In this task, the observation of the agent is defined asot=\{xt,x˙t,sin⁡\(θ\),cos⁡\(θ\),θ˙\}o\_\{t\}=\\\{x\_\{t\},\\dot\{x\}\_\{t\},\\sin\(\\theta\),\\cos\(\\theta\),\\dot\{\\theta\}\\\}\. Our method and the other methods that use a model\-based safe prior require the tracking errorete\_\{t\}as the additional input for the safe policy to generate a safe action\.ete\_\{t\}is defined as the difference between the current state𝐬t\\mathbf\{s\}\_\{t\}and the control equilibrium𝐬∗\\mathbf\{s\}^\{\*\}of the model\-based design,et=\{xt−x∗,x˙−x˙∗,θt−θ∗,θ˙t−θ˙∗\}e\_\{t\}=\\\{x\_\{t\}\-x^\{\*\},\\dot\{x\}\-\\dot\{x\}^\{\*\},\\theta\_\{t\}\-\\theta^\{\*\},\\dot\{\\theta\}\_\{t\}\-\\dot\{\\theta\}^\{\*\}\\\}\. The equilibrium is set as𝐬∗=\{0,0,0,0\}\\mathbf\{s\}^\{\*\}=\\\{0,0,0,0\\\}\. The control loop is running at 50Hz\.

We append the tracking errorete\_\{t\}to the observation space of the other model\-free methods to ensure all algorithms receive the same amount of information\. We adopt the reward function proposed in\(Caoet al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib115)\), formulated as

r=e−δ⋅d​\(𝐬t−𝐬^\)−β⋅𝐚t2,r=e^\{\-\\delta\\cdot d\(\\mathbf\{s\}\_\{t\}\-\\hat\{\\mathbf\{s\}\}\)\}\-\\beta\\cdot\\mathbf\{a\}\_\{t\}^\{2\},whereδ=5\\delta=5is a parameter that adjusts the smoothness of the exponential function;d​\(⋅\)d\(\\cdot\)is the Euclidean distance between the current state𝐬t\\mathbf\{s\}\_\{t\}and the control target𝐬^=\{0\.1,0,0,0\}\\mathbf\{\\hat\{s\}\}=\\\{0\.1,0,0,0\\\};β\\betais a parameter to balance the reward and the action penalty\. For more details about the task setting, we refer interested readers to\(Caoet al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib115)\)\.

Safety Constraints:In this task, the safety constraints are

\|xt\|≤xlim\|θt\|≤θlim,\|x\_\{t\}\|\\leq x\_\{\\text\{lim\}\}\\quad\|\\theta\_\{t\}\|\\leq\\theta\_\{\\text\{lim\}\},wherexlim=0\.5​mx\_\{\\text\{lim\}\}=0\.5~\\text\{m\}andθlim=0\.785​rad\\theta\_\{\\text\{lim\}\}=0\.785~\\text\{rad\}\.

Safety DesignThe safety envelope and safe policy are obtained from solving an LMI problem, as discussed in[Appendix˜B](https://arxiv.org/html/2606.31320#A2)\. The system model can be found at\(Florian,[2007](https://arxiv.org/html/2606.31320#bib.bib38)\)\. The linearized model and the code to calculate the matrix𝐏\\mathbf\{P\}and𝐅\\mathbf\{F\}are available in the attached supplementary files\. For more details of solving LMIs for the cartpole task, we refer the interested reader to\([Caoet al\.,](https://arxiv.org/html/2606.31320#bib.bib18); Seto and Sha,[1999](https://arxiv.org/html/2606.31320#bib.bib21)\)\.

#### C\.2\.2Glucose

Task DefinitionThe problem setting and simulation of this application are adopted from\(Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\)\. In the blood glucose regulation task, the goal is to regulate the blood glucose levelGGto minimize the Magni risk\(Foxet al\.,[2020](https://arxiv.org/html/2606.31320#bib.bib131)\)by controlling insulin injectionaIa\_\{I\}\. The dynamics of the glucose control problem are governed by the following ODEs\(Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\),

G˙\\displaystyle\\dot\{G\}=−p1​\(G−Gb\)−G​X\+Dt,\\displaystyle=\-p\_\{1\}\(G\-G\_\{b\}\)\-GX\+D\_\{t\},X˙\\displaystyle\\dot\{X\}=−p2​X\+p3​\(I−Ib\),\\displaystyle=\-p\_\{2\}X\+p\_\{3\}\(I\-I\_\{b\}\),I˙\\displaystyle\\dot\{I\}=−n​\(I−Ib\)\+aI\\displaystyle=\-n\(I\-I\_\{b\}\)\+a\_\{I\}Here,GGrepresents the amount of glucose in the blood, andIIrepresents the amount of insulin in the blood\.XXdescribes the delayed effect of insulin on lowering blood glucose, which is often unobservable\. In this task, the observation is theot=\{Gt,Δ​Gt,t\}o\_\{t\}=\\\{G\_\{t\},\\Delta G\_\{t\},t\\\}, whereΔ​Gt=Gt−Gt−1\\Delta G\_\{t\}=G\_\{t\}\-G\_\{t\-1\}andttis the total time passed after meal ingestion\. The equilibrium of the model\-based design is set to be the normal fasting level of glucose and insulin\.𝐬∗=\{G∗,X∗,I∗\}=\{138,0,7\}\\mathbf\{s\}^\{\*\}=\\\{G^\{\*\},X^\{\*\},I^\{\*\}\\\}=\\\{138,0,7\\\}\. The reward function is defined as

r=\{−\(3\.35506×\(\(ln0\.8353\)−3\.7932\)2if​10≤G≤1000,−1​e​3otherwise\.r=\\begin\{cases\}\-\(3\.35506\\times\(\(\\text\{ln\}^\{0\.8353\}\)\-3\.7932\)^\{2\}&\\text\{if \}10\\leq G\\leq 1000,\\\\ \-1e3&\\text\{otherwise \}\.\\end\{cases\}
Safety ConstraintsWe follow the safety constraints introduced in\(Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\)as:

Gmin≤Gt≤Gmax,G\_\{\\text\{min\}\}\\leq G\_\{t\}\\leq G\_\{\\text\{max\}\},whereGmin=10G\_\{\\text\{min\}\}=10andGmax=1000G\_\{\\text\{max\}\}=1000\.

Moded\-based DesignThe safety envelope and safe policy are obtained from solving an LMI problem, as discussed in[Appendix˜B](https://arxiv.org/html/2606.31320#A2)\. The system model can be found at\(Tianet al\.,[2024](https://arxiv.org/html/2606.31320#bib.bib125)\)\. The linearized model and the code to calculate the matrix𝐏\\mathbf\{P\}and𝐊\\mathbf\{K\}are available in the attached supplementary files\.

#### C\.2\.33D Quadrotor Goal Reaching

![Refer to caption](https://arxiv.org/html/2606.31320v1/images/quadrotor.png)\(a\)Setup of quadrotor goal reaching\. The red sphere represents the target zone\.
![Refer to caption](https://arxiv.org/html/2606.31320v1/images/quadruped.png)\(b\)Setup of quadruped navigation\. The target pick\-up zone is indicated as a blue spot, and the delivery zone is indicated as a yellow spot\. The terrain is randomized with uneven height\.

Figure 9:Experimental setup illustration of quadrotor and quadruped robotsTask DefinitionThe goal of this task is to control a quadrotor to reach a target goal position\{x^,y^,z^\}\\\{\\hat\{x\},\\hat\{y\},\\hat\{z\}\\\}by controlling the thrust input on each propeller\. The observation for the agent iso=\{x,y,z,θx,θy,θz,x˙,y˙,z˙,θx˙,θy˙,θz˙\}o=\\\{x,y,z,\\theta\_\{x\},\\theta\_\{y\},\\theta\_\{z\},\\dot\{x\},\\dot\{y\},\\dot\{z\},\\dot\{\\theta\_\{x\}\},\\dot\{\\theta\_\{y\}\},\\dot\{\\theta\_\{z\}\}\\\}\. The action space consists of a four\-dimensional trust input, denoted asa=\{u1,u2,u3,u4\}a=\\\{u\_\{1\},u\_\{2\},u\_\{3\},u\_\{4\}\\\}\. The equilibrium of the model\-based design iss∗=\{0,0,0,0,0,0,0,0,0,0,0,0\}s^\{\*\}=\\\{0,0,0,0,0,0,0,0,0,0,0,0\\\}\. The reward function is adopted from\(Yuanet al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib78)\)as:

r=e−α⋅\(‖x−x^‖2\+‖y−y^‖2\+‖z−z^‖2\)−β⋅‖a‖2,r=e^\{\-\\alpha\\cdot\(\\\|x\-\\hat\{x\}\\\|^\{2\}\+\\\|y\-\\hat\{y\}\\\|^\{2\}\+\\\|z\-\\hat\{z\}\\\|^\{2\}\)\-\\beta\\cdot\\\|a\\\|^\{2\}\},whereα=1\.0\\alpha=1\.0andβ=1​e−4\\beta=1e\-4are the weights to balance the distance\-related reward and action penalty\. In our case study, we set the initial position of the quadrotor as𝐬0x​y​z=\{1\.5,1\.5,1\.5\}\\mathbf\{s\}\_\{0\_\{xyz\}\}=\\\{1\.5,1\.5,1\.5\\\}and the target position of the quadrotor as𝐬^x​y​z=\{2\.5,2\.5,2\.5\}\\hat\{\\mathbf\{s\}\}\_\{xyz\}=\\\{2\.5,2\.5,2\.5\\\}\. The control loop is running at 50Hz\.

Safety ConstraintsThe safety constraint is defined as

xmin<xt<xmax,ymin<yt<ymax,zmin<zt<zmax\\displaystyle x\_\{\\text\{min\}\}<x\_\{t\}<x\_\{\\text\{max\}\},\\quad y\_\{\\text\{min\}\}<y\_\{t\}<y\_\{\\text\{max\}\},\\quad z\_\{\\text\{min\}\}<z\_\{t\}<z\_\{\\text\{max\}\}\(64\)wherexmin=−5\.0​m,ymin=−5\.0​m,zmin=0\.0​m,x\_\{\\text\{min\}\}=\-5\.0~\\text\{m\},y\_\{\\text\{min\}\}=\-5\.0~\\text\{m\},z\_\{\\text\{min\}\}=0\.0~\\text\{m\},andxmax=ymax=zmax=5\.0​mx\_\{\\text\{max\}\}=y\_\{\\text\{max\}\}=z\_\{\\text\{max\}\}=5\.0~\\text\{m\}, representing the allowable moving area in thex−y−zx\-y\-zspace\.

Safety DesignThe safety envelope and safe policy are obtained from solving an LMI problem, as discussed in[Appendix˜B](https://arxiv.org/html/2606.31320#A2)\. The system model can be found at\(Yuanet al\.,[2022](https://arxiv.org/html/2606.31320#bib.bib78)\)\. The linearized model and the code to calculate the matrix𝐏\\mathbf\{P\}and𝐅\\mathbf\{F\}are available in the attached supplementary files\.

#### C\.2\.4Quadruped Navigation on Uneven Terrain

Task DefinitionIn this task, we aim to train a safe RL policy to enable the quadruped robot to walk through the uneven terrain to finish a virtual package delivery problem\. The robot needs to first go to a package pick\-up zone \(A\) and then navigate a package drop zone \(B\)\. We assume a virtual package is automatically attached to the robot when the robot reaches zone A and detached when it reaches zone B\. The terrain is created unevenly by randomly placing blocks on the floor, with the maximum height of the unevenness to be 4 centimeters\. The control loop is running at 200Hz\.

The observation of the agent includes the pose of the robot in the world coordinates and the relative distance toward goal A for picking up, and the relative distance toward goal B for dropping off\. We additionally add the task phase ID,ii\(0 or 1\), for the picking up and delivery\. Overall,oois defined aso=\{x,y,z,θx,θy,θz,x˙,y˙,z˙,θx˙,θy˙,θz˙,xr​e​l,yr​e​l,zr​e​l,io=\\\{x,y,z,\\theta\_\{x\},\\theta\_\{y\},\\theta\_\{z\},\\dot\{x\},\\dot\{y\},\\dot\{z\},\\dot\{\\theta\_\{x\}\},\\dot\{\\theta\_\{y\}\},\\dot\{\\theta\_\{z\}\},x\_\{rel\},y\_\{rel\},z\_\{rel\},i\. The action space is defined as the desired acceleration asa=\{x¨,y¨,z¨,θx¨,θy¨,θ¨z\}a=\\\{\\ddot\{x\},\\ddot\{y\},\\ddot\{z\},\\ddot\{\\theta\_\{x\}\},\\ddot\{\\theta\_\{y\}\},\\ddot\{\\theta\}\_\{z\}\\\}\. The acceleration input is then mapped to the low\-level joint angles using a Model Predictive Controller \(MPC\)\(Yanget al\.,[2022b](https://arxiv.org/html/2606.31320#bib.bib132)\)\. The equilibrium point for the model\-based design is defined ass∗=\{0,0,0\.24,0,0,0,0\.26,0,0,0,0,0,0\}s^\{\*\}=\\\{0,0,0\.24,0,0,0,0\.26,0,0,0,0,0,0\\\}, meaning maintaining the heightz=0\.24z=0\.24and forward velocityx˙=0\.26​m/s\\dot\{x\}=0\.26~\\text\{m/s\}\. The reward is designed piece\-wise to incentivise the robot to move to A and B to finish the whole task, as

r​\(𝐬\)=wd​rd​\(𝐬,g\)\+wh​rh​\(𝐬,g\)\+wz​rz​\(𝐬\)\+R1​𝕀pickup\+R2​𝕀delivery\.r\(\\mathbf\{s\}\)=w\_\{d\}\\,r\_\{d\}\(\\mathbf\{s\},g\)\+w\_\{h\}\\,r\_\{h\}\(\\mathbf\{s\},g\)\+w\_\{z\}\\,r\_\{z\}\(\\mathbf\{s\}\)\+R\_\{1\}\\,\\mathbb\{I\}\_\{\\text\{pickup\}\}\+R\_\{2\}\\,\\mathbb\{I\}\_\{\\text\{delivery\}\}\.Each term of the reward is defined as:

positional reward:rd​\(𝐬,g\)=−‖𝐬x​y​z−𝐬^x​y​z‖,\\displaystyle\\text\{positional reward\}:r\_\{d\}\(\\mathbf\{s\},g\)=\-\\\|\\mathbf\{s\}\_\{xyz\}\-\\hat\{\\mathbf\{s\}\}\_\{xyz\}\\\|,directional reward:rh​\(𝐬,g\)=−\|θ−θg\|,\\displaystyle\\text\{directional reward\}:r\_\{h\}\(\\mathbf\{s\},g\)=\-\|\\theta\-\\theta\_\{g\}\|,height regulation reward:rz​\(𝐬\)=−\|z−zref\|,\\displaystyle\\text\{height regulation reward\}:r\_\{z\}\(\\mathbf\{s\}\)=\-\|z\-z\_\{\\text\{ref\}\}\|,𝕀pickup=\{1,if reaches the pickup zone \(A\)0,otherwise\\displaystyle\\mathbb\{I\}\_\{\\text\{pickup\}\}=\\begin\{cases\}1,&\\text\{if reaches the pickup zone \(A\)\}\\\\ 0,&\\text\{otherwise\}\\end\{cases\}𝕀delivery=\{1,if reaches the delivery zone \(B\)0,otherwise\\displaystyle\\mathbb\{I\}\_\{\\text\{delivery\}\}=\\begin\{cases\}1,&\\text\{if reaches the delivery zone \(B\)\}\\\\ 0,&\\text\{otherwise\}\\end\{cases\}
where the coefficient of each term is detailed as

wd=1\.0,wh=0\.25,wz=0\.25R1=2\.5,R2=50\.\\displaystyle w\_\{d\}=1\.0,\\quad w\_\{h\}=0\.25,\\quad w\_\{z\}=0\.25\\quad R\_\{1\}=2\.5,\\quad R\_\{2\}=50\.
Safety ConstraintsThe safety constraint in this task is mainly considered as the robot not falling and jumping too high while moving forward, as:

zmin≤zt≤zmaxz\_\{\\text\{min\}\}\\leq z\_\{t\}\\leq z\_\{\\text\{max\}\}wherezmin=0\.16​mz\_\{\\text\{min\}\}=0\.16~\\text\{m\}andzmax=0\.8​mz\_\{\\text\{max\}\}=0\.8~\\text\{m\}\.

Safety designThe safety envelope and safe policy are obtained from solving an LMI problem, as discussed in[Appendix˜B](https://arxiv.org/html/2606.31320#A2)\. The linearized model and the code to calculate the matrix𝐏\\mathbf\{P\}and𝐅\\mathbf\{F\}are available in the attached supplementary files\.

## Appendix DAdditional Experimental Results

In this section, we summarize the additional experimental results to provide more insights\.

### D\.1Experimental results for all methods on all considered experiments\.

We summarize the training progress and the accumulated safety violations in Fig[10](https://arxiv.org/html/2606.31320#A4.F10)\. TheΔmin=0\\Delta\_\{\\mathrm\{min\}\}=0is used for Simplex and AutoSafe as the default value\. The detailed ablation study onΔmin\\Delta\_\{\\mathrm\{min\}\}is shown in Table[4](https://arxiv.org/html/2606.31320#A4.T4)\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x13.png)Figure 10:Performance return curve and the accumulated safety violation for all algorithms in the considered case studies
### D\.2Divergence of the hard\-safety intervention

We visualize the training loss curves forAutoSafeandSimplex\. We observe that training of theSimplexis gradually diverging with a large critic loss, as shown in Fig\.[11](https://arxiv.org/html/2606.31320#A4.F11)\. We attribute this to the frequent interventions by the safe policy, which creates discontinuities in the data distribution, meaning the state–action transitions suddenly shift from those produced by the learning policy to those imposed by the safe controller\. This mismatch makes the data less smooth and harder for the neural network to approximate, causing unstable or diverging learning behavior\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x14.png)Figure 11:Critic Loss and Actor Loss of AutoSafe and Simplex\-Based Method
### D\.3Ablation on the setting of Temperaturepp

In this case study, we compare our learnable temperature setting against two heuristics, including linear and exponential increasing\. The result is shown in Fig\.[13](https://arxiv.org/html/2606.31320#A4.F13)\. We found that all methods work similarly regarding performance, except for the quadrotor case\. However, designing an effective scheduling scheme is non\-trivial, which requires repetitive manual tuning\. Moreover, the manual setting might introduce bias\. We visualize the evolution of the learnable sharpness parameterTTduring the policy learning in Fig\.[12](https://arxiv.org/html/2606.31320#A4.F12)\. It can be seen that the learned sharpness converges to different values across tasks, suggesting that it may not be trivial to manually set the "right" parameter using heuristics\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x15.png)Figure 12:The evolution of sharpness parameterppduring policy learning for studied applications![Refer to caption](https://arxiv.org/html/2606.31320v1/x16.png)Figure 13:Ablation Study of sharpness parameterpp: Learning\-based vs\. Schedule\-based\.
### D\.4Ablation of AdaLam

In this study, we investigate several ways of setting the weights between safe action and learning\-based action, as shown in Fig\.[14](https://arxiv.org/html/2606.31320#A4.F14)\. We found out that initializing theλ\\lambdato close to11at the beginning of the training enables safe interactions\. For simple tasks, such as cartpole and glucose, the agent could learn using the data generated by the safe policy\. However, we found that it is not effective in high\-dimensional cases\. For a learning\-based setting, the exploration is not effective; therefore, the performance of the policy is barely improved\. For the scheduled setting, we found that frequent safety violations occur whenλ\\lambdadecreases\. The frequent safety violations generate a lot of uninformative data, where the agent cannot learn to converge\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x17.png)Figure 14:Ablation of the adaptation forλ\\lambdain AdaLam
### D\.5Under varying constraints

In this section, we demonstrate that the current safety design can be readily extended to handle online constraints with slight modification\. Specifically, the offline recovery region can be reinterpreted and transformed into a closed\-form expression to accommodate new constraints while preserving local invariance guarantees\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x18.png)

![Refer to caption](https://arxiv.org/html/2606.31320v1/images/cartpole_vary.png)

Figure 15:State trajectories over an evaluation episode\. The dashed lines indicate the time\-varying constraints on the position imposed by each intruder\.##### CartPole

The DRL agent learns to drive the cart\-pole system toward a time\-varying target positionxt∗x\_\{t\}^\{\*\}while two moving obstacle constraints dynamically shrink the admissible state space\. The state space is𝒮⊂ℝ4\\mathcal\{S\}\\subset\\mathbb\{R\}^\{4\}, and the action space is𝒜⊂ℝ1\\mathcal\{A\}\\subset\\mathbb\{R\}^\{1\}\.

The system is subject to the following fixed safety constraints:

\|θt\|≤θmax,\|xt\|≤xmax\.\\displaystyle\|\\theta\_\{t\}\|\\leq\\theta\_\{\\max\},\\qquad\|x\_\{t\}\|\\leq x\_\{\\max\}\.\(65\)
In addition, two intruder\-induced time\-varying constraints impose dynamic bounds on the cart position:

xt≥ℓt,xt≤rt,\\displaystyle x\_\{t\}\\geq\\ell\_\{t\},\\qquad x\_\{t\}\\leq r\_\{t\},\(66\)whereℓt\\ell\_\{t\}andrtr\_\{t\}are updated online according to the motion of the intruders\.

As shown in Fig\.[15](https://arxiv.org/html/2606.31320#A4.F15),AutoSafelearns to safely drive the cartpole to reach the target goal under two moving objects constraints without safety violations\.

![Refer to caption](https://arxiv.org/html/2606.31320v1/x19.png)

![Refer to caption](https://arxiv.org/html/2606.31320v1/images/quadrotor_obstacle.png)

Figure 16:example evaluation trajectories with randomized obstacle positions\. The adapted safe recoverable region enables safe clearance for goal reaching\.
##### Quadrotor

A quadrotor must reach a goal position in 3D space while avoiding a static obstacle\. The state space is𝒮⊂ℝ12\\mathcal\{S\}\\subset\\mathbb\{R\}^\{12\}, and the action space is𝒜⊂ℝ4\\mathcal\{A\}\\subset\\mathbb\{R\}^\{4\}\.

The system is subject to the following fixed safety constraints:

\|ϕt\|≤ϕmax,\|ψt\|≤ψmax,\|ϑt\|≤ϑmax,\\displaystyle\|\\phi\_\{t\}\|\\leq\\phi\_\{\\max\},\\qquad\|\\psi\_\{t\}\|\\leq\\psi\_\{\\max\},\\qquad\|\\vartheta\_\{t\}\|\\leq\\vartheta\_\{\\max\},\(67\)and an altitude constraint

zt≥zmin\.\\displaystyle z\_\{t\}\\geq z\_\{\\min\}\.\(68\)
In addition, the obstacle is represented by a collection of time\-varying supporting hyperplane constraints:

ai,t⊤​pt≥bi,t,i=1,…,F,\\displaystyle a\_\{i,t\}^\{\\top\}p\_\{t\}\\geq b\_\{i,t\},\\qquad i=1,\\dots,F,\(69\)where each obstacle faceiidefines a supporting half\-space that is updated online according to the relative motion of the quadrotor\.

As shown in Fig\.[16](https://arxiv.org/html/2606.31320#A4.F16),AutoSafelearns to safely drive the quadrotor to reach the target goal while avoiding the obstacle\.

### D\.6Sensitivity Analysis onΔmin\\Delta\_\{\\mathrm\{min\}\}

In[Table˜4](https://arxiv.org/html/2606.31320#A4.T4),Δmin=0\\Delta\_\{\\mathrm\{min\}\}=0corresponds to the most aggressive setting, where the safety switch happens when the system exits the maximum safety recoverable region\.Δmin=1\.0\\Delta\_\{\\mathrm\{min\}\}=1\.0corresponds to the most conservative setting, whereλ=1\\lambda=1, meaning the system is fully controlled by theπsafe\\pi^\{\\text\{safe\}\}\. As expected, largerΔmin\\Delta\_\{\\min\}leads to fewer safety violations due to tighter safety enforcement, but may also reduce task performance by constraining exploration\.AutoSafeoutperformsSimplexin most tasks, while showing comparable performance on the Glucose Regulation benchmark\. AsΔmin→1\\Delta\_\{\\min\}\\rightarrow 1, both methods converge toward the same behavior\.

Table 4:Effect ofΔmin\\Delta\_\{\\mathrm\{min\}\}on performance and safety across four continuous control tasks\. Mean±\\pmstd over five random seeds\.
### D\.7Walltime comparison

Table 5:Computation Time Comparison\.Average wall\-clock time \(in milliseconds\) per step for inference and optimization steps\. Mean±\\pmstd computed over all logged episodes \(one seed\)\. Lower is better \(↓\\downarrow\)\. Most time\-consuming method per column isunderlined\.In this experiment, we observe that CBF \(optimization\-based\) incurs the highest inference cost due to the per\-step online optimization loop, and shows higher variance in action generation\. AdaLam requires higher optimization time, as the mixing parameter is trained separately from the Q\-network\. We see that SimplexRL achieves a relatively faster inference time because when the safe policy is activated, the inference switches from network forward pass to cheap matrix multiplication\. AutoSafe maintains low computational overhead for action inference and policy optimization comparable to the other baselines, while achieving strong performance in terms of return and safety violations\.