From Cumulative Constraints to Adaptive Runtime Safety Control for Nonstationary Reinforcement Learning

arXiv cs.LG Papers
safety reinforcement-learning runtime-monitoring adaptive-control nonstationary continual-learning

Summary

Proposes CPSS, a runtime safety mechanism that converts cumulative cost constraints into adaptive state-level thresholds for safe reinforcement learning in nonstationary environments, demonstrating reduced violations in highway merging scenarios.

arXiv:2605.18841v1 Announce Type: new Abstract: Safety in reinforcement learning is often specified through cumulative cost constraints, but these trajectory-level guarantees do not directly prevent unsafe individual decisions, especially under nonstationarity. In continual and nonstationary settings, the difficulty is amplified because the risk associated with the same action can vary across contexts, while a fixed state-level threshold may be either too conservative or too weak. We propose Constraint Projection Safety Shield (CPSS), a runtime mechanism that converts a cumulative safety budget into adaptive state-level control constraints during execution. CPSS tracks the remaining safety budget, projects it into a time-varying admissible risk threshold, and filters policy actions whose predicted safety cost exceeds the active threshold. The threshold is adjusted online using contextual signals so that enforcement becomes stricter in more demanding or rapidly changing regimes and less restrictive when the available safety budget is sufficient. We analyze the resulting shielded policy and show that the mechanism guarantees per-state threshold satisfaction for executed actions, induces finite-horizon cumulative cost bounds, and yields a performance degradation bound in terms of intervention frequency and per-step reward distortion. We evaluate CPSS in nonstationary highway merging scenarios using highway-env. Across multiple seeds, CPSS substantially reduces proximity-based safety violations and increases separation margins while intervening selectively rather than dominating the learned policy. These results support adaptive budget-to-threshold projection as a practical way to transform cumulative safety specifications into effective local safety control for continual reinforcement learning systems.
Original Article

Similar Articles

Safe Continual Reinforcement Learning under Nonstationarity via Adaptive Safety Constraints

arXiv cs.LG

Proposes LILAC+, a framework for safe continual reinforcement learning under nonstationarity that uses three adaptive safety mechanisms: context-based safety constraints, adaptation-speed constraints, and budget-to-state safety enforcement. Evaluations in simulated driving environments show reduced safety violations under distribution shift while maintaining competitive performance.

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

arXiv cs.AI

This paper proposes Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method for safe reinforcement learning that incorporates local constraint sensitivity to improve safety recovery and reduce oscillations near safety boundaries, achieving higher constrained returns on navigation and locomotion benchmarks.

Robust Peak-cost Constrained Reinforcement Learning

arXiv cs.LG

This paper studies robust peak-cost constrained reinforcement learning, addressing limitations of standard CMDPs by controlling the maximum cost along a trajectory and considering dynamics uncertainty. The authors show zero duality gap may not hold and propose a surrogate optimization framework with robust value estimation.

Configurable Reward Model for Balanced Safety Alignment

arXiv cs.CL

This paper introduces the Configurable Safety Reward Model (CSRM), a reward model that can be configured to accommodate heterogeneous and evolving safety requirements for LLM alignment. CSRM achieves state-of-the-art results on configurable safety benchmarks and improves the helpfulness-safety tradeoff.

Safe and Generalizable Hierarchical Multi-Agent RL via Constraint Manifold Control

arXiv cs.AI

This paper proposes a hierarchical multi-agent reinforcement learning framework that enforces hard safety constraints via a constraint manifold at the low level while enabling effective coordination through high-level policy learning, providing theoretical safety guarantees and achieving near-perfect safety rates with good generalization.