Boundary-Seeking Policy Gradient for Safe Reinforcement Learning

# Boundary-Seeking Policy Gradient for Safe Reinforcement Learning
Source: [https://arxiv.org/html/2608.10204](https://arxiv.org/html/2608.10204)
Chenhua Fan1†, Jiahui Zhu1†, Yuhang Zhang1, and Honghao Wei1∗†contributed equally,∗corresponding author1School of EECS, Washington State University, Pullman, WA 99163, USA\(E\-mails: \{chenhua\.fan,jiahui\.zhu, yuhang\.zhang1, honghao\.wei\}@wsu\.edu\)

###### Abstract

Safe reinforcement learning maximizes reward subject to safety constraints\. For Constrained Markov Decision Processes, the linear\-programming view over occupancy measures implies that whenever the constraint is active at optimality, the optimal policy lies exactly on the constraint boundary, yet standard gradient\-based methods do not exploit this structure and often settle in the feasible interior\. We introduce Boundary\-Seeking Policy Gradient \(BSPG\), a first\-order method whose update combines a tangential component that improves reward while preserving cost to first order with a signed, residual\-driven normal component that regulates the policy toward the active boundary from either side; the combined direction admits an algebraic Lagrangian form with an induced coefficient and no learned dual variable\. Under exact gradients and stated regularity conditions, the constraint residual converges to zero from either side with a finite\-horizonO​\(1/T\)O\(1/\\sqrt\{T\}\)bound, the tangential component is a reward\-ascent direction on the boundary, and any convergent parameter sequence is stationary on the active constraint set, satisfying the KKT conditions when the limit is also a local maximizer over the feasible set\. This complements existing analyses, which certify feasibility but do not characterize the constraint value at convergence\. On a standard Safety\-Gymnasium navigation task, BSPG attains higher reward while tracking the boundary more tightly than the compared baselines\.

## IIntroduction

Reinforcement learning \(RL\) has achieved remarkable success in sequential decision\-making tasks including game playing\[[1](https://arxiv.org/html/2608.10204#bib.bib1)\], autonomous driving\[[2](https://arxiv.org/html/2608.10204#bib.bib2)\], and robotics\[[3](https://arxiv.org/html/2608.10204#bib.bib3)\]\. In safety\-critical applications, however, maximizing reward alone is insufficient: a deployed policy must also satisfy operational constraints, a requirement commonly formalized through Constrained Markov Decision Processes \(CMDPs\)\[[4](https://arxiv.org/html/2608.10204#bib.bib4)\]\.

Existing constrained policy optimization follows several directions\.*Primal–dual methods*\[[5](https://arxiv.org/html/2608.10204#bib.bib5),[6](https://arxiv.org/html/2608.10204#bib.bib6),[7](https://arxiv.org/html/2608.10204#bib.bib7),[8](https://arxiv.org/html/2608.10204#bib.bib8),[9](https://arxiv.org/html/2608.10204#bib.bib9),[10](https://arxiv.org/html/2608.10204#bib.bib10),[11](https://arxiv.org/html/2608.10204#bib.bib11)\]enforce the constraint through a learned multiplier, so the policy’s location relative to an active boundary is controlled only indirectly through the dual dynamics\.*Primal and projection\-based methods*\[[12](https://arxiv.org/html/2608.10204#bib.bib12),[13](https://arxiv.org/html/2608.10204#bib.bib13),[14](https://arxiv.org/html/2608.10204#bib.bib14),[15](https://arxiv.org/html/2608.10204#bib.bib15)\], and switching schemes such as CRPO\[[16](https://arxiv.org/html/2608.10204#bib.bib16)\], restrict or correct updates by feasibility — a one\-sided condition that does not say how much of the remaining cost budget a feasible policy should use\.*Gradient\-manipulation methods*\[[17](https://arxiv.org/html/2608.10204#bib.bib17),[18](https://arxiv.org/html/2608.10204#bib.bib18),[19](https://arxiv.org/html/2608.10204#bib.bib19)\]coordinate the two objectives through gradient geometry, which captures local alignment but not where the policy sits relative to the cost limit\. In short, standard updates do not separate reward improvement along a cost level set from signed regulation across cost levels toward an active boundary\.

This separation matters because of a well\-known structural property: when the CMDP model is known, safe RL admits an exact linear\-programming solution over occupancy measures\[[4](https://arxiv.org/html/2608.10204#bib.bib4)\], and whenever the constraint is active at optimality, the optimal policy lies on the boundary of the feasible set\. A strictly feasible policy therefore retains cost slack that can be locally converted into reward when the reward and cost gradients are positively aligned; Section[IV](https://arxiv.org/html/2608.10204#S4)makes both the global statement and this local trade\-off precise\.

Motivated by this structure, we propose Boundary\-Seeking Policy Gradient \(BSPG\), which makes signed boundary regulation an explicit, separate component of the update\. A tangential term removes from the reward gradient its component along the cost gradient, preserving cost to first order while retaining reward ascent; a normal term proportional to the signed residual decreases cost when the policy is infeasible and increases cost toward the limit when it is strictly feasible\. The ideal update admits an algebraic Lagrangian representation with an induced coefficient and maintains no learned dual variable\. Our contributions are:

1. 1\.We characterize the boundary structure of constrained policy optimization: when no unconstrained reward maximizer is feasible, every constrained occupancy\-measure maximizer uses the full cost budget, this target transfers to any policy class attaining the exact constrained optimum, and we identify when local slack supports first\-order reward improvement\.
2. 2\.We propose BSPG and, for the ideal exact\-gradient update, prove two\-sided residual convergence with a finite\-horizonO​\(1/T\)O\(1/\\sqrt\{T\}\)bound, reward ascent of the tangential component, and a characterization of any convergent run: the limit is stationary on the active set and satisfies the KKT conditions when it is also a local maximizer over the feasible set\.
3. 3\.On a standard Safety\-Gymnasium navigation task, BSPG achieves higher reward with tighter boundary adherence than the compared baselines\.

![Refer to caption](https://arxiv.org/html/2608.10204v1/result/bspg_vs_espo_1x4_circular_fixed.png)Figure 1:Local update geometry of ESPO\[[18](https://arxiv.org/html/2608.10204#bib.bib18)\]andBSPG\. ESPO shapes its direction from the reward–cost gradient relation; BSPG adds a normal component set by the signed constraint residual\. For the ideal update \(δ^=δ\\hat\{\\delta\}=\\delta\), this component points towardℳ\\mathcal\{M\}from both the feasible \(δ^<0\\hat\{\\delta\}<0\) and infeasible \(δ^\>0\\hat\{\\delta\}\>0\) sides\.
## IIRelated Work

Primal–dual methodsenforce safety through a Lagrange multiplier balancing reward against violation\. PPO\- and TRPO\-Lagrangian\[[20](https://arxiv.org/html/2608.10204#bib.bib20)\]update the multiplier from the observed violation, with coupled dynamics that can be sensitive and oscillatory; PID\-Lagrangian\[[5](https://arxiv.org/html/2608.10204#bib.bib5)\]stabilizes the dual update at the cost of extra tuning, and RCPO\[[21](https://arxiv.org/html/2608.10204#bib.bib21)\]and CAL\[[22](https://arxiv.org/html/2608.10204#bib.bib22)\]refine cost estimation and the coupled updates\. In all cases the residual acts on the multiplier, which then reshapes the policy step: boundary regulation is not an explicit component of the update itself\.

Primal methodsrestrict or correct updates by feasibility\. CPO\[[12](https://arxiv.org/html/2608.10204#bib.bib12)\]solves a constrained local subproblem with guarantees but added computational cost\[[20](https://arxiv.org/html/2608.10204#bib.bib20)\]; PCPO\[[13](https://arxiv.org/html/2608.10204#bib.bib13)\], FOCOPS\[[14](https://arxiv.org/html/2608.10204#bib.bib14)\], and CUP\[[15](https://arxiv.org/html/2608.10204#bib.bib15)\]give first\-order and projection\-based variants; CRPO\[[16](https://arxiv.org/html/2608.10204#bib.bib16)\]alternates reward and constraint steps by feasibility status, and CPPO\[[23](https://arxiv.org/html/2608.10204#bib.bib23)\]adds an infeasibility\-recovery mechanism\. Feasibility, however, is one\-sided: a policy with large slack and one at the boundary satisfy the same inequality, and when slack is locally convertible into reward \(Section[IV](https://arxiv.org/html/2608.10204#S4)\), feasibility alone does not determine how much of the budget to use\.

Gradient\-manipulation methodsare closest to our geometric view\. PCRPO\[[17](https://arxiv.org/html/2608.10204#bib.bib17)\]selects or combines directions by the reward–cost gradient angle, GradS\[[19](https://arxiv.org/html/2608.10204#bib.bib19)\]extends this to multiple constraints, and ESPO\[[18](https://arxiv.org/html/2608.10204#bib.bib18)\]uses gradient interaction to improve efficiency\. But gradient geometry is not boundary location: two policies with similar gradient directions can hold very different slack, so geometry alone determines neither the direction nor the magnitude of motion across cost levels\. These methods can reach an active boundary, yet their updates do not use the signed residual as a separate normal correction regulating the policy toward it\.

## IIIPreliminaries

We consider a CMDP defined by the tuple\(𝒮,𝒜,P,r,c,γ,μ0,d\)\(\\mathcal\{S\},\\mathcal\{A\},P,r,c,\\gamma,\\mu\_\{0\},d\), where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,P​\(s′∣s,a\)P\(s^\{\\prime\}\\mid s,a\)is the transition kernel,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}andc:𝒮×𝒜→ℝc:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}are the reward and cost functions,γ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor,μ0\\mu\_\{0\}is the initial\-state distribution, andd∈ℝd\\in\\mathbb\{R\}is the cost threshold\. A policyπ:𝒮→Δ​\(𝒜\)\\pi:\\mathcal\{S\}\\to\\Delta\(\\mathcal\{A\}\)maps each state to a distribution over actions\. For a policyπ𝜽\\pi\_\{\\bm\{\\theta\}\}parameterized by𝜽∈ℝp\\bm\{\\theta\}\\in\\mathbb\{R\}^\{p\}, define the expected discounted reward and cost as

Jr​\(π𝜽\)\\displaystyle J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)=𝔼τ∼π𝜽​\[∑t=0∞γt​r​\(st,at\)\],\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\right\],\(1\)Jc​\(π𝜽\)\\displaystyle J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)=𝔼τ∼π𝜽​\[∑t=0∞γt​c​\(st,at\)\]\.\\displaystyle=\\mathbb\{E\}\_\{\\tau\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}c\(s\_\{t\},a\_\{t\}\)\\right\]\.\(2\)Safe RL aims to solve the following constrained optimization problem:

max𝜽⁡Jr​\(π𝜽\)​s\.t\.​Jc​\(π𝜽\)≤d\.\\max\_\{\\bm\{\\theta\}\}\\;J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)\\;\\;\\textit\{s\.t\.\}\\;\\;J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)\\leq d\.\(3\)To describe the constraint geometry in policy parameter space, define

δ​\(𝜽\)≜Jc​\(π𝜽\)−d,\\displaystyle\\delta\(\\bm\{\\theta\}\)\\triangleq J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)\-d,\(constraint residual\),\\displaystyle\\text\{\(constraint residual\)\},\(4\)ℱ≜\{𝜽∈ℝp:δ​\(𝜽\)≤0\},\\displaystyle\\mathcal\{F\}\\triangleq\\\{\\bm\{\\theta\}\\in\\mathbb\{R\}^\{p\}:\\delta\(\\bm\{\\theta\}\)\\leq 0\\\},\(feasible parameter set\),\\displaystyle\\text\{\(feasible parameter set\)\},\(5\)ℳ≜\{𝜽∈ℝp:δ​\(𝜽\)=0\},\\displaystyle\\mathcal\{M\}\\triangleq\\\{\\bm\{\\theta\}\\in\\mathbb\{R\}^\{p\}:\\delta\(\\bm\{\\theta\}\)=0\\\},\(active constraint set\)\.\\displaystyle\\text\{\(active constraint set\)\}\.\(6\)Since our method is a policy\-gradient algorithm, we next introduce the reward and cost gradients that determine the local update direction\. These quantities also provide the basis for the geometric decomposition developed later\. By the policy gradient theorem\[[24](https://arxiv.org/html/2608.10204#bib.bib24)\], define the normalized discounted state distributiondγπ𝜽​\(s\)=\(1−γ\)​∑t=0∞γt​Prπ𝜽⁡\(st=s\)d\_\{\\gamma\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s\)=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Pr\_\{\\pi\_\{\\bm\{\\theta\}\}\}\(s\_\{t\}=s\)\. Then

𝒈r\\displaystyle\\bm\{g\}\_\{r\}=11−γ​𝔼s∼dγπ𝜽,a∼π𝜽​\[Arπ𝜽​\(s,a\)​∇𝜽log⁡π𝜽​\(a∣s\)\],\\displaystyle=\\frac\{1\}\{1\-\\gamma\}\\mathbb\{E\}\_\{s\\sim d\_\{\\gamma\}^\{\\pi\_\{\\bm\{\\theta\}\}\},\\,a\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\\!\\left\[A\_\{r\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)\\,\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\(a\\mid s\)\\right\],\(7\)𝒈c\\displaystyle\\bm\{g\}\_\{c\}=11−γ​𝔼s∼dγπ𝜽,a∼π𝜽​\[Acπ𝜽​\(s,a\)​∇𝜽log⁡π𝜽​\(a∣s\)\],\\displaystyle=\\frac\{1\}\{1\-\\gamma\}\\mathbb\{E\}\_\{s\\sim d\_\{\\gamma\}^\{\\pi\_\{\\bm\{\\theta\}\}\},\\,a\\sim\\pi\_\{\\bm\{\\theta\}\}\}\\\!\\left\[A\_\{c\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)\\,\\nabla\_\{\\bm\{\\theta\}\}\\log\\pi\_\{\\bm\{\\theta\}\}\(a\\mid s\)\\right\],\(8\)whereArπ𝜽​\(s,a\)=Qrπ𝜽​\(s,a\)−Vrπ𝜽​\(s\)A\_\{r\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)=Q\_\{r\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)\-V\_\{r\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s\)is the reward advantage, andAcπ𝜽​\(s,a\)=Qcπ𝜽​\(s,a\)−Vcπ𝜽​\(s\)A\_\{c\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)=Q\_\{c\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s,a\)\-V\_\{c\}^\{\\pi\_\{\\bm\{\\theta\}\}\}\(s\)is the cost advantage\.

The key observation during policy improvements is that the reward and cost gradients may not be aligned\. For a local update𝜽′=𝜽\+α​𝒗\\bm\{\\theta\}^\{\\prime\}=\\bm\{\\theta\}\+\\alpha\\bm\{v\}, differentiability givesJk​\(𝜽′\)−Jk​\(𝜽\)=α​⟨𝒈k,𝒗⟩\+o​\(α\)J\_\{k\}\(\\bm\{\\theta\}^\{\\prime\}\)\-J\_\{k\}\(\\bm\{\\theta\}\)=\\alpha\\left\\langle\\bm\{g\}\_\{k\},\\,\\bm\{v\}\\right\\rangle\+o\(\\alpha\), so𝒗\\bm\{v\}improves reward to first order if⟨𝒈r,𝒗⟩\>0\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{v\}\\right\\rangle\>0and reduces cost if⟨𝒈c,𝒗⟩<0\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{v\}\\right\\rangle<0\. When⟨𝒈r,𝒈c⟩\>0\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\>0, a naive ascent step along𝒈r\\bm\{g\}\_\{r\}increases both reward and cost to first order, thegradient conflictnear an active constraint that our method is designed to handle\.

## IVBoundary Optimality Principle

We begin with a basic structural observation about the CMDP in \([3](https://arxiv.org/html/2608.10204#S3.E3)\) that motivates our algorithm design\. When the constraint is active at the optimum, the optimal constrained solution should lie on the boundary of the feasible set rather than in its strict interior\. Intuitively, if a feasible policyπ𝜽\\pi\_\{\\bm\{\\theta\}\}satisfies the constraint with strict inequality, i\.e\.,Jc​\(π𝜽\)<dJ\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)<d, then, by continuity ofJcJ\_\{c\}in𝜽\\bm\{\\theta\}, there exists a neighborhood around𝜽\\bm\{\\theta\}in which the constraint remains inactive\. In such a region, moving toward the feasible boundary can potentially improve reward without immediately violating the constraint\. The following theorem formalizes this observation\.

###### Theorem IV\.1\(Boundary Optimality\)

Consider a finite discounted CMDP\([3](https://arxiv.org/html/2608.10204#S3.E3)\) with finite state and action spaces over the classΠ\\Piof all stationary policies\. Suppose the feasible set\{π∈Π:Jc​\(π\)≤d\}\\\{\\pi\\in\\Pi:J\_\{c\}\(\\pi\)\\leq d\\\}is non\-empty and that no unconstrained maximizer ofJrJ\_\{r\}overΠ\\Piis feasible, i\.e\., everyπr⋆∈arg⁡maxπ∈Π⁡Jr​\(π\)\\pi\_\{r\}^\{\\star\}\\in\\arg\\max\_\{\\pi\\in\\Pi\}J\_\{r\}\(\\pi\)satisfiesJc​\(πr⋆\)\>dJ\_\{c\}\(\\pi\_\{r\}^\{\\star\}\)\>d\. Then every constrained\-optimal policyπ⋆∈arg⁡maxπ∈Π:Jc​\(π\)≤d⁡Jr​\(π\)\\pi^\{\\star\}\\in\\arg\\max\_\{\\pi\\in\\Pi:\\,J\_\{c\}\(\\pi\)\\leq d\}J\_\{r\}\(\\pi\)satisfiesJc​\(π⋆\)=dJ\_\{c\}\(\\pi^\{\\star\}\)=d\.

*Proof\.*For a stationary policyπ\\pi, define its \(normalized\) occupancy measureρπ​\(s,a\)=\(1−γ\)​∑t=0∞γt​Prπ,μ0⁡\(st=s,at=a\)\\rho\_\{\\pi\}\(s,a\)=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Pr\_\{\\pi,\\mu\_\{0\}\}\(s\_\{t\}=s,a\_\{t\}=a\)\. The setΛ=\{ρπ:π∈Π\}\\Lambda=\\\{\\rho\_\{\\pi\}:\\pi\\in\\Pi\\\}is a compact convex polytope, andJr,JcJ\_\{r\},J\_\{c\}are linear functionals ofρ\\rho:Jk​\(π\)=11−γ​∑s,aρπ​\(s,a\)​k​\(s,a\)J\_\{k\}\(\\pi\)=\\frac\{1\}\{1\-\\gamma\}\\sum\_\{s,a\}\\rho\_\{\\pi\}\(s,a\)\\,k\(s,a\)fork∈\{r,c\}k\\in\\\{r,c\\\}\[[4](https://arxiv.org/html/2608.10204#bib.bib4)\]; hence the maxima below are attained\.

Suppose, for contradiction, that some constrained\-optimalπ⋆\\pi^\{\\star\}satisfiesJc​\(π⋆\)<dJ\_\{c\}\(\\pi^\{\\star\}\)<d, and writeρ⋆=ρπ⋆\\rho^\{\\star\}=\\rho\_\{\\pi^\{\\star\}\}\. Fix any unconstrained maximizerπr⋆\\pi\_\{r\}^\{\\star\}with occupancy measureρr\\rho\_\{r\}; by hypothesisJc​\(ρr\)\>dJ\_\{c\}\(\\rho\_\{r\}\)\>d\. IfJr​\(ρr\)=Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{r\}\)=J\_\{r\}\(\\rho^\{\\star\}\), thenπ⋆\\pi^\{\\star\}would itself be a feasible unconstrained maximizer, contradicting the hypothesis; henceJr​\(ρr\)\>Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{r\}\)\>J\_\{r\}\(\\rho^\{\\star\}\)\. Forλ∈\(0,1\]\\lambda\\in\(0,1\]letρλ=\(1−λ\)​ρ⋆\+λ​ρr∈Λ\\rho\_\{\\lambda\}=\(1\-\\lambda\)\\rho^\{\\star\}\+\\lambda\\rho\_\{r\}\\in\\Lambda\(convexity\)\. By linearity,Jr​\(ρλ\)=\(1−λ\)​Jr​\(ρ⋆\)\+λ​Jr​\(ρr\)\>Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{\\lambda\}\)=\(1\-\\lambda\)J\_\{r\}\(\\rho^\{\\star\}\)\+\\lambda J\_\{r\}\(\\rho\_\{r\}\)\>J\_\{r\}\(\\rho^\{\\star\}\), andJc​\(ρλ\)=\(1−λ\)​Jc​\(ρ⋆\)\+λ​Jc​\(ρr\)≤dJ\_\{c\}\(\\rho\_\{\\lambda\}\)=\(1\-\\lambda\)J\_\{c\}\(\\rho^\{\\star\}\)\+\\lambda J\_\{c\}\(\\rho\_\{r\}\)\\leq dfor everyλ≤\(d−Jc​\(ρ⋆\)\)/\(Jc​\(ρr\)−Jc​\(ρ⋆\)\)\\lambda\\leq\\bigl\(d\-J\_\{c\}\(\\rho^\{\\star\}\)\\bigr\)/\\bigl\(J\_\{c\}\(\\rho\_\{r\}\)\-J\_\{c\}\(\\rho^\{\\star\}\)\\bigr\), a strictly positive threshold\. Thus, for smallλ\>0\\lambda\>0, the policy induced byρλ\\rho\_\{\\lambda\}is feasible with strictly larger reward thanπ⋆,\\pi^\{\\star\},a contradiction\. HenceJc​\(π⋆\)=dJ\_\{c\}\(\\pi^\{\\star\}\)=d\.■\\blacksquare

The proof, which relies only on the convexity of the occupancy\-measure polytope and the linearity ofJr,JcJ\_\{r\},J\_\{c\}in the occupancy measure\[[4](https://arxiv.org/html/2608.10204#bib.bib4)\]\.

###### Corollary IV\.2\(Boundary optimality under realizability\)

Suppose the assumptions of Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1)hold, the maximum below is attained, and

max𝜽∈ℱ⁡Jr​\(π𝜽\)=maxπ∈Π:Jc​\(π\)≤d⁡Jr​\(π\)\.\\max\_\{\\bm\{\\theta\}\\in\\mathcal\{F\}\}J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)\\;=\\;\\max\_\{\\pi\\in\\Pi:\\,J\_\{c\}\(\\pi\)\\leq d\}J\_\{r\}\(\\pi\)\.\(9\)Then every global maximizer of the parameterized problem \([3](https://arxiv.org/html/2608.10204#S3.E3)\) belongs toℳ\\mathcal\{M\}\.

Indeed, any global maximizer𝜽⋆\\bm\{\\theta\}^\{\\star\}attains the constrained value, so the induced policyπ𝜽⋆\\pi\_\{\\bm\{\\theta\}^\{\\star\}\}is constrained\-optimal inΠ\\Pi, and Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1)givesJc​\(π𝜽⋆\)=dJ\_\{c\}\(\\pi\_\{\\bm\{\\theta\}^\{\\star\}\}\)=d\. Thus, nonconvex parameterization may create additional local stationary points, but it does not change the active cost level of a realizable global optimum\.

Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1)provides the basic theoretical motivation for BSPG: when the constraint is active at the optimum, the constrained optimum of the underlying CMDP uses the full cost budget\. Thus, in safe RL, the objective is not merely to remain feasible, but to approach and optimize along the active constraint boundary, where the optimal constrained solution is attained\.

###### Proposition IV\.5

\(Local reward improvement from unused budget\) SupposeJr,JcJ\_\{r\},J\_\{c\}haveLL\-Lipschitz gradients near𝛉\\bm\{\\theta\}and𝐠c​\(𝛉\)≠𝟎\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}\)\\neq\\bm\{0\}\(cf\. Assumptions[1](https://arxiv.org/html/2608.10204#Thmassumption1)and[2](https://arxiv.org/html/2608.10204#Thmassumption2)\)\. Fixβ∈\(0,1\)\\beta\\in\(0,1\)and consider the fractional normal step

Δβ​\(𝜽\)=−β​δ​\(𝜽\)‖𝒈c​\(𝜽\)‖2​𝒈c​\(𝜽\)\.\\Delta\_\{\\beta\}\(\\bm\{\\theta\}\)\\;=\\;\-\\,\\beta\\,\\frac\{\\delta\(\\bm\{\\theta\}\)\}\{\\left\\\|\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}\)\\right\\\|^\{2\}\}\\,\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}\)\.\(10\)Then

δ​\(𝜽\+Δβ\)\\displaystyle\\delta\(\\bm\{\\theta\}\+\\Delta\_\{\\beta\}\)=\(1−β\)​δ​\(𝜽\)\+O​\(δ​\(𝜽\)2\),\\displaystyle=\(1\-\\beta\)\\,\\delta\(\\bm\{\\theta\}\)\+O\\\!\\bigl\(\\delta\(\\bm\{\\theta\}\)^\{2\}\\bigr\),\(11\)Jr​\(π𝜽\+Δβ\)−Jr​\(π𝜽\)\\displaystyle J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\+\\Delta\_\{\\beta\}\}\)\-J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)=−β​δ​\(𝜽\)​⟨𝒈r,𝒈c⟩‖𝒈c‖2\+O​\(δ​\(𝜽\)2\),\\displaystyle=\-\\,\\beta\\,\\delta\(\\bm\{\\theta\}\)\\,\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\+O\\\!\\bigl\(\\delta\(\\bm\{\\theta\}\)^\{2\}\\bigr\),where both remainders are bounded byL​β22​‖𝐠c‖2​δ​\(𝛉\)2\\frac\{L\\beta^\{2\}\}\{2\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\,\\delta\(\\bm\{\\theta\}\)^\{2\}\. Consequently, ifδ​\(𝛉\)<0\\delta\(\\bm\{\\theta\}\)<0,⟨𝐠r,𝐠c⟩\>0\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\>0, and

\|δ​\(𝜽\)\|<2L​β​min⁡\{\(1−β\)​‖𝒈c‖2β,⟨𝒈r,𝒈c⟩\},\|\\delta\(\\bm\{\\theta\}\)\|\\;<\\;\\frac\{2\}\{L\\beta\}\\,\\min\\\!\\left\\\{\\frac\{\(1\-\\beta\)\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\{\\beta\},\\;\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\\right\\\},\(12\)then𝛉\+Δβ\\bm\{\\theta\}\+\\Delta\_\{\\beta\}is still strictly feasible and attains a strictly larger reward, with first\-order gainβ​\|δ​\(𝛉\)\|​⟨𝐠r,𝐠c⟩/‖𝐠c‖2\\beta\\,\|\\delta\(\\bm\{\\theta\}\)\|\\,\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\.

*Proof\.*ByLL\-smoothness,δ​\(𝜽\+Δβ\)=δ​\(𝜽\)\+⟨𝒈c,Δβ⟩\+Rc\\delta\(\\bm\{\\theta\}\+\\Delta\_\{\\beta\}\)=\\delta\(\\bm\{\\theta\}\)\+\\left\\langle\\bm\{g\}\_\{c\},\\,\\Delta\_\{\\beta\}\\right\\rangle\+R\_\{c\}with\|Rc\|≤L2​‖Δβ‖2\|R\_\{c\}\|\\leq\\tfrac\{L\}\{2\}\\left\\\|\\Delta\_\{\\beta\}\\right\\\|^\{2\}\. Direct computation gives⟨𝒈c,Δβ⟩=−β​δ​\(𝜽\)\\left\\langle\\bm\{g\}\_\{c\},\\,\\Delta\_\{\\beta\}\\right\\rangle=\-\\beta\\,\\delta\(\\bm\{\\theta\}\)and‖Δβ‖=β​\|δ​\(𝜽\)\|/‖𝒈c‖\\left\\\|\\Delta\_\{\\beta\}\\right\\\|=\\beta\|\\delta\(\\bm\{\\theta\}\)\|/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|, hence\|Rc\|≤L​β22​‖𝒈c‖2​δ​\(𝜽\)2\|R\_\{c\}\|\\leq\\frac\{L\\beta^\{2\}\}\{2\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\delta\(\\bm\{\\theta\}\)^\{2\}\. Identically,Jr​\(π𝜽\+Δβ\)−Jr​\(π𝜽\)=⟨𝒈r,Δβ⟩\+Rr=−β​δ​\(𝜽\)​⟨𝒈r,𝒈c⟩/‖𝒈c‖2\+RrJ\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\+\\Delta\_\{\\beta\}\}\)\-J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)=\\left\\langle\\bm\{g\}\_\{r\},\\,\\Delta\_\{\\beta\}\\right\\rangle\+R\_\{r\}=\-\\beta\\,\\delta\(\\bm\{\\theta\}\)\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\+R\_\{r\}with\|Rr\|≤L​β22​‖𝒈c‖2​δ​\(𝜽\)2\|R\_\{r\}\|\\leq\\frac\{L\\beta^\{2\}\}\{2\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\delta\(\\bm\{\\theta\}\)^\{2\}\. Whenδ​\(𝜽\)<0\\delta\(\\bm\{\\theta\}\)<0, we have\(1−β\)​δ​\(𝜽\)<0\(1\-\\beta\)\\delta\(\\bm\{\\theta\}\)<0andβ​\|δ​\(𝜽\)\|​⟨𝒈r,𝒈c⟩/‖𝒈c‖2\>0\\beta\|\\delta\(\\bm\{\\theta\}\)\|\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\>0, and both strictly dominate the remainders exactly when \([12](https://arxiv.org/html/2608.10204#S4.E12)\) holds, yielding strict feasibility and a strict reward increase\.■\\blacksquare

Proposition[IV\.5](https://arxiv.org/html/2608.10204#S4.Thmtheorem5)makes the intended claim precise: not every interior policy incurs a reward loss; rather, under positive gradient alignment, a fractionβ\\betaof the unused cost budget can be converted into a first\-order reward gain ofβ​\|δ\|​⟨𝒈r,𝒈c⟩/‖𝒈c‖2\\beta\|\\delta\|\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}while preserving strict feasibility\. This result motivates the constraint residual as a diagnostic of boundary tracking\. The residual alone is not a universal surrogate for reward, because the local reward change also depends on the alignment and magnitudes of the reward and cost gradients\.

## VBoundary\-Seeking Policy Gradient

Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1)and Corollary[IV\.2](https://arxiv.org/html/2608.10204#S4.Thmtheorem2)place the global constrained optimum on the active constraint setℳ\\mathcal\{M\}under their stated conditions;ℳ\\mathcal\{M\}may also contain non\-optimal points\. A natural algorithmic strategy is therefore to improve reward alongℳ\\mathcal\{M\}while regulating the iterates towardℳ\\mathcal\{M\}\. Motivated by this, we propose Boundary\-Seeking Policy Gradient \(BSPG\) Algorithm \([algorithm1](https://arxiv.org/html/2608.10204#alg1)\) in this section, which implements this strategy through a gradient decomposition that separates the two goals into orthogonal components\.

### V\-AGradient Decomposition

BSPG seeks to improve reward along the constraint manifold while steering the iterates toward it when necessary\. These two roles naturally correspond to the tangential and normal directions ofℳ\\mathcal\{M\}, motivating the following decomposition\. At any𝜽\\bm\{\\theta\}with𝒈c≠𝟎\\bm\{g\}\_\{c\}\\neq\\bm\{0\}, the tangent space of the cost level set\{𝜽′:Jc​\(π𝜽′\)=Jc​\(π𝜽\)\}\\\{\\bm\{\\theta\}^\{\\prime\}:J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}^\{\\prime\}\}\)=J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)\\\}\(which coincides with the tangent space ofℳ\\mathcal\{M\}when𝜽∈ℳ\\bm\{\\theta\}\\in\\mathcal\{M\}\) isT𝜽​ℳ=\{𝒗∈ℝp:⟨𝒗,𝒈c⟩=0\}\.T\_\{\\bm\{\\theta\}\}\\mathcal\{M\}=\\\{\\bm\{v\}\\in\\mathbb\{R\}^\{p\}:\\left\\langle\\bm\{v\},\\,\\bm\{g\}\_\{c\}\\right\\rangle=0\\\}\.The cost gradient𝒈c\\bm\{g\}\_\{c\}is normal to the level set\{Jc=const\}\\\{J\_\{c\}=\\text\{const\}\\\}, which induces a natural splitting of any vector𝒈∈ℝp\\bm\{g\}\\in\\mathbb\{R\}^\{p\}into the tangential part and the normal part𝒈=𝒈⟂\+𝒈∥\\bm\{g\}=\\bm\{g\}^\{\\perp\}\+\\bm\{g\}^\{\\parallel\}where:

𝒈⟂\\displaystyle\\bm\{g\}^\{\\perp\}=𝒈−⟨𝒈,𝒈c⟩‖𝒈c‖2​𝒈c∈T𝜽​ℳ,\\displaystyle=\\bm\{g\}\-\\frac\{\\left\\langle\\bm\{g\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\\;\\in\\;T\_\{\\bm\{\\theta\}\}\\mathcal\{M\},𝒈∥\\displaystyle\\bm\{g\}^\{\\parallel\}=⟨𝒈,𝒈c⟩‖𝒈c‖2​𝒈c∈span​\(𝒈c\)\.\\displaystyle=\\frac\{\\left\\langle\\bm\{g\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\\;\\in\\;\\mathrm\{span\}\(\\bm\{g\}\_\{c\}\)\.By construction, a first\-order update along𝒈⟂\\bm\{g\}^\{\\perp\}leaves the cost objective unchanged, since⟨∇𝜽Jc,𝒈⟂⟩=⟨𝒈c,𝒈⟂⟩=0\\left\\langle\\nabla\_\{\\bm\{\\theta\}\}J\_\{c\},\\,\\bm\{g\}^\{\\perp\}\\right\\rangle=\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{g\}^\{\\perp\}\\right\\rangle=0\. We therefore define the*tangential reward gradient*as

𝒈r⟂=𝒈r−⟨𝒈r,𝒈c⟩‖𝒈c‖2​𝒈c\.\\displaystyle\\bm\{g\}\_\{r\}^\{\\perp\}=\\bm\{g\}\_\{r\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\.\(13\)
Hence, whenever𝒈r⟂≠𝟎\\bm\{g\}\_\{r\}^\{\\perp\}\\neq\\bm\{0\}, it yields a first\-order improvement in the reward objective while leaving the cost objective unchanged, irrespective of the angleθr​c\\theta\_\{rc\}\.

### V\-BThe BSPG Update

The tangential reward gradient𝒈r⟂\\bm\{g\}\_\{r\}^\{\\perp\}captures reward\-improving motion along the constraint manifold, but by construction it does not regulate the iterate’s distance toℳ\\mathcal\{M\}\. To control this distance, we add a normal component that attracts the iterate toward the manifold from either side\. Specifically, we define the boundary\-attraction term as

𝒈b=−δ^⋅𝒈c‖𝒈c‖\+ε,\\bm\{g\}\_\{b\}=\-\\hat\{\\delta\}\\cdot\\frac\{\\bm\{g\}\_\{c\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\},\(14\)
whereδ^\\hat\{\\delta\}is the estimated constraint residual defined in Section[V\-D](https://arxiv.org/html/2608.10204#S5.SS4)andε\>0\\varepsilon\>0is a numerical stabilizer\. Note thatδ=Jc−d\\delta=J\_\{c\}\-dis a residual in cost value, not a Euclidean distance in parameter space\. The resulting BSPG update is

𝒈up=𝒈r⟂\+η⋅𝒈b,\\bm\{g\}\_\{\\mathrm\{up\}\}=\\bm\{g\}\_\{r\}^\{\\perp\}\+\\eta\\cdot\\bm\{g\}\_\{b\},\(15\)
whereη\>0\\eta\>0is the boundary\-attraction coefficient, and the parameter update is𝜽t\+1=𝜽t\+αt​𝒈up,t\.\\bm\{\\theta\}\_\{t\+1\}=\\bm\{\\theta\}\_\{t\}\+\\alpha\_\{t\}\\bm\{g\}\_\{\\mathrm\{up\},t\}\.The update in \([15](https://arxiv.org/html/2608.10204#S5.E15)\) combines a tangential component and a normal component\. The term𝒈r⟂\\bm\{g\}\_\{r\}^\{\\perp\}performs reward ascent while preserving the cost objective to first order\. The term𝒈b\\bm\{g\}\_\{b\}provides signed attraction toward the boundary: whenδ^\>0\\hat\{\\delta\}\>0, the iterate is infeasible and𝒈b\\bm\{g\}\_\{b\}acts to decreaseJcJ\_\{c\}; whenδ^<0\\hat\{\\delta\}<0, the iterate lies in the feasible interior and𝒈b\\bm\{g\}\_\{b\}acts to increaseJcJ\_\{c\}\. In both cases, the strength of this attraction scales with\|δ^\|\|\\hat\{\\delta\}\|\.

Fig\.[1](https://arxiv.org/html/2608.10204#S1.F1)illustrates the resulting geometry against ESPO\[[18](https://arxiv.org/html/2608.10204#bib.bib18)\]: on the feasible side \(Fig\.[1](https://arxiv.org/html/2608.10204#S1.F1)b\),𝒈b\\bm\{g\}\_\{b\}points outward towardℳ\\mathcal\{M\}, reducing conservatism through a mechanism absent in ESPO \(Fig\.[1](https://arxiv.org/html/2608.10204#S1.F1)a\); on the infeasible side \(Fig\.[1](https://arxiv.org/html/2608.10204#S1.F1)d\),𝒈b\\bm\{g\}\_\{b\}restores feasibility while𝒈r⟂\\bm\{g\}\_\{r\}^\{\\perp\}continues reward improvement along the boundary, whereas ESPO \(Fig\.[1](https://arxiv.org/html/2608.10204#S1.F1)c\) folds the cost correction into a single combined direction\.

![Refer to caption](https://arxiv.org/html/2608.10204v1/result/small_bspg_vs_espo_trajectory.png)Figure 2:Update trajectories of BSPG and ESPO on a synthetic constrained problem; the example only visualizes the two update components\.Fig\.[2](https://arxiv.org/html/2608.10204#S5.F2)traces both methods on a synthetic problem whose unconstrained reward maximizer lies beyond the boundary: from the same feasible start, ESPO remains interior while BSPG reaches the active boundary and then improves reward along it\.

### V\-CImplicit Lagrangian Interpretation

Although BSPG is derived from a geometric decomposition into tangential and normal directions, its update also admits an equivalent Lagrangian interpretation\. In particular, the combined direction in \([15](https://arxiv.org/html/2608.10204#S5.E15)\) can be rewritten as a gradient step on a Lagrangian objective with an automatically induced multiplier\. Define the*implicit multiplier*as:

λt=⟨𝒈r,𝒈c⟩‖𝒈c‖2\+η​δ^‖𝒈c‖\+ε\.\\lambda\_\{t\}=\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\+\\frac\{\\eta\\hat\{\\delta\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\}\.\(16\)
Substituting \([13](https://arxiv.org/html/2608.10204#S5.E13)\) into \([15](https://arxiv.org/html/2608.10204#S5.E15)\) and collecting the terms along𝒈c\\bm\{g\}\_\{c\}yields

𝒈up\\displaystyle\\bm\{g\}\_\{\\mathrm\{up\}\}=𝒈r−⟨𝒈r,𝒈c⟩‖𝒈c‖2​𝒈c−η​δ^​𝒈c‖𝒈c‖\+ε\\displaystyle=\\bm\{g\}\_\{r\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\-\\eta\\hat\{\\delta\}\\frac\{\\bm\{g\}\_\{c\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\}=𝒈r−\(⟨𝒈r,𝒈c⟩‖𝒈c‖2\+η​δ^‖𝒈c‖\+ε\)⏟λt​𝒈c\\displaystyle=\\bm\{g\}\_\{r\}\-\\underbrace\{\\left\(\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\+\\frac\{\\eta\\hat\{\\delta\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\}\\right\)\}\_\{\\displaystyle\\lambda\_\{t\}\}\\bm\{g\}\_\{c\}\(17\)=∇𝜽\[Jr​\(π𝜽\)−λ​Jc​\(π𝜽\)\]\|λ=λt,\\displaystyle=\\nabla\_\{\\bm\{\\theta\}\}\\\!\\left\[J\_\{r\}\(\\pi\_\{\\bm\{\\theta\}\}\)\-\\lambda J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\}\)\\right\]\\\!\\bigg\|\_\{\\lambda=\\lambda\_\{t\}\},\(18\)
where the gradient in \([18](https://arxiv.org/html/2608.10204#S5.E18)\) is evaluated withλ\\lambdaheld fixed atλt\\lambda\_\{t\}\. Eq\. \([18](https://arxiv.org/html/2608.10204#S5.E18)\) shows that each BSPG step is equivalent to a gradient\-ascent step on the LagrangianL​\(θ,λ\)=Jr​\(πθ\)−λ​Jc​\(πθ\),L\(\\theta,\\lambda\)=J\_\{r\}\(\\pi\_\{\\theta\}\)\-\\lambda J\_\{c\}\(\\pi\_\{\\theta\}\),with the multiplierλt\\lambda\_\{t\}induced automatically by the local gradient geometry and the boundary signal\.

The implicit multiplier in \([16](https://arxiv.org/html/2608.10204#S5.E16)\) consists of two terms with distinct roles\. The first term,⟨𝒈r,𝒈c⟩/‖𝒈c‖2\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle/\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}, is the projection coefficient of the reward gradient onto the cost gradient\. The second term,η​δ^/\(‖𝒈c‖\+ε\)\\eta\\hat\{\\delta\}/\\\!\(\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\), introduces a boundary\-dependent correction that modulates the effective penalty according to the signed residual: in the feasible interior \(δ^<0\\hat\{\\delta\}<0\) it lowersλt\\lambda\_\{t\}, tilting the update toward higher cost, and under positive gradient alignment, higher reward, until the residual closes\. We emphasize thatλt\\lambda\_\{t\}is an algebraic coefficient rather than a learned dual variable, and it need not be nonnegative away from a KKT point\. If the iterates converge to a stationary point onℳ\\mathcal\{M\}, thenδ^→0\\hat\{\\delta\}\\to 0andλt\\lambda\_\{t\}approaches the equality\-stationarity coefficientλ∗\\lambda^\{\*\}of Theorem[VI\.4](https://arxiv.org/html/2608.10204#S6.Thmtheorem4), which is a valid KKT multiplier exactly when the limit is also locally optimal overℱ\\mathcal\{F\}\.

### V\-DBoundary Distance Estimation

The BSPG update in \([15](https://arxiv.org/html/2608.10204#S5.E15)\), equivalently the implicit multiplier in \([16](https://arxiv.org/html/2608.10204#S5.E16)\), depends on the constraint residualδ​\(𝜽t\)=Jc​\(π𝜽t\)−d\\delta\(\\bm\{\\theta\}\_\{t\}\)=J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\_\{t\}\}\)\-d, which is not directly observable during training\. We therefore estimate it by combining the learned cost\-value networkVcV\_\{c\}with a residual batch correction:

δ^t=1N​∑i=1NVc​\(s0i\)⏟critic at​s0\+\(c¯batch−1\|τ\|​∑\(s,a\)∈τVc​\(s\)\)⏟residual bias correction−d\.\\hat\{\\delta\}\_\{t\}=\\underbrace\{\\frac\{1\}\{N\}\\sum\_\{i=1\}^\{N\}V\_\{c\}\(s\_\{0\}^\{i\}\)\}\_\{\\text\{critic at \}s\_\{0\}\}\+\\underbrace\{\\left\(\\bar\{c\}\_\{\\mathrm\{batch\}\}\-\\frac\{1\}\{\|\\tau\|\}\\sum\_\{\(s,a\)\\in\\tau\}V\_\{c\}\(s\)\\right\)\}\_\{\\text\{residual bias correction\}\}\-d\.\(19\)
whereNNis the number of trajectories in the batch,s0is\_\{0\}^\{i\}are the initial states,c¯batch\\bar\{c\}\_\{\\mathrm\{batch\}\}is the mean discounted cost return from the batch, and\|τ\|\|\\tau\|is the number of state–action pairs\. The first term estimates the cost objective from the initial\-state distribution through the critic; the second is a batch calibration term that vanishes whenVcV\_\{c\}matches the empirical cost returns\. Because the two averages are taken under different state distributions, no general unbiasedness or variance\-reduction claim follows without additional assumptions on the critic and the sampling process; we use \([19](https://arxiv.org/html/2608.10204#S5.E19)\) as an implementation heuristic\. The theory in Section[VI](https://arxiv.org/html/2608.10204#S6)analyzes the exact residualδ^t=δt\\hat\{\\delta\}\_\{t\}=\\delta\_\{t\}and does not depend on this estimator\.

### V\-EPractical BSPG Implementation

Algorithm[1](https://arxiv.org/html/2608.10204#alg1)presents the practical implementation of BSPG built upon PPO\[[25](https://arxiv.org/html/2608.10204#bib.bib25)\]\. Lines 5–7 estimate the boundary distance\(Eq\.\([19](https://arxiv.org/html/2608.10204#S5.E19)\)\), while lines 11–16 compute the shaped update direction\(Eq\.\([13](https://arxiv.org/html/2608.10204#S5.E13)\),\([15](https://arxiv.org/html/2608.10204#S5.E15)\)\); the remaining steps follow the standard PPO procedure\. Relative to PPO, BSPG introduces only two additional hyperparameters: the boundary\-attraction coefficientη\\etaand the clipping thresholdδmax\\delta\_\{\\max\}forδ^\\hat\{\\delta\}\.

Algorithm 1BSPG1:Input:Policy

π𝜽\\pi\_\{\\bm\{\\theta\}\}, value networks

VV,

VcV\_\{c\}, threshold

dd, coefficient

η\\eta, stabilizer

ε\\varepsilon, clip range

εclip\\varepsilon\_\{\\mathrm\{clip\}\}, residual clip

δmax\\delta\_\{\\max\}, step size

α\\alpha
2:whilenot convergeddo

3:Collect rollout

τ∼π𝜽\\tau\\sim\\pi\_\{\\bm\{\\theta\}\}for

TTsteps

4:Compute GAE reward advantages

A^\\hat\{A\}using

VV; cost advantages

A^c\\hat\{A\}\_\{c\}using

VcV\_\{c\}
5://Boundary estimation \(Eq\.[19](https://arxiv.org/html/2608.10204#S5.E19)\)

6:

δ^←1N​∑iVc​\(s0i\)\+\(c¯batch−1\|τ\|​∑s∈τVc​\(s\)\)−d\\hat\{\\delta\}\\\!\\leftarrow\\\!\\frac\{1\}\{N\}\\sum\_\{i\}V\_\{c\}\(s\_\{0\}^\{i\}\)\\\!\+\\\!\\bigl\(\\bar\{c\}\_\{\\mathrm\{batch\}\}\\\!\-\\\!\\frac\{1\}\{\|\\tau\|\}\\sum\_\{s\\in\\tau\}V\_\{c\}\(s\)\\bigr\)\-d
7:

δ^←clip​\(δ^,−δmax,δmax\)\\hat\{\\delta\}\\leftarrow\\mathrm\{clip\}\\\!\\left\(\\hat\{\\delta\},\\,\-\\delta\_\{\\max\},\\,\\delta\_\{\\max\}\\right\)
8://Surrogate gradients \(PPO clipped objectives\)

9:

Lr​\(𝜽\)←PPO\-clip surrogate for reward using​A^L\_\{r\}\(\\bm\{\\theta\}\)\\leftarrow\\text\{PPO\-clip surrogate for reward using \}\\hat\{A\}
10:

Lc​\(𝜽\)←PPO\-clip surrogate for cost using​A^cL\_\{c\}\(\\bm\{\\theta\}\)\\leftarrow\\text\{PPO\-clip surrogate for cost using \}\\hat\{A\}\_\{c\}
11:

𝒈r←∇𝜽Lr​\(𝜽\)\\bm\{g\}\_\{r\}\\leftarrow\\nabla\_\{\\bm\{\\theta\}\}L\_\{r\}\(\\bm\{\\theta\}\);

𝒈c←∇𝜽Lc​\(𝜽\)\\bm\{g\}\_\{c\}\\leftarrow\\nabla\_\{\\bm\{\\theta\}\}L\_\{c\}\(\\bm\{\\theta\}\)
12:Normalize:

𝒈r^←𝒈r/\(‖𝒈r‖\+ε\)\\hat\{\\bm\{g\}\_\{r\}\}\\leftarrow\\bm\{g\}\_\{r\}/\(\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|\+\\varepsilon\);

𝒈c^←𝒈c/\(‖𝒈c‖\+ε\)\\hat\{\\bm\{g\}\_\{c\}\}\\leftarrow\\bm\{g\}\_\{c\}/\(\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\)
13://Gradient decomposition

14:

𝒈r⟂←𝒈r^−⟨𝒈r^,𝒈c^⟩‖𝒈c^‖2\+ε​𝒈c^\\bm\{g\}\_\{r\}^\{\\perp\}\\leftarrow\\hat\{\\bm\{g\}\_\{r\}\}\-\\dfrac\{\\left\\langle\\hat\{\\bm\{g\}\_\{r\}\},\\,\\hat\{\\bm\{g\}\_\{c\}\}\\right\\rangle\}\{\\left\\\|\\hat\{\\bm\{g\}\_\{c\}\}\\right\\\|^\{2\}\+\\varepsilon\}\\hat\{\\bm\{g\}\_\{c\}\}
15:// Boundary attraction

16:

𝒈b←−δ^⋅𝒈c^/\(‖𝒈c^‖\+ε\)\\bm\{g\}\_\{b\}\\leftarrow\-\\hat\{\\delta\}\\cdot\\hat\{\\bm\{g\}\_\{c\}\}/\(\\left\\\|\\hat\{\\bm\{g\}\_\{c\}\}\\right\\\|\+\\varepsilon\)
17:// Shaped gradient update

18:

𝜽←𝜽\+α​\(𝒈r⟂\+η​𝒈b\)\\bm\{\\theta\}\\leftarrow\\bm\{\\theta\}\+\\alpha\\,\(\\bm\{g\}\_\{r\}^\{\\perp\}\+\\eta\\,\\bm\{g\}\_\{b\}\)
19:Update

VVand

VcV\_\{c\}via MSE on GAE return targets

20:endwhile

## VITheoretical Analysis

We now establish the convergence properties of the ideal BSPG update with exact gradients andδ^t=δt\\hat\{\\delta\}\_\{t\}=\\delta\_\{t\}; the statements below do not claim convergence of the stochastic normalized PPO implementation\. Three results are proved: the constraint residual converges to zero from either side \(with a finite\-horizon rate\), the tangential component is a reward\-ascent direction on the boundary, and any convergent run is characterized at its limit\. The proofs of Theorems[VI\.1](https://arxiv.org/html/2608.10204#S6.Thmtheorem1)–[VI\.4](https://arxiv.org/html/2608.10204#S6.Thmtheorem4)are provided in Appendices[\-B](https://arxiv.org/html/2608.10204#A0.SS2)–[\-D](https://arxiv.org/html/2608.10204#A0.SS4)\. We begin by collecting the regularity assumptions\.

###### Assumption 1\(Smoothness and bounded gradients\)

There is a compact setΘ\\Thetacontaining all exact BSPG iterates and the line segments between consecutive iterates\. OnΘ\\Theta,JrJ\_\{r\}andJcJ\_\{c\}haveLL\-Lipschitz gradients, and‖𝐠r‖,‖𝐠c‖≤G\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|,\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\\leq G\.

###### Assumption 2\(Constraint regularity\)

There are constantsμ\>0\\mu\>0andD<∞D<\\inftysuch that‖𝐠c​\(𝛉\)‖≥μ\\left\\\|\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}\)\\right\\\|\\geq\\muand\|δ​\(𝛉\)\|≤D\|\\delta\(\\bm\{\\theta\}\)\|\\leq Dfor all𝛉∈Θ\\bm\{\\theta\}\\in\\Theta\.

###### Assumption 3\(Diminishing step sizes\)

The step size sequence\{αt\}t≥0\\\{\\alpha\_\{t\}\\\}\_\{t\\geq 0\}satisfies:∑t=0∞αt=∞,\\sum\_\{t=0\}^\{\\infty\}\\alpha\_\{t\}=\\infty,and∑t=0∞αt2<∞\.\\sum\_\{t=0\}^\{\\infty\}\\alpha\_\{t\}^\{2\}<\\infty\.

Assumptions[1](https://arxiv.org/html/2608.10204#Thmassumption1)and[3](https://arxiv.org/html/2608.10204#Thmassumption3)are standard in first\-order constrained policy optimization\[[16](https://arxiv.org/html/2608.10204#bib.bib16)\]; if0≤c​\(s,a\)≤cmax0\\leq c\(s,a\)\\leq c\_\{\\max\}and0≤d≤cmax/\(1−γ\)0\\leq d\\leq c\_\{\\max\}/\(1\-\\gamma\), the residual bound holds automatically withD=max⁡\{d,cmax/\(1−γ\)−d\}D=\\max\\\{d,\\,c\_\{\\max\}/\(1\-\\gamma\)\-d\\\}\. Assumption[2](https://arxiv.org/html/2608.10204#Thmassumption2)is a nondegeneracy \(constraint\-qualification\) condition excluding points where the cost gradient vanishes; it makes the local cost level sets regular and ensures the decomposition \([13](https://arxiv.org/html/2608.10204#S5.E13)\) is well defined at every iterate\.

We first show that BSPG drives the constraint residualJc​\(π𝜽t\)−dJ\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\_\{t\}\}\)\-dto zero\.

###### Theorem VI\.1\(Boundary Convergence\)

Under Assumptions[1](https://arxiv.org/html/2608.10204#Thmassumption1)–[3](https://arxiv.org/html/2608.10204#Thmassumption3), letδt≜δ​\(𝛉t\)=Jc​\(π𝛉t\)−d\\delta\_\{t\}\\triangleq\\delta\(\\bm\{\\theta\}\_\{t\}\)=J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\_\{t\}\}\)\-ddenote the signed constraint violation at iterationtt\. The sequence\{δt\}\\\{\\delta\_\{t\}\\\}generated by exact\-gradient BSPG satisfies:

V​\(𝜽t\+1\)≤V​\(𝜽t\)−μ′​αt​δt2\+CR​αt2,V\(\\bm\{\\theta\}\_\{t\+1\}\)\\leq V\(\\bm\{\\theta\}\_\{t\}\)\-\\mu^\{\\prime\}\\alpha\_\{t\}\\,\\delta\_\{t\}^\{2\}\+C\_\{R\}\\alpha\_\{t\}^\{2\},\(20\)whereV​\(𝛉\)=12​δ​\(𝛉\)2V\(\\bm\{\\theta\}\)=\\tfrac\{1\}\{2\}\\delta\(\\bm\{\\theta\}\)^\{2\},μ′=η​μ2G\+ε\\mu^\{\\prime\}=\\frac\{\\eta\\mu^\{2\}\}\{G\+\\varepsilon\}, andCR=\(G2\+L​D\)​\(G2\+η2​D2\)C\_\{R\}=\(G^\{2\}\+LD\)\(G^\{2\}\+\\eta^\{2\}D^\{2\}\)\. Consequently:

∑t=0∞αt​δt2<∞andδt→0\.\\sum\_\{t=0\}^\{\\infty\}\\alpha\_\{t\}\\,\\delta\_\{t\}^\{2\}<\\infty\\qquad\\text\{and\}\\qquad\\delta\_\{t\}\\to 0\.\(21\)Thus,Jc​\(π𝛉t\)→dJ\_\{c\}\(\\pi\_\{\\bm\{\\theta\}\_\{t\}\}\)\\to dfrom either side of the boundary\.

###### Corollary VI\.2\(Finite\-horizon residual bound\)

LetAT=∑t=0T−1αtA\_\{T\}=\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}and drawτ∈\{0,…,T−1\}\\tau\\in\\\{0,\\ldots,T\-1\\\}withPr⁡\(τ=t\)=αt/AT\\Pr\(\\tau=t\)=\\alpha\_\{t\}/A\_\{T\}\. Then

𝔼​\[δτ2\]≤V0\+CR​∑t=0T−1αt2μ′​AT\.\\mathbb\{E\}\[\\delta\_\{\\tau\}^\{2\}\]\\;\\leq\\;\\frac\{V\_\{0\}\+C\_\{R\}\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}^\{2\}\}\{\\mu^\{\\prime\}A\_\{T\}\}\.\(22\)In particular, for constantαt=α¯/T\\alpha\_\{t\}=\\bar\{\\alpha\}/\\sqrt\{T\}over a fixed horizonTT, the bound isO​\(1/T\)O\(1/\\sqrt\{T\}\)\.

The corollary follows by summing \([20](https://arxiv.org/html/2608.10204#S6.E20)\) overt<Tt<Tand dividing byμ′​AT\\mu^\{\\prime\}A\_\{T\}; see Appendix[\-B](https://arxiv.org/html/2608.10204#A0.SS2)\.

The Lyapunov functionV=12​δ2V=\\tfrac\{1\}\{2\}\\delta^\{2\}is symmetric in the sign of the residual, so the ideal normal component regulates the target cost level from either side\. This differs from a standard projected dual update: a negative residual can decrease the dual variable only until it reaches zero, after which the boundary signal is lost, whereas BSPG continues to apply a residual\-proportional normal correction toward the prescribed cost level\.

It is worth noting that the Lyapunov functionV=12​δ2V=\\frac\{1\}\{2\}\\delta^\{2\}is symmetric in the sign ofδ\\delta, so the boundary\-seeking mechanism operates identically whether the current iterate is feasible or infeasible\. This contrasts with Lagrangian methods, where the multiplier update acts only on positive violations\.

The boundary\-attraction term drivesδ→0\\delta\\to 0; however, it does not by itself guarantee reward improvement\. The following theorem shows that the tangential component𝒈r⟂\\bm\{g\}\_\{r\}^\{\\perp\}is a valid ascent direction forJrJ\_\{r\}\.

###### Theorem VI\.3\(Reward Ascent\)

For any𝛉\\bm\{\\theta\}with𝐠c≠𝟎\\bm\{g\}\_\{c\}\\neq\\bm\{0\}:

⟨𝒈r,𝒈r⟂⟩=‖𝒈r⟂‖2≥0\.\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\rangle=\\left\\\|\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\\|^\{2\}\\geq 0\.\(23\)Equality holds iff𝐠r∥𝐠c\\bm\{g\}\_\{r\}\\parallel\\bm\{g\}\_\{c\}\. On the manifoldℳ\\mathcal\{M\}\(whereδ=0\\delta=0and𝐠up=𝐠r⟂\\bm\{g\}\_\{\\mathrm\{up\}\}=\\bm\{g\}\_\{r\}^\{\\perp\}\), the BSPG update is a strict ascent direction forJrJ\_\{r\}unless𝐠r∥𝐠c\\bm\{g\}\_\{r\}\\parallel\\bm\{g\}\_\{c\}, i\.e\., unless𝛉\\bm\{\\theta\}is a first\-order stationary point of the equality\-constrained problemmax⁡Jr\\max J\_\{r\}subject toJc=dJ\_\{c\}=d\.

We next characterize convergent runs of the exact update\.

###### Theorem VI\.4\(Convergent\-limit characterization\)

Under Assumptions[1](https://arxiv.org/html/2608.10204#Thmassumption1)–[3](https://arxiv.org/html/2608.10204#Thmassumption3), suppose the exact\-gradient BSPG iterates converge:𝛉t→𝛉∗\\bm\{\\theta\}\_\{t\}\\to\\bm\{\\theta\}^\{\*\}\. Then𝛉∗∈ℳ\\bm\{\\theta\}^\{\*\}\\in\\mathcal\{M\}and

𝒈r​\(𝜽∗\)=λ∗​𝒈c​\(𝜽∗\),λ∗=⟨𝒈r​\(𝜽∗\),𝒈c​\(𝜽∗\)⟩‖𝒈c​\(𝜽∗\)‖2,\\bm\{g\}\_\{r\}\(\\bm\{\\theta\}^\{\*\}\)=\\lambda^\{\*\}\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}^\{\*\}\),\\qquad\\lambda^\{\*\}=\\frac\{\\left\\langle\\bm\{g\}\_\{r\}\(\\bm\{\\theta\}^\{\*\}\),\\,\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}^\{\*\}\)\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}^\{\*\}\)\\right\\\|^\{2\}\},\(24\)i\.e\.,𝛉∗\\bm\{\\theta\}^\{\*\}is a first\-order stationary point for maximizingJrJ\_\{r\}on the active constraint set, and complementary slacknessλ∗​\(Jc​\(π𝛉∗\)−d\)=0\\lambda^\{\*\}\(J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}^\{\*\}\}\)\-d\)=0holds\. If, in addition,𝛉∗\\bm\{\\theta\}^\{\*\}is a local maximizer ofJrJ\_\{r\}onℱ\\mathcal\{F\}, thenλ∗≥0\\lambda^\{\*\}\\geq 0and𝛉∗\\bm\{\\theta\}^\{\*\}satisfies the KKT conditions of \([3](https://arxiv.org/html/2608.10204#S3.E3)\)\.

## VIIExperiments

Evaluation Setups\.We evaluate our method onSafetyPointGoal1\-v0using the OmniSafe\[[27](https://arxiv.org/html/2608.10204#bib.bib27)\]framework\. This task provides a standard constrained navigation setting for studying the reward–cost tradeoff in safe policy optimization\.

Baselines\.We compare BSPG against two closely related constrained RL baselines:CRPO\[[16](https://arxiv.org/html/2608.10204#bib.bib16)\]andESPO\[[18](https://arxiv.org/html/2608.10204#bib.bib18)\]\. CRPO is a constraint\-driven policy optimization method that switches update behavior according to constraint satisfaction, emphasizing feasibility control\. ESPO is a recent first\-order safe policy optimization method that directly balances reward improvement and cost reduction through gradient\-based policy updates\.

Evaluation Metrics\.We report three primary metrics throughout training: \(i\)average episodic reward, where higher values indicate better task performance; \(ii\)average episodic cost, where lower values indicate better constraint control; and \(iii\)boundary proximity, the empirical residual magnitude\|C^−d\|\|\\widehat\{C\}\-d\|, whereC^\\widehat\{C\}is the evaluation cost statistic; smaller values mean that the reported policy uses the prescribed budget more fully\. The third metric is our primary diagnostic and, to our knowledge, has not been explicitly reported in prior Safe\-RL work\. By Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1), smaller boundary proximity indicates tighter tracking of the active constraint boundary; by Proposition[IV\.5](https://arxiv.org/html/2608.10204#S4.Thmtheorem5), under positive reward–cost gradient alignment, the remaining residual corresponds to locally recoverable reward\. The residual alone is not a universal surrogate for reward, since the local reward change also depends on the gradient alignment and magnitudes\.

As shown in Fig\.[3](https://arxiv.org/html/2608.10204#S7.F3), BSPG attains the highest episodic reward on SafetyPointGoal1\-v0\. At the same time, BSPG maintains episodic cost near the constraint limit, while both baselines remain more conservative and settle well inside the feasible region\. This behavior is most clearly illustrated in the boundary\-proximity plot, where BSPG remains consistently closest to zero, indicating tighter tracking of the active constraint boundary\. Taken together, these results support the proposed boundary\-regulation mechanism in the evaluated task; they do not by themselves imply a general performance ordering across environments\.

![Refer to caption](https://arxiv.org/html/2608.10204v1/result/reward_cost_2000dpi_256c.png)Figure 3:Results on SafetyPointGoal1\-v0\.
## VIIIConclusion

In this paper, we proposed Boundary\-Seeking Policy Gradient \(BSPG\), a safe reinforcement learning method that explicitly drives policy updates toward the active constraint boundary while improving reward along it\. By combining a tangential reward term with a signed boundary\-attraction term, BSPG encourages boundary\-aligned optimization rather than conservative interior solutions\. On the theory side, occupancy\-measure analysis identifies the active boundary as the global constrained target when no unconstrained reward maximizer is feasible, and the realizability corollary transfers this target to a policy class attaining the exact constrained value\. For the ideal exact\-gradient update, BSPG drives the signed residual to zero from either side with a finite\-horizonO​\(1/T\)O\(1/\\sqrt\{T\}\)bound, and any convergent run reaches a stationary point on the active constraint set, satisfying the KKT conditions when the limit is locally optimal over the feasible set\. Empirically, BSPG attains a strong reward–safety trade\-off in the evaluated task by making fuller use of the safety budget\.

## Acknowledgement

This work is partially supported by NSF ECCS Award \#2534263\.

## References

- \[1\]D\. Silver, T\. Hubert, J\. Schrittwieser, I\. Antonoglou, M\. Lai, A\. Guez, M\. Lanctot, L\. Sifre, D\. Kumaran, T\. Graepel, T\. Lillicrap, K\. Simonyan, and D\. Hassabis, “A general reinforcement learning algorithm that masters chess, shogi, and go through self\-play,”*Science*, vol\. 362, no\. 6419, pp\. 1140–1144, 2018\.
- \[2\]Q\. Li, Z\. Peng, L\. Feng, Q\. Zhang, Z\. Xue, and B\. Zhou, “Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning,”*IEEE transactions on pattern analysis and machine intelligence*, vol\. 45, no\. 3, pp\. 3461–3475, 2022\.
- \[3\]J\. Kober, J\. A\. Bagnell, and J\. Peters, “Reinforcement learning in robotics: A survey,”*The International Journal of Robotics Research*, vol\. 32, no\. 11, pp\. 1238–1274, 2013\.
- \[4\]E\. Altman,*Constrained Markov decision processes*\. CRC Press, 1999, vol\. 7\.
- \[5\]A\. Stooke, J\. Achiam, and P\. Abbeel, “Responsive safety in reinforcement learning by pid lagrangian methods,” in*Int\. Conf\. Machine Learning \(ICML\)*\. PMLR, 2020, pp\. 9133–9143\.
- \[6\]S\. Paternain, M\. Calvo\-Fullana, L\. F\. O\. Chamon, and A\. Ribeiro, “Safe policies for reinforcement learning via primal\-dual methods,”*IEEE Trans\. Autom\. Control*, pp\. 1–1, 2022\.
- \[7\]H\. Wei, X\. Liu, and L\. Ying, “Triple\-Q: a model\-free algorithm for constrained reinforcement learning with sublinear regret and zero constraint violation,” in*Int\. Conf\. Artificial Intelligence and Statistics \(AISTATS\)*, 2022\.
- \[8\]H\. Wei, A\. Ghosh, N\. Shroff, L\. Ying, and X\. Zhou, “Provably efficient model\-free algorithms for non\-stationary CMDPs,” in*Int\. Conf\. Artificial Intelligence and Statistics \(AISTATS\)*\. PMLR, 2023, pp\. 6527–6570\.
- \[9\]H\. Zhang, X\. Peng, H\. Wei, and X\. Liu, “Safe and efficient: A primal\-dual method for offline convex cmdps under partial data coverage,” in*Advances Neural Information Processing Systems \(NeurIPS\)*, 2024\.
- \[10\]D\. Ding, X\. Wei, Z\. Yang, Z\. Wang, and M\. Jovanovic, “Provably efficient safe exploration via primal\-dual policy optimization,” in*Int\. Conf\. Artificial Intelligence and Statistics \(AISTATS\)*, vol\. 130\. PMLR, 2021, pp\. 3304–3312\.
- \[11\]J\. Zhu, K\. Yu, D\. Lee, X\. Liu, and H\. Wei, “An optimistic algorithm for online cmdps with anytime adversarial constraints,” in*Int\. Conf\. Machine Learning \(ICML\)*, ser\. ICML’25\. JMLR\.org, 2025\.
- \[12\]J\. Achiam, D\. Held, A\. Tamar, and P\. Abbeel, “Constrained policy optimization,” in*Int\. Conf\. Machine Learning \(ICML\)*, vol\. 70\. JMLR, 2017, pp\. 22–31\.
- \[13\]T\.\-Y\. Yang, J\. Rosca, K\. Narasimhan, and P\. J\. Ramadge, “Projection\-based constrained policy optimization,” in*Int\. Conf\. on Learning Representations \(ICLR\)*, 2020\.
- \[14\]Y\. Zhang, Q\. Vuong, and K\. Ross, “First order constrained optimization in policy space,”*Advances in Neural Information Processing Systems*, vol\. 33, pp\. 15 338–15 349, 2020\.
- \[15\]L\. Yang, J\. Ji, J\. Dai, L\. Zhang, B\. Zhou, P\. Li, Y\. Yang, and G\. Pan, “Constrained update projection approach to safe policy optimization,”*Advances in Neural Information Processing Systems*, vol\. 35, pp\. 9111–9124, 2022\.
- \[16\]T\. Xu, Y\. Liang, and G\. Lan, “Crpo: A new approach for safe reinforcement learning with convergence guarantee,” in*Int\. Conf\. Machine Learning \(ICML\)*, M\. Meila and T\. Z\. 0001, Eds\., vol\. 139\. PMLR, 2021, pp\. 11 480–11 491\.
- \[17\]S\. Gu, B\. Sel, Y\. Ding, L\. Wang, Q\. Lin, M\. Jin, and A\. Knoll, “Balance reward and safety optimization for safe reinforcement learning: A perspective of gradient manipulation,” in*AAAI Conf\. Artificial Intelligence*, vol\. 38, no\. 19, 2024, pp\. 21 099–21 106\.
- \[18\]S\. Gu, L\. Shi, Y\. Ding, A\. Knoll, C\. Spanos, A\. Wierman, and M\. Jin, “Enhancing efficiency of safe reinforcement learning via sample manipulation,”*Advances in Neural Information Processing Systems*, vol\. 37, pp\. 17 247–17 285, 2024\.
- \[19\]Y\. Yao, Z\. Liu, Z\. Cen, P\. Huang, T\. Zhang, W\. Yu, and D\. Zhao, “Gradient shaping for multi\-constraint safe reinforcement learning,” in*6th annual learning for dynamics & control conference*\. PMLR, 2024, pp\. 25–39\.
- \[20\]A\. Ray, J\. Achiam, and D\. Amodei, “Benchmarking safe exploration in deep reinforcement learning,”*arXiv preprint arXiv:1910\.01708*, vol\. 7, no\. 1, p\. 2, 2019\.
- \[21\]C\. Tessler, D\. J\. Mankowitz, and S\. Mannor, “Reward constrained policy optimization,”*arXiv preprint arXiv:1805\.11074*, 2018\.
- \[22\]Z\. Wu, B\. Tang, Q\. Lin, C\. Yu, S\. Mao, Q\. Xie, X\. Wang, and D\. Wang, “Off\-policy primal\-dual safe reinforcement learning,”*arXiv preprint arXiv:2401\.14758*, 2024\.
- \[23\]C\. Xuan, F\. Zhang, F\. Yin, and H\.\-K\. Lam, “Constrained proximal policy optimization,”*arXiv preprint arXiv:2305\.14216*, 2023\.
- \[24\]R\. S\. Sutton, D\. McAllester, S\. Singh, and Y\. Mansour, “Policy gradient methods for reinforcement learning with function approximation,” in*Advances Neural Information Processing Systems \(NeurIPS\)*, ser\. NIPS’99, 1999\.
- \[25\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov, “Proximal policy optimization algorithms,” 2017\.
- \[26\]V\. S\. Borkar,*Stochastic approximation: a dynamical systems viewpoint*\. Springer, 2009, vol\. 48\.
- \[27\]J\. Ji, J\. Zhou, B\. Zhang, J\. Dai, X\. Pan, R\. Sun, W\. Huang, Y\. Geng, M\. Liu, and Y\. Yang, “Omnisafe: An infrastructure for accelerating safe reinforcement learning research,”*Journal of Machine Learning Research*, vol\. 25, no\. 285, pp\. 1–6, 2024\.

### \-AProof of Theorem[IV\.1](https://arxiv.org/html/2608.10204#S4.Thmtheorem1)

For a stationary policyπ\\pi, define its \(normalized\) occupancy measureρπ​\(s,a\)=\(1−γ\)​∑t=0∞γt​Prπ,μ0⁡\(st=s,at=a\)\\rho\_\{\\pi\}\(s,a\)=\(1\-\\gamma\)\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}\\Pr\_\{\\pi,\\mu\_\{0\}\}\(s\_\{t\}=s,a\_\{t\}=a\)\. The setΛ=\{ρπ:π∈Π\}\\Lambda=\\\{\\rho\_\{\\pi\}:\\pi\\in\\Pi\\\}is a compact convex polytope, andJr,JcJ\_\{r\},J\_\{c\}are linear functionals ofρ\\rho:Jk​\(π\)=11−γ​∑s,aρπ​\(s,a\)​k​\(s,a\)J\_\{k\}\(\\pi\)=\\frac\{1\}\{1\-\\gamma\}\\sum\_\{s,a\}\\rho\_\{\\pi\}\(s,a\)\\,k\(s,a\)fork∈\{r,c\}k\\in\\\{r,c\\\}\[[4](https://arxiv.org/html/2608.10204#bib.bib4)\]; hence the maxima below are attained\.

Suppose, for contradiction, that some constrained\-optimalπ⋆\\pi^\{\\star\}satisfiesJc​\(π⋆\)<dJ\_\{c\}\(\\pi^\{\\star\}\)<d, and writeρ⋆=ρπ⋆\\rho^\{\\star\}=\\rho\_\{\\pi^\{\\star\}\}\. Fix any unconstrained maximizerπr⋆\\pi\_\{r\}^\{\\star\}with occupancy measureρr\\rho\_\{r\}; by hypothesisJc​\(ρr\)\>dJ\_\{c\}\(\\rho\_\{r\}\)\>d\. IfJr​\(ρr\)=Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{r\}\)=J\_\{r\}\(\\rho^\{\\star\}\), thenπ⋆\\pi^\{\\star\}would itself be a feasible unconstrained maximizer, contradicting the hypothesis; henceJr​\(ρr\)\>Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{r\}\)\>J\_\{r\}\(\\rho^\{\\star\}\)\. Forλ∈\(0,1\]\\lambda\\in\(0,1\]letρλ=\(1−λ\)​ρ⋆\+λ​ρr∈Λ\\rho\_\{\\lambda\}=\(1\-\\lambda\)\\rho^\{\\star\}\+\\lambda\\rho\_\{r\}\\in\\Lambda\(convexity\)\. By linearity,Jr​\(ρλ\)=\(1−λ\)​Jr​\(ρ⋆\)\+λ​Jr​\(ρr\)\>Jr​\(ρ⋆\)J\_\{r\}\(\\rho\_\{\\lambda\}\)=\(1\-\\lambda\)J\_\{r\}\(\\rho^\{\\star\}\)\+\\lambda J\_\{r\}\(\\rho\_\{r\}\)\>J\_\{r\}\(\\rho^\{\\star\}\), andJc​\(ρλ\)=\(1−λ\)​Jc​\(ρ⋆\)\+λ​Jc​\(ρr\)≤dJ\_\{c\}\(\\rho\_\{\\lambda\}\)=\(1\-\\lambda\)J\_\{c\}\(\\rho^\{\\star\}\)\+\\lambda J\_\{c\}\(\\rho\_\{r\}\)\\leq dfor everyλ≤\(d−Jc​\(ρ⋆\)\)/\(Jc​\(ρr\)−Jc​\(ρ⋆\)\)\\lambda\\leq\\bigl\(d\-J\_\{c\}\(\\rho^\{\\star\}\)\\bigr\)/\\bigl\(J\_\{c\}\(\\rho\_\{r\}\)\-J\_\{c\}\(\\rho^\{\\star\}\)\\bigr\), a strictly positive threshold\. Thus, for smallλ\>0\\lambda\>0, the policy induced byρλ\\rho\_\{\\lambda\}is feasible with strictly larger reward thanπ⋆,\\pi^\{\\star\},a contradiction\. HenceJc​\(π⋆\)=dJ\_\{c\}\(\\pi^\{\\star\}\)=d\.■\\blacksquare

### \-BProof of Theorem[VI\.1](https://arxiv.org/html/2608.10204#S6.Thmtheorem1)

DefineV​\(𝜽\)=12​δ​\(𝜽\)2V\(\\bm\{\\theta\}\)=\\tfrac\{1\}\{2\}\\delta\(\\bm\{\\theta\}\)^\{2\}\. By the chain rule,∇𝜽V=δ​𝒈c\\nabla\_\{\\bm\{\\theta\}\}V=\\delta\\bm\{g\}\_\{c\}\. The functionVVisLVL\_\{V\}\-smooth withLV=G2\+L​DL\_\{V\}=G^\{2\}\+LD: for any𝜽,𝜽′∈ℝp\\bm\{\\theta\},\\bm\{\\theta\}^\{\\prime\}\\in\\mathbb\{R\}^\{p\},

‖∇𝜽V​\(𝜽′\)−∇𝜽V​\(𝜽\)‖≤\\displaystyle\\left\\\|\\nabla\_\{\\bm\{\\theta\}\}V\(\\bm\{\\theta\}^\{\\prime\}\)\\\!\-\\\!\\nabla\_\{\\bm\{\\theta\}\}V\(\\bm\{\\theta\}\)\\right\\\|\\leq\(\|δ′−δ\|⋅‖𝒈c′‖\+\|δ\|⋅‖𝒈c′−𝒈c‖\)\\displaystyle\\bigl\(\|\\delta^\{\\prime\}\-\\delta\|\\\!\\cdot\\\!\\left\\\|\\bm\{g\}\_\{c\}^\{\\prime\}\\right\\\|\+\|\\delta\|\\\!\\cdot\\\!\\left\\\|\\bm\{g\}\_\{c\}^\{\\prime\}\-\\bm\{g\}\_\{c\}\\right\\\|\\bigr\)≤\\displaystyle\\leq\(G2\+L​D\)​‖𝜽′−𝜽‖,\\displaystyle\(G^\{2\}\+LD\)\\left\\\|\\bm\{\\theta\}^\{\\prime\}\-\\bm\{\\theta\}\\right\\\|,\(25\)using\|δ′−δ\|≤G​‖𝜽′−𝜽‖\|\\delta^\{\\prime\}\-\\delta\|\\leq G\\left\\\|\\bm\{\\theta\}^\{\\prime\}\-\\bm\{\\theta\}\\right\\\|\(mean value theorem and Assumption[1](https://arxiv.org/html/2608.10204#Thmassumption1)\),‖𝒈c‖≤G\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\\leq G\(Assumption[1](https://arxiv.org/html/2608.10204#Thmassumption1)\),‖𝒈c′−𝒈c‖≤L​‖𝜽′−𝜽‖\\left\\\|\\bm\{g\}\_\{c\}^\{\\prime\}\-\\bm\{g\}\_\{c\}\\right\\\|\\leq L\\left\\\|\\bm\{\\theta\}^\{\\prime\}\-\\bm\{\\theta\}\\right\\\|\(Assumption[1](https://arxiv.org/html/2608.10204#Thmassumption1)\), and\|δ\|≤D\|\\delta\|\\leq D\(Assumption[2](https://arxiv.org/html/2608.10204#Thmassumption2)\); the segment condition in Assumption[1](https://arxiv.org/html/2608.10204#Thmassumption1)ensures these bounds hold along the update segments\. By the descent lemma:

Vt\+1≤Vt\+αt​δt​⟨𝒈ct,𝒈upt⟩\+LV​αt22​‖𝒈upt‖2\.V\_\{t\+1\}\\leq V\_\{t\}\+\\alpha\_\{t\}\\delta\_\{t\}\\left\\langle\\bm\{g\}\_\{c\}^\{t\},\\,\\bm\{g\}\_\{\\mathrm\{up\}\}^\{t\}\\right\\rangle\+\\frac\{L\_\{V\}\\alpha\_\{t\}^\{2\}\}\{2\}\\left\\\|\\bm\{g\}\_\{\\mathrm\{up\}\}^\{t\}\\right\\\|^\{2\}\.\(26\)
Since𝒈up=𝒈r⟂\+η​𝒈b\\bm\{g\}\_\{\\mathrm\{up\}\}=\\bm\{g\}\_\{r\}^\{\\perp\}\+\\eta\\bm\{g\}\_\{b\}with exact gradients:

*Tangential term\.*⟨𝒈c,𝒈r⟂⟩=⟨𝒈c,𝒈r−⟨𝒈r,𝒈c⟩‖𝒈c‖2​𝒈c⟩=⟨𝒈c,𝒈r⟩−⟨𝒈r,𝒈c⟩=0\.\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\rangle=\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{g\}\_\{r\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\\right\\rangle=\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{g\}\_\{r\}\\right\\rangle\-\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle=0\.

*Boundary term\.*𝒈b=−δt​𝒈c/\(‖𝒈c‖\+ε\)\\bm\{g\}\_\{b\}=\-\\delta\_\{t\}\\bm\{g\}\_\{c\}/\(\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\)\(exactδt\\delta\_\{t\}, exact𝒈c\\bm\{g\}\_\{c\}\), so:

δt​⟨𝒈c,𝒈b⟩=−δt2​‖𝒈c‖2‖𝒈c‖\+ε≤−δt2​μ2G\+ε,\\delta\_\{t\}\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{g\}\_\{b\}\\right\\rangle=\-\\delta\_\{t\}^\{2\}\\frac\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\}\\leq\-\\delta\_\{t\}^\{2\}\\frac\{\\mu^\{2\}\}\{G\+\\varepsilon\},\(27\)where we used‖𝒈c‖≥μ\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\\geq\\mu\(Assumption[2](https://arxiv.org/html/2608.10204#Thmassumption2)\) and‖𝒈c‖≤G\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\\leq G\(Assumption[1](https://arxiv.org/html/2608.10204#Thmassumption1)\)\. Therefore:

δt​⟨𝒈ct,𝒈upt⟩=η​δt​⟨𝒈ct,𝒈bt⟩≤−η​μ2G\+ε⏟μ′​δt2\.\\delta\_\{t\}\\left\\langle\\bm\{g\}\_\{c\}^\{t\},\\,\\bm\{g\}\_\{\\mathrm\{up\}\}^\{t\}\\right\\rangle=\\eta\\delta\_\{t\}\\left\\langle\\bm\{g\}\_\{c\}^\{t\},\\,\\bm\{g\}\_\{b\}^\{t\}\\right\\rangle\\leq\-\\underbrace\{\\frac\{\\eta\\mu^\{2\}\}\{G\+\\varepsilon\}\}\_\{\\mu^\{\\prime\}\}\\delta\_\{t\}^\{2\}\.\(28\)
We know that‖𝒈up‖2≤2​‖𝒈r⟂‖2\+2​η2​‖𝒈b‖2≤2​G2\+2​η2​D2≜C1\\left\\\|\\bm\{g\}\_\{\\mathrm\{up\}\}\\right\\\|^\{2\}\\leq 2\\left\\\|\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\\|^\{2\}\+2\\eta^\{2\}\\left\\\|\\bm\{g\}\_\{b\}\\right\\\|^\{2\}\\leq 2G^\{2\}\+2\\eta^\{2\}D^\{2\}\\triangleq C\_\{1\}, since‖𝒈r⟂‖≤‖𝒈r‖≤G\\left\\\|\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\\|\\leq\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|\\leq Gand‖𝒈b‖=\|δt\|​‖𝒈c‖/\(‖𝒈c‖\+ε\)≤\|δt\|≤D\\left\\\|\\bm\{g\}\_\{b\}\\right\\\|=\|\\delta\_\{t\}\|\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|/\(\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\+\\varepsilon\)\\leq\|\\delta\_\{t\}\|\\leq D\.

Substituting the two bounds above into \([26](https://arxiv.org/html/2608.10204#A0.E26)\):

Vt\+1≤Vt−μ′​αt​δt2\+CR​αt2,CR=LV​C12\.V\_\{t\+1\}\\leq V\_\{t\}\-\\mu^\{\\prime\}\\alpha\_\{t\}\\delta\_\{t\}^\{2\}\+C\_\{R\}\\alpha\_\{t\}^\{2\},\\quad C\_\{R\}=\\tfrac\{L\_\{V\}C\_\{1\}\}\{2\}\.\(29\)This is \([20](https://arxiv.org/html/2608.10204#S6.E20)\)\.

Finally, summing from0toT−1T\-1:μ′​∑t=0T−1αt​δt2≤V0\+CR​∑t=0∞αt2<∞\\mu^\{\\prime\}\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}\\delta\_\{t\}^\{2\}\\leq V\_\{0\}\+C\_\{R\}\\sum\_\{t=0\}^\{\\infty\}\\alpha\_\{t\}^\{2\}<\\infty\. LettingT→∞T\\to\\inftygives∑αt​δt2<∞\\sum\\alpha\_\{t\}\\delta\_\{t\}^\{2\}<\\infty\.

To showδt→0\\delta\_\{t\}\\to 0: the recursionVt\+1≤\(1−2​μ′​αt\)​Vt\+CR​αt2V\_\{t\+1\}\\leq\(1\-2\\mu^\{\\prime\}\\alpha\_\{t\}\)V\_\{t\}\+C\_\{R\}\\alpha\_\{t\}^\{2\}is a Robbins–Monro recursion with summable perturbation\. Sinceαt→0\\alpha\_\{t\}\\to 0, there existsNNwith0≤1−2​μ′​αt≤10\\leq 1\-2\\mu^\{\\prime\}\\alpha\_\{t\}\\leq 1for allt≥Nt\\geq N; iterating fromNNand using1−x≤e−x1\-x\\leq e^\{\-x\},∏s=Nt\(1−2​μ′​αs\)≤exp⁡\(−2​μ′​∑s=Ntαs\)→0\\prod\_\{s=N\}^\{t\}\(1\-2\\mu^\{\\prime\}\\alpha\_\{s\}\)\\leq\\exp\(\-2\\mu^\{\\prime\}\\sum\_\{s=N\}^\{t\}\\alpha\_\{s\}\)\\to 0, and the convolution withαs2\\alpha\_\{s\}^\{2\}vanishes \(standard argument\), soVt→0V\_\{t\}\\to 0andδt→0\\delta\_\{t\}\\to 0\.

*Proof of Corollary[VI\.2](https://arxiv.org/html/2608.10204#S6.Thmtheorem2)\.*Summing \([20](https://arxiv.org/html/2608.10204#S6.E20)\) overt=0,…,T−1t=0,\\ldots,T\-1givesμ′​∑t=0T−1αt​δt2≤V0\+CR​∑t=0T−1αt2\\mu^\{\\prime\}\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}\\delta\_\{t\}^\{2\}\\leq V\_\{0\}\+C\_\{R\}\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}^\{2\}\. Since𝔼​\[δτ2\]=1AT​∑t=0T−1αt​δt2\\mathbb\{E\}\[\\delta\_\{\\tau\}^\{2\}\]=\\frac\{1\}\{A\_\{T\}\}\\sum\_\{t=0\}^\{T\-1\}\\alpha\_\{t\}\\delta\_\{t\}^\{2\}, dividing byμ′​AT\\mu^\{\\prime\}A\_\{T\}yields \([22](https://arxiv.org/html/2608.10204#S6.E22)\)\. Withαt=α¯/T\\alpha\_\{t\}=\\bar\{\\alpha\}/\\sqrt\{T\},AT=α¯​TA\_\{T\}=\\bar\{\\alpha\}\\sqrt\{T\}and∑tαt2=α¯2\\sum\_\{t\}\\alpha\_\{t\}^\{2\}=\\bar\{\\alpha\}^\{2\}, giving theO​\(1/T\)O\(1/\\sqrt\{T\}\)rate\.■\\blacksquare

### \-CProof of Theorem[VI\.3](https://arxiv.org/html/2608.10204#S6.Thmtheorem3)

Compute:⟨𝒈r,𝒈r⟂⟩=⟨𝒈r,𝒈r−⟨𝒈r,𝒈c⟩‖𝒈c‖2​𝒈c⟩=‖𝒈r‖2−⟨𝒈r,𝒈c⟩2‖𝒈c‖2\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\rangle=\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{r\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}\\right\\rangle=\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|^\{2\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle^\{2\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\. Also:‖𝒈r⟂‖2=‖𝒈r‖2−2​⟨𝒈r,𝒈c⟩2‖𝒈c‖2\+⟨𝒈r,𝒈c⟩2‖𝒈c‖2=‖𝒈r‖2−⟨𝒈r,𝒈c⟩2‖𝒈c‖2\\left\\\|\\bm\{g\}\_\{r\}^\{\\perp\}\\right\\\|^\{2\}=\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|^\{2\}\-2\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle^\{2\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\+\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle^\{2\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}=\\left\\\|\\bm\{g\}\_\{r\}\\right\\\|^\{2\}\-\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle^\{2\}\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\. Both expressions are equal and non\-negative by Cauchy–Schwarz\. Equality holds iff‖𝒈r⟂‖=0\\\|\\bm\{g\}\_\{r\}^\{\\perp\}\\\|=0iff𝒈r∥𝒈c,\\bm\{g\}\_\{r\}\\parallel\\bm\{g\}\_\{c\},confirming that𝒈r=α​𝒈c\\bm\{g\}\_\{r\}=\\alpha\\bm\{g\}\_\{c\}for some scalarα\\alpha\.

### \-DProof of Theorem[VI\.4](https://arxiv.org/html/2608.10204#S6.Thmtheorem4)

By Theorem[VI\.1](https://arxiv.org/html/2608.10204#S6.Thmtheorem1),δt→0\\delta\_\{t\}\\to 0; since𝜽t→𝜽∗\\bm\{\\theta\}\_\{t\}\\to\\bm\{\\theta\}^\{\*\}andδ\\deltais continuous,δ​\(𝜽∗\)=0\\delta\(\\bm\{\\theta\}^\{\*\}\)=0, i\.e\.,𝜽∗∈ℳ\\bm\{\\theta\}^\{\*\}\\in\\mathcal\{M\}\.

Stationarity\.Suppose, for contradiction, that𝒖≜𝒈r⟂​\(𝜽∗\)≠𝟎\\bm\{u\}\\triangleq\\bm\{g\}\_\{r\}^\{\\perp\}\(\\bm\{\\theta\}^\{\*\}\)\\neq\\bm\{0\}\. Since‖𝒈c‖≥μ\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|\\geq\\muonΘ\\Theta\(Assumption[2](https://arxiv.org/html/2608.10204#Thmassumption2)\), the map𝜽↦𝒈r⟂​\(𝜽\)\\bm\{\\theta\}\\mapsto\\bm\{g\}\_\{r\}^\{\\perp\}\(\\bm\{\\theta\}\)is continuous at𝜽∗\\bm\{\\theta\}^\{\*\}, and the normal component vanishes in the limit,‖η​𝒈b​\(𝜽t\)‖≤η​\|δt\|→0\\left\\\|\\eta\\bm\{g\}\_\{b\}\(\\bm\{\\theta\}\_\{t\}\)\\right\\\|\\leq\\eta\|\\delta\_\{t\}\|\\to 0\. Hence𝒈up​\(𝜽t\)=𝒈r⟂​\(𝜽t\)\+η​𝒈b​\(𝜽t\)→𝒖\\bm\{g\}\_\{\\mathrm\{up\}\}\(\\bm\{\\theta\}\_\{t\}\)=\\bm\{g\}\_\{r\}^\{\\perp\}\(\\bm\{\\theta\}\_\{t\}\)\+\\eta\\bm\{g\}\_\{b\}\(\\bm\{\\theta\}\_\{t\}\)\\to\\bm\{u\}, and there existsNNsuch that⟨𝒖,𝒈up​\(𝜽t\)⟩≥12​‖𝒖‖2,∀t≥N\.\\left\\langle\\bm\{u\},\\,\\bm\{g\}\_\{\\mathrm\{up\}\}\(\\bm\{\\theta\}\_\{t\}\)\\right\\rangle\\;\\geq\\;\\tfrac\{1\}\{2\}\\left\\\|\\bm\{u\}\\right\\\|^\{2\},\\qquad\\forall t\\geq N\.Projecting the recursion𝜽t\+1−𝜽t=αt​𝒈up​\(𝜽t\)\\bm\{\\theta\}\_\{t\+1\}\-\\bm\{\\theta\}\_\{t\}=\\alpha\_\{t\}\\bm\{g\}\_\{\\mathrm\{up\}\}\(\\bm\{\\theta\}\_\{t\}\)onto𝒖\\bm\{u\}and summing fromNNtoT−1T\-1:⟨𝒖,𝜽T−𝜽N⟩=∑t=NT−1αt​⟨𝒖,𝒈up​\(𝜽t\)⟩≥12​‖𝒖‖2​∑t=NT−1αt→T→∞∞,\\left\\langle\\bm\{u\},\\,\\bm\{\\theta\}\_\{T\}\-\\bm\{\\theta\}\_\{N\}\\right\\rangle=\\sum\_\{t=N\}^\{T\-1\}\\alpha\_\{t\}\\left\\langle\\bm\{u\},\\,\\bm\{g\}\_\{\\mathrm\{up\}\}\(\\bm\{\\theta\}\_\{t\}\)\\right\\rangle\\;\\geq\\;\\tfrac\{1\}\{2\}\\left\\\|\\bm\{u\}\\right\\\|^\{2\}\\sum\_\{t=N\}^\{T\-1\}\\alpha\_\{t\}\\;\\xrightarrow\[T\\to\\infty\]\{\}\\;\\infty,since∑tαt=∞\\sum\_\{t\}\\alpha\_\{t\}=\\infty\(Assumption[3](https://arxiv.org/html/2608.10204#Thmassumption3)\)\. This contradicts𝜽T→𝜽∗\\bm\{\\theta\}\_\{T\}\\to\\bm\{\\theta\}^\{\*\}, whose left side converges to the finite value⟨𝒖,𝜽∗−𝜽N⟩\\left\\langle\\bm\{u\},\\,\\bm\{\\theta\}^\{\*\}\-\\bm\{\\theta\}\_\{N\}\\right\\rangle\. Therefore𝒈r⟂​\(𝜽∗\)=𝟎\\bm\{g\}\_\{r\}^\{\\perp\}\(\\bm\{\\theta\}^\{\*\}\)=\\bm\{0\}, and since‖𝒈c​\(𝜽∗\)‖≥μ\>0\\left\\\|\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}^\{\*\}\)\\right\\\|\\geq\\mu\>0we may solve for𝒈r\\bm\{g\}\_\{r\}:𝒈r\(𝜽∗\)=⟨𝒈r,𝒈c⟩‖𝒈c‖2𝒈c=:λ∗𝒈c\(𝜽∗\)\\bm\{g\}\_\{r\}\(\\bm\{\\theta\}^\{\*\}\)=\\frac\{\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{g\}\_\{c\}\\right\\rangle\}\{\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\}\\bm\{g\}\_\{c\}=:\\lambda^\{\*\}\\bm\{g\}\_\{c\}\(\\bm\{\\theta\}^\{\*\}\)\. Complementary slacknessλ∗​\(Jc​\(π𝜽∗\)−d\)=0\\lambda^\{\*\}\(J\_\{c\}\(\\pi\_\{\\bm\{\\theta\}^\{\*\}\}\)\-d\)=0is immediate from𝜽∗∈ℳ\\bm\{\\theta\}^\{\*\}\\in\\mathcal\{M\}\.

Dual feasibility under local optimality\.Suppose𝜽∗\\bm\{\\theta\}^\{\*\}is a local maximizer ofJrJ\_\{r\}onℱ\\mathcal\{F\}and, for contradiction,λ∗<0\\lambda^\{\*\}<0\. From stationarity,𝒈r=λ∗​𝒈c\\bm\{g\}\_\{r\}=\\lambda^\{\*\}\\bm\{g\}\_\{c\}\. Consider the direction𝒗=−𝒈c\\bm\{v\}=\-\\bm\{g\}\_\{c\}:⟨𝒈r,𝒗⟩=−λ∗​‖𝒈c‖2\>0\\left\\langle\\bm\{g\}\_\{r\},\\,\\bm\{v\}\\right\\rangle=\-\\lambda^\{\*\}\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}\>0\(reward increases to first order\),⟨𝒈c,𝒗⟩=−‖𝒈c‖2<0\\left\\langle\\bm\{g\}\_\{c\},\\,\\bm\{v\}\\right\\rangle=\-\\left\\\|\\bm\{g\}\_\{c\}\\right\\\|^\{2\}<0\(cost strictly decreases, so the point enters the feasible interior sinceJc​\(π𝜽∗\)=dJ\_\{c\}\(\\pi\_\{\\bm\{\\theta\}^\{\*\}\}\)=d\)\. Hence, for sufficiently smallε0\>0\\varepsilon\_\{0\}\>0, the point𝜽∗−ε0​𝒈c∈ℱ\\bm\{\\theta\}^\{\*\}\-\\varepsilon\_\{0\}\\bm\{g\}\_\{c\}\\in\\mathcal\{F\}attains strictly higher reward, contradicting local maximality\. Thereforeλ∗≥0\\lambda^\{\*\}\\geq 0, and together with stationarity, primal feasibility, and complementary slackness,𝜽∗\\bm\{\\theta\}^\{\*\}satisfies the KKT conditions of \([3](https://arxiv.org/html/2608.10204#S3.E3)\)\.■\\blacksquare