Robust Peak-cost Constrained Reinforcement Learning

# Robust Peak-cost Constrained Reinforcement Learning
Source: [https://arxiv.org/html/2607.15457](https://arxiv.org/html/2607.15457)
Shilpa Mukhopadhyay1, Sourav Ganguly1, Santosh Mohan Rajkumar3, Honghao Wei2, Debdipta Goswami3and Arnob Ghosh11Dept\. of Electrical and Computer Engineering, NJIT, Newark, NJ, USA\(E\-mails: \{sm3934,sg2786,arnob\.ghosh\}@njit\.edu\)2School of Electrical Engineering and Computer Science Washington State University, Pullman, WA, USA\(E\-mail: honghao\.wei@wsu\.edu\)3Department of Mechanical and Aerospace Engineering, The Ohio State University, Columbus, OH, USA\(E\-mail: \{rajkumar\.36,goswami\.78\}@osu\.edu\)

###### Abstract

We study robust peak\-cost constrained reinforcement learning \(RP\-CRL\), where the objective is to maximize expected reward while controlling the maximum cost encountered along a trajectory\. This setting is motivated by safety\-critical applications in which a single large violation can be catastrophic and therefore cannot be adequately captured by the standard CMDP framework based on expected cumulative cost\. Existing reachability\-constrained RL methods adopt Lagrangian\-based approaches, yet the underlying duality properties of peak\-cost constrained MDPs remain unclear\. We show that, unlike standard CMDPs, peak\-cost constrained MDPs may not admit zero duality gap\. We further consider a robust formulation to address simulator\-to\-real\-world mismatch in the transition dynamics\. To solve this problem, we develop a surrogate optimization framework and a robust value estimation method based on integral probability metrics\. We prove that, with appropriate hyperparameter choices, the surrogate solution attains the same robust reward value as the original problem while violating the constraint by at mostϵ\\epsilon\. Experiments show that the proposed method effectively enforces safety under dynamics perturbations while retaining strong reward performance\.

## IIntroduction

Safety is a fundamental requirement in intelligent decision\-making problems\. Motivated by this, recent reinforcement learning \(RL\) research has extensively studied constrained Markov decision processes \(CMDPs\)\[[2](https://arxiv.org/html/2607.15457#bib.bib10),[14](https://arxiv.org/html/2607.15457#bib.bib19),[13](https://arxiv.org/html/2607.15457#bib.bib18),[25](https://arxiv.org/html/2607.15457#bib.bib13)\]\. In the standard CMDP framework, safety is enforced by requiring the*expected cumulative cost*to remain below a prescribed threshold\. However, this formulation is insufficient for many practical applications, where the primary concern is not the total accumulated cost, but rather the*largest cost incurred along a trajectory*\. In such settings, even a single large violation can be catastrophic, regardless of how small the cumulative cost is overall\. For example, a trajectory may incur small costs at most time steps but experience one severe safety violation, making it significantly more dangerous than another trajectory whose maximum cost remains uniformly bounded\.

To address this limitation, recent works\[[28](https://arxiv.org/html/2607.15457#bib.bib4),[11](https://arxiv.org/html/2607.15457#bib.bib7)\]have considered reachability\-constrained reinforcement learning \(RCRL\), where the goal is to maximize expected reward subject to a peak\-cost constraint, namely that the maximum cost encountered along a trajectory remains below a specified threshold \(often0\)\. These works develop Lagrangian\-based methods for solving the resulting optimization problem\. However, unlike the standard CMDP setting, where zero duality gap is well established, such a guarantee has not been shown for peak\-cost constrained MDPs\. As a result, it remains unclear whether these Lagrangian\-based approaches can recover the true optimal solution\.

An additional limitation of existing work is that it does not account for uncertainty in the transition dynamics\[[28](https://arxiv.org/html/2607.15457#bib.bib4),[11](https://arxiv.org/html/2607.15457#bib.bib7)\]\. In practice, policies are often trained in simulation and then deployed in the real world, where the true dynamics may differ from those of the simulator\. Consequently, a policy that satisfies the peak\-cost constraint in simulation may still violate the same constraint after deployment due to model mismatch\. This motivates the following central questions:

Does a peak\-cost constrained MDP admit zero duality gap? How can we develop an algorithm for the robust peak\-cost constrained problem that bridges the sim\-to\-real gap?

Our Contributions:

- •We show that, unlike the standard CMDP, a peak\-cost constrained MDP may fail to admit zero duality gap, even in a two\-state MDP\. The key reason is that convexity with respect to the state\-action occupancy measure breaks down\.
- •We formulate a robust peak\-cost constrained problem and propose a surrogate optimization framework\. We show that, with appropriate hyperparameter choices, the optimal solution of the surrogate problem achieves the same robust reward value as the original problem while violating the constraint by at most anϵ\\epsilon\-amount\.
- •We develop an effective approach for estimating the robust value function based on an integral probability metric\. Empirically, we show that the proposed method can effectively satisfy the constraint even when the environment is perturbed\.

### I\-ARelated Works

CMDP: Constrained Markov Decision Processes \(CMDPs\) are a foundational framework for solving constrained reinforcement learning \(CRL\) problems\[[2](https://arxiv.org/html/2607.15457#bib.bib10)\]\. CMDPs leverage the convexity of the state\-action occupancy measure, enabling the use of primal\-dual methods to solve the optimization problem\[[20](https://arxiv.org/html/2607.15457#bib.bib11),[24](https://arxiv.org/html/2607.15457#bib.bib12),[25](https://arxiv.org/html/2607.15457#bib.bib13),[30](https://arxiv.org/html/2607.15457#bib.bib14)\]\. These methods have well\-established convergence guarantees and have been extensively studied in the literature\[[13](https://arxiv.org/html/2607.15457#bib.bib18),[14](https://arxiv.org/html/2607.15457#bib.bib19),[8](https://arxiv.org/html/2607.15457#bib.bib20)\]\. Beyond primal\-dual methods, linear programming \(LP\)\-based and model\-based approaches have also been explored, directly addressing the primal problem\[[27](https://arxiv.org/html/2607.15457#bib.bib15),[1](https://arxiv.org/html/2607.15457#bib.bib16),[7](https://arxiv.org/html/2607.15457#bib.bib17)\]\. However, peak\-constained objective is different compared to the CMDP\.

RCMDP: Unlike standard CMDPs, the optimization problem for RCMDPs is non\-convex in the state\-action occupancy measure, which complicates the use of traditional Lagrangian or primal\-dual methods\[[26](https://arxiv.org/html/2607.15457#bib.bib21)\]\. Some studies have attempted to address this challenge using primal\-dual methods under strong duality assumptions, but these approaches often lack iteration complexity guarantees\[[29](https://arxiv.org/html/2607.15457#bib.bib22)\]\. Recent work has introduced epigraph reformulations to address the limitations of primal\-dual methods, but these approaches require computationally expensive binary search procedures and are sensitive to noise in policy value estimation\[[16](https://arxiv.org/html/2607.15457#bib.bib2)\]\.\[[12](https://arxiv.org/html/2607.15457#bib.bib1)\]avoids these limitations by jointly optimizing for robustness and constraint satisfaction, eliminating the need for binary search with iteration guarantees\. However, these methods do not consider peak cost constraint which we consider\.

Feasible Sets and Reachability Analysis for Safety and Optimality in CMDPs: Characterizing feasible sets is a critical and ongoing challenge in safe control and reinforcement learning\[[5](https://arxiv.org/html/2607.15457#bib.bib23)\]\. Feasible sets, often represented using safety certificates like control barrier functions \(CBFs\)\[[6](https://arxiv.org/html/2607.15457#bib.bib24),[6](https://arxiv.org/html/2607.15457#bib.bib24),[17](https://arxiv.org/html/2607.15457#bib.bib25)\], define the set of states from which the system can recover to safety\. However, energy\-based safety certificates often result in overly conservative or inaccurate feasible sets, limiting their applicability\[[18](https://arxiv.org/html/2607.15457#bib.bib26)\]\. Hamilton Jacobian \(HJ\) reachability analysis provides a more rigorous approach to deriving feasible sets but is computationally expensive due to the non\-trivial partial differential equations involved\[[3](https://arxiv.org/html/2607.15457#bib.bib27)\]\. Recently, RL\-based approaches are considered to find policy that will avoid unsafe sets\[[9](https://arxiv.org/html/2607.15457#bib.bib9),[10](https://arxiv.org/html/2607.15457#bib.bib8),[23](https://arxiv.org/html/2607.15457#bib.bib28)\], however, they did not consider on maximizing reward as an objective\.\[[28](https://arxiv.org/html/2607.15457#bib.bib4),[11](https://arxiv.org/html/2607.15457#bib.bib7)\]learns a single policy that balances safety and optimality,considering peak cost constraint optimization\. They used Lagrangian framework to solve the problem, however, they did not provide any strong duality guarantee\. Further, none of the above work considered robust optimization framework bridging the sim\-to\-real gap while being trained on the simulator model which might be different from the real environment\.

## IIPreliminaries and Problem Formulation

### II\-AConstrained Markov Decision Processes

A Constrained Markov Decision Process \(CMDP\) is defined by

M=⟨𝒮,𝒜,P,r,c,γ,d0⟩,M=\\langle\\mathcal\{S\},\\mathcal\{A\},P,r,c,\\gamma,d\_\{0\}\\rangle,where𝒮\\mathcal\{S\}and𝒜\\mathcal\{A\}denote the state and action spaces,P\(⋅∣s,a\)P\(\\cdot\\mid s,a\)is the \(possibly unknown\) transition kernel,r:𝒮×𝒜→ℝr:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}is the reward function,c:𝒮×𝒜→ℝc:\\mathcal\{S\}\\times\\mathcal\{A\}\\to\\mathbb\{R\}is the cost function,γ∈\(0,1\)\\gamma\\in\(0,1\)is the discount factor, andd0d\_\{0\}is the initial state distribution\. A policy is denoted byπ\(⋅∣s\):𝒮→Δ\(𝒜\)\\pi\(\\cdot\\mid s\):\\mathcal\{S\}\\rightarrow\\Delta\(\\mathcal\{A\}\)\.

For a given transition modelPPand policyπ\\pi, letVrP,π​\(s\)V\_\{r\}^\{P,\\pi\}\(s\)andVcP,π​\(s\)V\_\{c\}^\{P,\\pi\}\(s\)denote the discounted negative reward and cost value functions, respectively:

VrP,π\(s\)=−𝔼\[∑t=0∞γtr\(st,at\)\|s0=s,π,P\],V\_\{r\}^\{P,\\pi\}\(s\)=\-\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}r\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=s,\\pi,P\\right\],VcP,π\(s\)=𝔼\[∑t=0∞γtc\(st,at\)\|s0=s,π,P\]\.V\_\{c\}^\{P,\\pi\}\(s\)=\\mathbb\{E\}\\\!\\left\[\\sum\_\{t=0\}^\{\\infty\}\\gamma^\{t\}c\(s\_\{t\},a\_\{t\}\)\\,\\middle\|\\,s\_\{0\}=s,\\pi,P\\right\]\.The corresponding objective values from the initial distributiond0d\_\{0\}are

JrP​\(π\)=𝔼s∼d0​\[VrP,π​\(s\)\],JcP​\(π\)=𝔼s∼d0​\[VcP,π​\(s\)\]\.J^\{P\}\_\{r\}\(\\pi\)=\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\\!\\left\[V\_\{r\}^\{P,\\pi\}\(s\)\\right\],\\quad J^\{P\}\_\{c\}\(\\pi\)=\\mathbb\{E\}\_\{s\\sim d\_\{0\}\}\\\!\\left\[V\_\{c\}^\{P,\\pi\}\(s\)\\right\]\.
A standard CMDP seeks to maximize reward while keeping the expected discounted cost below a budgetbb\. Corresponding to discounted negative reward, the CMDP objective becomes:

minπ⁡JrP​\(π\)s\.t\.JcP​\(π\)≤b\.\\min\_\{\\pi\}\\;J^\{P\}\_\{r\}\(\\pi\)\\qquad\\text\{s\.t\.\}\\qquad J^\{P\}\_\{c\}\(\\pi\)\\leq b\.When Slater’s condition holds, the standard CMDP admits zero duality gap\[[21](https://arxiv.org/html/2607.15457#bib.bib30)\], and therefore Lagrangian\-based methods provide an effective solution approach\.

### II\-BPeak\-Cost Constraints

Although CMDPs are widely used, a major limitation is that the constraint is expressed only through the*expected cumulative discounted cost*\. In many applications, however, controlling only the cumulative cost is insufficient; instead, one must ensure that the*maximum*cost encountered along a trajectory remains below a desired threshold\.

Concrete example \(reachability and persistent safety\)In many safety\-critical settings, we require that the system never enter an unsafe region\. Letc​\(s\)c\(s\)be a state\-based safety function such thatc​\(s\)≤bc\(s\)\\leq bfor safe states andc​\(s\)\>bc\(s\)\>bfor unsafe states\. Then, for a policyπ\\pi, safety along a trajectoryτ=\(s0,s1,…\)∼π\\tau=\(s\_\{0\},s\_\{1\},\\dots\)\\sim\\pirequiresmaxt∈ℕ⁡cπ​\(st\)≤0\\max\_\{t\\in\\mathbb\{N\}\}c^\{\\pi\}\(s\_\{t\}\)\\leq 0with probability11wherebbis some safety threshold\. That is, the maximum safety violation encountered along the trajectory must remain non\-positive\.

Motivated by this, we consider the following peak\-constrained problem:

minπ⁡JrP​\(π\)s\.t\.maxs∈𝒮π,P⁡c​\(s\)≤b,\\min\_\{\\pi\}\\;J\_\{r\}^\{P\}\(\\pi\)\\qquad\\text\{s\.t\.\}\\qquad\\max\_\{s\\in\\mathcal\{S\}^\{\\pi,P\}\}c\(s\)\\leq b,where𝒮π,P=\{s∈𝒮:Pr⁡\(∃t≥0,st=s\|s0∼d0,π,P\)\>0\}\\mathcal\{S\}^\{\\pi,P\}=\\\{s\\in\\mathcal\{S\}:\\Pr\(\\exists t\\geq 0,s\_\{t\}=s\|s\_\{0\}\\sim d\_\{0\},\\pi,P\)\>0\\\}\. We assume that the initial states always give you at least one policy which is feasible\. Often times, we consider an episodic setup with lengthTTthenttwill be upper bounded byTT\.

Note that the peak constraint is difficult to evaluate as one needs to predict the future states at a given timett\. Reference\[[9](https://arxiv.org/html/2607.15457#bib.bib9)\]first observed that, for a fixed policyπ\\pi, this peak\-cost constraint can be represented through a value function using a discounted recursion\. In particular, define

Qc,peakπ,P​\(s,a\)\\displaystyle Q\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s,a\)=\(1−γ\)​c​\(s\)\\displaystyle=\(1\-\\gamma\)c\(s\)\+γ​max⁡\{c​\(s\),𝔼P\(⋅∣s,a\)​\[Vc,peakπ,P​\(⋅\)\]\}\\displaystyle\+\\gamma\\max\\left\\\{c\(s\),\\,\\mathbb\{E\}\_\{P\(\\cdot\\mid s,a\)\}\\left\[V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(\\cdot\)\\right\]\\right\\\}\(1\)and

Vc,peakπ,P\(s\)=⟨π\(⋅∣s\),Qc,peakπ,P\(s,⋅\)⟩\.V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)=\\langle\\pi\(\\cdot\\mid s\),Q\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s,\\cdot\)\\rangle\.Forγ<1\\gamma<1, the operator induced by \([1](https://arxiv.org/html/2607.15457#S2.E1)\) is a contraction, and henceVc,peakπV\_\{c,\\mathrm\{peak\}\}^\{\\pi\}can be computed for a given policyπ\\piby fixed\-point iteration\. For finite\-horizon episodic setup, we can simply takeγ=1\\gamma=1\.

This leads to the following modified optimization problem:

minπ⁡JrP​\(π\)s\.t\.maxs∈𝒮π,P⁡\[Vc,peakπ,P​\(s\)\]≤b\.\\displaystyle\\min\_\{\\pi\}\\;J^\{P\}\_\{r\}\(\\pi\)\\qquad\\text\{s\.t\.\}\\qquad\\max\_\{s\\in\\mathcal\{S\}^\{\\pi,P\}\}\\\!\\left\[V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\\right\]\\leq b\.\(2\)A related formulation was also considered by\[[28](https://arxiv.org/html/2607.15457#bib.bib4)\]\. They proposed the following Lagrangian\-based objective:

minπ⁡maxβ≥0⁡\[Vrπ,P​\(s\)\+β​\(s\)​Vc,peakπ,P​\(s\)\],\\displaystyle\\min\_\{\\pi\}\\max\_\{\\beta\\geq 0\}\\;\\\!\\left\[V\_\{r\}^\{\\pi,P\}\(s\)\+\\beta\(s\)V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\\right\],\(3\)whereβ​\(s\)\\beta\(s\)is a state\-dependent Lagrange multiplier used to penalize constraint violations\.

### II\-CLimitations of the Lagrangian Approach in\[[28](https://arxiv.org/html/2607.15457#bib.bib4)\]

While\[[28](https://arxiv.org/html/2607.15457#bib.bib4)\]adopts the Lagrangian formulation in \([3](https://arxiv.org/html/2607.15457#S2.E3)\), it does not establish a strong\-duality guarantee\. This is a critical issue because the peak\-cost constrained problem is fundamentally different from the standard CMDP: the constraint is no longer based on an additive discounted cumulative cost, but instead depends on the maximum cost encountered along a trajectory\. In particular, the reachability set𝒮π,P\\mathcal\{S\}^\{\\pi,P\}inherently depends on the policyπ\\pi\.

In this paper, we show that the peak\-cost constrained problem does*not*necessarily admit zero duality gap\. To the best of our knowledge, this is the first result showing that, unlike the standard CMDP, the peak\-constrained problem may fail to satisfy strong duality\. The key reason is that the problem is no longer convex in the state\-action occupancy measure, due to its dependence on the trajectory\-wise maximum cost\.

###### Theorem 1

\[informal\] The problem stated in \([2](https://arxiv.org/html/2607.15457#S2.E2)\) may have a non\-zero duality gap even if the Slater’s condition is satisfied\.

The proof of Theorem[1](https://arxiv.org/html/2607.15457#Thmtheorem1)is in Appendix[\-A](https://arxiv.org/html/2607.15457#A0.SS1)\.

### II\-DLimitations and a Robust Formulation

Problem \([2](https://arxiv.org/html/2607.15457#S2.E2)\) does not account for the mismatch between the simulated environment used for training and the real environment in which the learned policy is ultimately deployed\. In practice, policies are often trained in simulation and then transferred to the real world, where the transition dynamics may differ from those of the simulator\. As a result, a policy that appears feasible in simulation may violate the safety constraints after deployment\. This motivates a robust formulation in which the learned policy must remain feasible under the worst\-case model within a prescribed uncertainty set\. Accordingly, we consider the following robust counterpart:

minπ⁡maxP∈𝒫⁡JrP​\(π\)s\.t\.​maxP∈𝒫⁡maxs∈𝒮π,P⁡\[Vc,peakπ,P​\(s\)\]≤b\.\\displaystyle\\min\_\{\\pi\}\\max\_\{P\\in\\mathcal\{P\}\}\\;J\_\{r\}^\{P\}\(\\pi\)\\qquad\\text\{s\.t\. \}\\max\_\{P\\in\\mathcal\{P\}\}\\max\_\{s\\in\\mathcal\{S\}^\{\\pi,P\}\}\\\!\\left\[V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\\right\]\\leq b\.\(4\)This formulation seeks to maximize the worst\-case expected cumulative reward while ensuring that the worst\-case peak constraint remains below the prescribed threshold\.

For the uncertainty set, one natural choice is the rectangular uncertainty set

𝒫\(s,a\)=\{P∈Δ\(S\):D\(P,P0\(⋅∣s,a\)\)≤ρ\},\\mathcal\{P\}\_\{\(s,a\)\}=\\left\\\{P\\in\\Delta\(S\):D\\\!\\left\(P,P\_\{0\}\(\\cdot\\mid s,a\)\\right\)\\leq\\rho\\right\\\},whereD​\(⋅,⋅\)D\(\\cdot,\\cdot\)measures the distance between two probability distributions, andP0P\_\{0\}is the nominal model encountered in simulator\. Rectangular uncertainty sets are particularly attractive because they often lead to tractable robust dynamic programming formulations, whereas evaluating the robust value function under non\-rectangular uncertainty sets is, in general, NP\-hard\[[15](https://arxiv.org/html/2607.15457#bib.bib31)\]\.

For continuous state spaces, one may instead consider uncertainty sets defined through an integral probability metric \(IPM\)\[[31](https://arxiv.org/html/2607.15457#bib.bib5)\], which admits an efficient representation of the worst\-case value function under linear function approximation\. In particular, the IPM between two distributionsppandqqis defined as

dF​\(p,q\)=supf∈F\{p⊤​f−q⊤​f\},F⊆ℝ\|S\|\.d\_\{F\}\(p,q\)=\\sup\_\{f\\in F\}\\left\\\{p^\{\\top\}f\-q^\{\\top\}f\\right\\\},\\qquad F\\subseteq\\mathbb\{R\}^\{\|S\|\}\.Under suitable choices ofFF, the IPM recovers important distributional distances; for example, under special conditions it reduces to the total variation distance\[[19](https://arxiv.org/html/2607.15457#bib.bib6)\]\. Moreover, IPM\-based uncertainty sets yield a closed\-form expression for the robust Bellman operator under linear approximation for value function \(i\.e\.,Vc,peakπ​\(s\)=Ψ​\(s\)T​wπ,cV\_\{c,\\mathrm\{peak\}\}^\{\\pi\}\(s\)=\\Psi\(s\)^\{T\}w^\{\\pi,c\}\)

LP​Vc,peak,wπ=\(1−γ\)​c​\(s,a\)\+γ​Vc,peak,wπ\+ρ​‖wc,2:d‖2\.L\_\{P\}V\_\{c,\\mathrm\{peak\},w\}^\{\\pi\}=\(1\-\\gamma\)c\(s,a\)\+\\gamma V\_\{c,\\mathrm\{peak\},w\}^\{\\pi\}\+\\rho\\\|w\_\{c,2:d\}\\\|\_\{2\}\.\(5\)Here,LPL\_\{P\}is the worst case robust Bellman operator, defined asLpVc,peakπ=c\(s,a\)\+supP⟨P\(⋅\|s,a\),Vc,peakπ,P\(⋅\)⟩L\_\{p\}V\_\{c,\\mathrm\{peak\}\}^\{\\pi\}=c\(s,a\)\+\\sup\_\{P\}\\langle P\(\\cdot\|s,a\),V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(\\cdot\)\\rangle

## IIISolution Methodology

In this section, we discuss our problem to solve peak\-cost constrained robust MDP as described in \([4](https://arxiv.org/html/2607.15457#S2.E4)\)\.

### III\-ARobust Peak\-Cost Constrained MDP Formulation

Note that since the problem in \([2](https://arxiv.org/html/2607.15457#S2.E2)\) may not have a zero duality gap we have to consider a surrogate problem\. Inspired from\[[12](https://arxiv.org/html/2607.15457#bib.bib1)\], we consider the following surrogate problem forβ\>0\\beta\>0\.

minπ⁡max⁡\{maxP⁡Jrπ,P/β,\(maxP,s∈𝒮π,P⁡Vc,peakπ,P​\(s\)−b\)\}\\min\_\{\\pi\}\\max\\\{\\max\_\{P\}J\_\{r\}^\{\\pi,P\}/\\beta,\(\\max\_\{P,s\\in\\mathcal\{S\}^\{\\pi,P\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\-b\)\\\}\(6\)\[[12](https://arxiv.org/html/2607.15457#bib.bib1)\]adopted this surrogate problem for the robust CMDP problem, here, we apply for the robust peak cost constrained problem\. Note that when the policyπ\\piis infeasible andβ\\betais large enough, then, it would try to optimize the constraint value function to make it satisfy the constraint\. On the other hand, when the policy is feasible then it would optimize the objective if the reward is positive if the peak\-cost constraint is satisfied\.

The effectiveness of the surrogate problem in \([6](https://arxiv.org/html/2607.15457#S3.E6)\) is shown in the following proposition\.

###### Proposition 1

Letπ^∗\\hat\{\\pi\}^\{\*\}be an optimal solution of \([6](https://arxiv.org/html/2607.15457#S3.E6)\), thenJrπ∗^≥Jrπ∗J\_\{r\}^\{\\hat\{\\pi^\{\*\}\}\}\\geq J\_\{r\}^\{\\pi^\{\*\}\}whereπ∗\\pi^\{\*\}is an optimal solution for \([4](https://arxiv.org/html/2607.15457#S2.E4)\)\. Further ifβ≥Cmax/ϵ\\beta\\geq C\_\{\\text\{max\}\}/\\epsilon, then\(maxP,s∈𝒮π,P⁡Vc,peakπ,P​\(s\)−b\)\+≤ϵ\(\\max\_\{P,s\\in\\mathcal\{S\}^\{\\pi,P\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\-b\)\_\{\+\}\\leq\\epsilon\.

The above proof is adapted from Proposition 1 in\[[12](https://arxiv.org/html/2607.15457#bib.bib1)\], and has been shown in Appendix[\-A](https://arxiv.org/html/2607.15457#A0.SS1)\.\[[12](https://arxiv.org/html/2607.15457#bib.bib1)\]also provides the convergence rate guarantee for a robust natural policy gradient which solves \([6](https://arxiv.org/html/2607.15457#S3.E6)\) for the finite state\-space\. Our goal is to apply for the large state\-space problem, hence, we do not consider the natural policy gradient\. We rather consider an robust actor\-critic\-based approach, and we use PPO\-method\[[22](https://arxiv.org/html/2607.15457#bib.bib32)\]for the actor\.

## IVProposed Algorithm

In this section, we present our approach for solving the robust peak\-constrained MDP problem formulated in \([6](https://arxiv.org/html/2607.15457#S3.E6)\)\. Our algorithm is based on the Actor\-Critic framework modified for the peak\-constrained, and the robust value function\.

### IV\-AActor–Critic Optimization

To account for robustness, we adapt the temporal\-difference \(TD\) errors for both the reward critic and the peak\-cost critic using the IPM\-based robustification in \([5](https://arxiv.org/html/2607.15457#S2.E5)\)\. Specifically, we copnsider the following advantage function

δt,r\\displaystyle\\delta\_\{t,r\}=rt\+γ​Vr​\(st\+1\)−Vr​\(st\)\+ρr​‖wr,2:d‖2,\\displaystyle=r\_\{t\}\+\\gamma V\_\{r\}\(s\_\{t\+1\}\)\-V\_\{r\}\(s\_\{t\}\)\+\\rho\_\{r\}\\\|w\_\{r,2:d\}\\\|\_\{2\},\(7\)δt,c\\displaystyle\\delta\_\{t,c\}=\(1−γ\)​ct\+γ​max⁡\(ct,Vc,peak​\(st\+1\)\)\\displaystyle=\\big\(1\-\\gamma\\big\)c\_\{t\}\+\\gamma\\max\\\!\\big\(c\_\{t\},\\,V\_\{c,\\mathrm\{peak\}\}\(s\_\{t\+1\}\)\\big\)−Vc,peak​\(st\)\+ρc​‖wc,2:d‖2\.\\displaystyle\\qquad\\qquad\\qquad\-V\_\{c,\\mathrm\{peak\}\}\(s\_\{t\}\)\+\\rho\_\{c\}\\\|w\_\{c,2:d\}\\\|\_\{2\}\.\(8\)Note that the added regularization is due to the robust value function metric as discussed in \([5](https://arxiv.org/html/2607.15457#S2.E5)\)\. For smooth approximation, instead of max operator, we use Log\-Sum\-Exponential\(LSE\)\.

Using these TD errors, we construct the generalized advantage estimates \(GAEs\) for the reward objective and 1\-step advantage estimate for peak\-cost objective as

At,r\\displaystyle A\_\{t,r\}=∑l=0∞\(γ​λ\)l​δt\+l,r,\\displaystyle=\\sum\_\{l=0\}^\{\\infty\}\(\\gamma\\lambda\)^\{l\}\\delta\_\{t\+l,r\},\(9\)At,c\\displaystyle A\_\{t,c\}=δt,c\.\\displaystyle=\\delta\_\{t,c\}\.\(10\)
Hereλ\\lambdais the GAE decay\. The corresponding value targets for the reward and peak\-cost critics are given by

vtarget,r​\(st\)\\displaystyle v\_\{\\text\{target\},r\}\(s\_\{t\}\)=At,r\+Vr​\(st\),\\displaystyle=A\_\{t,r\}\+V\_\{r\}\(s\_\{t\}\),\(11\)vtarget,c​\(st\)\\displaystyle v\_\{\\text\{target\},c\}\(s\_\{t\}\)=At,c\+Vc,peak​\(st\)\\displaystyle=A\_\{t,c\}\+V\_\{c,\\mathrm\{peak\}\}\(s\_\{t\}\)\(12\)
The actor is updated using the PPO\-style clipped objective\[[22](https://arxiv.org/html/2607.15457#bib.bib32)\]depending on whether we are optimizing the objective or the constraint

Lactor=𝔼t​\[min⁡\(rt​\(θ\)​At,j,clip​\(rt​\(θ\),1−ϵ,1\+ϵ\)​At,j\)\],\\displaystyle L\_\{\\text\{actor\}\}=\\mathbb\{E\}\_\{t\}\\\!\\left\[\\min\\\!\\left\(r\_\{t\}\(\\theta\)\\,A\_\{t,j\},\\;\\mathrm\{clip\}\\\!\\big\(r\_\{t\}\(\\theta\),\\,1\-\\epsilon,\\,1\+\\epsilon\\big\)\\,A\_\{t,j\}\\right\)\\right\],\(13\)wherert​\(θ\)r\_\{t\}\(\\theta\)denotes the policy ratio andAt,jA\_\{t,j\}is the advantage signal used for the policy update, forj=r,cj=r,c\.

The robust critic losses for the reward and peak\-cost value functions are defined as

LVr\\displaystyle L\_\{V\_\{r\}\}=\(Vr​\(st\)−vtarget,r​\(st\)\)2,\\displaystyle=\\Big\(V\_\{r\}\(s\_\{t\}\)\-v\_\{\\text\{target\},r\}\(s\_\{t\}\)\\Big\)^\{2\},\(14\)LVc,peak\\displaystyle L\_\{V\_\{c,\\mathrm\{peak\}\}\}=\(Vc,peak​\(st\)−vtarget,c​\(st\)\)2\.\\displaystyle=\\Big\(V\_\{c,\\mathrm\{peak\}\}\(s\_\{t\}\)\-v\_\{\\text\{target\},c\}\(s\_\{t\}\)\\Big\)^\{2\}\.\(15\)
Algorithm[1](https://arxiv.org/html/2607.15457#algorithm1)summarizes the complete training procedure for the robust peak\-constrained MDP\.

Input:Initial policy parameters

θ0\\theta\_\{0\}, initial value function parameters

ϕr,0\\phi\_\{r,0\}and

ϕc,0\\phi\_\{c,0\}
Parameters:*Discount factorγ\\gamma, Peak\-Cost budgetbb, GAE decayλ\\lambda, Lagrangian multiplierβ\\beta, PPO clipping parameterϵ\\epsilon, learning ratesα*actor*\\alpha\_\{\\text\{actor\}\},α*critic,r*\\alpha\_\{\\text\{critic,r\}\},α*critic,c*\\alpha\_\{\\text\{critic,c\}\}Reward/Cost Selectorψ\(\.\)\\psi\(\.\)*

Initialize:Policy

πθ\\pi\_\{\\theta\}with parameters

θ0\\theta\_\{0\}, robust value functions

VrV\_\{r\}and

Vc,peakV\_\{c,\\mathrm\{peak\}\}with parameters

ϕr,0\\phi\_\{r,0\}and

ϕc,0\\phi\_\{c,0\};

for*t=0,1,…,T−1t=0,1,\\dots,T\-1*do

Collect a set of trajectories

𝒟t=\{τi\}i=1N\\mathcal\{D\}\_\{t\}=\\\{\\tau\_\{i\}\\\}\_\{i=1\}^\{N\}by running policy

πθ\\pi\_\{\\theta\}in the environment;

A=1\|Dt\|​∑s∈DtVr​\(s\)βA=\\frac\{1\}\{\|D\_\{t\}\|\}\\sum\_\{s\\in D\_\{t\}\}\\frac\{V\_\{r\}\(s\)\}\{\\beta\};

B=maxs∈Dt⁡\(Vc,peak​\(s\)−b\)B=\\max\_\{s\\in D\_\{t\}\}\\left\(V\_\{c,\\mathrm\{peak\}\}\(s\)\-b\\right\);

F=arg⁡max⁡\{A,B\}F=\\arg\\max\\\{A,B\\\};

Estimate Advantage & Update Critic:

ψ​\(F\)\\psi\(F\);

Update Policy using Eqn[13](https://arxiv.org/html/2607.15457#S4.E13)and update

θt\\theta\_\{t\}using SGD with learning rate

αactor\\alpha\_\{\\text\{actor\}\};

Output:Optimized policy

πθ\\pi\_\{\\theta\}

Algorithm 1RP\-CRLInput:Choice function F, Discount factor

γ\\gamma, GAE decay

λ\\lambda,PPO clipping parameter

ϵ\\epsilon, learning rates

αactor\\alpha\_\{\\text\{actor\}\},

αcritic,r\\alpha\_\{\\text\{critic,r\}\},

αcritic,c\\alpha\_\{\\text\{critic,c\}\}
if*F=0F=0*then

Update Reward Critic:;

Compute TD errors

δt,r\\delta\_\{t,r\}using Eqn[7](https://arxiv.org/html/2607.15457#S4.E7);

Estimate advantages

At,rA\_\{t,r\}using Eqn[9](https://arxiv.org/html/2607.15457#S4.E9);

else if*F=1F=1*then

Update Cost Critic:;

Compute TD errors

δt,c\\delta\_\{t,c\}using Eqn[8](https://arxiv.org/html/2607.15457#S4.E8);

Estimate advantages

At,cA\_\{t,c\}using Eqn[10](https://arxiv.org/html/2607.15457#S4.E10);

Estimate targets

vt​a​r​g​e​t,rv\_\{target,r\}using Eqn[11](https://arxiv.org/html/2607.15457#S4.E11);

Estimate reward critic loss

LVrL\_\{V\_\{r\}\}using Eqn[14](https://arxiv.org/html/2607.15457#S4.E14);

Update critic parameters

ϕr\\phi\_\{r\}using SGD with learning rate

αcritic,r\\alpha\_\{\\text\{critic,r\}\};

Estimate targets

vt​a​r​g​e​t,cv\_\{target,c\}using Eqn[12](https://arxiv.org/html/2607.15457#S4.E12);

Estimate cost critic loss

LVc,peakL\_\{V\_\{c,\\mathrm\{peak\}\}\}using Eqn[15](https://arxiv.org/html/2607.15457#S4.E15);

Update critic parameters

ϕc\\phi\_\{c\}using SGD with learning rate

αcritic,c\\alpha\_\{\\text\{critic,c\}\};

Algorithm 2Reward/Cost Selectorψ\\psi\. Note that we take a sample average of the reward value function scaled by1/β1/\\beta\(AA\), if it is higher than the maximum value function corresponding to the state sampled from the trajectory, then we update the actor based on the average reward value function, otherwise we update the actor based on the constraint value function to minimize the constraint violation\.

## VExperiments

### V\-ABaselines

We compare RP\-CRL with two baselines:

1. 1\.Primal Dual method for Peak\-Cost Constraint which is used in\[[28](https://arxiv.org/html/2607.15457#bib.bib4),[11](https://arxiv.org/html/2607.15457#bib.bib7)\]\.
2. 2\.Using Surrogate Objective in \([6](https://arxiv.org/html/2607.15457#S3.E6)\) trained in Non\-Perturbed environment

RP\-CRL refers to Surrogate Objective trained on Perturbed Environment\.

### V\-BOpenAI Gym Environment Results

The CartPole environment is a widely used reinforcement learning \(RL\) benchmark introduced in OpenAI Gym\[[4](https://arxiv.org/html/2607.15457#bib.bib29)\]\. It consists of a cart moving along a frictionless track, with a pole attached to the cart by an unactuated joint\. The agent applies forces to the cart to move it left or right, aiming to balance the pole upright while keeping the cart within specified boundaries\.

Constrained CartPole EnvironmentIn our work, we use a modified version of the standard CartPole environment by introducingsafety and cost constraints\. The state space remains the same as the standard CartPole and is represented as:Observation Space=\{\(x,x˙,θ,θ˙\)∣x,x˙,θ,θ˙∈ℝ\}\.\\text\{Observation\\ Space\}=\\\{\(x,\\dot\{x\},\\theta,\\dot\{\\theta\}\)\\mid x,\\dot\{x\},\\theta,\\dot\{\\theta\}\\in\\mathbb\{R\}\\\}\.wherexxis the cart position,x˙\\dot\{x\}is the cart velocity,θ\\thetais the pole angle in radians, andθ˙\\dot\{\\theta\}is the pole angular velocity\. The observation space is continuous and unbounded\. The action space differs from the standard environment\. Instead of discrete actions \(\{0,1\}\\\{0,1\\\}\), we use acontinuous action spacewhere the agent can apply a forceF∈\[−10\.0,10\.0\]F\\in\[\-10\.0,10\.0\]\.

The dynamics of the CartPole system are given by the following equations\. The angular acceleration of the pole is:

θ¨=g​sin⁡θ\+cos⁡θ​\(−F−mp​l​θ˙2​sin⁡θ\)/\(mc\+mp\)l​\(43−mp​cos2⁡θmc\+mp\),\\ddot\{\\theta\}=\\tfrac\{g\\sin\\theta\+\\cos\\theta\\left\(\-F\-m\_\{p\}l\\dot\{\\theta\}^\{2\}\\sin\\theta\\right\)/\(m\_\{c\}\+m\_\{p\}\)\}\{l\\left\(\\tfrac\{4\}\{3\}\-\\tfrac\{m\_\{p\}\\cos^\{2\}\\theta\}\{m\_\{c\}\+m\_\{p\}\}\\right\)\},and the horizontal acceleration of the cart is:

x¨=F\+mp​l​\(θ˙2​sin⁡θ−θ¨​cos⁡θ\)mc\+mp\.\\ddot\{x\}=\\tfrac\{F\+m\_\{p\}l\\left\(\\dot\{\\theta\}^\{2\}\\sin\\theta\-\\ddot\{\\theta\}\\cos\\theta\\right\)\}\{m\_\{c\}\+m\_\{p\}\}\.For our experiments,g=9\.8​m/s2g=9\.8\\,\\text\{m/s\}^\{2\}is the gravitational acceleration,mc=1\.0​kgm\_\{c\}=1\.0\\,\\text\{kg\}is the cart’s mass,mp=0\.1​kgm\_\{p\}=0\.1\\,\\text\{kg\}is the pole’s mass, andl=0\.5​ml=0\.5\\,\\text\{m\}is the half\-length of the pole\. The time step for simulation isτ=0\.02​s\\tau=0\.02\\,\\text\{s\}\.

The constrained environment enforces safety by requiring the cart’s position to remain within safe zonex∈\[−1,1\]x\\in\[\-1,1\]\. In general the pole angle should stay withinx∈\[−2\.4,2\.4\]x\\in\[\-2\.4,2\.4\]andθ∈\[−12∘,12∘\]\\theta\\in\[\-12^\{\\circ\},12^\{\\circ\}\]\. If these limits are exceeded before 450 steps, the episode terminates early, and anearly termination penaltyof10\.010\.0is added to the total cost\. The agent incurs a cost when the cart’s position exceeds\|x\|\>1\.0\|x\|\>1\.0, with the cost proportional to the absolute distance from the center:

cost=\{\|x\|if​\|x\|\>1\.010if early termination\\text\{cost\}=\\begin\{cases\}\|x\|&\\text\{if \}\|x\|\>1\.0\\\\ 10&\\text\{if early termination\}\\end\{cases\}The objective is to maximize cumulative rewards while keeping the peak cost below a predefined baseline\. The maximum episode length is 500 steps\.

Perturbed CartPole EnvironmentTo model transition uncertainty, we perturb gravity during training asg′=g\+𝒩​\(0,0\.5\)g^\{\\prime\}=g\+\\mathcal\{N\}\(0,0\.5\), requiring the policy to remain robust under uncertain dynamics\.

Training DetailsThe architecture comprises three main networks: the Actor, the Critic, and the Peak\-Cost Critic, each structured as a feedforward neural network with three layers \(two hidden layers and one output layer\)\. The Actor network features two hidden layers, each containing 64 neurons with a ReLU activation function, and it outputs parameters for the Gaussian distribution\. Similarly, the Peak\-Cost network consists of two hidden layers with 64 neurons each, utilizing a ReLU activation function, and it produces a single value representing the state value estimate\. The Peak\-Cost Critic follows the same architecture as the Critic, also featuring two hidden layers with 64 neurons and a ReLU activation function, outputting a single cost estimate\. All the Actor and Critic networks use a learning rate of 3e\-4\.

### V\-CExperiments and Analysis

For the experiments \([1](https://arxiv.org/html/2607.15457#S5.I2.i1)and[2](https://arxiv.org/html/2607.15457#S5.I2.i2)\), we used Lagrangian multiplierβ=25\\beta=25, baselineb=2\.0b=2\.0, a warm start of 300 episodes\(for RP\-CRL\)\.

1. 1\.Effectiveness of our Surrogate Objective ![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/New_PD2_rewards.png)\(a\)Returns ![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/New_PD2_max_costs.png)\(b\)Peak\-Cost Figure 1:Comparison of Primal Dual\(PD\) and Surrogate Objective trained in non\-perturbed environment, and RP\-CRL trained in perturbed CartPole environment\. Here we want to maximize return while restrict peak\-cost below baseline\. RP\-CRL has learned peak\-cost constraint and has higher rewards than PD\.Fig[1](https://arxiv.org/html/2607.15457#S5.F1)compares the training of Primal\-Dual \(PD\), Surrogate Objective in \([6](https://arxiv.org/html/2607.15457#S3.E6)\) on a non\-perturbed CartPole environment and RP\-CRL\(Surrogate Objective trained on a perturbed CartPole environment\)\. The Surrogate objective based methods successfully enforce the peak\-cost constraint and achieve higher returns compared to PD\. PD learns lower returns and has higher peak\-cost than baseline because it learns a sub\-optimal policy\.
2. 2\.Inference in Perturbed Environment ![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/gravity_comparison2_rewards_legend_horizontal.png) ![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/gravity_comparison2_rewards.png)\(a\)Returns ![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/gravity_comparison2_max_costs.png)\(b\)Peak\-Cost Figure 2:Comparison of inference performances of PD trained on non\-perturbed\(NP\) environment, Surrogate Objective \([6](https://arxiv.org/html/2607.15457#S3.E6)\) trained on NP environment and RP\-CRL trained in perturbed \(P\) environment\.RP\-CRL performs best with a peak\-cost of zero\. PD fails to adapt to a perturbed environment at inference\.The inference is done with a perturbation ofg′=g\+𝒩​\(0,2\.0\)g^\{\\prime\}=g\+\\mathcal\{N\}\(0,2\.0\)\. The comparison in Fig[2](https://arxiv.org/html/2607.15457#S5.F2)highlights the superior performance of RP\-CRL trained in a perturbed \(P\) environment, achieving the best results with a peak\-cost of zero\. In contrast, Surrogate Objective, trained in non\-perturbed \(NP\) environments, has a peak\-cost below baseline and PD fails to meet the peak\-cost constraint and adapt to the robustness required during inference, evident from its rewards as well\.

### V\-DTraining across Different Cost Functions

We perform a study of RP\-CRL learning two different cost functions to investigate the generalizability of training our proposed approach\.

Cost1 \(C1\): This cost penalizes deviations from the center and heavily penalizes early termination due to exceeding position or angle limits\. A cost of equal to position error is incurred when the cart’s position exceeds a distance of 1 from origin\. If the cart’s position is greater than 2\.4 or pole angle is greater than12∘12^\{\\circ\}before the termination limit of 450 steps, a penalty of10\.010\.0is added to the cost\. The baseline used for this cost isb=2b=2\. Green line in Fig[1](https://arxiv.org/html/2607.15457#S5.F1)shows training plots using C1 for RP\-CRL\.

Cost2 \(C2\): This cost incentivizes staying within safe position and angle limits to avoid termination\. It penalizes deviations in position greater than 1\.0 from origin and pole angle greater than8∘8^\{\\circ\}\. These costs increase as they approach termination thresholds \(position\>2\.4\>2\.4, angle\>12∘\>12^\{\\circ\}\)\. The baseline used for this cost isb=0\.5b=0\.5\. Fig[3](https://arxiv.org/html/2607.15457#S5.F3)shows training plots using C2\.

![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/New_Ours2_C2_rewards.png)\(a\)Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/New_Ours2_C2_max_costs.png)\(b\)Peak\-Cost

Figure 3:Learning RP\-CRL on C2\. The policy learns the peak\-cost below baseline\.\.These results demonstrate the robustness and adaptability of RP\-CRL in effectively learning across different cost functions\.

### V\-ETraining across Different Levels of Perturbations

We perform a study of RP\-CRL learning two different perturbation levels in the perturbed CartPole environment\. This experiment is conducted to investigate how good RP\-CRL learns when perturbations in environment are different\. Fig[4](https://arxiv.org/html/2607.15457#S5.F4)shows the training curves for different perturbation levels\. We perturb the gravityg′=g\+𝒩​\(0,σ\)g^\{\\prime\}=g\+\\mathcal\{N\}\(0,\\sigma\)\. For comparison, we considerσ=\{0\.5,2\}\\sigma=\\\{0\.5,2\\\}\. The learning curves demonstrate RP\-CRL learns different perturbation levels effectively\.

![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/Pert_rewards.png)\(a\)Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/Pert_max_costs.png)\(b\)Peak\-Cost

Figure 4:Comparison of training curves of RP\-CRL for different perturbation levels in the environment\. The perturbation is introduced in gravity\. RP\-CRL learns maximized returns and bounded peak\-cost below baseline for both perturbation levels\.
### V\-FMujoco Environment Results

![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/ant_training_rewards.png)\(a\)Ant: Training Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/HC_training_rewards.png)\(b\)HalfCheetah: Training Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/humanoid_training_rewards.png)\(c\)Humanoid: Training Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/Swimmer_training_rewards.png)\(d\)Swimmer: Training Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/ant_training_max_costs.png)\(e\)Ant: Training Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/HC_training_max_costs.png)\(f\)HalfCheetah: Training Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/humanoid_training_max_costs.png)\(g\)Humanoid: Training Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/Swimmer_training_max_costs.png)\(h\)Swimmer: Training Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/ant_inference_rewards.png)\(i\)Ant: Inference Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/HC_inference_rewards.png)\(j\)HalfCheetah: Inference Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/humanoid_inference_rewards.png)\(k\)Humanoid: Inference Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/swimmer_inference_rewards.png)\(l\)Swimmer: Inference Returns
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/ant_inference_max_costs.png)\(m\)Ant: Inference Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/HC_inference_max_costs.png)\(n\)HalfCheetah: Inference Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/humanoid_inference_max_costs.png)\(o\)Humanoid: Inference Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/swimmer_inference_max_costs.png)\(p\)Swimmer: Inference Max Cost
![Refer to caption](https://arxiv.org/html/2607.15457v1/figures/HC_inference_rewards_legend_horizontal.png)

Figure 5:Training and inference performance across four MuJoCo environments\. Rows show training returns, training maximum costs, inference returns, and inference maximum costs, respectively\. We observe that RP\-CRL has better performance at perturbed inference, maintains persistent safety at perturbed inference, and the performance learning curve has lower standard deviation for RP\-CRL\.We further evaluate RP\-CRL on four MuJoCo environments: Ant, HalfCheetah, Humanoid, and Swimmer\. For Ant, HalfCheetah, and Swimmer, the cost is defined using the maximum absolute action torque\. The agent incurs zero cost when the maximum absolute action torque is below the threshold0\.50\.5; otherwise, the cost is the amount by which this maximum torque exceeds0\.50\.5\. For Humanoid, the cost is defined as the energy consumption, computed as the sum of squared actions\. During training, Ant and Humanoid are trained with gravity perturbations up to0\.70\.7, HalfCheetah with gravity perturbations up to2\.02\.0, and Swimmer with viscosity perturbations up to0\.70\.7\.

Fig\.[5](https://arxiv.org/html/2607.15457#S5.F5)compares the training behavior of RP\-CRL with the surrogate objective trained without robustness\. During training, RP\-CRL achieves smoother learning curves with lower variance while maintaining the peak\-cost constraint more consistently across environments\. This indicates that training under perturbed dynamics improves stability without sacrificing constraint satisfaction\.

For inference, we evaluate the learned policies under stronger perturbations, with gravity perturbed up to2\.02\.0\(Ant, HalfCheetah, Humanoid\) and viscosity perturbed up to2\.02\.0\(Swimmer\)\. The inference results show that RP\-CRL achieves better performance under perturbed dynamics and maintains persistent safety compared to the non\-robust surrogate objective\. These results demonstrate that robustness\-aware training improves both return and safety generalization under environment uncertainty\.

## VIConclusion and Future Work

In this work, we address the challenges of peak\-cost constrained MDPs, demonstrating that they may not exhibit zero duality gap, unlike standard CMDPs\. We propose a robust surrogate approach that achieves near\-optimal solutions while ensuring constraint satisfaction within an acceptable marginbb\. Through extensive experiments on a perturbed CartPole and Mujoco environments, we show that RP\-CRL outperforms existing methods in terms of both constraint satisfaction and reward optimization, even under uncertain dynamics\.

The characterization of the convergence guarantees of the proposed approach has been left for the future\. Extending this framework to more complex real\-world systems constitutes an important future research direction\.

## Acknowledgement

This work is partially supported by NSF ECCS Award \#2534262\.

## References

- \[1\]\(2017\)Constrained policy optimization\.InInternational conference on machine learning,pp\. 22–31\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[2\]E\. Altman\(2021\)Constrained markov decision processes\.Routledge\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3),[§I](https://arxiv.org/html/2607.15457#S1.p1.1)\.
- \[3\]S\. Bansal, M\. Chen, S\. Herbert, and C\. J\. Tomlin\(2017\)Hamilton\-jacobi reachability: a brief overview and recent advances\.In2017 IEEE 56th annual conference on decision and control \(CDC\),pp\. 2242–2253\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[4\]G\. Brockman, V\. Cheung, L\. Pettersson, J\. Schneider, J\. Schulman, J\. Tang, and W\. Zaremba\(2016\)Openai gym\.arXiv preprint arXiv:1606\.01540\.Cited by:[§V\-B](https://arxiv.org/html/2607.15457#S5.SS2.p1.1)\.
- \[5\]L\. Brunke, M\. Greeff, A\. W\. Hall, Z\. Yuan, S\. Zhou, J\. Panerati, and A\. P\. Schoellig\(2022\)Safe learning in robotics: from learning\-based control to safe reinforcement learning\.Annual Review of Control, Robotics, and Autonomous Systems5\(1\),pp\. 411–444\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[6\]J\. J\. Choi, D\. Lee, K\. Sreenath, C\. J\. Tomlin, and S\. L\. Herbert\(2021\)Robust control barrier–value functions for safety\-critical control\.In2021 60th IEEE Conference on Decision and Control \(CDC\),pp\. 6814–6821\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[7\]Y\. Chow, O\. Nachum, E\. Duenez\-Guzman, and M\. Ghavamzadeh\(2018\)A lyapunov\-based approach to safe reinforcement learning\.Advances in neural information processing systems31\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[8\]D\. Ding, K\. Zhang, T\. Basar, and M\. Jovanovic\(2020\)Natural policy gradient primal\-dual method for constrained markov decision processes\.Advances in Neural Information Processing Systems33,pp\. 8378–8390\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[9\]J\. F\. Fisac, A\. K\. Akametalu, M\. N\. Zeilinger, S\. Kaynama, J\. Gillula, and C\. J\. Tomlin\(2018\)A general safety framework for learning\-based control in uncertain robotic systems\.IEEE Transactions on Automatic Control64\(7\),pp\. 2737–2752\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1),[§II\-B](https://arxiv.org/html/2607.15457#S2.SS2.p4.2)\.
- \[10\]J\. F\. Fisac, N\. F\. Lugovoy, V\. Rubies\-Royo, S\. Ghosh, and C\. J\. Tomlin\(2019\)Bridging hamilton\-jacobi safety analysis and reinforcement learning\.In2019 International Conference on Robotics and Automation \(ICRA\),pp\. 8550–8556\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[11\]M\. Ganai, Z\. Gong, C\. Yu, S\. Herbert, and S\. Gao\(2023\)Iterative reachability estimation for safe reinforcement learning\.Advances in Neural Information Processing Systems36,pp\. 69764–69797\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1),[§I](https://arxiv.org/html/2607.15457#S1.p2.1),[§I](https://arxiv.org/html/2607.15457#S1.p3.1),[item 1](https://arxiv.org/html/2607.15457#S5.I1.i1.p1.1)\.
- \[12\]S\. Ganguly, K\. Panaganti, A\. Ghosh, and A\. WiermanEfficient policy optimization in robust constrained mdps with iteration complexity guarantees\.InThe Thirty\-ninth Annual Conference on Neural Information Processing Systems,Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p2.1),[§III\-A](https://arxiv.org/html/2607.15457#S3.SS1.p1.1),[§III\-A](https://arxiv.org/html/2607.15457#S3.SS1.p1.3),[§III\-A](https://arxiv.org/html/2607.15457#S3.SS1.p3.1)\.
- \[13\]A\. Ghosh, X\. Zhou, and N\. Shroff\(2023\)Achieving sub\-linear regret in infinite horizon average reward constrained mdp with linear function approximation\.InThe Eleventh International Conference on Learning Representations,Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3),[§I](https://arxiv.org/html/2607.15457#S1.p1.1)\.
- \[14\]A\. Ghosh, X\. Zhou, and N\. Shroff\(2024\)Towards achieving sub\-linear regret and hard constraint violation in model\-free rl\.InInternational Conference on Artificial Intelligence and Statistics,pp\. 1054–1062\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3),[§I](https://arxiv.org/html/2607.15457#S1.p1.1)\.
- \[15\]G\. N\. Iyengar\(2005\)Robust dynamic programming\.Mathematics of Operations Research30\(2\),pp\. 257–280\.Cited by:[§II\-D](https://arxiv.org/html/2607.15457#S2.SS4.p2.2)\.
- \[16\]T\. Kitamura, T\. Kozuno, W\. Kumagai, K\. Hoshino, Y\. Hosoe, K\. Kasaura, M\. Hamaya, P\. Parmas, and Y\. MatsuoNear\-optimal policy identification in robust constrained markov decision processes via epigraph form\.InThe Thirteenth International Conference on Learning Representations,Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p2.1)\.
- \[17\]Y\. Luo and T\. Ma\(2021\)Learning barrier certificates: towards safe reinforcement learning with zero training\-time violations\.Advances in Neural Information Processing Systems34,pp\. 25621–25632\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[18\]H\. Ma, C\. Liu, S\. E\. Li, S\. Zheng, and J\. Chen\(2022\)Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning\.InLearning for Dynamics and Control Conference,pp\. 97–109\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[19\]A\. Müller\(1997\)Integral probability metrics and their generating classes of functions\.Advances in applied probability29\(2\),pp\. 429–443\.Cited by:[§II\-D](https://arxiv.org/html/2607.15457#S2.SS4.p3.4)\.
- \[20\]S\. Paternain, M\. Calvo\-Fullana, L\. F\. Chamon, and A\. Ribeiro\(2022\)Safe policies for reinforcement learning via primal\-dual methods\.IEEE Transactions on Automatic Control68\(3\),pp\. 1321–1336\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[21\]S\. Paternain, L\. Chamon, M\. Calvo\-Fullana, and A\. Ribeiro\(2019\)Constrained reinforcement learning has zero duality gap\.Advances in Neural Information Processing Systems32\.Cited by:[§II\-A](https://arxiv.org/html/2607.15457#S2.SS1.p3.2)\.
- \[22\]J\. Schulman, F\. Wolski, P\. Dhariwal, A\. Radford, and O\. Klimov\(2017\)Proximal policy optimization algorithms\.arXiv preprint arXiv:1707\.06347\.Cited by:[§III\-A](https://arxiv.org/html/2607.15457#S3.SS1.p3.1),[§IV\-A](https://arxiv.org/html/2607.15457#S4.SS1.p4.4)\.
- \[23\]O\. So, C\. Ge, and C\. Fan\(2024\)Solving minimum\-cost reach avoid using reinforcement learning\.Advances in Neural Information Processing Systems37,pp\. 30951–30984\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1)\.
- \[24\]A\. Stooke, J\. Achiam, and P\. Abbeel\(2020\)Responsive safety in reinforcement learning by pid lagrangian methods\.InInternational conference on machine learning,pp\. 9133–9143\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[25\]C\. Tessler, D\. J\. Mankowitz, and S\. MannorReward constrained policy optimization\.InInternational Conference on Learning Representations,Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3),[§I](https://arxiv.org/html/2607.15457#S1.p1.1)\.
- \[26\]Y\. Wang, F\. Miao, and S\. Zou\(2022\)Robust constrained reinforcement learning\.arXiv preprint arXiv:2209\.06866\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p2.1)\.
- \[27\]T\. Xu, Y\. Liang, and G\. Lan\(2021\)Crpo: a new approach for safe reinforcement learning with convergence guarantee\.InInternational Conference on Machine Learning,pp\. 11480–11491\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[28\]D\. Yu, H\. Ma, S\. Li, and J\. Chen\(2022\)Reachability constrained reinforcement learning\.InInternational conference on machine learning,pp\. 25636–25655\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p3.1),[§I](https://arxiv.org/html/2607.15457#S1.p2.1),[§I](https://arxiv.org/html/2607.15457#S1.p3.1),[§II\-B](https://arxiv.org/html/2607.15457#S2.SS2.p5.3),[§II\-C](https://arxiv.org/html/2607.15457#S2.SS3.6.2),[§II\-C](https://arxiv.org/html/2607.15457#S2.SS3.p1.2),[item 1](https://arxiv.org/html/2607.15457#S5.I1.i1.p1.1)\.
- \[29\]Z\. Zhang, K\. Panaganti, L\. Shi, Y\. Sui, A\. Wierman, and Y\. Yue\(2024\)Distributionally robust constrained reinforcement learning under strong duality\.arXiv preprint arXiv:2406\.15788\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p2.1)\.
- \[30\]L\. Zheng and L\. Ratliff\(2020\)Constrained upper confidence reinforcement learning\.InLearning for Dynamics and Control,pp\. 620–629\.Cited by:[§I\-A](https://arxiv.org/html/2607.15457#S1.SS1.p1.1.3)\.
- \[31\]R\. Zhou, T\. Liu, M\. Cheng, D\. Kalathil, P\. Kumar, and C\. Tian\(2023\)Natural actor\-critic for robust reinforcement learning with function approximation\.Advances in neural information processing systems36,pp\. 97–133\.Cited by:[§II\-D](https://arxiv.org/html/2607.15457#S2.SS4.p3.2)\.

### \-AProof of Theorem[1](https://arxiv.org/html/2607.15457#Thmtheorem1)and Proposition[1](https://arxiv.org/html/2607.15457#Thmproposition1)

Formal Statement of Theorem[1](https://arxiv.org/html/2607.15457#Thmtheorem1): Consider a discounted MDP with discount factorγ=0\.9\\gamma=0\.9, one nonterminal statess, and one absorbing terminal state⊥\\bot\. At statess, there are two actions:

- •actiona0a\_\{0\}: rewardr​\(s,a0\)=4r\(s,a\_\{0\}\)=4, costc​\(s,a0\)=0c\(s,a\_\{0\}\)=0, and transition P\(⊥∣s,a0\)=1;P\(\\bot\\mid s,a\_\{0\}\)=1;
- •actiona1a\_\{1\}: rewardr​\(s,a1\)=8r\(s,a\_\{1\}\)=8, costc​\(s,a1\)=12c\(s,a\_\{1\}\)=\\tfrac\{1\}\{2\}, and transition P​\(s∣s,a1\)=1\.P\(s\\mid s,a\_\{1\}\)=1\.

Letπp\\pi\_\{p\}denote the stationary randomized policy that choosesa0a\_\{0\}with probabilityp∈\[0,1\]p\\in\[0,1\]anda1a\_\{1\}with probability1−p1\-p\. Define the hard\-max modified constraint value

Vcπp\(s\)=∑aπp\(a∣s\)\(\(1−γ\)c\(s,a\)\\displaystyle V\_\{c\}^\{\\pi\_\{p\}\}\(s\)=\\sum\_\{a\}\\pi\_\{p\}\(a\\mid s\)\\Bigl\(\(1\-\\gamma\)c\(s,a\)\+γmax\{c\(s,a\),𝔼\[Vcπp\(s′\)∣s,a\]\}\),\\displaystyle\+\\gamma\\max\\\{c\(s,a\),\\mathbb\{E\}\[V\_\{c\}^\{\\pi\_\{p\}\}\(s^\{\\prime\}\)\\mid s,a\]\\\}\\Bigr\),\(16\)withVcπp​\(⊥\)=0V\_\{c\}^\{\\pi\_\{p\}\}\(\\bot\)=0\.

Consider the constrained optimization problem

P⋆:=maxp∈\[0,1\]⁡Jr​\(πp\)s\.t\.Vcπp​\(s\)≤b,P^\{\\star\}:=\\max\_\{p\\in\[0,1\]\}J\_\{r\}\(\\pi\_\{p\}\)\\qquad\\text\{s\.t\.\}\\qquad V\_\{c\}^\{\\pi\_\{p\}\}\(s\)\\leq b,with thresholdb=0\.4b=0\.4, whereJr​\(πp\)J\_\{r\}\(\\pi\_\{p\}\)is the standard discounted reward value starting fromss\. Here, note that peak cost\-constraint across the trajectory is the same as here because of the two states with one state being terminal\.

Then the associated Lagrangian dual

D⋆:=infν≥0supp∈\[0,1\]\{Jr​\(πp\)−ν​\(Vcπp​\(s\)−b\)\}D^\{\\star\}:=\\inf\_\{\\nu\\geq 0\}\\ \\sup\_\{p\\in\[0,1\]\}\\Bigl\\\{J\_\{r\}\(\\pi\_\{p\}\)\-\\nu\\bigl\(V\_\{c\}^\{\\pi\_\{p\}\}\(s\)\-b\\bigr\)\\Bigr\\\}satisfies

D⋆\>P⋆\.D^\{\\star\}\>P^\{\\star\}\.More precisely,

P⋆=1807≈25\.7143,D⋆=64\.8,P^\{\\star\}=\\tfrac\{180\}\{7\}\\approx 25\.7143,\\qquad D^\{\\star\}=64\.8,so that

D⋆−P⋆=136835≈39\.0857\>0\.D^\{\\star\}\-P^\{\\star\}=\\tfrac\{1368\}\{35\}\\approx 39\.0857\>0\.Hence the peak\-cost\-constraint problem admits a strict duality gap\.

###### Proof:

LetJr​\(πp\)J\_\{r\}\(\\pi\_\{p\}\)denote the discounted reward value starting fromss\. By the Bellman equation,

Jr​\(πp\)=p⋅4\+\(1−p\)​\(8\+γ​Jr​\(πp\)\)\.J\_\{r\}\(\\pi\_\{p\}\)=p\\cdot 4\+\(1\-p\)\\bigl\(8\+\\gamma J\_\{r\}\(\\pi\_\{p\}\)\\bigr\)\.Sinceγ=0\.9\\gamma=0\.9, this becomes

Jr​\(πp\)=4​p\+\(1−p\)​\(8\+0\.9​Jr​\(πp\)\)\.J\_\{r\}\(\\pi\_\{p\}\)=4p\+\(1\-p\)\(8\+0\.9J\_\{r\}\(\\pi\_\{p\}\)\)\.Rearranging gives

Jr​\(πp\)​\(1−0\.9​\(1−p\)\)=8−4​p,J\_\{r\}\(\\pi\_\{p\}\)\\bigl\(1\-0\.9\(1\-p\)\\bigr\)=8\-4p,hence

Jr​\(πp\)=8−4​p0\.1\+0\.9​p\.J\_\{r\}\(\\pi\_\{p\}\)=\\tfrac\{8\-4p\}\{0\.1\+0\.9p\}\.
Next, letV​\(p\):=Vcπp​\(s\)V\(p\):=V\_\{c\}^\{\\pi\_\{p\}\}\(s\)\. By definition of the modified hard\-max constraint,

V​\(p\)\\displaystyle V\(p\)=p​\(\(1−γ\)⋅0\+γ​max⁡\{0,0\}\)\+\\displaystyle=p\\Bigl\(\(1\-\\gamma\)\\cdot 0\+\\gamma\\max\\\{0,0\\\}\\Bigr\)\+\(1−p\)​\(\(1−γ\)​12\+γ​max⁡\{12,V​\(p\)\}\)\.\\displaystyle\\qquad\(1\-p\)\\Bigl\(\(1\-\\gamma\)\\tfrac\{1\}\{2\}\+\\gamma\\max\\\{\\tfrac\{1\}\{2\},V\(p\)\\\}\\Bigr\)\.\(17\)The first term is zero, so

V​\(p\)=\(1−p\)​\(0\.1⋅12\+0\.9​max⁡\{12,V​\(p\)\}\)\.V\(p\)=\(1\-p\)\\Bigl\(0\.1\\cdot\\tfrac\{1\}\{2\}\+0\.9\\max\\\{\\tfrac\{1\}\{2\},V\(p\)\\\}\\Bigr\)\.
We now solve this fixed\-point equation\. Suppose first thatV​\(p\)≤12V\(p\)\\leq\\tfrac\{1\}\{2\}\. Then

max⁡\{12,V​\(p\)\}=12,\\max\\\{\\tfrac\{1\}\{2\},V\(p\)\\\}=\\tfrac\{1\}\{2\},and thus

V​\(p\)=\(1−p\)​\(0\.05\+0\.9⋅12\)=\(1−p\)⋅12=1−p2\.V\(p\)=\(1\-p\)\\Bigl\(0\.05\+0\.9\\cdot\\tfrac\{1\}\{2\}\\Bigr\)=\(1\-p\)\\cdot\\tfrac\{1\}\{2\}=\\tfrac\{1\-p\}\{2\}\.Since1−p2≤12\\tfrac\{1\-p\}\{2\}\\leq\\tfrac\{1\}\{2\}for allp∈\[0,1\]p\\in\[0,1\], this is self\-consistent\. Therefore,

Vcπp​\(s\)=1−p2,∀p∈\[0,1\]\.V\_\{c\}^\{\\pi\_\{p\}\}\(s\)=\\tfrac\{1\-p\}\{2\},\\qquad\\forall p\\in\[0,1\]\.
We now solve the primal problem

maxp∈\[0,1\]⁡Jr​\(πp\)s\.t\.1−p2≤0\.4\.\\max\_\{p\\in\[0,1\]\}J\_\{r\}\(\\pi\_\{p\}\)\\qquad\\text\{s\.t\.\}\\qquad\\tfrac\{1\-p\}\{2\}\\leq 0\.4\.The constraint is equivalent to

1−p≤0\.8⇔p≥0\.2\.1\-p\\leq 0\.8\\iff p\\geq 0\.2\.Moreover,

Jr​\(πp\)=8−4​p0\.1\+0\.9​p,J\_\{r\}\(\\pi\_\{p\}\)=\\tfrac\{8\-4p\}\{0\.1\+0\.9p\},and differentiating yields

dd​p​Jr​\(πp\)=−4​\(0\.1\+0\.9​p\)−\(8−4​p\)​\(0\.9\)\(0\.1\+0\.9​p\)2\\displaystyle\\tfrac\{d\}\{dp\}J\_\{r\}\(\\pi\_\{p\}\)=\\tfrac\{\-4\(0\.1\+0\.9p\)\-\(8\-4p\)\(0\.9\)\}\{\(0\.1\+0\.9p\)^\{2\}\}=−7\.6\(0\.1\+0\.9​p\)2<0\.\\displaystyle=\\tfrac\{\-7\.6\}\{\(0\.1\+0\.9p\)^\{2\}\}<0\.\(18\)HenceJr​\(πp\)J\_\{r\}\(\\pi\_\{p\}\)is strictly decreasing inpp, so the primal optimum is attained at the smallest feasiblepp, namelyp⋆=0\.2p^\{\\star\}=0\.2\. Therefore,

P⋆=Jr​\(π0\.2\)=8−4​\(0\.2\)0\.1\+0\.9​\(0\.2\)=7\.20\.28=1807\.P^\{\\star\}=J\_\{r\}\(\\pi\_\{0\.2\}\)=\\tfrac\{8\-4\(0\.2\)\}\{0\.1\+0\.9\(0\.2\)\}=\\tfrac\{7\.2\}\{0\.28\}=\\tfrac\{180\}\{7\}\.
Now consider the Lagrangian

L​\(p,ν\)=Jr​\(πp\)−ν​\(Vcπp​\(s\)−0\.4\),ν≥0\.L\(p,\\nu\)=J\_\{r\}\(\\pi\_\{p\}\)\-\\nu\\bigl\(V\_\{c\}^\{\\pi\_\{p\}\}\(s\)\-0\.4\\bigr\),\\qquad\\nu\\geq 0\.Substituting the formulas forJr​\(πp\)J\_\{r\}\(\\pi\_\{p\}\)andVcπp​\(s\)V\_\{c\}^\{\\pi\_\{p\}\}\(s\)gives

L​\(p,ν\)=8−4​p0\.1\+0\.9​p−ν​\(1−p2−0\.4\)\\displaystyle L\(p,\\nu\)=\\tfrac\{8\-4p\}\{0\.1\+0\.9p\}\-\\nu\\left\(\\tfrac\{1\-p\}\{2\}\-0\.4\\right\)=8−4​p0\.1\+0\.9​p−ν​\(0\.1−0\.5​p\)\.\\displaystyle=\\tfrac\{8\-4p\}\{0\.1\+0\.9p\}\-\\nu\(0\.1\-0\.5p\)\.\(19\)
Fixν≥0\\nu\\geq 0\. The derivative with respect toppis

∂∂p​L​\(p,ν\)=−7\.6\(0\.1\+0\.9​p\)2\+ν2,\\tfrac\{\\partial\}\{\\partial p\}L\(p,\\nu\)=\-\\tfrac\{7\.6\}\{\(0\.1\+0\.9p\)^\{2\}\}\+\\tfrac\{\\nu\}\{2\},and the second derivative is

∂2∂p2​L​\(p,ν\)=13\.68\(0\.1\+0\.9​p\)3\>0\.\\tfrac\{\\partial^\{2\}\}\{\\partial p^\{2\}\}L\(p,\\nu\)=\\tfrac\{13\.68\}\{\(0\.1\+0\.9p\)^\{3\}\}\>0\.Thus for every fixedν\\nu, the functionp↦L​\(p,ν\)p\\mapsto L\(p,\\nu\)is convex on\[0,1\]\[0,1\]\. Therefore its supremum over\[0,1\]\[0,1\]is attained at one of the endpoints:

g​\(ν\):=supp∈\[0,1\]L​\(p,ν\)=max⁡\{L​\(0,ν\),L​\(1,ν\)\}\.g\(\\nu\):=\\sup\_\{p\\in\[0,1\]\}L\(p,\\nu\)=\\max\\\{L\(0,\\nu\),L\(1,\\nu\)\\\}\.
Evaluating the endpoints,

L​\(0,ν\)=80\.1−0\.1​ν=80−0\.1​ν,L\(0,\\nu\)=\\tfrac\{8\}\{0\.1\}\-0\.1\\nu=80\-0\.1\\nu,and

L​\(1,ν\)=41\+0\.4​ν=4\+0\.4​ν\.L\(1,\\nu\)=\\tfrac\{4\}\{1\}\+0\.4\\nu=4\+0\.4\\nu\.Hence

g​\(ν\)=max⁡\{80−0\.1​ν,4\+0\.4​ν\}\.g\(\\nu\)=\\max\\\{80\-0\.1\\nu,\\;4\+0\.4\\nu\\\}\.
The minimum of this maximum of two affine functions occurs when they are equal:

80−0\.1​ν=4\+0\.4​ν⇔76=0\.5​ν⇔ν⋆=152\.80\-0\.1\\nu=4\+0\.4\\nu\\iff 76=0\.5\\nu\\iff\\nu^\{\\star\}=152\.Substituting back,

D⋆=g​\(ν⋆\)=80−0\.1​\(152\)=64\.8\.D^\{\\star\}=g\(\\nu^\{\\star\}\)=80\-0\.1\(152\)=64\.8\.
Finally,

D⋆−P⋆=64\.8−1807=3245−1807\\displaystyle D^\{\\star\}\-P^\{\\star\}=64\.8\-\\tfrac\{180\}\{7\}=\\tfrac\{324\}\{5\}\-\\tfrac\{180\}\{7\}=2268−90035\\displaystyle=\\tfrac\{2268\-900\}\{35\}=136835\>0\.\\displaystyle=\\tfrac\{1368\}\{35\}\>0\.Thus the duality gap is strict\. ∎

Proof of Proposition[1](https://arxiv.org/html/2607.15457#Thmproposition1):

###### Proof:

Part \(i\): Reward Optimality\.Sinceπ∗\\pi^\{\*\}is feasible for problem \([4](https://arxiv.org/html/2607.15457#S2.E4)\), we have:maxP∈𝒫,s∈𝒮π∗⁡Vc,peakπ∗,P​\(s\)≤b,\\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\pi^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\pi^\{\*\},P\}\(s\)\\leq b,which implies: the surrogate objective evaluated atπ∗\\pi^\{\*\}satisfies:

max⁡\{maxP∈𝒫⁡Jrπ∗,Pβ,maxP∈𝒫,s∈𝒮π∗⁡Vc,peakπ∗,P​\(s\)−b\}\\displaystyle\\max\\left\\\{\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\pi^\{\*\},P\}\}\{\\beta\},\\ \\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\pi^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\pi^\{\*\},P\}\(s\)\-b\\right\\\}=maxP∈𝒫⁡Jrπ∗,Pβ,\\displaystyle=\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\pi^\{\*\},P\}\}\{\\beta\},\(20\)since the second term inside the max is non\-positive\. Sinceπ^∗\\hat\{\\pi\}^\{\*\}is the minimizer of the surrogate problem \([6](https://arxiv.org/html/2607.15457#S3.E6)\), we have:

max⁡\{maxP∈𝒫⁡Jrπ^∗,Pβ,maxP∈𝒫,s∈𝒮π^∗⁡Vc,peakπ^∗,P​\(s\)−b\}\\displaystyle\\max\\left\\\{\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\hat\{\\pi\}^\{\*\},P\}\}\{\\beta\},\\ \\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\hat\{\\pi\}^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\hat\{\\pi\}^\{\*\},P\}\(s\)\-b\\right\\\}≥maxP∈𝒫⁡Jrπ∗,Pβ\.\\displaystyle\\geq\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\pi^\{\*\},P\}\}\{\\beta\}\.\(21\)Since the left\-hand side is greater than or equal tomaxP∈𝒫⁡Jrπ^∗,Pβ\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\hat\{\\pi\}^\{\*\},P\}\}\{\\beta\}, the result follows for the minimization objective\.

Part \(ii\): Constraint Violation Bound\.From the optimality ofπ^∗\\hat\{\\pi\}^\{\*\}and the bound established in Part \(i\):

max⁡\{maxP∈𝒫⁡Jrπ^∗,Pβ,maxP∈𝒫,s∈𝒮π^∗⁡Vc,peakπ^∗,P​\(s\)−b\}\\displaystyle\\max\\left\\\{\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\hat\{\\pi\}^\{\*\},P\}\}\{\\beta\},\\ \\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\hat\{\\pi\}^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\hat\{\\pi\}^\{\*\},P\}\(s\)\-b\\right\\\}≥maxP∈𝒫⁡Jrπ∗,Pβ\.\\displaystyle\\geq\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\pi^\{\*\},P\}\}\{\\beta\}\.\(22\)In particular, the second term inside the max satisfies:

maxP∈𝒫,s∈𝒮π^∗⁡Vc,peakπ^∗,P​\(s\)−b≥maxP∈𝒫⁡Jrπ∗,Pβ\.\\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\hat\{\\pi\}^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\hat\{\\pi\}^\{\*\},P\}\(s\)\-b\\geq\\frac\{\\max\_\{P\\in\\mathcal\{P\}\}J\_\{r\}^\{\\pi^\{\*\},P\}\}\{\\beta\}\.\(23\)Let us consider the cost valuesci∈\[0,Cmax\]c\_\{i\}\\in\[0,C\_\{\\text\{max\}\}\]\. SinceVc,peakπ,P​\(s\)∈\[0,Cmax\]V\_\{c,\\mathrm\{peak\}\}^\{\\pi,P\}\(s\)\\in\[0,C\_\{\\text\{max\}\}\]for allπ,P,s\\pi,P,sby definition of the peak\-cost value function, andJrπ∗,P≤Cm​a​xJ\_\{r\}^\{\\pi^\{\*\},P\}\\leq C\_\{max\}, we obtain:

maxP∈𝒫,s∈𝒮π^∗⁡Vc,peakπ^∗,P​\(s\)−b≤Cmaxβ\.\\max\_\{P\\in\\mathcal\{P\},\\ s\\in\\mathcal\{S\}^\{\\hat\{\\pi\}^\{\*\}\}\}V\_\{c,\\mathrm\{peak\}\}^\{\\hat\{\\pi\}^\{\*\},P\}\(s\)\-b\\leq\\frac\{C\_\{\\text\{max\}\}\}\{\\beta\}\.\(24\)Settingβ≥Cmax/ϵ\\beta\\geq C\_\{\\text\{max\}\}/\\epsiloncompletes the proof of part \(ii\)\. ∎